# Streaming output text When deploying a finetuned (SFT, DPO) model with custom inference script

**URL:** <https://discuss.huggingface.co/t/streaming-output-text-when-deploying-a-finetuned-sft-dpo-model-with-custom-inference-script/116214>\
**Category:** Amazon SageMaker\
**Created:** [November 7, 2024, 2:39pm UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-a-finetuned-sft-dpo-model-with-custom-inference-script/116214 "2024-11-07T14:39:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![xnaxi](https://avatars.discourse-cdn.com/v4/letter/x/c77e96/32.png) [@xnaxi](https://discuss.huggingface.co/u/xnaxi)\
**Post date:** [November 7, 2024, 2:39pm UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-a-finetuned-sft-dpo-model-with-custom-inference-script/116214/1 "2024-11-07T14:39:21Z")

</div>

I have a finetuned mistral instruct V2 model that has been fine-tuned using SFT followed by DPO mechanisms. I want to deploy this on sagemaker with custom inference script. While doing this I also want to stream the output via sagmaker. Is this possible. I have seen a comment last November saying - One cannot use the llm container with a custom inference script. ([Deploying custom inference script with llama2 finetuned model - #2 by philschmid](https://discuss.huggingface.co/t/deploying-custom-inference-script-with-llama2-finetuned-model/63280/2))

Any idea if we can do this now…any help would be appreciated

---

<div class="post-metadata">

**Author:** ![xnaxi](https://avatars.discourse-cdn.com/v4/letter/x/c77e96/32.png) [@xnaxi](https://discuss.huggingface.co/u/xnaxi)\
**Post date:** [November 8, 2024, 7:10am UTC](https://discuss.huggingface.co/t/streaming-output-text-when-deploying-a-finetuned-sft-dpo-model-with-custom-inference-script/116214/2 "2024-11-08T07:10:04Z")

</div>

@philschmid…Would appreciate your help