# Deploying a conversational pipeline on AWS

**URL:** <https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280>\
**Category:** Amazon SageMaker\
**Created:** [January 1, 2022, 4:55am UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280 "2022-01-01T04:55:44Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![JB2022](https://avatars.discourse-cdn.com/v4/letter/j/bbce88/32.png) [@JB2022](https://discuss.huggingface.co/u/JB2022)\
**Post date:** [January 1, 2022, 4:55am UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/1 "2022-01-01T04:55:44Z")

</div>

I am following the instructions for deploying a custom `Microsoft/DialoGPT-medium` on AWS Sagemaker. As a first step, I am using the instructions included on the model hub, but they do not work and appear to be out of date.

The code that follows leads to this error:

```auto
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "ConversationalPipeline expects a Conversation or list of Conversations as an input"
}

```

Code from the documentation:

```auto
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'microsoft/DialoGPT-medium',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.6.1',
	pytorch_version='1.7.1',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?",
	}
})

```

I can make those lists into `Conversation` objects, but even still I get an error. I am not even sure they will be json serializable anymore.

The larger issue is that I want to use a finetuned `Microsoft/DialoGPT-medium` that I trained on my local machine. I am following a Hugging Face youtube tutorial for that one, but once again the code presented in the video does not work out of the box.

```auto
# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz", # path to your trained SageMaker model
   role=role, # IAM role with permissions to create an endpoint
   transformers_version="4.6", # Transformers version used
   pytorch_version="1.7", # PyTorch version used
   py_version='py36', # Python version used
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
   initial_instance_count=1,
   instance_type="ml.m5.xlarge"
)

```

This fails with the following error:

```auto
ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{
  "code": 400,
  "type": "InternalServerException",
  "message": "(\"You need to define one of the following [\u0027feature-extraction\u0027, \u0027text-classification\u0027, \u0027token-classification\u0027, \u0027question-answering\u0027, \u0027table-question-answering\u0027, \u0027fill-mask\u0027, \u0027summarization\u0027, \u0027translation\u0027, \u0027text2text-generation\u0027, \u0027text-generation\u0027, \u0027zero-shot-classification\u0027, \u0027conversational\u0027, \u0027image-classification\u0027] as env \u0027TASK\u0027.\", 403)"

```

This is my fourth time deploying a Hugging Face model to AWS Sagemaker, and the process is so incredibly complex and non-intuitive. I feel like the entire process is held up by toothpicks. Thanks for any light you can shed.

---

<div class="post-metadata">

**Author:** ![CyranoB](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyranob/32/3632_2.png) [@CyranoB](https://discuss.huggingface.co/u/CyranoB)\
**Post date:** [January 5, 2022, 10:29pm UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/2 "2022-01-05T22:29:33Z")

</div>

I think you forgot to add the env task when creating your HuggingFaceModel:

```auto
huggingface_model = HuggingFaceModel(
   model_data="s3://xxxxxxxxxxxxxxx/model.tar.gz", # path to your trained SageMaker model
   role=role, # IAM role with permissions to create an endpoint
   transformers_version="4.6", # Transformers version used
   pytorch_version="1.7", # PyTorch version used
   py_version='py36', # Python version used
   env={ 'HF_TASK':'conversational' },
)

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [January 7, 2022, 8:06am UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/3 "2022-01-07T08:06:27Z")

</div>

Hey @JB2022,

we added support for `conversational` pipeline with a later release. Can you use instead of `transformers_version="4.6"` =\> `4.12` and for `pytorch_version="1.7"` =\> `1.9`.

You can find the whole list of available containers here: [Reference](https://huggingface.co/docs/sagemaker/reference#inference-dlc-overview)

Then your fist code snippet should work.

```python
from sagemaker.huggingface import HuggingFaceModel
import sagemaker

role = sagemaker.get_execution_role()
# Hub Model configuration. https://huggingface.co/models
hub = {
	'HF_MODEL_ID':'microsoft/DialoGPT-medium',
	'HF_TASK':'conversational'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
	transformers_version='4.12',
	pytorch_version='1.9',
	py_version='py36',
	env=hub,
	role=role, 
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
	initial_instance_count=1, # number of instances
	instance_type='ml.m5.xlarge' # ec2 instance type
)

predictor.predict({
	'inputs': {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?",
	}
})

```

---

<div class="post-metadata">

**Author:** ![rachelcorey](https://avatars.discourse-cdn.com/v4/letter/r/258eb7/32.png) [@rachelcorey](https://discuss.huggingface.co/u/rachelcorey)\
**Post date:** [January 12, 2022, 3:46pm UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/4 "2022-01-12T15:46:44Z")

</div>

@JB2022 I would LOVE to have a chat with you… you say you’ve done this four times?! Please tell me how? 🙏 Any chance at all you could DM me on Twitter? My username / twitter page is on my Huggingface profile.

---

<div class="post-metadata">

**Author:** ![Kamaljp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kamaljp/32/15013_2.png) [@Kamaljp](https://discuss.huggingface.co/u/Kamaljp)\
**Post date:** [April 13, 2023, 3:36am UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/5 "2023-04-13T03:36:28Z")

</div>

I also faced the same issue today, and tried @CyranoB idea and decided to add the env variable.  
hub = {  
‘HF\_TASK’:‘text-classification’  
}  
Then attach this value to the HuggingFaceModel class as below  
huggingface\_model = HuggingFaceModel(  
model\_data=“s3://xxxxxxxxxxxxxxx/model.tar.gz”,  
role=role,  
transformers\_version=“4.6”,  
pytorch\_version=“1.7”,  
py\_version=‘py36’,  
env=hub,  
)

## This did not work… So I tried the next option. Where I decided to provide the model\_id too

hub = {  
‘HF\_MODEL\_ID’:‘distilbert-base-uncased-finetuned-sst-2-english’,  
‘HF\_TASK’:‘text-classification’  
}  
Then deployed. After that the predict method worked.  
**Ps:** I have not reviewed the underlying code of predict method.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [April 13, 2023, 7:53am UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/6 "2023-04-13T07:53:05Z")

</div>

Hello @Kamaljp could you share how you created the `model.tar.gz`? Also you normally don’t need to add the `HF_TASK` when deploying a `model.tar.gz`.  
You should also update the versions you use to the latest ones, which are 4.26.0, 1.13.1, py39

---

<div class="post-metadata">

**Author:** ![hooman650](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hooman650/32/15594_2.png) [@hooman650](https://discuss.huggingface.co/u/hooman650)\
**Post date:** [April 23, 2023, 9:37pm UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/7 "2023-04-23T21:37:08Z")

</div>

hi @philschmid Is there an easy way to add streaming to conversational endpoints using a bit of customizations? Streaming is quite handy as the models are now larger (Flan-t5 , etc), for this SSE probably is required. I saw this ([GitHub - huggingface/text-generation-inference: Large Language Model Text Generation Inference](https://github.com/huggingface/text-generation-inference)) but it seems to be an over kill. Any advice is really appreciated.

Cheers

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [April 24, 2023, 5:46am UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/8 "2023-04-24T05:46:53Z")

</div>

The SageMaker platform currently doesn’t support SSE (Server Sent Events), which is needed for streaming.

---

<div class="post-metadata">

**Author:** ![Lorius2](https://avatars.discourse-cdn.com/v4/letter/l/c68b51/32.png) [@Lorius2](https://discuss.huggingface.co/u/Lorius2)\
**Post date:** [July 8, 2023, 5:41pm UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/10 "2023-07-08T17:41:36Z")

</div>

hi @philschmid! I see you mentioned that SSE aren’t currently supported by SageMaker Endpoints, however, I was just wondering if this has changed now with the LLM inference container you released for aws ([Introducing the Hugging Face LLM Inference Container for Amazon SageMaker](https://huggingface.co/blog/sagemaker-huggingface-llm)).

Currently looking into options for hosting LLMs with streaming enabled so any inputs are welcome.

Cheers

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [July 13, 2023, 1:00pm UTC](https://discuss.huggingface.co/t/deploying-a-conversational-pipeline-on-aws/13280/11 "2023-07-13T13:00:29Z")

</div>

Inference endpoints is supporting streaming see: [Deploy LLMs with Hugging Face Inference Endpoints](https://huggingface.co/blog/inference-endpoints-llm)