# ModelError when I run predict after deploying gpt-j for question answering **URL:** \ **Category:** Amazon SageMaker\ **Created:** [February 24, 2023, 10:50pm UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-gpt-j-for-question-answering/32472 "2023-02-24T22:50:13Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![arjunrd01](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@arjunrd01](https://discuss.huggingface.co/u/arjunrd01)\ **Post date:** [February 24, 2023, 10:50pm UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-gpt-j-for-question-answering/32472/1 "2023-02-24T22:50:13Z")
Here’s my code: ```auto from sagemaker.huggingface import HuggingFaceModel hub = { 'HF_MODEL_ID':'EleutherAI/gpt-j-6B', # model_id from hf.co/models 'HF_TASK':'question-answering' # NLP task you want to use for predictions } # create Hugging Face Model Class huggingface_model = HuggingFaceModel( env=hub, role=role, # iam role with permissions to create an Endpoint transformers_version="4.6", # transformers version used pytorch_version="1.7", # pytorch version used py_version="py36", # python version of the DLC ) predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.g4dn.12xlarge" ) predictor.predict({ 'inputs': "How are you?" }) ``` ## And here’s the error thrown on predict ModelError Traceback (most recent call last) in 1 predictor.predict({ ----\> 2 ‘inputs’: “How are you?” 3 }) /opt/conda/lib/python3.7/site-packages/sagemaker/predictor.py in predict(self, data, initial\_args, target\_model, target\_variant, inference\_id) 159 data, initial\_args, target\_model, target\_variant, inference\_id 160 ) → 161 response = self.sagemaker\_session.sagemaker\_runtime\_client.invoke\_endpoint(\*\*request\_args) 162 return self.\_handle\_response(response) 163 /opt/conda/lib/python3.7/site-packages/botocore/client.py in \_api\_call(self, \*args, \*\*kwargs) 355 “%s() only accepts keyword arguments.” % py\_operation\_name) 356 # The “self” in this scope is referring to the BaseClient. → 357 return self.\_make\_api\_call(operation\_name, kwargs) 358 359 \_api\_call. **name** = str(py\_operation\_name) /opt/conda/lib/python3.7/site-packages/botocore/client.py in \_make\_api\_call(self, operation\_name, api\_params) 674 error\_code = parsed\_response.get(“Error”, {}).get(“Code”) 675 error\_class = self.exceptions.from\_code(error\_code) → 676 raise error\_class(parsed\_response, operation\_name) 677 else: 678 return parsed\_response ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ “code”: 400, “type”: “InternalServerException”, “message”: “\u0027gptj\u0027” } ". See [https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2023-02-24-22-12-33-444](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/huggingface-pytorch-inference-2023-02-24-22-12-33-444) in account 748895571360 for more information. Lastly here are the cloudwatch logs, pls help! Thank you so much 2023-02-24 22:33:03,967 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error 2023-02-24 22:33:03,967 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2023-02-24 22:33:03,967 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 207, in handle 2023-02-24 22:33:03,967 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.initialize(context) 2023-02-24 22:33:03,967 [INFO] W-9000-EleutherAI\_\_gpt-j-6B com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 2 2023-02-24 22:33:03,967 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 78, in initialize 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model\_dir) 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 105, in load 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = get\_pipeline(task=os.environ[“HF\_TASK”], model\_dir=model\_dir, device=self.device) 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/transformers\_utils.py”, line 230, in get\_pipeline 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = pipeline(task=task, model=model\_dir, tokenizer=model\_dir, device=device, \*\*kwargs) 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/pipelines/ **init**.py”, line 389, in pipeline 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - framework, model = infer\_framework\_from\_model(model, targeted\_task, revision=revision, task=task) 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/pipelines/base.py”, line 88, in infer\_framework\_from\_model 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model = model\_class.from\_pretrained(model, \*\*model\_kwargs) 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/models/auto/auto\_factory.py”, line 376, in from\_pretrained 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - pretrained\_model\_name\_or\_path, return\_unused\_kwargs=True, \*\*kwargs 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/models/auto/configuration\_auto.py”, line 432, in from\_pretrained 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config\_class = CONFIG\_MAPPING[config\_dict[“model\_type”]] 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: ‘gptj’ 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/mms/service.py”, line 108, in predict 2023-02-24 22:33:03,968 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self.\_entry\_point(input\_batch, self.context) 2023-02-24 22:33:03,968 [INFO] W-9000-EleutherAI\_\_gpt-j-6B ACCESS\_LOG - /169.254.178.2:56264 “POST /invocations HTTP/1.1” 400 5 2023-02-24 22:33:03,969 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 231, in handle 2023-02-24 22:33:03,969 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400) 2023-02-24 22:33:03,969 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: ‘gptj’ : 400 ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [February 25, 2023, 7:31am UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-gpt-j-for-question-answering/32472/2 "2023-02-25T07:31:19Z")
To my knowledge, GPT-J doesn’t support question-answering. You can find suitable Q&A models here: [Models - Hugging Face](https://huggingface.co/models?pipeline_tag=question-answering) Vice versa, you can see the supported task for a given model in the model card, e.g. for GPT-J: [EleutherAI/gpt-j-6B · Hugging Face](https://huggingface.co/EleutherAI/gpt-j-6B) As you can see in the top left, GPT-J only supports text generation. Hope that helps. Cheers Heiko ---
**Author:** ![arjunrd01](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@arjunrd01](https://discuss.huggingface.co/u/arjunrd01)\ **Post date:** [February 27, 2023, 10:11pm UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-gpt-j-for-question-answering/32472/3 "2023-02-27T22:11:36Z")
Hey there thanks for responding! I changed HF\_TASK to text-generation but am still running into an issue. I’ve pasted the cloudwatch logs below. Thanks again for your help! 2023-02-27 21:52:35,102 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error 2023-02-27 21:52:35,102 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2023-02-27 21:52:35,102 [INFO] W-9000-EleutherAI\_\_gpt-j-6B com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 2 2023-02-27 21:52:35,102 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 207, in handle 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.initialize(context) 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 78, in initialize 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model\_dir) 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 105, in load 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = get\_pipeline(task=os.environ[“HF\_TASK”], model\_dir=model\_dir, device=self.device) 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/transformers\_utils.py”, line 230, in get\_pipeline 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf\_pipeline = pipeline(task=task, model=model\_dir, tokenizer=model\_dir, device=device, \*\*kwargs) 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/pipelines/ **init**.py”, line 389, in pipeline 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - framework, model = infer\_framework\_from\_model(model, targeted\_task, revision=revision, task=task) 2023-02-27 21:52:35,103 [INFO] W-9000-EleutherAI\_\_gpt-j-6B ACCESS\_LOG - /169.254.178.2:46284 “POST /invocations HTTP/1.1” 400 5 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/pipelines/base.py”, line 88, in infer\_framework\_from\_model 2023-02-27 21:52:35,103 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - model = model\_class.from\_pretrained(model, \*\*model\_kwargs) 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/transformers/models/auto/auto\_factory.py”, line 376, in from\_pretrained 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config\_class = CONFIG\_MAPPING[config\_dict[“model\_type”]] 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: ‘gptj’ 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - During handling of the above exception, another exception occurred: 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/mms/service.py”, line 108, in predict 2023-02-27 21:52:35,104 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - ret = self.\_entry\_point(input\_batch, self.context) 2023-02-27 21:52:35,105 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File “/opt/conda/lib/python3.6/site-packages/sagemaker\_huggingface\_inference\_toolkit/handler\_service.py”, line 231, in handle 2023-02-27 21:52:35,105 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise PredictionException(str(e), 400) 2023-02-27 21:52:35,105 [INFO] W-EleutherAI\_\_gpt-j-6B-3-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - mms.service.PredictionException: ‘gptj’ : 400 ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [February 28, 2023, 6:04am UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-gpt-j-for-question-answering/32472/4 "2023-02-28T06:04:39Z")
Can you please try the framework versions of the latest available [Inference DLC](https://huggingface.co/docs/sagemaker/reference#inference-dlc-overview)? I.e. ```auto huggingface_model = HuggingFaceModel( env=hub, role=role, # iam role with permissions to create an Endpoint transformers_version="4.17", # transformers version used pytorch_version="1.10", # pytorch version used py_version="py38", # python version of the DLC ) ``` ---
**Author:** ![arjunrd01](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@arjunrd01](https://discuss.huggingface.co/u/arjunrd01)\ **Post date:** [February 28, 2023, 11:30pm UTC](https://discuss.huggingface.co/t/modelerror-when-i-run-predict-after-deploying-gpt-j-for-question-answering/32472/5 "2023-02-28T23:30:56Z")
I’m assuming I would have to switch my python version in sagemaker studio to be able to specify py38 rather than py36 without throwing an error. How exactly should I go about that? Thanks again for your help, I’m really grateful