# Deploying Open AI's whisper on Sagemaker **URL:** \ **Category:** Amazon SageMaker\ **Created:** [October 21, 2022, 8:09am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761 "2022-10-21T08:09:15Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\ **Post date:** [October 21, 2022, 8:09am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/1 "2022-10-21T08:09:15Z")
Hi there! I’m trying to deploy Open AI’s [whisper-large](https://huggingface.co/openai/whisper-large) model, using the suggested code snippet on the hub. The actual deployment of the model succeeds on an `ml.m5.xlarge` instance. But when I try to invoke the endpoint as follows, I get the error below. What am I doing wrong? ```auto aws = boto3.Session( region_name="region-goes-here", aws_access_key_id="access-key-goes-here", aws_secret_access_key="secret-access-key-goes-here", ) runtime = aws.client('runtime.sagemaker') with open("some_audio_file.mp3", "rb") as file: audio_file = file.read() response = runtime.invoke_endpoint( EndpointName=ENDPOINT, Body=audio_file, ) ``` ```auto botocore.errorfactory.ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (400) from primary with message "{ "code": 400, "type": "InternalServerException", "message": "\u0027whisper\u0027" } ``` ---
**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\ **Post date:** [October 21, 2022, 8:15am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/2 "2022-10-21T08:15:11Z")
Looking at the logs in CloudWatch, I see the following traceback: ```auto 2022-10-21T08:12:59,629 [INFO] W-9000-openai__whisper-large com.amazonaws.ml.mms.wlm.WorkerThread - Backend response time: 0 2022-10-21T08:12:59,629 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error 2022-10-21T08:12:59,629 [INFO] W-9000-openai__whisper-large ACCESS_LOG - /169.254.178.2:33784 "POST /invocations HTTP/1.1" 400 5 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last): 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 219, in handle 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.initialize(context) 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 77, in initialize 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - self.model = self.load(self.model_dir) 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 104, in load 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf_pipeline = get_pipeline(task=os.environ["HF_TASK"], model_dir=model_dir, device=self.device) 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/sagemaker_huggingface_inference_toolkit/transformers_utils.py", line 272, in get_pipeline 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - hf_pipeline = pipeline(task=task, model=model_dir, device=device, **kwargs) 2022-10-21T08:12:59,630 [INFO] W-openai __whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/transformers/pipelines/__ init__.py", line 541, in pipeline 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config = AutoConfig.from_pretrained(model, revision=revision, _from_pipeline=task, **model_kwargs) 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 657, in from_pretrained 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - config_class = CONFIG_MAPPING[config_dict["model_type"]] 2022-10-21T08:12:59,630 [INFO] W-openai __whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 372, in__ getitem__ 2022-10-21T08:12:59,630 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise KeyError(key) 2022-10-21T08:12:59,631 [INFO] W-openai__whisper-large-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: 'whisper' ``` Could it have something to do with the version of the `transformers` package? I deployed it with version 4.17.0. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [October 21, 2022, 8:42am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/3 "2022-10-21T08:42:00Z")
hey @thusken, `whisper` was added in `transformers==4.23.1` you need to update the version then it should work. ---
**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\ **Post date:** [October 21, 2022, 9:15am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/4 "2022-10-21T09:15:12Z")
Hi @philschmid , Thanks for the reply! Although when I try to deploy to Sagemaker with a newer transformers version, I get the error: ```auto Unsupported huggingface version: 4.23.1. You may need to upgrade your SDK version (pip install -U sagemaker) for newer huggingface versions. Supported huggingface version(s): 4.6.1, 4.10.2, 4.11.0, 4.12.3, 4.17.0, 4.6, 4.10, 4.11, 4.12, 4.17. ``` Doing a `pip install -U sagemaker` doesn’t help. Any tips? ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [October 21, 2022, 2:43pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/5 "2022-10-21T14:43:32Z")
Hi @thusken - I assume your using the SageMaker Hugginface DLC, is that right? If so, the latest available version for those is currently 4.17, see [here](https://huggingface.co/docs/sagemaker/reference#inference-dlc-overview). What you can do, however, is to provide a `requirements.txt` file that will let you specify dependencies you want to be installed in the container. If you put `transformers==4.23.1` into that requirements.txt file then this transformers version will be installed. See [here](https://github.com/aws/sagemaker-huggingface-inference-toolkit#-user-defined-codemodules) for more info. Hope that helps. Cheers Heiko ---
**Author:** ![BatuhanYilmaz](https://avatars.discourse-cdn.com/v4/letter/b/ec9cab/32.png) [@BatuhanYilmaz](https://discuss.huggingface.co/u/BatuhanYilmaz)\ **Post date:** [October 26, 2022, 12:00pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/6 "2022-10-26T12:00:31Z")
Hey @thusken , I’m also trying to deploy a Whisper powered app. Could you make it work? ---
**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\ **Post date:** [October 26, 2022, 2:19pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/7 "2022-10-26T14:19:59Z")
I haven’t had time since last week to look at this, so unfortunately I can’t help you here. Also, thanks for the reply @marshmellow77 ! I guess that approach also involves creating a separate `inference.py` file as shown in the example? So far I just deployed the model straight from the hub. ---
**Author:** ![BatuhanYilmaz](https://avatars.discourse-cdn.com/v4/letter/b/ec9cab/32.png) [@BatuhanYilmaz](https://discuss.huggingface.co/u/BatuhanYilmaz)\ **Post date:** [October 27, 2022, 11:39am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/8 "2022-10-27T11:39:08Z")
I see. Thanks for the reply. ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [October 28, 2022, 3:38pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/9 "2022-10-28T15:38:12Z")
To use the latest version of the `transformers` library in a SageMaker DLC you don’t have to provide a custome inference script, just a `requirements.txt` file with a line that says `transformers==4.23.1`. The DLC will then install/update to the specified transformers version. This [example](https://github.com/huggingface/notebooks/blob/main/sagemaker/17_custom_inference_script/sagemaker-notebook.ipynb) & this [documentation](https://github.com/aws/sagemaker-huggingface-inference-toolkit#-user-defined-codemodules) show how to do that, hope that helps. ---
**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\ **Post date:** [November 1, 2022, 2:00pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/10 "2022-11-01T14:00:24Z")
Thanks for the input @marshmellow77 ! I managed to get a few steps further in deploying the model, using the examples you linked. For those interested, here’s my current deployment code snippet: ```nohighlight # create Hugging Face Model Class huggingface_model = HuggingFaceModel( model_data=s3_location, # path to your model and script role=role, # iam role with permissions to create an Endpoint transformers_version='4.17.0', pytorch_version='1.10.2', py_version='py38', # python version used env={ 'HF_MODEL_ID':'openai/whisper-large', 'HF_TASK':'automatic-speech-recognition' } ) # deploy the endpoint endpoint predictor = huggingface_model.deploy( initial_instance_count=1, instance_type="ml.p2.xlarge" ) ``` Here, the `s3_location` variable contains the location of the model archive: ```auto repository = "openai/whisper-large" model_id=repository.split("/")[-1] s3_location=f"s3://{sess.default_bucket()}/custom_inference/{model_id}/model.tar.gz" ``` This model archive is created by the following bash commands (executed from a Sagemaker notebook): ```auto !git lfs install !git clone https://huggingface.co/$repository !cp -r code/ $model_id/code/ %cd $model_id !tar zcvf model.tar.gz * !aws s3 cp model.tar.gz $s3_location ``` Note that you need to add a `requirements.txt` file with `transformers==4.23.1` in the `code` folder after you execute the `git clone` command for this to work. The endpoint is now able to load the Whisper model, which of course a big step forward, but I’m as of now not yet able to properly call the endpoint. For example, if I load a small audio file and try to predict it as follows, ```auto from transformers.pipelines.automatic_speech_recognition import ffmpeg_read SAMPLING_RATE = 1000 with open("some_audio_file.mp3", "rb") as file: audio_file = file.read() audio_nparray = ffmpeg_read(audio_file, SAMPLING_RATE) predictor.predict({ "raw": audio_nparray, "sampling_rate": SAMPLING_RATE }) ``` the following error is raised: ```auto { "code": 400, "type": "InternalServerException", "message": "expected np.ndarray (got list)" } ``` Even though `audio_nparray` is actually a NumPy array: ```auto type(audio_nparray) numpy.ndarray ``` ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [November 4, 2022, 8:21am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/11 "2022-11-04T08:21:13Z")
@thusken I wasn’t able to reproduce your error message, but I wanted to let you know that your approach of sending an NP array to the endpoint is probably not the best idea. The reason for that is that SM endpoints have a payload size limit of 6 MB: ![Screenshot 2022-11-04 at 08.10.24](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de5fe0a497820374cb222b1147cd731ef9c4e68b.png) (from [Amazon SageMaker endpoints and quotas - AWS General Reference](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html)) What you could do instead is using a [DataSerializer](https://sagemaker.readthedocs.io/en/stable/api/inference/serializers.html#sagemaker.serializers.DataSerializer) to stream the raw file to the endpoint. You would have to sepcify the serializer when deploying the model, see [here](https://sagemaker.readthedocs.io/en/stable/frameworks/huggingface/sagemaker.huggingface.html#sagemaker.huggingface.model.HuggingFaceModel.deploy). ---
**Author:** ![sohoha](https://avatars.discourse-cdn.com/v4/letter/s/3bc359/32.png) [@sohoha](https://discuss.huggingface.co/u/sohoha)\ **Post date:** [November 4, 2022, 8:29am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/12 "2022-11-04T08:29:37Z")
@thusken According to this [notebook](https://github.com/huggingface/notebooks/blob/main/sagemaker/20_automatic_speech_recognition_inference/sagemaker-notebook.ipynb), you should specify DataSerializer to serialize data. ```python hub = { 'HF_MODEL_ID':'openai/whisper-base', 'HF_TASK':'automatic-speech-recognition' } # create Hugging Face Model Class huggingface_model = HuggingFaceModel( model_data = s3_location, transformers_version='4.17.0', pytorch_version='1.10.2', py_version='py38', env=hub, role=role, ) # deploy model to SageMaker Inference audio_serializer = DataSerializer(content_type='audio/x-audio') predictor = huggingface_model.deploy( initial_instance_count=1, # number of instances instance_type='ml.m5.xlarge', # ec2 instance type serializer=audio_serializer ) ``` Based on that [issue](https://github.com/aws/sagemaker-huggingface-inference-toolkit/issues/73), you can use serializers. > The Hugging Face inference toolkit supports all the `transformers` pipelines with their default inputs. The Toolkit implements several serializers to parse binary data, e.g., audio or images to the matching format for the transformers pipeline, e.g., ` PIL` or `np`. So, for inference below code worked in my environment. ```python audio_path = "sample1.flac" res = predictor.predict(data=audio_path) print(res) ``` ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [November 4, 2022, 8:32am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/13 "2022-11-04T08:32:59Z")
Great example @sohoha , thanks for sharing 👍 ---
**Author:** ![thusken](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thusken/32/27787_2.png) [@thusken](https://discuss.huggingface.co/u/thusken)\ **Post date:** [November 22, 2022, 10:19am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/14 "2022-11-22T10:19:05Z")
Sorry for the late reply, but this indeed gets it working! Thanks for the useful information! ---
**Author:** ![amitkayal](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@amitkayal](https://discuss.huggingface.co/u/amitkayal)\ **Post date:** [November 25, 2022, 1:49pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/15 "2022-11-25T13:49:15Z")
@sohoha Thanks a lot for all your guidance. I tried the same approach but unfortunately it is failing with "error **An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (413) from primary and could not load the entire response body.". I then looked into CW and see the error** I did more analysis and dont see any code folder into the path $model\_id/code/ . Does this means the repo structure has been changed now? @thusken did you face similar error? > 2022-11-25T13:30:27,599 [INFO] W-openai\_\_whisper-base-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - raise KeyError(key) > 2022-11-25T13:30:27,599 [INFO] W-openai\_\_whisper-base-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - KeyError: ‘whisper’ ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [November 25, 2022, 2:51pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/16 "2022-11-25T14:51:53Z")
@amitkayal, this error usually appears if you use a transformers version that doesn’t support the Whisper model, see also the beginning of this thread. Can you confirm you upgraded transformer library to a version that support Whisper (see also this post: [Deploying Open AI's whisper on Sagemaker - #9 by marshmellow77](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/9))? ---
**Author:** ![amitkayal](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@amitkayal](https://discuss.huggingface.co/u/amitkayal)\ **Post date:** [November 25, 2022, 3:21pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/17 "2022-11-25T15:21:05Z")
> [@marshmellow77](#): > > 4.23.1 @marshmellow77 @thusken Yes, I am using transformer version 4.23.1 in my requirements.txt file. Do I also need inference.py file? I believe we dont need that. ## The log has error “”" ModelError Traceback (most recent call last) /tmp/ipykernel\_11082/3757015611.py in ----\> 1 res = predictor.predict(data=audio\_path) 2 print(res) ~/anaconda3/envs/amazonei\_pytorch\_latest\_p37/lib/python3.7/site-packages/sagemaker/predictor.py in predict(self, data, initial\_args, target\_model, target\_variant, inference\_id) 159 data, initial\_args, target\_model, target\_variant, inference\_id 160 ) → 161 response = self.sagemaker\_session.sagemaker\_runtime\_client.invoke\_endpoint(\*\*request\_args) 162 return self.\_handle\_response(response) 163 ~/anaconda3/envs/amazonei\_pytorch\_latest\_p37/lib/python3.7/site-packages/botocore/client.py in \_api\_call(self, \*args, \*\*kwargs) 528 ) 529 # The “self” in this scope is referring to the BaseClient. → 530 return self.\_make\_api\_call(operation\_name, kwargs) 531 532 \_api\_call. **name** = str(py\_operation\_name) ~/anaconda3/envs/amazonei\_pytorch\_latest\_p37/lib/python3.7/site-packages/botocore/client.py in \_make\_api\_call(self, operation\_name, api\_params) 958 error\_code = parsed\_response.get(“Error”, {}).get(“Code”) 959 error\_class = self.exceptions.from\_code(error\_code) → 960 raise error\_class(parsed\_response, operation\_name) 961 else: 962 return parsed\_response ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (413) from primary and could not load the entire response body. See [https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/vanila-openai-whisper-tiny-2022-11-25-15-29-02](https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/vanila-openai-whisper-tiny-2022-11-25-15-29-02) in account 015267333920 for more information. '“”" However, i could not use transformer 4.23.1 version in the following code for HuggingFaceModel object as this is throwing me error "“Unsupported huggingface version: 4.23.1. You may need to upgrade your SDK version (pip install -U sagemaker) for newer huggingface versions. Supported huggingface version(s): 4.6.1, 4.10.2, 4.11.0, 4.12.3, 4.17.0, 4.6, 4.10, 4.11, 4.12, 4.17”. Do you think this is causing issue? _from sagemaker.huggingface.model import HuggingFaceModel_ _from sagemaker.serializers import DataSerializer_ _hub = {_ - ‘HF\_MODEL\_ID’:‘openai/whisper-base’,\* - ‘HF\_TASK’:‘automatic-speech-recognition’\* _}_ _huggingface\_model = HuggingFaceModel(_ - env=hub, # configuration for loading model from Hub\* - role=role, # iam role with permissions to create an Endpoint\* - model\_data = s3\_location,\* - transformers\_version=“4.23.1”, # transformers version used\* - pytorch\_version=“1.10.2”, # pytorch version used\* - py\_version=‘py38’, # python version used\* _)_ Thanks ---
**Author:** ![amitkayal](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@amitkayal](https://discuss.huggingface.co/u/amitkayal)\ **Post date:** [November 28, 2022, 10:16am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/18 "2022-11-28T10:16:08Z")
I found the issue and RCA is that input size was more than 6MB and so this issue happened. How I can do workaround this? Is batch transform only way to avoid the issue? I also wanted to know how i can call this endpoint later as that time i will not have predictor object available. Thanks ---
**Author:** ![sohoha](https://avatars.discourse-cdn.com/v4/letter/s/3bc359/32.png) [@sohoha](https://discuss.huggingface.co/u/sohoha)\ **Post date:** [November 29, 2022, 7:16am UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/19 "2022-11-29T07:16:23Z")
Sorry for late response. Though you should customize inference.py, you can use real-time endpoint. When inference, you upload your audio file to S3 and pass that path to the endpoint. Then, in inference.py you can implement logics to download specified file from S3 and recognize speech. Or, you can split your audio file into less than 6MB and merge the results after inference. ---
**Author:** ![yugaljain1999](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yugaljain1999/32/12790_2.png) [@yugaljain1999](https://discuss.huggingface.co/u/yugaljain1999)\ **Post date:** [November 29, 2022, 3:53pm UTC](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761/20 "2022-11-29T15:53:38Z")
Hello @marshmellow77 @sohoha I tried replicating this approach but every time whenever even I upload video or audio of size less than 6MB of length 1 minute, it just output transcription of 30 seconds. Can we vary this 30 second limitation atleast for 6MB of input video size? Hoping to get reply back soon. Thanks [Next page](https://discuss.huggingface.co/t/deploying-open-ais-whisper-on-sagemaker/24761.md?page=2)