# How are the inputs tokenized when model deployment?

**URL:** <https://discuss.huggingface.co/t/how-are-the-inputs-tokenized-when-model-deployment/9692>\
**Category:** Amazon SageMaker\
**Created:** [September 2, 2021, 1:39pm UTC](https://discuss.huggingface.co/t/how-are-the-inputs-tokenized-when-model-deployment/9692 "2021-09-02T13:39:10Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 2, 2021, 3:19pm UTC](https://discuss.huggingface.co/t/how-are-the-inputs-tokenized-when-model-deployment/9692/6 "2021-09-02T15:19:46Z")

</div>

Training and Inference are two completely different things. You are using the same tokenizer, but not the same configuration.

First of all, it is not possible to predict a longer sentence than `512` with the model you use. Meaning you can either use a model, which supports a longer input sequence, e.g. [longofrmer](https://huggingface.co/allenai/longformer-base-4096) or you can truncate your inputs in advance, so sending only inputs smaller than \< 512.

Additionally, you could send in the `parameters` key of your request configuration to automatically truncate any incoming sequence, meaning the inference pipeline would automatically cut after 512 tokens.

```python
long_sentence = "...." # longer than 512 tokens
sentiment_input= {
   {'inputs':long_sentence,
    'parameters': {'truncation':True}
   }
predictor.predict(sentiment_input)

```

---

_[View the full topic](https://discuss.huggingface.co/t/how-are-the-inputs-tokenized-when-model-deployment/9692)._