# Longformer and sentiment analysis

**URL:** <https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416>\
**Category:** Beginners\
**Created:** [August 23, 2021, 5:51pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416 "2021-08-23T17:51:44Z")\
**Posts on this page:** 1\
**Showing post:** 6

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [August 24, 2021, 7:00pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/6 "2021-08-24T19:00:48Z")

</div>

Models like BERT, RoBERTa, etc. all take a max sequence length of 512 tokens. Note that these models use _subword_ tokenization, which means that a given word might be tokenized into several tokens, so in practice these models can take in less than 500 words. So if you really want to use the pipeline API with a very long text, you can use models like [LongFormer](https://huggingface.co/transformers/model_doc/longformer.html) or [BigBird](https://huggingface.co/transformers/model_doc/bigbird.html), which can handle 4096 tokens in a single forward pass.

As an alternative (if you still want to use a model like BERT/RoBERTa), you can implement it using a sliding window approach. You can do this using the tokenizer, like so:

```auto
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "a very long text"
encoding = tokenizer(text, return_overflowing_tokens=True, stride=128)

```

This will create multiple training examples (encodings) for a given text, by sliding a window (each having 128 tokens of overlap) across the text. You can then feed each training example through the model, and average the predictions.

---

_[View the full topic](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416)._