# Certain words don't work with bert?

**URL:** <https://discuss.huggingface.co/t/certain-words-dont-work-with-bert/6562>\
**Category:** Beginners\
**Created:** [June 3, 2021, 11:30pm UTC](https://discuss.huggingface.co/t/certain-words-dont-work-with-bert/6562 "2021-06-03T23:30:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![close](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@close](https://discuss.huggingface.co/u/close)\
**Post date:** [June 3, 2021, 11:30pm UTC](https://discuss.huggingface.co/t/certain-words-dont-work-with-bert/6562/1 "2021-06-03T23:30:01Z")

</div>

hi, I was trying to run bert but was getting the error “IndexError: index out of range in self”. after troubleshooting for a couple of days I figured out it was the word “screwing” that was breaking my code. is this a bug or is there certain words you cant use with bert? or am I just doing something wrong? thanks.

here’s the code example:

import transformers  
import torch  
import torch.nn as nn

class SentimentClassifier(nn.Module):  
def **init** (self, n\_classes):  
super(SentimentClassifier, self). **init** ()  
self.bert = transformers.BertModel.from\_pretrained(‘bert-base-cased’)  
self.drop = nn.Dropout(p=0.3)  
self.out = nn.Linear(self.bert.config.hidden\_size, n\_classes)

```
def forward(self, input_ids, mask):
    output = self.bert(
      input_ids=input_ids,
      attention_mask=mask
    )
    output = self.drop(output['pooler_output'])
    return self.out(output)

```

# this doesn’t work

batch\_sentences = [  
‘screwing’,  
]

# running this works

batch\_sentences2 = [  
‘this is a test sentence’,  
‘another one’  
]

bert\_model = transformers.BertModel.from\_pretrained(‘bert-base-uncased’)  
tokenizer = transformers.BertTokenizer.from\_pretrained(‘bert-base-uncased’)

encoded\_inputs = tokenizer(batch\_sentences, padding=True, truncation=True, add\_special\_tokens=True)  
samples = torch.tensor(encoded\_inputs[‘input\_ids’])  
targets = torch.zeros(samples.shape[0]).long()  
mask = (samples != 0)

print(samples.shape)  
model = SentimentClassifier(3)  
EPOCHS = 10  
optimizer = transformers.AdamW(model.parameters(), lr=2e-5, correct\_bias=False)  
total\_steps = 1 \* EPOCHS  
scheduler = transformers.get\_linear\_schedule\_with\_warmup(  
optimizer,  
num\_warmup\_steps=0,  
num\_training\_steps=total\_steps  
)

criterion = nn.CrossEntropyLoss()  
model = model.train()

for i in range(EPOCHS):  
print(i)  
preds = model(input\_ids=samples, mask=mask)  
loss = criterion(preds, targets)  
print(loss)  
loss.backward()  
nn.utils.clip\_grad\_norm\_(model.parameters(), max\_norm=1.0)  
optimizer.step()  
scheduler.step()  
optimizer.zero\_grad()

---

<div class="post-metadata">

**Author:** ![ehalit](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ehalit/32/28854_2.png) [@ehalit](https://discuss.huggingface.co/u/ehalit)\
**Post date:** [June 15, 2021, 5:36am UTC](https://discuss.huggingface.co/t/certain-words-dont-work-with-bert/6562/2 "2021-06-15T05:36:20Z")

</div>

I hope you’ve figured out the solution already but it seems that the tokenizer you use `bert-base-uncased` and the model initialized in the `SentimentClassifier` class `bert-base-cased` do not match. There may be overlaps in the vocabulary of the cased and uncased tokenizers which may seem working fine in some cases but the same token ids can decode to totally different text sequences with different tokenizers. In general, using an uncased tokenizer for a cased model or vice versa should always be erroneous, even if there is no error message showing up.

---

<div class="post-metadata">

**Author:** ![close](https://avatars.discourse-cdn.com/v4/letter/c/e9c0ed/32.png) [@close](https://discuss.huggingface.co/u/close)\
**Post date:** [June 15, 2021, 12:38pm UTC](https://discuss.huggingface.co/t/certain-words-dont-work-with-bert/6562/3 "2021-06-15T12:38:09Z")

</div>

yeah, I tried switching so that both were uncased but it didn’t fix it. I ended up just going through the dataset with a batch size of one and if the index error was raised, I would delete the sentence from the dataset. probably not the best way to deal with it but it worked.