# WordLevel error: Missing \[UNK\] token from the vocabulary

**URL:** <https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107>\
**Category:** 🤗Tokenizers\
**Created:** [March 26, 2021, 12:24am UTC](https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107 "2021-03-26T00:24:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Athena](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/athena/32/1417_2.png) [@Athena](https://discuss.huggingface.co/u/Athena)\
**Post date:** [March 26, 2021, 12:24am UTC](https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107/1 "2021-03-26T00:24:49Z")

</div>

Hi, I am trying to train a basic Word Level tokenizer based on a file `data.txt` containing

```auto
5174 5155 4749 4814 4832 4761 4523 4999 4860 4699 5024 4788 [UNK]

```

When I run my code

```auto
from tokenizers import Tokenizer
from tokenizers.models import WordLevel

tokenizer = Tokenizer(WordLevel(unk_token='[UNK]'))
tokenizer.train(files=['data.txt'])
tokenizer.encode('5155')

```

I get the error

> Exception: WordLevel error: Missing [UNK] token from the vocabulary

Why is it still missing despite having `[UNK]` in `data.txt` and also setting `unk_token='[UNK]'`?

Any help is very appreciated!

---

<div class="post-metadata">

**Author:** ![lesscomfortable](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lesscomfortable/32/2106_2.png) [@lesscomfortable](https://discuss.huggingface.co/u/lesscomfortable)\
**Post date:** [June 17, 2021, 8:48pm UTC](https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107/2 "2021-06-17T20:48:16Z")

</div>

Hi Athena, I’m having the same issue… did you find the root of the problem?

---

<div class="post-metadata">

**Author:** ![echizen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/echizen/32/2576_2.png) [@echizen](https://discuss.huggingface.co/u/echizen)\
**Post date:** [August 26, 2021, 8:02pm UTC](https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107/3 "2021-08-26T20:02:29Z")

</div>

I am experiencing this too

---

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](https://discuss.huggingface.co/u/antoine2323231)\
**Post date:** [September 21, 2022, 10:50am UTC](https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107/4 "2022-09-21T10:50:30Z")

</div>

Having the same issue…

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [October 27, 2022, 9:09am UTC](https://discuss.huggingface.co/t/wordlevel-error-missing-unk-token-from-the-vocabulary/5107/5 "2022-10-27T09:09:12Z")

</div>

```auto
from tokenizers import Tokenizer
from tokenizers.models import WordLevel
from tokenizers.trainers import WordLevelTrainer

tokenizer = Tokenizer(WordLevel(unk_token='[UNK]'))

########## Specify [UNK] here ############
trainer = WordLevelTrainer(
    special_tokens=['[UNK]']
)
##########################################

files = ['./datasets/AAABBBCCC.txt']
tokenizer.train(files, trainer) # <--- specify trainer
tokenizer.encode('41').ids

```