# Reused tokenizer returns unk

**URL:** <https://discuss.huggingface.co/t/reused-tokenizer-returns-unk/23358>\
**Category:** 🤗Tokenizers\
**Created:** [September 19, 2022, 8:40pm UTC](https://discuss.huggingface.co/t/reused-tokenizer-returns-unk/23358 "2022-09-19T20:40:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![EmnaBou](https://avatars.discourse-cdn.com/v4/letter/e/ee7513/32.png) [@EmnaBou](https://discuss.huggingface.co/u/EmnaBou)\
**Post date:** [September 19, 2022, 8:40pm UTC](https://discuss.huggingface.co/t/reused-tokenizer-returns-unk/23358/1 "2022-09-19T20:40:44Z")

</div>

Hello  
I’m training a tokenizer from an old one (Bert based)  
the new tokenizer returns [UNK] for words already exist in vocabulary and run correctly with the old tokenizer

```auto
from transformers import AutoTokenizer

old_tokenizer = AutoTokenizer.from_pretrained("asafaya/bert-base-arabic")

tokens = old_tokenizer.tokenize('مع')
tokens

```

returns [‘مع’]  
while with the new tokenizer

```auto
new_tokenizer= old_tokenizer.train_new_from_iterator(training_corpus, 10)
tokens = new_tokenizer.tokenize('مع')
tokens

```

returns [‘[UNK]’]

Can any one help me please !

---

<div class="post-metadata">

**Author:** ![KashMkj18](https://avatars.discourse-cdn.com/v4/letter/k/b77776/32.png) [@KashMkj18](https://discuss.huggingface.co/u/KashMkj18)\
**Post date:** [March 14, 2024, 9:41pm UTC](https://discuss.huggingface.co/t/reused-tokenizer-returns-unk/23358/2 "2024-03-14T21:41:51Z")

</div>

I am also facing similar issue. Were you able to sort it our ?