# Tokenizers offset issue

**URL:** <https://discuss.huggingface.co/t/tokenizers-offset-issue/22759>\
**Category:** Beginners\
**Created:** [September 8, 2022, 12:10pm UTC](https://discuss.huggingface.co/t/tokenizers-offset-issue/22759 "2022-09-08T12:10:02Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Najm](https://avatars.discourse-cdn.com/v4/letter/n/dec6dc/32.png) [@Najm](https://discuss.huggingface.co/u/Najm)\
**Post date:** [September 8, 2022, 12:10pm UTC](https://discuss.huggingface.co/t/tokenizers-offset-issue/22759/1 "2022-09-08T12:10:03Z")

</div>

Hi, I am new to Transformers and have problem with understanding the tokens offseting in Tokenizers.  
I want to use a third-party tokenizer in Spacy and want to import a pre-trained tokenizer directly from “vocab.json” file. The tokenizer that I want to use is “gottbert-base”, a German language model.  
Can someone tell me why offset information is different in the below codes? I want to use the code in variation 1 but for me offseting in variation 2 is correct. How can I solve this issue?  
Variation 1 :

```auto
from tokenizers import ByteLevelBPETokenizer
bpe = ByteLevelBPETokenizer("gottbert/vocab.json", "gottbert/merges.txt")
text = "Ich habe keine Rückmeldung von euch! Schicken Sie mir die Terminbestätigung."
tokens = bpe.encode(text)
print(tokens.tokens)
print(tokens.offsets)

```

Variation 2:

```auto
from transformers import AutoTokenizer

tokenizer_pretrained = AutoTokenizer.from_pretrained("uklfr/gottbert-base", from_slow=True)
text = "Ich habe keine Rückmeldung von euch! Schicken Sie mir die Terminbestätigung."
tokenized = tokenizer_pretrained(text, return_offsets_mapping=True)
print(tokenized.encodings[0].tokens)
print(tokenized.offset_mapping)

```

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1a4aa92fb8d767a73ac4f5bc03018c7957dfe5b6.png)