# Tokenizer.encode not returning encodings

**URL:** <https://discuss.huggingface.co/t/tokenizer-encode-not-returning-encodings/10616>\
**Category:** 🤗Tokenizers\
**Created:** [October 8, 2021, 1:25pm UTC](https://discuss.huggingface.co/t/tokenizer-encode-not-returning-encodings/10616 "2021-10-08T13:25:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![astarostap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astarostap/32/2836_2.png) [@astarostap](https://discuss.huggingface.co/u/astarostap)\
**Post date:** [October 8, 2021, 1:25pm UTC](https://discuss.huggingface.co/t/tokenizer-encode-not-returning-encodings/10616/1 "2021-10-08T13:25:23Z")

</div>

How do you get token encodings? The method isn’t working in this case.

I need the token offsets in order to translate my labels to a normal list of token tags (my labels are in this format [{start\_index: int, end\_index: int, tag: str} …]

Thank you!

Maybe it’s because this is a pre-trained fast tokenizer?

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/8e33d23a0fd30ab720b61e9fd81a66ee58c3d133.png)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 8, 2021, 2:03pm UTC](https://discuss.huggingface.co/t/tokenizer-encode-not-returning-encodings/10616/2 "2021-10-08T14:03:13Z")

</div>

You need to use the tokenizer directly on your text, not the `encode` method:

```auto
tokenizer("hello world")

```

---

<div class="post-metadata">

**Author:** ![astarostap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astarostap/32/2836_2.png) [@astarostap](https://discuss.huggingface.co/u/astarostap)\
**Post date:** [October 9, 2021, 2:03am UTC](https://discuss.huggingface.co/t/tokenizer-encode-not-returning-encodings/10616/3 "2021-10-09T02:03:14Z")

</div>

Thank you Sylvain! That worked, I also had to pass return\_offset\_mapping=True

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/0/0eaa072193fee9375a1d6902927dd6c45540a93b.png)