# Xlm-Roberta Tokenizing

**URL:** <https://discuss.huggingface.co/t/xlm-roberta-tokenizing/3267>\
**Category:** 🤗Transformers\
**Created:** [January 18, 2021, 5:43pm UTC](https://discuss.huggingface.co/t/xlm-roberta-tokenizing/3267 "2021-01-18T17:43:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Seva](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@Seva](https://discuss.huggingface.co/u/Seva)\
**Post date:** [January 18, 2021, 5:43pm UTC](https://discuss.huggingface.co/t/xlm-roberta-tokenizing/3267/1 "2021-01-18T17:43:01Z")

</div>

Hello everyone. How would you solve the following problem:  
I have a word and a sentence with this word, I also have the position of this word. How would you universally for all languages find the position of a tokenized word in a tokenized sentence?  
I really have a problem with this one, it’s mainly with a Chinese language

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [January 19, 2021, 5:59am UTC](https://discuss.huggingface.co/t/xlm-roberta-tokenizing/3267/2 "2021-01-19T05:59:50Z")

</div>

cc @Narsil, @anthony

---

<div class="post-metadata">

**Author:** ![Narsil](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/narsil/32/40029_2.png) [@Narsil](https://discuss.huggingface.co/u/Narsil)\
**Post date:** [January 19, 2021, 8:52am UTC](https://discuss.huggingface.co/t/xlm-roberta-tokenizing/3267/3 "2021-01-19T08:52:13Z")

</div>

Hi @Seva,

If you use `tokenizers` (which are FastTokenizers within `transformers`.) then you have offsets whenever you call `encode` (if you use `transformers` you need to actually call `tokenizer._tokenizer.encode` to call the underlying `tokenizers` object).

```python
tokenizer = AutoTokenizer.from_pretrained('roberta-large')

encoded = tokenizer._tokenizer.encode("This is a test")
encoded.offsets
# [(0, 0), (0, 4), (5, 7), (8, 9), (10, 14), (0, 0)]
# <s>, "This", "is", "a", "test", </s>

```

So from that you should be able to recover your information.  
This information is available through `Fast Tokenizer` method calls too.

---

<div class="post-metadata">

**Author:** ![anthony](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anthony/32/2505_2.png) [@anthony](https://discuss.huggingface.co/u/anthony)\
**Post date:** [January 19, 2021, 5:24pm UTC](https://discuss.huggingface.co/t/xlm-roberta-tokenizing/3267/4 "2021-01-19T17:24:03Z")

</div>

You can have a look at [the doc for `BatchEncoding` in `transformers`](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.BatchEncoding) which is the type returned from the tokenizers methods (`encode_plus`, `batch_encode_plus`, ` __call__ `).

There are a bunch of helpers available to easily map between characters, words, and tokens.