# Why does PreTrainedTokenizerFast return a list instead of tokenizers.Encoding instance?

**URL:** <https://discuss.huggingface.co/t/why-does-pretrainedtokenizerfast-return-a-list-instead-of-tokenizers-encoding-instance/31184>\
**Category:** Beginners\
**Created:** [February 6, 2023, 11:53am UTC](https://discuss.huggingface.co/t/why-does-pretrainedtokenizerfast-return-a-list-instead-of-tokenizers-encoding-instance/31184 "2023-02-06T11:53:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vprokopev](https://avatars.discourse-cdn.com/v4/letter/v/5e9695/32.png) [@vprokopev](https://discuss.huggingface.co/u/vprokopev)\
**Post date:** [February 6, 2023, 11:53am UTC](https://discuss.huggingface.co/t/why-does-pretrainedtokenizerfast-return-a-list-instead-of-tokenizers-encoding-instance/31184/1 "2023-02-06T11:53:05Z")

</div>

```auto
from transformers import AutoTokenizer
from tokenizers import Tokenizer
from tokenizers.models import BPE, WordPiece
from tokenizers.trainers import BpeTrainer, WordPieceTrainer
from tokenizers.pre_tokenizers import Whitespace
tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))
trainer = WordPieceTrainer(vocab_size=5000, 
                           special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.pre_tokenizer = Whitespace()
tokenizer.train_from_iterator(texts, trainer)
output = tokenizer.encode("Hello, y'all! How are you 😁 ?")
print(type(output))
tokenizer1 = AutoTokenizer.from_pretrained("bert-base-cased")
output = tokenizer1.encode("Hello, y'all! How are you 😁 ?")
print(type(output))

```

Out:  
\<class ‘tokenizers.Encoding’\>  
\<class ‘list’\>

Isn’t it supposed to be the same object?

Also, how do I get tokens from pretrained AutoTokenizer?

Thank you!

---

<div class="post-metadata">

**Author:** ![vprokopev](https://avatars.discourse-cdn.com/v4/letter/v/5e9695/32.png) [@vprokopev](https://discuss.huggingface.co/u/vprokopev)\
**Post date:** [February 6, 2023, 12:00pm UTC](https://discuss.huggingface.co/t/why-does-pretrainedtokenizerfast-return-a-list-instead-of-tokenizers-encoding-instance/31184/2 "2023-02-06T12:00:48Z")

</div>

I think I found an answer in one of the tutorial videos. I should use tokenizer.beckend\_tokenizer that is an instance of tokenizers.Tokenizer class.