# How to create a tokenizers from a custom pretrained tokenizer?

**URL:** <https://discuss.huggingface.co/t/how-to-create-a-tokenizers-from-a-custom-pretrained-tokenizer/3909>\
**Category:** 🤗Transformers\
**Created:** [February 24, 2021, 1:20am UTC](https://discuss.huggingface.co/t/how-to-create-a-tokenizers-from-a-custom-pretrained-tokenizer/3909 "2021-02-24T01:20:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![johanbjorck](https://avatars.discourse-cdn.com/v4/letter/j/3be4f8/32.png) [@johanbjorck](https://discuss.huggingface.co/u/johanbjorck)\
**Post date:** [February 24, 2021, 1:20am UTC](https://discuss.huggingface.co/t/how-to-create-a-tokenizers-from-a-custom-pretrained-tokenizer/3909/1 "2021-02-24T01:20:25Z")

</div>

I have created a custom tokenizer from the tokenizers library, roughly following ([The tokenization pipeline — tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/pipeline.html)). However, these tokenizers do not have utilities like transforming encoded sentences to torch tensors and so on. For this, I’d want to use the PreTrainedTokenizerFast class. It exposes an interface for getting the tokenizers of various pretrained models from google/facebook/etc, but I want to use my own tokenizer.

How do I create a PreTrainedTokenizerFast from my own tokenizer?

thanks!!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [February 24, 2021, 4:14pm UTC](https://discuss.huggingface.co/t/how-to-create-a-tokenizers-from-a-custom-pretrained-tokenizer/3909/2 "2021-02-24T16:14:59Z")

</div>

cc @anthony on this.