# How to create a Huggingface tokenizer from a non-Huggingface tokenizer?

**URL:** <https://discuss.huggingface.co/t/how-to-create-a-huggingface-tokenizer-from-a-non-huggingface-tokenizer/5983>\
**Category:** 🤗Tokenizers\
**Created:** [May 4, 2021, 6:01pm UTC](https://discuss.huggingface.co/t/how-to-create-a-huggingface-tokenizer-from-a-non-huggingface-tokenizer/5983 "2021-05-04T18:01:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ChaosNLP](https://avatars.discourse-cdn.com/v4/letter/c/9de0a6/32.png) [@ChaosNLP](https://discuss.huggingface.co/u/ChaosNLP)\
**Post date:** [May 4, 2021, 6:01pm UTC](https://discuss.huggingface.co/t/how-to-create-a-huggingface-tokenizer-from-a-non-huggingface-tokenizer/5983/1 "2021-05-04T18:01:51Z")

</div>

Hello,

The title is self-explanatory. I have access to a published Bert model with its custom tokenizer. I have the vocab file, and python functions that receive a text, tokenize it according to the vocab, and do some post-processing and convert them to IDs acceptable by Bert model.

I would like to transform the custom tokenizer, which is a hassle to work with, to a 🤗 tokenizer so I can use all of amazing functionalities that 🤗 and other 🤗 -based libraries provide. I already managed to transform the Bert model to a 🤗 BertModel, but tokenzier seems to be trickier.

Is there a way I can somehow transform this non-🤗 tokenizer to a 🤗 tokenizer? Is _PreTrainedTokenizer_ what I need to use?