# Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error **URL:** \ **Category:** 🤗Tokenizers\ **Created:** [November 5, 2024, 7:46pm UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874 "2024-11-05T19:46:18Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![antoshka1608](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/antoshka1608/32/32213_2.png) [@antoshka1608](https://discuss.huggingface.co/u/antoshka1608)\ **Post date:** [November 5, 2024, 7:46pm UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/1 "2024-11-05T19:46:18Z")
I’m encountering an issue when trying to load my custom tokenizer from a model repository on the Hugging Face Hub. Despite following the documentation for custom tokenizers. > **[antoshka1608/wordpiece-tokenizer-v1 at main](https://huggingface.co/antoshka1608/wordpiece-tokenizer-v1/tree/main)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. To load the tokenizer, I’m using: from transformers import AutoTokenizer tokenizer = AutoTokenizer.from\_pretrained(“antoshka1608/wordpiece-tokenizer-v1”, use\_fast=False, trust\_remote\_code=True) When loading the tokenizer, it downloads `tokenizer_config.json` and `vocab.json` but then fails with the `Tokenizer class BaseTokenizer does not exist or is not currently imported` error. Has anyone else encountered this issue or have suggestions on what might be going wrong? Any guidance on troubleshooting this would be greatly appreciated! Thank you for your help! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 6, 2024, 4:14am UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/2 "2024-11-06T04:14:47Z")
That error usually occurs when the transoformers library is out of date, but it’s hard to imagine that you’re using a version so old that BaseTokeniker isn’t defined. ```auto pip install -U transformers ``` There could be some other cause, such as the manual now being out of date. ---
**Author:** ![antoshka1608](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/antoshka1608/32/32213_2.png) [@antoshka1608](https://discuss.huggingface.co/u/antoshka1608)\ **Post date:** [November 6, 2024, 7:21am UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/3 "2024-11-06T07:21:11Z")
Sorry, for, probably, misleading you, its not transformer’s tokenizer, but my custom ( i did it, inherited from PreTrainedTokenizer) And yes, i reinstalled transformers mant times, doesnt work. After your reply, i thought that this is might be issue with name duplication, but no. With new name of my tokenizer the same error(but of course with new tokenizer name ) . ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 6, 2024, 7:53am UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/4 "2024-11-06T07:53:46Z")
> **[CohereForAI/c4ai-command-r-plus · ValueError: Tokenizer class CohereTokenizer...](https://huggingface.co/CohereForAI/c4ai-command-r-plus/discussions/25#66556dfe904cd973f5e0c246)** > > Running this example script gives an error Oh, I see. The name is the same as the existing one, and the tokenizer is also a custom one. Have you encountered this bug? Also, in the case of HF in general, there are cases where information that was true at the time is now false in the manual, so I think the easiest thing to do is to refer to the py or json of someone else’s model that is working. It’s tough if there is no similar model… The next best thing is to refer to the code of the library itself. ---
**Author:** ![antoshka1608](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/antoshka1608/32/32213_2.png) [@antoshka1608](https://discuss.huggingface.co/u/antoshka1608)\ **Post date:** [November 6, 2024, 8:03am UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/5 "2024-11-06T08:03:35Z")
Yeah, thank you for you respond, anyway!! Actually, if i import my tokenizer directly from directory of my project. tokenizer = CustomBaseTokenizer.from\_pretrained(hugging\_face\_name) It works good, but probably it takes just vocab file and doesnt use another files, so we cant check anything here. Ill try your idea about other’s works later. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [November 6, 2024, 8:10am UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/6 "2024-11-06T08:10:09Z")
Many of the HF libraries use hard-coded file names, so sometimes they work and sometimes they don’t. If it works locally but not online, the problem is often with the file names, their placement, or the YAML part of README.md (the de facto repository configuration file). Also, in your case, since you’re using a gated model, there’s a chance that the error is occurring because you’re failing to pass the token. Even when calling the tokenizer, you need a token to read the repo. If it seems like a bug in the library, you’ll need to find a way to make the error more visible and identify the bug itself. Or you could find the part that’s causing the problem and bypass it. In this case, I’ll go read the github…🙄 ---
**Author:** ![antoshka1608](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/antoshka1608/32/32213_2.png) [@antoshka1608](https://discuss.huggingface.co/u/antoshka1608)\ **Post date:** [November 6, 2024, 8:17am UTC](https://discuss.huggingface.co/t/issue-with-loading-custom-tokenizer-tokenizer-class-basetokenizer-does-not-exist-or-is-not-currently-imported-error/115874/7 "2024-11-06T08:17:38Z")
Yep, going to do a deep dive later.