# Character level tokenizer with specific order

**URL:** <https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150>\
**Category:** 🤗Tokenizers\
**Created:** [February 5, 2025, 2:54pm UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150 "2025-02-05T14:54:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [February 5, 2025, 2:54pm UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150/1 "2025-02-05T14:54:36Z")

</div>

Hello everyone,

I trained a character-level model using a custom tokenizer class (not from Hugging Face). Now I want to create a HF tokenizer so users can consume for inference. However, I only see two character-level tokenizers: Canine and ByT5. These tokenizers use “ord” to get the id, instead of a specific order from the vocabulary (e.g., using sort and enumerate). I know ord is more robust, but unfortunately i didn’t train my model with this type of tokenization.

I know I can create a tokenizer class inheriting from PreTrainedTokenizer, but how do I upload it to HF? Do i need to create a PR? I don’t want force users to install my  
library to get the tokenizer.

For reference, my original tokenizer is quite simple:

class Tokenizer(object):  
“”“Convert between strings and index”“”  
def **init** (self, protein\_alphabet=MSA\_ALPHABET, pad=MSA\_PAD, mask=MASK, all\_aas=MSA\_AAS, gap=GAP, start=START,  
stop=STOP, sep=SEP):  
self.alphabet = list(“”.join(protein\_alphabet))  
self.all\_aas = list(“”.join(all\_aas))  
self.pad = pad  
self.mask = mask  
self.gap = gap  
self.start = start  
self.stop = stop  
self.sep = sep  
self.a\_to\_i = {u: i for i, u in enumerate(self.alphabet)}  
self.i\_to\_a = np.array(self.alphabet)

```
def tokenize(self, seq):
    return np.array([self.a_to_i[a] for a in seq[0]]) # for nested lists

def tokenizeMSA(self, seq):
    return np.array([self.a_to_i[a] for a in seq]) # not nested

def untokenize(self, x):
    if torch.is_tensor(x):
        return "".join([self.i_to_a[int(t.item())] for t in x])
    else:
        return "".join([self.i_to_a[t] for t in x])

```

Thanks !

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 5, 2025, 3:14pm UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150/2 "2025-02-05T15:14:52Z")

</div>

Hello. I found a guide for adding models (not tokenizers but maybe similar case). It seems that the procedure is done on github. It doesn’t seem to be as difficult as I thought.

> **[How to add a model to 🤗 Transformers?](https://huggingface.co/docs/transformers/add_new_model)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [February 5, 2025, 5:03pm UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150/3 "2025-02-05T17:03:26Z")

</div>

I see, thank you. So there is no simple way of reusing what’s already available in HF library ?

---

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [February 5, 2025, 5:16pm UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150/4 "2025-02-05T17:16:40Z")

</div>

@John6666 Do you think it’s possible to upload the code in my HF repo and use trust\_remote\_code to use the tokenizer?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 6, 2025, 6:29am UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150/5 "2025-02-06T06:29:39Z")

</div>

I see. It probably works. Unless you want to incorporate it as a new architecture into the code of the Hugging Face library itself, it seems like you can use it from AutoClass by writing **tokenizer\_config.json** and setting **trust\_remote\_code=True**.

> **[Building custom models](https://huggingface.co/docs/transformers/custom_models)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

[AutoTokenizer で chiTra トークナイザを読み込む #transformers - Qiita](https://qiita.com/mh-northlander/items/0b543edfec2e341bd4a0) (in Japanese)

---

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [February 7, 2025, 4:49pm UTC](https://discuss.huggingface.co/t/character-level-tokenizer-with-specific-order/139150/6 "2025-02-07T16:49:09Z")

</div>

@John6666  
I’m trying this but failing with this error: [Custom Tokenizer Error - Please Help!](https://discuss.huggingface.co/t/custom-tokenizer-error-please-help/139539)

Any idea?