# Adding atomic / indivisible tokens to BPE tokenizer

**URL:** <https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486>\
**Category:** 🤗Tokenizers\
**Created:** [July 2, 2025, 7:55am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486 "2025-07-02T07:55:53Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![GuyShur](https://avatars.discourse-cdn.com/v4/letter/g/ce73a5/32.png) [@GuyShur](https://discuss.huggingface.co/u/GuyShur)\
**Post date:** [July 2, 2025, 7:55am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/1 "2025-07-02T07:55:53Z")

</div>

Hi all,

I am trying to create a BPE tokenizer that has added “atomic” tokens before training, meaning words of length \> 1 that should always be tokenized as one indivisible unit (but possibly merged with other words).

For example, my input text can look like: “HKLDCPHY\<RR\>\<QQ\>PDHGIVMN”, where \<RR\> and \<QQ\> must be treated essentially as single characters. After tokenization they can be their own tokens or merged to something like PHY\<RR\> or \<RR\>\<QQ\> but they should never be split (something like [“PHY”, “\<RR”, “\>\<”, “QQ”, “\>”, “PDH”,…] is bad, for example).

Is there a way to do this?

I would encode the important substrings to single chars but I have a lot of them and past the ASCII characters I am sometimes getting “” appended to tokens containing the rest of the characters, plus I really want the tokens to be Interpretable.

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 2, 2025, 11:43am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/2 "2025-07-02T11:43:04Z")

</div>

Special tokens seem to be considered atomic. However, the implementation of special tokens is quite complex (it has been revised and changed over a long period of time), so it would be safer to search for information while working on it.

> [@Are special\_tokens the only tokens guaranteed to be atomic?](https://discuss.huggingface.co/t/are-special-tokens-the-only-tokens-guaranteed-to-be-atomic/4124):
>
> I’m building a BPE tokenizer from scratch, and I’d like to add some tokens to its vocabulary that are never broken apart and never included in merges. The second part of that (no tokens in the merges) is easy – the tokens don’t appear in the training text I give the tokenizer. But for the first part (never breaking my special token, e.g. [Y], into its [,Y, and] components), I’m less sure of the correct process. I had though I could just add the tokens to the tokenizer using add\_tokens() befo…

> <https://github.com/huggingface/tokenizers/issues/985>
>
> Hi,
> 
> I want to train a tokenizer with code like the following
> \`\`\`
> \# I am not… sure about the correct way, so I try to add '\<sep\>' in every possible way.
> 
> trainer = BpeTrainer(special\_tokens=\["\<unk\>", "\<pad\>", '\<sep\>'\], vocab\_size=vocab\_size, )
> tokenizer.add\_special\_tokens(\['\<sep\>'\])
> tokenizer.add\_tokens(\['\<sep\>'\])
> tokenizer.train\_from\_iterator(trainer=trainer, iterator=iterator\_over\_seqs)
> \`\`\`
> An example sequence is \`ABCD\<sep\>EFGH\`. 
> However the trained vocabulary contains token \`'\<'\`, \`'\>'\` and \`'e'\`, \`'ep'\`, \`'p'\`, \`'s'\`, \`'sep'\`, which are undesired.
> 
> So I'm wondering what should I do to let the tokenizer taking the \`'\<sep\>'\` as a single special token?
> 
> Thanks

---

<div class="post-metadata">

**Author:** ![Felicitywood](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/felicitywood/32/49463_2.png) [@Felicitywood](https://discuss.huggingface.co/u/Felicitywood)\
**Post date:** [July 2, 2025, 11:51am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/3 "2025-07-02T11:51:31Z")

</div>

If I add atomic tokens as special ones, will it stop them from being split?

---

<div class="post-metadata">

**Author:** ![NuralNexus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nuralnexus/32/56230_2.png) [@NuralNexus](https://discuss.huggingface.co/u/NuralNexus)\
**Post date:** [July 2, 2025, 1:02pm UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/4 "2025-07-02T13:02:48Z")

</div>

Yes what you want is a (BPE) tokenizer with added pre-tokenized atomic substrings that Are treated as indivisible units (not split at any point), Can be merged with surrounding tokens (e.g. PHY),  
Are multi-character, but always considered atomic,  
Stay interpretable (not replaced with Unicode hacks or private-use glyphs). Here is how to do this cleanly with modern tokenization tools particularly 🤗 tokenizers (Rust-backed and flexible). That reserves a vocabulary of predefined atomic substrings  
And learns merges on top of those atomic substrings Step-by-Step Implementation with tokenizers Define Your Atomic Units. atomic\_units = [“”, “”, “”, “”, “HKLDCPHY”] These must never be split even if they’re multi-character. Pre-tokenize Your Corpus with Atomic Units as Special Tokens  
Before training, wrap your corpus in a custom pre-tokenizer that, Scans input for atomic units,  
Inserts them as single “tokens” before BPE starts learning. from tokenizers import Tokenizer, models, pre\_tokenizers, trainers, decoders  
from tokenizers.pre\_tokenizers import PreTokenizer  
import re

class AtomicUnitPreTokenizer:  
def **init** (self, atomic\_units):  
self.atomic\_units = sorted(atomic\_units, key=len, reverse=True)  
self.pattern = re.compile(“|”.join(re.escape(a) for a in self.atomic\_units))

```
def pre_tokenize_str(self, input_str):
    def _split(match):
        # Return the atomic unit as its own token
        return f" {match.group(0)} "

    # Inject spaces around atomic units so tokenizers can treat them as isolated
    split_str = self.pattern.sub(_split, input_str)
    # Then split on whitespace
    return [(token, (0, 0)) for token in split_str.strip().split()] plug this into HuggingFace’s tokenizer pipeline Initialize and Train the Tokenizer

```

from tokenizers import Tokenizer  
from tokenizers.models import BPE  
from tokenizers.trainers import BpeTrainer

# Your atomic units

atomic\_units = [“”, “”, “”, “”]

# Tokenizer with custom pre-tokenizer

tokenizer = Tokenizer(BPE(unk\_token=“[UNK]”))  
tokenizer.pre\_tokenizer = AtomicUnitPreTokenizer(atomic\_units)

trainer = BpeTrainer(vocab\_size=1000, special\_tokens=[“[UNK]”] + atomic\_units)

# Corpus containing your sequences

corpus = [“HKLDCPHYPDHGIVMN”, “ABCDE”]  
tokenizer.train\_from\_iterator(corpus, trainer=trainer) Test the Tokenization  
encoded = tokenizer.encode(“HKLDCPHYPDHGIVMN”)  
print(encoded.tokens)  
Good output:

[‘HKLDCPHY’, ‘’, ‘’, ‘PDHGIVMN’]

# Or even: [‘PHY’, ‘’, ‘’, ‘PDH’] — as long as and aren’t split

Bad output:

[‘\<’, ‘RR’, ‘\>’, ‘\<’, ‘QQ’, ‘\>’]  
That’s should be a way to successfully forced atomic tokens into the model without hacking the Unicode space or losing interpretability. The key is treating “” and others as “pre-tokenized” units before training. You don’t need to substitute in weird ASCII or Unicode you’re just teaching the tokenizer to respect and treat them as atomic substrings.  
This integrates well with HuggingFace transformers via PreTrainedTokenizerFast. Add to Special Tokens If you want , , etc. to be treated as special tokens later (e.g. for generation), you can  
tokenizer.add\_special\_tokens([“”, “”, “”, “”])  
If you just look to converting to transformers tokenizer  
from transformers import PreTrainedTokenizerFast  
hf\_tokenizer = PreTrainedTokenizerFast(tokenizer\_object=tokenizer)  
hf\_tokenizer.add\_special\_tokens({“additional\_special\_tokens”: atomic\_units})  
You don’t need to encode them into Unicode glyphs or sacrifice interpretability.  
You just need: A custom pre-tokenizer that isolates atomic unitsA BPE tokenizer trained on top of those preserved tokens Add to special\_tokens for further control

---

<div class="post-metadata">

**Author:** ![GuyShur](https://avatars.discourse-cdn.com/v4/letter/g/ce73a5/32.png) [@GuyShur](https://discuss.huggingface.co/u/GuyShur)\
**Post date:** [July 3, 2025, 9:11am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/5 "2025-07-03T09:11:07Z")

</div>

Thanks @John6666,

Adding these as special tokens works to keep them from fragmenting, but I would actually like these to participate in merges, and special tokens don’t do that. Is there a way to achieve this outcome?

---

<div class="post-metadata">

**Author:** ![GuyShur](https://avatars.discourse-cdn.com/v4/letter/g/ce73a5/32.png) [@GuyShur](https://discuss.huggingface.co/u/GuyShur)\
**Post date:** [July 3, 2025, 9:27am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/6 "2025-07-03T09:27:52Z")

</div>

> [@NuralNexus](#):
>
> with surrounding tokens (e.g. PHY),

Thank you @NuralNexus!

I tried to recreate your code because it’s a bit mangled and this is what I have:

```auto
from tokenizers import Tokenizer, models, pre_tokenizers, trainers, decoders
from tokenizers.pre_tokenizers import PreTokenizer
import re

class AtomicUnitPreTokenizer:
    def __init__ (self, atomic_units):
        self.atomic_units = sorted(atomic_units, key=len, reverse=True)
        # Create regex pattern that matches any of the atomic units
        self.pattern = re.compile("|".join(re.escape(a) for a in self.atomic_units))

    def pre_tokenize_str(self, input_str):
        tokens = []
        last_end = 0

        # Find all atomic units in the string
        for match in self.pattern.finditer(input_str):
            start, end = match.span()

            # Add any text before the atomic unit
            if start > last_end:
                before_text = input_str[last_end:start]
                if before_text:
                    tokens.append((before_text, (last_end, start)))

            # Add the atomic unit as a single token
            tokens.append((match.group(0), (start, end)))
            last_end = end

        # Add any remaining text after the last atomic unit
        if last_end < len(input_str):
            remaining_text = input_str[last_end:]
            if remaining_text:
                tokens.append((remaining_text, (last_end, len(input_str))))

        return tokens

# Define your atomic units - these will never be split
atomic_units = ["<RR>", "<QQ>", "<PP>", "<XX>"]

# Initialize tokenizer with BPE model
tokenizer = Tokenizer(models.BPE(unk_token="[UNK]"))

# Set up the pre-tokenizer to handle atomic units
tokenizer.pre_tokenizer = AtomicUnitPreTokenizer(atomic_units)

# Set up the trainer
trainer = trainers.BpeTrainer(
    vocab_size=1000, special_tokens=["[UNK]"] + atomic_units, min_frequency=2
)

# Example corpus containing your sequences
corpus = [
    "HKLDCPHY<RR><QQ>PDHGIVMN",
    "ABCDE<PP><XX>FGHIJK",
    "MNOP<RR>QRST<QQ>UVWX",
    "YZAB<PP>CDEF<XX>GHIJ",
]

corpus += ["<QQ><RR>"] * 100

# Train the tokenizer
tokenizer.train_from_iterator(corpus, trainer=trainer)

# Test the tokenization
test_string = "HKLDCPHY<RR><QQ>PDHGIVMN"
encoded = tokenizer.encode(test_string)
print(f"Input: {test_string}")
print(f"Tokens: {encoded.tokens}")
print(f"IDs: {encoded.ids}")

# Test with another example
test_string2 = "ABC<PP><XX>DEF<RR>GHI"
encoded2 = tokenizer.encode(test_string2)
print(f"\nInput: {test_string2}")
print(f"Tokens: {encoded2.tokens}")
print(f"IDs: {encoded2.ids}")

# Optional: Convert to HuggingFace format for use with transformers
from transformers import PreTrainedTokenizerFast

hf_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
hf_tokenizer.add_special_tokens({"additional_special_tokens": atomic_units})

print(f"\nHuggingFace tokenizer test:")
print(f"Tokens: {hf_tokenizer.tokenize(test_string)}")
print(f"IDs: {hf_tokenizer.encode(test_string)}")

```

First of all is this what you intended?

I actually already tried a similar thing to this but ran into an error which I’m running into again now with this code:

` TypeError: argument 'pretok': 'AtomicUnitPreTokenizer' object cannot be converted to 'PreTokenizer'`

I tried debugging it but not much luck, any idea how to progress?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 3, 2025, 10:09am UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/7 "2025-07-03T10:09:28Z")

</div>

Hmm… Like this?

```py
#tokenizer.pre_tokenizer = AtomicUnitPreTokenizer(atomic_units)
tokenizer.pre_tokenizer = PreTokenizer.custom(AtomicUnitPreTokenizer(atomic_units))

```

> <https://github.com/huggingface/tokenizers/issues/269>
>
> Hello, 
> 
> I was trying to subclass \`PreTokenizer\` class from \`tokenizers.pre\_to…kenizers\` in order to create my own custom pre-tokenizer for splitting the text at word level, but this doesn't seem to work since I'm getting the following:
> 
> \`\`\`python
> TypeError: type 'PreTokenizer' is not an acceptable base type.
> \`\`\`
> 
> Any ideas how I can do this? Thanks in advance.

> <https://github.com/huggingface/tokenizers/issues/1634>
>
> Hey,
> 
> I want to train a Tokenizer that operates on a custom PreTokenizer. I tr…ied a mix of \[this documentation post\](https://huggingface.co/docs/tokenizers/pipeline) and \[this example\](https://github.com/huggingface/tokenizers/blob/b24a2fc1781d5da4e6ebcd3ecb5b91edffc0a05f/bindings/python/examples/custom\_components.py). My resulting code looks like this:
> 
> \`\`\`python
> class GlyLESPreTokenizer:
> def \_\_init\_\_(self, \*args, \*\*kwargs):
> pass
> 
> def \_\_new\_\_(cls, \*args, \*\*kwargs):
> return super().\_\_new\_\_(cls)
>     
> def glyles\_split(self, iupac: str):
> iuapc = iupac.strip().replace(" ", "")
> token = CommonTokenStream(GlyLESLexer(InputStream(data="{" + iupac + "}")))
> GlyLESParser(token).start()
> idx = 0
> output = \[\]
> for i in range(1, len(token.tokens) - 2):
> txt = str(token.tokens\[i\].text)
> output.append((txt, (idx, idx + len(txt))))
> idx += len(txt)
> return output
>         
> def pre\_tokenize\_str(self, input\_: str):
> return self.glyles\_split(input\_)
> 
> 
> iupac = "QuiNAlaAc(b1-4)GalNAcA(a1-4)GalOAc(a1-2)QuiNAlaAc"
> 
> \# This returns a list of 33 token
> GlyLESPreTokenizer().pre\_tokenize\_str(iupac)
> 
> \# This however only returns a list with one token that is the entire input string.
> pre\_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer()).pre\_tokenize\_str(iupac)
> \`\`\`
> 
> The final idea is to use it in such setting:
> \`\`\`python
> tokenizer = Tokenizer(models.Model())
> tokenizer.normalizer = normalizers.Strip()
> tokenizer.pre\_tokenizer = pre\_tokenizers.PreTokenizer.custom(GlyLESPreTokenizer())
> \`\`\`
> 
> Can someone help me to understand how to use the \`pre\_tokenizers.PreTokenizer.custom\` method to inject a custom, python-written PreTokenizer into a Tokenizer? Unfortunately, it is far beyond the scope of the project to convert the logic from \[GlyLES\](https://github.com/kalininalab/GlyLES) to RUST, so it has to be a Python PreTokenizer-class that is somehow injected into the Tokenizer.
> 
> Thank you for any help, comment, or feedback in advance.
> Roman

> [@How to add additional custom pre-tokenization processing?](https://discuss.huggingface.co/t/how-to-add-additional-custom-pre-tokenization-processing/1637):
>
> I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. I am looking at the pretokenizer function /huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py\_src/tokenizers/pre\_to…

---

<div class="post-metadata">

**Author:** ![GuyShur](https://avatars.discourse-cdn.com/v4/letter/g/ce73a5/32.png) [@GuyShur](https://discuss.huggingface.co/u/GuyShur)\
**Post date:** [July 3, 2025, 2:04pm UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/8 "2025-07-03T14:04:36Z")

</div>

Thanks @John6666 that did get me closer!

I tried:

```auto
from tokenizers import Tokenizer, models, trainers, PreTokenizedString, NormalizedString
from tokenizers.pre_tokenizers import PreTokenizer, BertPreTokenizer
import re

class AtomicUnitPreTokenizer:
    def __init__ (self, atomic_units):
        self.atomic_units = sorted(atomic_units, key=len, reverse=True)
        # Create regex pattern that matches any of the atomic units
        self.pattern = re.compile("|".join(re.escape(a) for a in self.atomic_units))
        print(f"Atomic units regex pattern: {self.pattern.pattern}")

    def split(self, _, norm: NormalizedString):
        # split the input string into tokens wrapped in < xx > and everything else
        tokens = []
        start = 0
        input_str = str(norm)
        for match in self.pattern.finditer(input_str):
            # Add everything before the match as a token
            if start < match.start():
                tokens.append(input_str[start : match.start()])
            # Add the matched atomic unit as a token
            tokens.append(input_str[match.start() : match.end()])
            start = match.end()
        # Add any remaining part of the string as a token
        if start < len(input_str):
            tokens.append(input_str[start:])

        print(f"Split tokens: {tokens}")
        return [NormalizedString(token) for token in tokens]

    def pre_tokenize(self, pre_tokenized: PreTokenizedString):
        return pre_tokenized.split(self.split)

# Define your atomic units - these will never be split
atomic_units = ["<RR>", "<QQ>"]
# Initialize tokenizer with BPE model
tokenizer = Tokenizer(models.BPE(unk_token="[UNK]"))
# Set up the pre-tokenizer to handle atomic units
tokenizer.pre_tokenizer = PreTokenizer.custom(AtomicUnitPreTokenizer(atomic_units))

# Set up the trainer
trainer = trainers.BpeTrainer(
    vocab_size=1000, special_tokens=["[UNK]"], min_frequency=1
)

# Example corpus containing your sequences
corpus = ["<RR><QQ>" for _ in range(3)]

# add the atomic units as regular tokens
for unit in atomic_units:
    tokenizer.add_tokens([unit])

print(f"Vocabulary before training: {tokenizer.get_vocab()}")

# Train the tokenizer
tokenizer.train_from_iterator(corpus, trainer=trainer)

print(f"Vocabulary after training: {tokenizer.get_vocab()}")
# Test the tokenization

test_string = "<RR><QQ>"
encoded = tokenizer.encode(test_string, is_pretokenized=False)
print(f"\nInput: {test_string}")
print(f"Tokens: {encoded.tokens}")
print(f"IDs: {encoded.ids}")
# Replacing with base tokenizer because custom pre-tokenizers can't be serialized
tokenizer.pre_tokenizer = BertPreTokenizer()
from transformers import PreTrainedTokenizerFast

hf_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
hf_tokenizer.add_special_tokens({"additional_special_tokens": atomic_units})

print(f"\nHuggingFace tokenizer test:")
print(f"Tokens: {hf_tokenizer.tokenize(test_string)}")
print(f"IDs: {hf_tokenizer.encode(test_string)}")

```

But unfortunately not much luck:

```auto
Atomic units regex pattern: <RR>|<QQ>
Vocabulary before training: {'<QQ>': 1, '<RR>': 0}
[00:00:00] Pre-processing sequences █████████████████████████████████████████████████████████████ 3 / 0Split tokens: ['<RR>', '<QQ>']
Split tokens: ['<RR>', '<QQ>']
Split tokens: ['<RR>', '<QQ>']
[00:00:00] Pre-processing sequences █████████████████████████████████████████████████████████████ 0 / 0[00:00:00] Tokenize words █████████████████████████████████████████████████████████████ 2 / 2
[00:00:00] Count pairs █████████████████████████████████████████████████████████████ 2 / 2
[00:00:00] Compute merges █████████████████████████████████████████████████████████████ 6 / 6
Vocabulary after training: {'<RR>': 0, 'Q': 3, '[UNK]': 0, 'R>': 8, 'Q>': 7, '>': 2, 'R': 4, '<': 1, '<R': 6, '<Q': 5, '<QQ>': 1}

Input: <RR><QQ>
Tokens: ['<RR>', '<QQ>']
IDs: [0, 1]

HuggingFace tokenizer test:
Tokens: ['<RR>', '<QQ>']
IDs: [10, 9]

```

Calling tokenizer.encode doesn’t cause the print statement in the pre\_tokenizer, I wonder if it doesn’t pre-tokenize the examples properly.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 3, 2025, 2:17pm UTC](https://discuss.huggingface.co/t/adding-atomic-indivisible-tokens-to-bpe-tokenizer/161486/9 "2025-07-03T14:17:13Z")

</div>

Hmm… One dirty hack.

```py
hf_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer)
hf_tokenizer._tokenizer.pre_tokenizer = AtomicUnitPreTokenizer(atomic_units) # Added
hf_tokenizer.add_special_tokens({"additional_special_tokens": atomic_units})

```