# Change Gemma tokenizer unused token

**URL:** <https://discuss.huggingface.co/t/change-gemma-tokenizer-unused-token/80867>\
**Category:** Beginners\
**Created:** [April 9, 2024, 10:07am UTC](https://discuss.huggingface.co/t/change-gemma-tokenizer-unused-token/80867 "2024-04-09T10:07:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Severino](https://avatars.discourse-cdn.com/v4/letter/s/d6d6ee/32.png) [@Severino](https://discuss.huggingface.co/u/Severino)\
**Post date:** [April 9, 2024, 10:07am UTC](https://discuss.huggingface.co/t/change-gemma-tokenizer-unused-token/80867/1 "2024-04-09T10:07:08Z")

</div>

I want to add a new token to the gemma tokenizer let’s call it `<newtoken>`. I would like to take advantage of the fact that the vocabulary of this tokenizer has unused tokens (`<unused0>`, `<unused1>`, …).  
The first thing I tried was to `add_token("<newtoken>")` but this only append the token to the end of the vocabulary instead of replacing the unused ones.  
Is there an expected way to achieve the replacement of the unused tokens? Something like: `add_token("<newtoken>", replace_unused=True)`

---

<div class="post-metadata">

**Author:** ![ksopyla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ksopyla/32/38870_2.png) [@ksopyla](https://discuss.huggingface.co/u/ksopyla)\
**Post date:** [January 9, 2025, 10:19am UTC](https://discuss.huggingface.co/t/change-gemma-tokenizer-unused-token/80867/2 "2025-01-09T10:19:36Z")

</div>

I have just ask bing copilot and give me code snippet

```auto
from transformers import AutoTokenizer

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Define the mapping of old tokens to new tokens
token_mapping = {
    "<unused0>": "<NEW_TOKEN1>",
    "<unused1>": "<NEW_TOKEN2>"
}

# Update the tokenizer's vocabulary
for old_token, new_token in token_mapping.items():
    if old_token in tokenizer.get_vocab():
        token_id = tokenizer.convert_tokens_to_ids(old_token)
        tokenizer.add_tokens([new_token])
        new_token_id = tokenizer.convert_tokens_to_ids(new_token)
        tokenizer.vocab[new_token] = new_token_id
        del tokenizer.vocab[old_token]

# Save the updated tokenizer
tokenizer.save_pretrained("./updated_tokenizer")

```

I haven’t tested it and can’t confirm it works.

There is also github issue: [how to replace the existing token in a tokenizer · Issue #27974 · huggingface/transformers](https://github.com/huggingface/transformers/issues/27974)