# Why do I get 'Ġ' when adding emojis to the tokenizer?

**URL:** <https://discuss.huggingface.co/t/why-do-i-get-g-when-adding-emojis-to-the-tokenizer/7056>\
**Category:** Beginners\
**Created:** [June 22, 2021, 10:19am UTC](https://discuss.huggingface.co/t/why-do-i-get-g-when-adding-emojis-to-the-tokenizer/7056 "2021-06-22T10:19:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon58275033](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@anon58275033](https://discuss.huggingface.co/u/anon58275033)\
**Post date:** [June 22, 2021, 10:19am UTC](https://discuss.huggingface.co/t/why-do-i-get-g-when-adding-emojis-to-the-tokenizer/7056/1 "2021-06-22T10:19:11Z")

</div>

Hello,

I have added custom tokens to my tokenizer, which are emojis. This is the code I have used, which adds the new tokens:

```auto
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint)

num_added_toks = tokenizer.add_tokens(['👏'])
print('We have added', num_added_toks, 'tokens')
model.resize_token_embeddings(len(tokenizer)) # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e. the length of the tokenizer

```

Output:

```auto
We have added 1 tokens
Embedding(50270, 768)

```

Though, when I try to tokenize a phrase using this code:

```auto
print(tokenizer.tokenize('Congrats 👏'))

```

I get this output with that strange `'Ġ'` symbol:

```auto
['Cong', 'rats', 'Ġ', '👏']

```

---

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](https://discuss.huggingface.co/u/nbroad)\
**Post date:** [June 27, 2021, 11:25pm UTC](https://discuss.huggingface.co/t/why-do-i-get-g-when-adding-emojis-to-the-tokenizer/7056/2 "2021-06-27T23:25:32Z")

</div>

[https://github.com/pytorch/fairseq/issues/1716](https://github.com/pytorch/fairseq/issues/1716)