# Roberta pretokenizer - split punctuation?

**URL:** <https://discuss.huggingface.co/t/roberta-pretokenizer-split-punctuation/79248>\
**Category:** Beginners\
**Created:** [March 29, 2024, 5:30am UTC](https://discuss.huggingface.co/t/roberta-pretokenizer-split-punctuation/79248 "2024-03-29T05:30:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![AngledLuffa](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@AngledLuffa](https://discuss.huggingface.co/u/AngledLuffa)\
**Post date:** [March 29, 2024, 5:30am UTC](https://discuss.huggingface.co/t/roberta-pretokenizer-split-punctuation/79248/1 "2024-03-29T05:30:38Z")

</div>

When using the ByteLevelBPETokenizer to build a tokenizer for a new Roberta model, I found that the tokenizer has quite a few tokens in it which are letters in the alphabet with a period or other punctuation attached. I took a look at the ByteLevelBPETokenizer implementation:

> <https://github.com/huggingface/tokenizers/blob/main/bindings/python/py_src/tokenizers/implementations/byte_level_bpe.py>

It appears the pretokenizer used is always

```auto
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=add_prefix_space)

```

The character level one has an option to pretokenize on more than just whitespace:

```auto
if split_on_whitespace_only:
    tokenizer.pre_tokenizer = pre_tokenizers.WhitespaceSplit()
else:
    tokenizer.pre_tokenizer = pre_tokenizers.BertPreTokenizer()

```

Is there a recommended solution in the byte level version which will pretokenize the punctuation separately from the rest of the text? Is that something I should do myself before building the Roberta model (and possibly before creating the transformer itself)?

---

<div class="post-metadata">

**Author:** ![KeelyPowers](https://avatars.discourse-cdn.com/v4/letter/k/7ba0ec/32.png) [@KeelyPowers](https://discuss.huggingface.co/u/KeelyPowers)\
**Post date:** [March 30, 2024, 12:13am UTC](https://discuss.huggingface.co/t/roberta-pretokenizer-split-punctuation/79248/2 "2024-03-30T00:13:11Z")

</div>

You can write your own pre-tokenizer or use a pre-tokenizer that will split the text into parts based on the presence of punctuation. For example, you can use `BertPreTokenizer`, which splits text into tokens, taking punctuation into account. After this you can use `ByteLevelBPETokenizer` for additional tokenization. This way, you can more precisely control the tokenization process and avoid creating extra tokens that are letters of the alphabet with a period or other punctuation marks. However, keep in mind that the approach may require additional coding and testing to ensure it is effective and suits your needs.

---

<div class="post-metadata">

**Author:** ![AngledLuffa](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@AngledLuffa](https://discuss.huggingface.co/u/AngledLuffa)\
**Post date:** [March 30, 2024, 12:28am UTC](https://discuss.huggingface.co/t/roberta-pretokenizer-split-punctuation/79248/3 "2024-03-30T00:28:51Z")

</div>

Thanks. That seems like a good approach - so basically I just need to process the text before passing it to the tokenizer builder? It would be a little more convenient for it to be a built in option such as with the Bert training, but it isn’t too hard to work around, at least.