# Punctuation and Spaces in RoBERTa Tokenizer for NER with Pre-tokenized Data

**URL:** <https://discuss.huggingface.co/t/punctuation-and-spaces-in-roberta-tokenizer-for-ner-with-pre-tokenized-data/13765>\
**Category:** 🤗Transformers\
**Created:** [January 16, 2022, 10:52pm UTC](https://discuss.huggingface.co/t/punctuation-and-spaces-in-roberta-tokenizer-for-ner-with-pre-tokenized-data/13765 "2022-01-16T22:52:13Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sapirw](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@sapirw](https://discuss.huggingface.co/u/sapirw)\
**Post date:** [January 16, 2022, 10:52pm UTC](https://discuss.huggingface.co/t/punctuation-and-spaces-in-roberta-tokenizer-for-ner-with-pre-tokenized-data/13765/1 "2022-01-16T22:52:13Z")

</div>

Hi,  
I’m trying to use RoBERTa for NER using [CoNLL-2003 dataset](https://huggingface.co/datasets/conll2003).  
My question is about the tokenizer definition and use at [run\_ner.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py).  
An example of an output of that exact tokenizer is:  
Input: `['SOCCER', '-', 'JAPAN', 'GET', 'LUCKY', 'WIN', ',', 'CHINA', 'IN', 'SURPRISE', 'DEFEAT', '.']`  
Output: `['ĠSO', 'CC', 'ER', 'Ġ-', 'ĠJ', 'AP', 'AN', 'ĠGET', 'ĠL', 'UCK', 'Y', 'ĠWIN', 'Ġ,', 'ĠCH', 'INA', 'ĠIN', 'ĠSUR', 'PR', 'ISE', 'ĠDE', 'FE ', 'AT', 'Ġ.']`

RoBERTa tokenizer outputs different tokens for punctuation marks depending on whether there is a I think it would have made more sense if the tokens for “.” were ‘.’ instead of ‘Ġ.’ (but the output `offset_mapping` should stay the same.).

My question is - why is this ignored in run\_ner.py?

Thanks