# Dealing with escape characters when using models built on the BERT architecture **URL:** \ **Category:** Beginners\ **Created:** [March 30, 2026, 9:42am UTC](https://discuss.huggingface.co/t/dealing-with-escape-characters-when-using-models-built-on-the-bert-architecture/174792 "2026-03-30T09:42:51Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![malkolmhorlen](https://avatars.discourse-cdn.com/v4/letter/m/5fc32e/32.png) [@malkolmhorlen](https://discuss.huggingface.co/u/malkolmhorlen)\ **Post date:** [March 30, 2026, 9:42am UTC](https://discuss.huggingface.co/t/dealing-with-escape-characters-when-using-models-built-on-the-bert-architecture/174792/1 "2026-03-30T09:42:51Z")
Hi! I am currently working on a research project and I will be using a model built on the BERT architecture for the first time. My question is, should I remove any escape characters (e.g., \n) in the pre-processing stage, or would this instead worsen the performance of the model. Thank you! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 30, 2026, 11:48am UTC](https://discuss.huggingface.co/t/dealing-with-escape-characters-when-using-models-built-on-the-bert-architecture/174792/2 "2026-03-30T11:48:50Z")
Results from actual testing with the (now) latest version of Transformers v5: * * * Usually, **no**. For a standard BERT-family model, you generally should **not remove actual newline characters like `\n` by default**. In the original BERT tokenizer, tab, newline, and carriage return are treated as whitespace, and the text-cleaning step converts whitespace-like characters into ordinary spaces before tokenization. Hugging Face’s `BertNormalizer` describes the same behavior: BERT-style normalization cleans text and standardizes whitespace before subword tokenization. ([GitHub](https://github.com/google-research/bert/blob/master/tokenization.py "tokenization.py - google-research/bert")) ## The main idea BERT does not read raw text “as-is.” It reads the output of a tokenizer pipeline. In that pipeline, **normalization and pre-tokenization happen before WordPiece tokenization** , so whitespace handling is already part of how the model was designed to see text. Hugging Face’s tokenizer docs are explicit that pre-tokenization creates an **upper bound** on what final tokens can be. ([Hugging Face](https://huggingface.co/docs/tokenizers/pipeline "The tokenization pipeline")) That leads to the core rule: - **Real newlines** are usually safe to keep. - **Literal escaped strings** like `\\n` are a different case and may need cleanup. - **Deleting boundaries** can be harmful if it merges words together. A practical way to think about it is that the tokenizer is part of the model contract, not a disposable preprocessing detail. ## The first distinction you need to make There are two different things people often call “escape characters”: ### 1. A real newline character This is an actual line break in the text. Example: ```plaintext This movie was not good ``` ### 2. A literal backslash-plus-letter sequence This is visible text that contains `\` and `n`. Example: ```plaintext This movie was not \n good ``` For BERT, these are **not the same**. A real newline is treated like whitespace. A literal `\` is not whitespace. In the original BERT code, whitespace characters are handled by `_is_whitespace`, while non-letter and non-number ASCII symbols are treated separately as punctuation-like characters. ([GitHub](https://github.com/google-research/bert/blob/master/tokenization.py "tokenization.py - google-research/bert")) So the answer depends on **which one your data actually contains**. ## Why removing real `\n` usually does not help Because BERT already normalizes it. The original BERT preprocessing logic treats `\n`, `\t`, and `\r` as whitespace, and Hugging Face’s BERT normalizer says the same thing in current docs: BERT-style text normalization includes cleaning text and standardizing whitespace before subword splitting. ([GitHub](https://github.com/google-research/bert/blob/master/tokenization.py "tokenization.py - google-research/bert")) That means if your text contains normal line breaks, manually deleting them often adds little or nothing. In many cases, the tokenizer will effectively turn them into ordinary spacing anyway. If the line break was just formatting, leaving it alone is usually fine. ## When removing them can make things worse ### If you delete a boundary and merge words This is the real danger. If preprocessing turns something like: ```plaintext not good ``` into: ```plaintext notgood ``` then the model no longer sees two ordinary words separated by whitespace. It sees a merged string, and WordPiece can split that merged string into very different subword units. Since BERT is a WordPiece model, that changes the input token IDs the model actually receives. The BERT paper and model docs both describe BERT as using WordPiece tokenization. ([GitHub](https://github.com/google-research/bert "google-research/bert: TensorFlow code and pre-trained ...")) This is the important point: **the problem is usually not the existence of the newline itself. The problem is deleting it in a way that destroys a boundary.** ### If layout carries meaning Sometimes line breaks are not just visual formatting. They can carry weak structure: - list items - addresses - clinical notes - dialogue turns - legal clauses - titles versus body text If you flatten everything aggressively, you may remove useful structure. The tokenizer pipeline docs matter here because pre-tokenization defines the pieces within which later tokenization happens. ([Hugging Face](https://huggingface.co/docs/tokenizers/pipeline "The tokenization pipeline")) ### If your task depends on alignment For token classification, NER, or extractive QA, preprocessing can create alignment problems. Hugging Face’s token-classification docs explicitly show that you need to realign labels to tokens with `word_ids()`, and fast tokenizers keep offset mappings to track character spans from the original text. If you rewrite the text carelessly before tokenization, labels and spans can drift. ([Hugging Face](https://huggingface.co/docs/transformers/tasks/token_classification "Token classification")) ## When you should clean more aggressively You should clean more aggressively when the text contains **artifacts** , not real content. Typical examples: - broken JSON export - scraped text with literal `\\n` - malformed control characters - OCR junk - text where visible backslashes are accidental formatting residue If your data literally contains `\\n` because of serialization, then replacing it with a space or a real newline often makes sense. If you leave it untouched, BERT will not interpret it as layout. It will treat the backslash and following characters as ordinary visible text. ([GitHub](https://github.com/google-research/bert/blob/master/tokenization.py "tokenization.py - google-research/bert")) But if `\\n` is meaningful domain content, such as in code, regexes, logs, or markup, then converting or deleting it would be wrong. ## The safest practical default For a first BERT-based research project, this is the best default policy: ### Keep by default - ordinary words and punctuation - real newlines, tabs, and carriage returns - natural document structure ### Clean carefully - malformed encodings - invisible junk control characters - literal `\\n`, `\\t`, `\\r` only when they are accidental artifacts ### Preserve explicitly if structure matters If line boundaries are important for your task, do not assume default whitespace normalization is enough. Represent them deliberately with a marker such as `[NL]` or `` and use that consistently in training and inference. That follows directly from how tokenizer pipelines define boundaries before final tokenization. ([Hugging Face](https://huggingface.co/docs/tokenizers/pipeline "The tokenization pipeline")) ## What this means by task ### Text classification For sentiment, topic classification, or similar sentence/document classification, the safest starting point is **minimal cleanup**. Real `\n` usually does not need special handling. ([GitHub](https://github.com/google-research/bert/blob/master/tokenization.py "tokenization.py - google-research/bert")) ### Token classification and NER Be more conservative. Alignment matters, and label mapping can break if you rewrite text too aggressively. ([Hugging Face](https://huggingface.co/docs/transformers/tasks/token_classification "Token classification")) ### Extractive QA Also be conservative. Span boundaries matter, so avoid unnecessary text rewriting. Offset-aware tokenization exists for a reason. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter6/3 "Fast tokenizers' special powers")) ### Code-like or log-like text Do not assume `\\n` is noise. In that setting, it may be meaningful content. ## The short recommendation For your case, the best default is: - **Do not strip actual `\n` by default.** - **Do clean literal `\\n` only when it is clearly a serialization artifact.** - **Avoid any preprocessing that can merge adjacent words.** - **Keep preprocessing identical across training, validation, test, and inference.** ([GitHub](https://github.com/google-research/bert/blob/master/tokenization.py "tokenization.py - google-research/bert")) ## Final answer So the answer is: **Removing real newline characters is usually unnecessary and can sometimes be harmful if it destroys useful boundaries or structure.** **Cleaning literal escaped artifacts like `\\n` is often correct, but only when they are accidental export noise rather than meaningful text.** That is the safest, most technically accurate default for BERT-family models.