# Fine-tuning a masked language model

**URL:** <https://discuss.huggingface.co/t/fine-tuning-a-masked-language-model/14337>\
**Category:** Beginners\
**Created:** [February 2, 2022, 2:31pm UTC](https://discuss.huggingface.co/t/fine-tuning-a-masked-language-model/14337 "2022-02-02T14:31:04Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![rs15](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@rs15](https://discuss.huggingface.co/u/rs15)\
**Post date:** [February 2, 2022, 2:31pm UTC](https://discuss.huggingface.co/t/fine-tuning-a-masked-language-model/14337/1 "2022-02-02T14:31:04Z")

</div>

Hi everyone.

I was going through the \* [Fine-tuning a masked language model](https://huggingface.co/course/chapter7/3?fw=pt) section in the course and I can’t understand one thing there. I’m talking about the following piece of code:

```auto
concatenated_examples = {
    k: sum(tokenized_samples[k], []) for k in tokenized_samples.keys()
}
total_length = len(concatenated_examples["input_ids"])
print(f"'>>> Concatenated reviews length: {total_length}'")

```

From what I know, BERT can only have one [CLS] token at the beginning. So, if we concat all these texts and split them into chunks, we will get multiple [CLS] tokens (which is not BERT-like)

```auto
[CLS] ... [SEP] [CLS] ... [SEP] ...

```

in a sequence. Why does it work? Is there any paper that describes this behavior or maybe any other source?

Thank you.