# Overlapping data between pre-training and fine-tuning stages

**URL:** <https://discuss.huggingface.co/t/overlapping-data-between-pre-training-and-fine-tuning-stages/10624>\
**Category:** 🤗Transformers\
**Created:** [October 8, 2021, 5:50pm UTC](https://discuss.huggingface.co/t/overlapping-data-between-pre-training-and-fine-tuning-stages/10624 "2021-10-08T17:50:14Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![fozyurt](https://avatars.discourse-cdn.com/v4/letter/f/c68b51/32.png) [@fozyurt](https://discuss.huggingface.co/u/fozyurt)\
**Post date:** [October 8, 2021, 5:50pm UTC](https://discuss.huggingface.co/t/overlapping-data-between-pre-training-and-fine-tuning-stages/10624/1 "2021-10-08T17:50:14Z")

</div>

Hi

I am currently pre-training a RoBERTa model for my own data. If some part of the data that is used for language modeling task during the pre-training process is also used in fine-tuning process, does this cause a biased results at the end of the fine-tuning process ?