# Format of data during pre-training

**URL:** <https://discuss.huggingface.co/t/format-of-data-during-pre-training/1388>\
**Category:** 🤗Datasets\
**Created:** [October 6, 2020, 3:23pm UTC](https://discuss.huggingface.co/t/format-of-data-during-pre-training/1388 "2020-10-06T15:23:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shainaraza](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shainaraza/32/2854_2.png) [@shainaraza](https://discuss.huggingface.co/u/shainaraza)\
**Post date:** [October 6, 2020, 3:23pm UTC](https://discuss.huggingface.co/t/format-of-data-during-pre-training/1388/1 "2020-10-06T15:23:15Z")

</div>

What should be the format of the data for pre-training? could it be any raw data (e.g., news articles) in my case and then after I fine-tune, then I need to define it for a specific task e.g., classification?

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [October 7, 2020, 1:13pm UTC](https://discuss.huggingface.co/t/format-of-data-during-pre-training/1388/2 "2020-10-07T13:13:09Z")

</div>

It comes in many forms and usually needs processing/adaptation for model input. Look into transformers examples to start.

If you are asking about the datasets project specifically - it makes it even simpler to do the above. There are many prepared datasets ready to go. Read through documentation.