# GPT-2 fine-tuning

**URL:** <https://discuss.huggingface.co/t/gpt-2-fine-tuning/42990>\
**Category:** Beginners\
**Created:** [June 12, 2023, 10:56pm UTC](https://discuss.huggingface.co/t/gpt-2-fine-tuning/42990 "2023-06-12T22:56:30Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![nash0823](https://avatars.discourse-cdn.com/v4/letter/n/5f9b8f/32.png) [@nash0823](https://discuss.huggingface.co/u/nash0823)\
**Post date:** [June 12, 2023, 10:56pm UTC](https://discuss.huggingface.co/t/gpt-2-fine-tuning/42990/1 "2023-06-12T22:56:30Z")

</div>

Hi everyone! First of all, I’m new to huggingface, transformers and NLP in general. My question is in regards to fine-tuning.

I have a dataset of scientific documents in a csv (about 50,000) with titles and abstracts. I tried training it on a gpt2 model and even used methods like freezing the first layer, training it on five epochs then unfreezing all layers and fine-tuning it.

The problem is that once the training is completed, when I use it to generate text, it seems to “forget” everything else and only generate text that are solely related to my dataset. For instance, if I ask it to complete a query like “the weather outside is very sunny and…” it gives me very scientific answers as if it “forgot” everything from before I fine-tuned it with my dataset.

Is there any way around this? Sorry if its a stupid question but I’m very new to this and would love to learn more about how to use it the proper way