# Continue pre-training GPT2

**URL:** <https://discuss.huggingface.co/t/continue-pre-training-gpt2/34692>\
**Category:** Research\
**Created:** [March 26, 2023, 7:29am UTC](https://discuss.huggingface.co/t/continue-pre-training-gpt2/34692 "2023-03-26T07:29:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![IdoAmit198](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/idoamit198/32/12110_2.png) [@IdoAmit198](https://discuss.huggingface.co/u/IdoAmit198)\
**Post date:** [March 26, 2023, 7:29am UTC](https://discuss.huggingface.co/t/continue-pre-training-gpt2/34692/1 "2023-03-26T07:29:53Z")

</div>

Hi guys,

Since 2019, when OpenAI introduced to us GPT2, a lot has changed and new methods/optimization schemes emerged.  
I believe GPT2 is sub-optimal considering the jump NLP made since then.

Therefore, I’m trying to continue pre-training GPT2 (small, medium, large), and would love to hear from your experience!

- I’m using the openwebtext dataset, do any of you recommend a better/richer one?
- Did any of you try distillation to continue pre-train GPT2?
- Any other SOTA trick/optimization method you do recommend?

---

<div class="post-metadata">

**Author:** ![jonathantiedchen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jonathantiedchen/32/42927_2.png) [@jonathantiedchen](https://discuss.huggingface.co/u/jonathantiedchen)\
**Post date:** [March 10, 2025, 12:14pm UTC](https://discuss.huggingface.co/t/continue-pre-training-gpt2/34692/2 "2025-03-10T12:14:46Z")

</div>

Hi @IdoAmit198. Do you have any updated on this? I was wondering the same but find limited resources on that. I want toncontinuous pre-train GPT2 on OpenWebMath.