# GPT model loss diverges while fine-tuning

**URL:** <https://discuss.huggingface.co/t/gpt-model-loss-diverges-while-fine-tuning/14638>\
**Category:** Models\
**Created:** [February 11, 2022, 8:55pm UTC](https://discuss.huggingface.co/t/gpt-model-loss-diverges-while-fine-tuning/14638 "2022-02-11T20:55:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![entropy](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@entropy](https://discuss.huggingface.co/u/entropy)\
**Post date:** [February 11, 2022, 8:55pm UTC](https://discuss.huggingface.co/t/gpt-model-loss-diverges-while-fine-tuning/14638/1 "2022-02-11T20:55:29Z")

</div>

I’m trying to fine-tune the distilled GPT model on a new dataset and I’m having issues with the loss diverging during training. Can anyone thing of why this might be happening? I’ve lowered the learning rate to `1e-7` which feels extremely low, and the issue persists

 ![Screen Shot 2022-02-11 at 2.24.28 AM](https://us1.discourse-cdn.com/hellohellohello/original/2X/3/37f0fdcd115e61bb09a0bd5bef344a42e037c885.png)