# GPT loss increasing

**URL:** <https://discuss.huggingface.co/t/gpt-loss-increasing/7034>\
**Category:** Models\
**Created:** [June 21, 2021, 11:21am UTC](https://discuss.huggingface.co/t/gpt-loss-increasing/7034 "2021-06-21T11:21:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![treeofknowledge](https://avatars.discourse-cdn.com/v4/letter/t/2acd7d/32.png) [@treeofknowledge](https://discuss.huggingface.co/u/treeofknowledge)\
**Post date:** [June 21, 2021, 11:21am UTC](https://discuss.huggingface.co/t/gpt-loss-increasing/7034/1 "2021-06-21T11:21:47Z")

</div>

Hi! I’m finetuning a GPT-2 model (360M params = size M) on a huge training set (dozens of GB) with a low learning rate (15e-6, or 0.000015). The loss value dips at the start, and then gradually increases for a long time. I haven’t seen it start to fall again yet. Is this behavior normal/expected?

 ![graph](https://us1.discourse-cdn.com/hellohellohello/original/2X/3/31c62dba89968d70282d0a1b02144e676e470482.png)

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [June 21, 2021, 2:55pm UTC](https://discuss.huggingface.co/t/gpt-loss-increasing/7034/2 "2021-06-21T14:55:44Z")

</div>

hey @treeofknowledge this plot suggests your learning rate is too high after 25k training steps so you could try using a learning rate scheduler like the default provided in `transformers.Trainer` (see [here](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#transformers.Trainer.create_scheduler))
