# Bert trainning loss suddenly explodes

**URL:** <https://discuss.huggingface.co/t/bert-trainning-loss-suddenly-explodes/8684>\
**Category:** 🤗Transformers\
**Created:** [July 26, 2021, 8:11am UTC](https://discuss.huggingface.co/t/bert-trainning-loss-suddenly-explodes/8684 "2021-07-26T08:11:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hong-555](https://avatars.discourse-cdn.com/v4/letter/h/dc4da7/32.png) [@hong-555](https://discuss.huggingface.co/u/hong-555)\
**Post date:** [July 26, 2021, 8:11am UTC](https://discuss.huggingface.co/t/bert-trainning-loss-suddenly-explodes/8684/1 "2021-07-26T08:11:58Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/a797909cb7b611293f7fcdf55a9ae89ebc594f4b.png)

Training Chinese and English language modeling MLM bert model  
–num\_train\_epochs 500   
–per\_device\_train\_batch\_size 8   
–learning\_rate 1e-4   
–warmup\_steps 5000   
–max\_seq\_length 512 \

---

<div class="post-metadata">

**Author:** ![adnenabdessaied](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adnenabdessaied/32/17916_2.png) [@adnenabdessaied](https://discuss.huggingface.co/u/adnenabdessaied)\
**Post date:** [July 25, 2023, 11:07am UTC](https://discuss.huggingface.co/t/bert-trainning-loss-suddenly-explodes/8684/2 "2023-07-25T11:07:34Z")

</div>

Hey! This is probably too late. But this is most likely due to a high learning rate. You could reduce the learning rate and/or use gradient clipping. The latter will not have a significant change in performance, but will prevent a bad mini-batch from messing up the training of your model. Good luck!