# Training loss no drop for MT5ForSequenceClassification

**URL:** <https://discuss.huggingface.co/t/training-loss-no-drop-for-mt5forsequenceclassification/73552>\
**Category:** 🤗Transformers\
**Created:** [February 17, 2024, 5:46am UTC](https://discuss.huggingface.co/t/training-loss-no-drop-for-mt5forsequenceclassification/73552 "2024-02-17T05:46:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![forrestbao](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/forrestbao/32/23472_2.png) [@forrestbao](https://discuss.huggingface.co/u/forrestbao)\
**Post date:** [February 17, 2024, 5:46am UTC](https://discuss.huggingface.co/t/training-loss-no-drop-for-mt5forsequenceclassification/73552/1 "2024-02-17T05:46:22Z")

</div>

Hi,

I am using mT5 for sequence classification for the first time. Basically I am doing cross-lingual NLI using XNLI. I am using the official `MT5ForSequenceClassification` method, which attaches a FC network to the decoder’s output. The same code worked well for mBART and T5 (english, french and german only). But after I switched to mT5 (i literally just changed the model name string), I noticed that the training loss never dropped. I have tried various learning rates from 1e-3 to 1e-6. No luck.

Has anyone had a similar experience? Any suggestions ?

---

<div class="post-metadata">

**Author:** ![astarostin1983](https://avatars.discourse-cdn.com/v4/letter/a/e495f1/32.png) [@astarostin1983](https://discuss.huggingface.co/u/astarostin1983)\
**Post date:** [June 23, 2024, 12:08am UTC](https://discuss.huggingface.co/t/training-loss-no-drop-for-mt5forsequenceclassification/73552/2 "2024-06-23T00:08:57Z")

</div>

Hi,  
Did you manage to solve this problem? It seems that I’ve met the same issue. Moreover, I tried to model my classification problem on the same data as generative problem using MT5ForConditionalGeneration and forcing it to generate correct classification labels as text and surprisingly it worked quite good. But straightforward approach doesn’t work, loss doesn’t decrease. I use ‘google/mt5-base’ model.

Regards,  
Anatoly

---

<div class="post-metadata">

**Author:** ![forrestbao](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/forrestbao/32/23472_2.png) [@forrestbao](https://discuss.huggingface.co/u/forrestbao)\
**Post date:** [January 1, 2025, 3:55am UTC](https://discuss.huggingface.co/t/training-loss-no-drop-for-mt5forsequenceclassification/73552/3 "2025-01-01T03:55:22Z")

</div>

I think so. I don’t remember what I did (I tried a bunch of things not related to this issue) but the problem is gone. I probably solved the problem by using a larger model and did full finetuning instead of LoRA.