# Bert Multi-lingual fine-tuning for multilabel classification

**URL:** <https://discuss.huggingface.co/t/bert-multi-lingual-fine-tuning-for-multilabel-classification/14112>\
**Category:** Intermediate\
**Created:** [January 25, 2022, 7:31pm UTC](https://discuss.huggingface.co/t/bert-multi-lingual-fine-tuning-for-multilabel-classification/14112 "2022-01-25T19:31:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MajTad](https://avatars.discourse-cdn.com/v4/letter/m/dfb087/32.png) [@MajTad](https://discuss.huggingface.co/u/MajTad)\
**Post date:** [January 25, 2022, 7:31pm UTC](https://discuss.huggingface.co/t/bert-multi-lingual-fine-tuning-for-multilabel-classification/14112/1 "2022-01-25T19:31:43Z")

</div>

Hi,  
I’m trying to make French email sentences multilabel classification with certain categories such as _commitmemt, proposition, meeting, request, subjectif, etc._ .

The first problem I faced is that I don’t have labeled sentences rather I have french emails as dataset). Based on this I found the BC3 dataset (English emails) which has sentences annotated with some of labels listed above. So I came up with this approah; **First fine tune a bert multilingual on this BC3 dataset on multilabel classification task and then make a zero-shot transfert learning with the finetuned model (or simply use it in inference) on sentences of my French emails.** What do you think about this approach?

So I started by prepropcessing the BC3 dataset and obtain 848 sentences, each of them with their occurences annotations according to each categorty. On the image below, the last 5 columns represent the number of time each annotator labeled a sentence for a specific label.

 ![76fd1904c3443215cb02a4c02a2d83de2875f494](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/22488730438df1badc730790987dbac64a18656a.png)

_ **Are those 848 samples enough to fine tune a Bert multilingual model?** _

I try to fine tune by representing category as on the image below .  
 ![46a0ee9c7436a29ba3629fc31009a497fd89aee3](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/46a0ee9c7436a29ba3629fc31009a497fd89aee3.png)

With one epoch, BATCH\_SIZE = 4, the loss function did’t converge, rather it oscillates between 0.79 and 0.34.

What kind of advices would you give the solve this kind of problem?

Thanks.