# NER on multiple languages

**URL:** <https://discuss.huggingface.co/t/ner-on-multiple-languages/401>\
**Category:** 🤗Transformers\
**Created:** [July 20, 2020, 6:11pm UTC](https://discuss.huggingface.co/t/ner-on-multiple-languages/401 "2020-07-20T18:11:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![goutham794](https://avatars.discourse-cdn.com/v4/letter/g/8baadc/32.png) [@goutham794](https://discuss.huggingface.co/u/goutham794)\
**Post date:** [July 20, 2020, 6:11pm UTC](https://discuss.huggingface.co/t/ner-on-multiple-languages/401/1 "2020-07-20T18:11:15Z")

</div>

I want to do NER task on news articles that are in dozens of languages. Is the best option to go for [`xlm-roberta-large-finetuned-conll03-english`](https://huggingface.co/xlm-roberta-large-finetuned-conll03-english)? I read that XLM models fine-tuned for a language work well in other languages as well. My main issue is that this model is too big. Should I go for language specific smaller models if I already know what language I’m dealing with?  
Also I’m curious, why does the [`xlm-roberta-large-finetuned-conll03-german`](https://huggingface.co/xlm-roberta-large-finetuned-conll03-german) have so much more downloads than the English one?

---

<div class="post-metadata">

**Author:** ![stefan-it](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefan-it/32/20175_2.png) [@stefan-it](https://discuss.huggingface.co/u/stefan-it)\
**Post date:** [August 6, 2020, 11:36am UTC](https://discuss.huggingface.co/t/ner-on-multiple-languages/401/2 "2020-08-06T11:36:46Z")

</div>

Hi @goutham794,

you could train a multi-lingual NER model on the WikiANN dataset (or better: use the train/dev/test partitioned from [GitHub - afshinrahimi/mmner: Massively Multilingual Transfer for NER](https://github.com/afshinrahimi/mmner)).

But fine-tuning one big multi-lingual NER model could be very complicated (fine-tuning instabilities). And you should keep in mind, that WikiANN only has three label types.

If you already know what languages you want to cover, then a better way would be to train “mono-lingual” models + just search for NER datasets for your desired languages. Good resource is:

> **[GitHub - juand-r/entity-recognition-datasets: A collection of corpora for...](https://github.com/juand-r/entity-recognition-datasets)**
>
> A collection of corpora for named entity recognition (NER) and entity recognition tasks. These annotated datasets cover a variety of languages, domains and entity types. - GitHub - juand-r/entity-r...