# Tiny mBART doc/info **URL:** \ **Category:** 🤗Transformers\ **Created:** [July 21, 2020, 3:22pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414 "2020-07-21T15:22:12Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 21, 2020, 3:22pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/1 "2020-07-21T15:22:12Z")
I could not find any documentation/info for the `sshleifer/tiny-mbart` model. How big it is? How it was trained? What is the peformance, etc.? Did I miss something? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 21, 2020, 3:34pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/2 "2020-07-21T15:34:48Z")
AFAIK, i think this model is created for testing purpose. Pinging @sshleifer for confirmation. ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 21, 2020, 3:36pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/3 "2020-07-21T15:36:27Z")
…any chance for a production ready smaller mBART? Like `mbart-base`? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 21, 2020, 3:41pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/4 "2020-07-21T15:41:19Z")
not that I know of. ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [July 21, 2020, 4:25pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/5 "2020-07-21T16:25:50Z")
How small do you want @martin-avrios ? For english-romanian or cc25? I will distill anything en-ro if you can figure out TPU 🙂 `tiny-mBART` is just for testing purposes, it’s randomly initialized. ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 21, 2020, 4:40pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/6 "2020-07-21T16:40:12Z")
For cc25. I am working with mostly german and some french and italian docs and if I don’t freeze embeds than `mbart-large-cc25` does not fit into the 16GB V100 (which is the largest Google Cloud has for a single GPU). Not even with batch size 1. So I thought I try a smaller mBART. But yes, TPU would be even more awesome because I could experiment with a lot more then. ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [July 28, 2020, 7:19pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/7 "2020-07-28T19:19:04Z")
Yeah I have been running everything with `--freeze_embeds`. I’m also trying to figure out how to trim the embeddings, as most of them aren’t used, but blocked on > > > Similar to #474, I want to restrict my vocabulary, and then save a new model fil…e that uses the restricted vocabulary. > > > I tried to do this by saving a vocabulary, modifying it, and then figuring out how to save the restricted model, but I found that even without any modification, running \`spm\_export\_vocab\` follow by \`spm\_encode --vocabulary\` produces different results. > > For example, > \`\`\`bash > echo "Șeful ONU declară că nu există soluții militare în Siria" | spm\_encode --model enro\_trimmed/sentence.bpe.model > = \> ▁Ș e ful ▁ONU ▁de cla ră ▁că ▁nu ▁există ▁solu ții ▁militare ▁în ▁Siria > \`\`\` > > \`\`\`bash > spm\_export\_vocab --model enro\_trimmed/sentence.bpe.model --output=sp\_vocab.txt > echo "Șeful ONU declară că nu există soluții militare în Siria" | spm\_encode --model enro\_trimmed/sentence.bpe.model --vocabulary sp\_vocab.txt > =\> ▁ Ș e f u l ▁ O N U ▁ d e c l a r ă ▁ c ă ▁ n u ▁ e x i s t ă ▁ s o l u ț i i ▁ m i l i t a r e ▁ î n ▁ S i r i a > \`\`\` > Is this expected behavior? > My end goal is that in python, \`spm.encode\_as\_ids\` only produces ids \< length of the restricted vocab, so if there is a more direct way to achieve that objective I would love to know it! > > Thanks! I’m also happy to finetune mbart-cc25 on a public dataset for you, if that would help. Also, have you tried using Marian? ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 30, 2020, 6:53am UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/8 "2020-07-30T06:53:20Z")
Have not tried Marian yet but it seems interesting. It’s for translation, right? But since I used translation mode for my problem it could definitely work. Also I found excellent pre-trained models on TF Hub but they are not fine-tunable (according to the page). TransformerXL pre-trained on Wiki40B (a new dataset in 40 languages), separate model for each language. At least for me this would be the ultimate model. Seq2seq, unlimited sequence length and 41 languages. See [https://tfhub.dev/google/collections/wiki40b-lm/1](https://tfhub.dev/google/collections/wiki40b-lm/1) ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 3, 2020, 10:47pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/9 "2020-08-03T22:47:03Z")
yes its for translation. Try it out! It’s much smaller/faster than bart and we have 1100 language pairs: [https://huggingface.co/models?search=Helsinki-NLP](https://huggingface.co/models?search=Helsinki-NLP) ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 3, 2020, 10:47pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/10 "2020-08-03T22:47:35Z")
I’ve never actually finetuned it so let me know if there are any bugs! ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [August 4, 2020, 3:25pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/11 "2020-08-04T15:25:27Z")
I noticed there is a de-de pair which is exactly what I need but I wonder who else needs this and what for? Looks like somebody already tried summarization in german? ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 6, 2020, 8:03pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/12 "2020-08-06T20:03:01Z")
link to model? ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [August 6, 2020, 8:30pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/13 "2020-08-06T20:30:15Z")
[https://huggingface.co/Helsinki-NLP/opus-mt-de-de](https://huggingface.co/Helsinki-NLP/opus-mt-de-de) ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 7, 2020, 2:49pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/14 "2020-08-07T14:49:22Z")
I believe that is an accident, not a summarization model. ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 7, 2020, 9:01pm UTC](https://discuss.huggingface.co/t/tiny-mbart-doc-info/414/15 "2020-08-07T21:01:34Z")
According to Jörg Tiedemann, the author, it’s a paraphraser, rather than a summarizer. He writes, > There are texts with alternative translations into the same language, which I used for training intralingual models like this one. They are maybe not very useful at this moment as they probably just copy the input text. That this only obtains 40 BLEU is not that strange as this is tested with paraphrased sentences. Note also that this model is really rather a paraphrase model than a summarisation model as they seem to use it in the discussion