# Export M2M100 model to ONNX **URL:** \ **Category:** 🤗Transformers\ **Created:** [May 9, 2022, 6:53am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694 "2022-05-09T06:53:33Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![NNDam](https://avatars.discourse-cdn.com/v4/letter/n/d26b3c/32.png) [@NNDam](https://discuss.huggingface.co/u/NNDam)\ **Post date:** [May 9, 2022, 6:53am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/1 "2022-05-09T06:53:33Z")
I’ve port `facebook/m2m100_418M` to ONNX for translation task using [this](https://huggingface.co/docs/transformers/serialization) but when visualize by netron, it required 4 inputs: **input\_ids** , **attention\_mask** , **decoder\_input\_ids** , **decoder\_attention\_mask** and I don’t know how to inference with ONNX-runtime. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/ade61b5ea66b2da674003666e46cc552957ea84c.png) How can I solve this problem ? Thanks in advance for your help. ---
**Author:** ![double](https://avatars.discourse-cdn.com/v4/letter/d/8baadc/32.png) [@double](https://discuss.huggingface.co/u/double)\ **Post date:** [May 16, 2022, 4:25pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/2 "2022-05-16T16:25:23Z")
Did you find a solution? ---
**Author:** ![omoekan](https://avatars.discourse-cdn.com/v4/letter/o/45deac/32.png) [@omoekan](https://discuss.huggingface.co/u/omoekan)\ **Post date:** [June 30, 2022, 4:50pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/3 "2022-06-30T16:50:26Z")
I have the same issue. Have you found a solution yet? ---
**Author:** ![Jour](https://avatars.discourse-cdn.com/v4/letter/j/e9a140/32.png) [@Jour](https://discuss.huggingface.co/u/Jour)\ **Post date:** [July 7, 2022, 12:31pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/4 "2022-07-07T12:31:16Z")
I tried to convert this model with onnx by adding this type of the task `python3.8 -m transformers.onnx --model=facebook/m2m100_418M onnx/ --feature=seq2seq-lm-with-past`, but in this case it says that it needs 54 inputs, otherwise I have the same problem. I know that the model needs the input and output language but I can’t really understand how to use the model with onnx. An example would be welcome 😉 I also looked for indications in the commit of the model: [M2M100 support for ONNX export by michaelbenayoun · Pull Request #15193 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/15193). I think it can be useful. ---
**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\ **Post date:** [July 8, 2022, 9:29am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/5 "2022-07-08T09:29:33Z")
cc @lewtun ---
**Author:** ![echoRG](https://avatars.discourse-cdn.com/v4/letter/e/df788c/32.png) [@echoRG](https://discuss.huggingface.co/u/echoRG)\ **Post date:** [August 22, 2022, 1:48am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/6 "2022-08-22T01:48:25Z")
Also having the same question, Could I have an example for this m2m-100 onnx model? It will be very helpful. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [August 22, 2022, 2:52pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/7 "2022-08-22T14:52:05Z")
Hi folks, the best way to run inference with ONNX models is via the `optimum` library. This library allows you to inject ONNX models directly in the `pipeline()` function from `transformers` and thus skip all the annoying pre- and post-processing steps 🙂 Here’s a demo for M2M100 based on the [docs](https://huggingface.co/docs/optimum/onnxruntime/modeling_ort#optimum.onnxruntime.ORTModelForSeq2SeqLM.forward.example-2): ```auto from transformers import AutoTokenizer, pipeline from optimum.onnxruntime import ORTModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("facebook/m2m100_418M") # `from_transformers` will export the model to ONNX on-the-fly 🤯 model = ORTModelForSeq2SeqLM.from_pretrained("facebook/m2m100_418M", from_transformers=True) onnx_translation = pipeline("translation_en_to_de", model=model, tokenizer=tokenizer) text = "My name is Lewis." # returns [{'translation_text': 'Mein Name ist Lewis.'}] pred = onnx_translation(text) ``` Hope that helps! ---
**Author:** ![awaiskaleem](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/awaiskaleem/32/1682_2.png) [@awaiskaleem](https://discuss.huggingface.co/u/awaiskaleem)\ **Post date:** [November 30, 2022, 1:37pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/8 "2022-11-30T13:37:15Z")
Running into following error when I run code as is from @lewtun `AttributeError: type object 'FeaturesManager' has no attribute 'determine_framework'` Using following version: torch → ‘1.10.0’ transformers → ‘4.20.1’ ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [November 30, 2022, 1:52pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/9 "2022-11-30T13:52:08Z")
cc @fxmarty who might be able to take a look 🙏 ---
**Author:** ![awaiskaleem](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/awaiskaleem/32/1682_2.png) [@awaiskaleem](https://discuss.huggingface.co/u/awaiskaleem)\ **Post date:** [November 30, 2022, 1:57pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/10 "2022-11-30T13:57:12Z")
Thanks, also, not sure where is target language ‘de’ mentioned above in tokenizer/model. Greatly appreciate your help. ---
**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\ **Post date:** [December 1, 2022, 4:30pm UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/11 "2022-12-01T16:30:49Z")
Hi @awaiskaleem , `transformers==4.20.1` is 5 months old. Could you try to update (`pip install --upgrade transformers`)? Current supported stable version is `4.20.0`. The code snippet from @lewtun works well for me with `transformers==4.20.0` and `optimum==1.5.1`. Additionally @NNDam , @double @omoekan , @Jour , @echoRG , @awaiskaleem , I wanted to let you know that the ONNX export through `transformers.onnx` will likely soon rely on a soft dependency to `optimum.exporters` where all things export will be maintained. You can check [the documentation here](https://huggingface.co/docs/optimum/main/en/exporters/onnx/usage_guides/export_a_model). Now, specifically for M2M100, keep in mind that it is a seq2seq (translation) model! Hence, it uses both an encoder and decoder, as detailed in [transformers doc](https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/m2m_100#overview). In transformers, the standard use is to `model.generate(**inputs)`. However, by default the ONNX export can not handle the loop that there is in the decoder: [transformers/utils.py at d51e7c7e8265d69db506828dce77eb4ef9b72157 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/d51e7c7e8265d69db506828dce77eb4ef9b72157/src/transformers/generation/utils.py#L2786-L2884) . Hence, when exporting to ONNX in a single file, unless you do some manual surgery on the ONNX graph, the model will be hardly usable. The solution that is currently explored & in use in [Optimum’s ORTModelForSeq2SeqLM](https://huggingface.co/docs/optimum/main/en/onnxruntime/package_reference/modeling_ort#optimum.onnxruntime.ORTModelForSeq2SeqLM) leveraging ONNX Runtime is to use two ONNX files: one for the encoder, and one for the decoder. Using Optimum main (not yet in the stable release, but you can expect it next week), `python -m optimum.exporters.onnx --model valhalla/m2m100_tiny_random --for-ort m2m100_tiny_onnx_ort`, we obtain two models: - an encoder expecting the `input_ids`, `attention_mask` - a decoder expecting `encoder_attention_mask`, `input_ids` and `encoder_hidden_states`. This follows closely transformers [decoder](https://huggingface.co/docs/transformers/v4.24.0/en/model_doc/m2m_100#transformers.M2M100Model) and [generate](https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate). ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/f5dd4dffa92298cc26edb68530ecfbb2b6b0ea48.png) ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/44f857bc2edc63c85ce62843e5981016c0fd76ae.png) So if you would like to use these exported ONNX models outside of Optimum, I simply recommend to use the above command to export and handle yourself the models then. But `ORTModelForSeq2SeqLM` is meant to save you the hassle. If you want to try it right away, feel free to try the dev version: `pip install -U git+https://github.com/huggingface/optimum.git@main` Edit 2022-12-27: Feel free to have a look at the latest release notes which includes the feature: [Release v1.6.0: Optimum CLI, Stable Diffusion ONNX export, BetterTransformer & ONNX support for more architectures · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/releases/tag/v1.6.0) ---
**Author:** ![ahmedbr](https://avatars.discourse-cdn.com/v4/letter/a/50afbb/32.png) [@ahmedbr](https://discuss.huggingface.co/u/ahmedbr)\ **Post date:** [May 29, 2023, 7:56am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/12 "2023-05-29T07:56:12Z")
This partially worked for me. I mean I was able to load the infer the model successfully but the text wasn’t translated into “de” in your example. The result I got was as following: ``` pred: [{'translation_text': 'de: My name is Lewis.'}] ``` The model I’m using is: “facebook/nllb-200-distilled-600M” ---
**Author:** ![luckyt](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@luckyt](https://discuss.huggingface.co/u/luckyt)\ **Post date:** [June 14, 2023, 8:52am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/13 "2023-06-14T08:52:39Z")
Hi there! Thanks for the insights, I have a couple of questions about the architecture of the ORT seq2seq model that I hope you could clarify. Firstly, I’m curious why Optimum requires the encoder and decoder to be loaded from two separate ONNX files, instead of a single ONNX file? I’m guessing (from a quick glance at the source code) that it’s because it utilizes two ORT inference sessions for the encoder and decoder instead of using a single session for the entire model – is there a rationale for this design? Secondly, you mentioned that the ONNX export of the model faces difficulties with the generate loop in the decoder, while the transformers model seems to handle it fine. I’m wondering what specifically in the generate loop makes it challenging for the ORT model to handle? ---
**Author:** ![fxmarty](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fxmarty/32/23782_2.png) [@fxmarty](https://discuss.huggingface.co/u/fxmarty)\ **Post date:** [June 15, 2023, 5:52am UTC](https://discuss.huggingface.co/t/export-m2m100-model-to-onnx/17694/14 "2023-06-15T05:52:15Z")
@ahmedbr Coud you fill a bug report with a reproduction script on [Issues · huggingface/optimum · GitHub](https://github.com/huggingface/optimum/issues) so that I can have a look at it? @luckyt The main reason is because you normally want to run the encoder only once, while you’d like to loop over the decoder when generating. You could say, ok why not wrap everything into a single ONNX, with say an `If` node to decide whether or not to run the encoder? Something like this with subgraphs: ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/d/3df0ea80f2417f048c3a2397be9c7706032950cc.png) This could be doable actually. The issue with that is that usability is a bit harder, as the encoder and decoder do not have the same inputs/outputs. So you would need to create fake input/outputs, which theoretically works, but may lead into errors and be a bit unintuitive. About generation, what is slightly challenging is that inputs/outputs are fixed with ONNX, and more importantly when exporting we use `torch.jit.trace` that can not handle controlflows, that are typically use to handle the without/with past (use KV cache or not) case. In the first step of the generation, you don’t use the KV cache, while in later steps you do. See [transformers/src/transformers/models/t5/modeling\_t5.py at v4.30.2 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/v4.30.2/src/transformers/models/t5/modeling_t5.py#L502) & [How does the ONNX exporter work for GenerationModel with `past_key_value`?](https://discuss.huggingface.co/t/how-does-the-onnx-exporter-work-for-generationmodel-with-past-key-value/31316)