# What should decoder\_input\_ids be when pre-training mBART?

**URL:** <https://discuss.huggingface.co/t/what-should-decoder-input-ids-be-when-pre-training-mbart/159819>\
**Category:** Models\
**Created:** [June 18, 2025, 3:13pm UTC](https://discuss.huggingface.co/t/what-should-decoder-input-ids-be-when-pre-training-mbart/159819 "2025-06-18T15:13:32Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Philomath868](https://avatars.discourse-cdn.com/v4/letter/p/b487fb/32.png) [@Philomath868](https://discuss.huggingface.co/u/Philomath868)\
**Post date:** [June 18, 2025, 3:13pm UTC](https://discuss.huggingface.co/t/what-should-decoder-input-ids-be-when-pre-training-mbart/159819/1 "2025-06-18T15:13:32Z")

</div>

A simple question: when pre-training (denoising) with mBART-50, should decoder\_input\_ids be based upon **labels** (the correct text) or upon **input\_ids** (the corrupted text)?

In some reference code I found, decoder\_input\_ids appears to be input\_ids shifted right: [batch[“decoder\_input\_ids”] = self.shift\_tokens\_right(batch[“input\_ids”])](https://github.com/morganmcg1/rotobart/blob/cd29e458d0adfa270d47e11614427935541aef0c/data_collator.py#L247).

However, from many other sources, it seems that decoder\_input\_ids is always based on labels, from which the model learns via cross-attention. (See also [possible mistake in documentation · Issue #11357 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/11357) where it is claimed that the mention of input\_ids is a bug, as labels is actually used).

So which one is it? I would be immensely grateful for any clarification or guidance!