# XLMProphetNet returning different results when using padding

**URL:** <https://discuss.huggingface.co/t/xlmprophetnet-returning-different-results-when-using-padding/43421>\
**Category:** 🤗Transformers\
**Created:** [June 15, 2023, 8:06pm UTC](https://discuss.huggingface.co/t/xlmprophetnet-returning-different-results-when-using-padding/43421 "2023-06-15T20:06:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![alexayala](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@alexayala](https://discuss.huggingface.co/u/alexayala)\
**Post date:** [June 15, 2023, 8:06pm UTC](https://discuss.huggingface.co/t/xlmprophetnet-returning-different-results-when-using-padding/43421/1 "2023-06-15T20:06:24Z")

</div>

Issue also available here: [XLMProphetNet returning different results when using padding · Issue #24289 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/24289)

I’m noticing a behavior with the XLMProphetNet model where when I use padding for the same decoder\_input\_ids then the resulting decoder output differs. This results in a different loss being computed for a batch versus each sample individually without padding.

**Code to reproduce:**

```auto
from transformers import XLMProphetNetTokenizer, XLMProphetNetForConditionalGeneration
import torch

tokenizer = XLMProphetNetTokenizer.from_pretrained("microsoft/xprophetnet-large-wiki100-cased-xglue-ntg")
model = XLMProphetNetForConditionalGeneration.from_pretrained("microsoft/xprophetnet-large-wiki100-cased-xglue-ntg").eval()

enc_input = tokenizer("test", return_tensors="pt")
input_ids = enc_input.input_ids
attention_mask = enc_input.attention_mask

dec_input_ids = torch.tensor([[model.config.decoder_start_token_id]], dtype=torch.int64)
dec_attention_mask = torch.tensor([[1]], dtype=torch.int64)

dec_input_ids_pad = torch.tensor([[model.config.decoder_start_token_id, model.config.pad_token_id]], dtype=torch.int64)
dec_attention_mask_pad = torch.tensor([[1, 0]], dtype=torch.int64)

out1 = model(
    input_ids=input_ids, attention_mask=attention_mask,
    decoder_input_ids=dec_input_ids, decoder_attention_mask=dec_attention_mask
)

out2 = model(
    input_ids=input_ids, attention_mask=attention_mask,
    decoder_input_ids=dec_input_ids_pad, decoder_attention_mask=dec_attention_mask_pad
)

torch.isclose(out1.logits, out2.logits[:, 0], atol=1e-1).all() # false

```

Is this expected? During training I’m noticing that runs that don’t use any padding (accumulation iterations) are converging whereas any of my runs that use padding are failing to train properly.

---

<div class="post-metadata">

**Author:** ![alexayala](https://avatars.discourse-cdn.com/v4/letter/a/a183cd/32.png) [@alexayala](https://discuss.huggingface.co/u/alexayala)\
**Post date:** [June 16, 2023, 6:59pm UTC](https://discuss.huggingface.co/t/xlmprophetnet-returning-different-results-when-using-padding/43421/2 "2023-06-16T18:59:47Z")

</div>

@patrickvonplaten Do you have any insight on why this is happening? Thanks!