# Understanding the encoder-decoder loss calculation VS CLM loss

**URL:** <https://discuss.huggingface.co/t/understanding-the-encoder-decoder-loss-calculation-vs-clm-loss/74175>\
**Category:** Beginners\
**Created:** [February 21, 2024, 4:56pm UTC](https://discuss.huggingface.co/t/understanding-the-encoder-decoder-loss-calculation-vs-clm-loss/74175 "2024-02-21T16:56:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![SammyC](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sammyc/32/22824_2.png) [@SammyC](https://discuss.huggingface.co/u/SammyC)\
**Post date:** [February 21, 2024, 4:56pm UTC](https://discuss.huggingface.co/t/understanding-the-encoder-decoder-loss-calculation-vs-clm-loss/74175/1 "2024-02-21T16:56:39Z")

</div>

Hello!

I’m trying to understand th encoder-decoder loss, from modeling\_encoder\_decoder.py line 640, the loss is computed “independent from decoder (as some shift the logits inside them)”: why do we NOT need to shift the logits for next token prediction just like in CLM (e.g. modeling\_bert.py line 1255)?

In the loss implemented in modeling\_bert.py line 1255:

```auto
# we are doing next-token prediction; shift prediction scores and input ids by one 
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
labels = labels[:, 1:].contiguous()
loss_fct = CrossEntropyLoss()
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))

```

we align the predicted\_scores with the right-shifted labels that are one step ahead by `labels=labels[:,1:]` and `shifted_prediction_scores = prediction_scores[:, :-1, :]`, so we are predicting the next token. However, from the loss in modeling\_encoder\_decoder.py line 640:

```auto
logits = decoder_outputs.logits if return_dict else decoder_outputs[0]
loss_fct = CrossEntropyLoss()
loss = loss_fct(logits.reshape(-1, self.decoder.config.vocab_size), labels.view(-1))

```

it is asking the model to learn to reconstruct the shifted labels instead of predicting the next. If my understanding here is correct, how would it be able to generate next tokens from a decoder\_start\_token (like in any seq2seq model)?

The reason I’m asking this is because I’m implementing a custom encoder-decoder following modeling\_encoder\_decoder.py and am training everything from scratch, and during generation it is always outputing the decoder\_start\_token, and I think it’s because the model learns to reconstruct the decoder\_input\_id rather than predicting the next. Since the first token it sees is the decoder\_start\_token, it reconstructs it, append to decoder\_input\_id, generate with the updated decoder\_input\_id, take the last item in the output\_sequence (which is still a decoder\_start\_token), append again and so on. In the end I just get a sequence of decoder\_start\_token.

Could someone please help me understand the procedure or point out where I made a mistake?

Thank you very much :))