# Bug in Summarization tutorial

**URL:** <https://discuss.huggingface.co/t/bug-in-summarization-tutorial/60566>\
**Category:** Site Feedback\
**Created:** [October 31, 2023, 1:07pm UTC](https://discuss.huggingface.co/t/bug-in-summarization-tutorial/60566 "2023-10-31T13:07:18Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hacker1337](https://avatars.discourse-cdn.com/v4/letter/h/919ad9/32.png) [@Hacker1337](https://discuss.huggingface.co/u/Hacker1337)\
**Post date:** [October 31, 2023, 1:07pm UTC](https://discuss.huggingface.co/t/bug-in-summarization-tutorial/60566/1 "2023-10-31T13:07:18Z")

</div>

Hello, I was reproducing the tutorial [Summarization](https://huggingface.co/docs/transformers/tasks/summarization)

The code seems to contain the same problem, that is discussed here [Decoding error while using DataCollatorForSeq2Seq · Issue #24433 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/24433)  
Forgetting to replace -100-s in prediction labels will lead to error  
`OverflowError: out of range integral type conversion attempted`  
And in “compute metrics” function this replacement is done only with labels, not with predictions.

```auto
def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]
    result["gen_len"] = np.mean(prediction_lens)

    return {k: round(v, 4) for k, v in result.items()}

```

This is a very tricky bug and probably it would be great to change something in the library, so that people won’t have to do it every time manually. Cuz, as you can see it leads to this hard-to-debug problem.  
It’s hard to debug because it throws an error only if the padding was used, which happens pretty randomly.  
By default generation length is 20, which is the reason, why most of the time the notebook from the tutorial executes without the error.  
But after increasing max\_gen\_len e.g. to 100 it fails much more often.

```auto
training_args = Seq2SeqTrainingArguments(
	...
    generation_max_length=100,
)

```

---

<div class="post-metadata">

**Author:** ![alpecevit](https://avatars.discourse-cdn.com/v4/letter/a/9e8a1a/32.png) [@alpecevit](https://discuss.huggingface.co/u/alpecevit)\
**Post date:** [March 21, 2024, 12:54pm UTC](https://discuss.huggingface.co/t/bug-in-summarization-tutorial/60566/2 "2024-03-21T12:54:54Z")

</div>

Hey @Hacker1337  
Did you find a solution for this? Having the same problem.

---

<div class="post-metadata">

**Author:** ![Hacker1337](https://avatars.discourse-cdn.com/v4/letter/h/919ad9/32.png) [@Hacker1337](https://discuss.huggingface.co/u/Hacker1337)\
**Post date:** [March 21, 2024, 3:48pm UTC](https://discuss.huggingface.co/t/bug-in-summarization-tutorial/60566/3 "2024-03-21T15:48:28Z")

</div>

Yes, solution is very simple. As described in the github issue, you have to simply replace -100 with padding token in predictions as well as was done with labels.

Insert this before using `predictions` values.  
`predictions = np.where(predictions != -100, predictions, tokenizer.pad_token_id)`