# Bug in gradient accumulation training\_step in huggingface Trainer?

**URL:** <https://discuss.huggingface.co/t/bug-in-gradient-accumulation-training-step-in-huggingface-trainer/114995>\
**Category:** 🤗Transformers\
**Created:** [November 1, 2024, 11:13am UTC](https://discuss.huggingface.co/t/bug-in-gradient-accumulation-training-step-in-huggingface-trainer/114995 "2024-11-01T11:13:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nguyen-brat](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nguyen-brat](https://discuss.huggingface.co/u/nguyen-brat)\
**Post date:** [November 1, 2024, 11:13am UTC](https://discuss.huggingface.co/t/bug-in-gradient-accumulation-training-step-in-huggingface-trainer/114995/1 "2024-11-01T11:13:24Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/b/7bc1b5af83a217ca9ea73ca32aa280efe0ab0d3c.png)  
Hello, when I use huggingface trainer and apply gradient accumulation to loss it is significantly bigger than when I not use it, so I checked the code but there is a confusion in line 3604 of Trainer class in training\_step function, they multiple loss by gradient\_accumulation\_steps then when return they just device it by the same gradient\_accumulation\_steps so what is the purpose of this after all ? (my transformers version is 4.46.1)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 2, 2024, 1:14am UTC](https://discuss.huggingface.co/t/bug-in-gradient-accumulation-training-step-in-huggingface-trainer/114995/2 "2024-11-02T01:14:47Z")

</div>

The github version has been slightly changed…  
I don’t know if it’s fixed or not.

> <https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L3645>

I don’t think the part in question has changed…  
I wonder what is going right…

---

<div class="post-metadata">

**Author:** ![nguyen-brat](https://avatars.discourse-cdn.com/v4/letter/n/ecae2f/32.png) [@nguyen-brat](https://discuss.huggingface.co/u/nguyen-brat)\
**Post date:** [November 2, 2024, 2:40am UTC](https://discuss.huggingface.co/t/bug-in-gradient-accumulation-training-step-in-huggingface-trainer/114995/3 "2024-11-02T02:40:26Z")

</div>

when I update the dev version from GitHub directly, the loss doesn’t multiply with the number of gradient accumulations anymore, it seems like some dev has messed it up in the 4.46.1 version 😅 . But do you have any idea why when the custom loss function is provided we need to multiply the loss by the gradient accumulation\_steps? Moreover, it is crazy that when I update to 4.47.0.dev0 version the training time is reduced from 14 hours to 5 hours, I really don’t know what is going on here 😅

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 2, 2024, 3:03am UTC](https://discuss.huggingface.co/t/bug-in-gradient-accumulation-training-step-in-huggingface-trainer/114995/4 "2024-11-02T03:03:33Z")

</div>

No, I have no idea!😆  
I know I’ve never used the trainer function, but that’s not the point, this means that this is probably a tough issue.

I guess they are trying to fix…  
The following commitments, for example, are just that.  
But it’s probably not completely fixed yet. They might.  
Even today, there are easy bugs here and there in various libraries that are routinely fixed. If information is not smoothly provided from the user side to the developer side, bugs will not be noticed and fixed…

> <https://github.com/huggingface/transformers/commit/dca93ca076c68372dcf3ad1239a2119afdda629c>
>
> \* replace total\_batched\_samples with step while counting grad accum step
> 
> \* re…move unused variable
> 
> \* simplify condition for update step
> 
> \* fix format by ruff
> 
> \* simplify update step condition using accelerator.sync\_gradients
> 
> \* simplify update condition using do\_sync\_step
> 
> \* remove print for test
> 
> \---------
> 
> Co-authored-by: Zach Mueller \<muellerzr@gmail.com\>

But I don’t think we have an issue on this question yet…  
I haven’t even had a github account for a week, so I can’t say for sure…

> **[Issues · huggingface/transformers](https://github.com/huggingface/transformers/issues)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - Issues · huggingface/transformers

[https://github.com/huggingface/transformers/pulls](https://github.com/huggingface/transformers/pulls)
