# BERTology compute\_heads\_importance without zero grad

**URL:** <https://discuss.huggingface.co/t/bertology-compute-heads-importance-without-zero-grad/1391>\
**Category:** Intermediate\
**Created:** [October 7, 2020, 12:50am UTC](https://discuss.huggingface.co/t/bertology-compute-heads-importance-without-zero-grad/1391 "2020-10-07T00:50:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ricardo-moura](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ricardo-moura/32/431_2.png) [@ricardo-moura](https://discuss.huggingface.co/u/ricardo-moura)\
**Post date:** [October 7, 2020, 12:50am UTC](https://discuss.huggingface.co/t/bertology-compute-heads-importance-without-zero-grad/1391/1 "2020-10-07T00:50:01Z")

</div>

Hello,  
analyzing the code from [https://github.com/huggingface/transformers/blob/master/examples/bertology/run\_bertology.py](https://github.com/huggingface/transformers/blob/master/examples/bertology/run_bertology.py) I could understand that the head\_importance variable is being incremented by the head\_mask’s gradients. However, these gradients are not being turned to zero after each iteration. In PyTorch the gradients are accumulated and there is a recommendation to turn it to zero after each iteration. Wouldn’t be adequate to call head\_mask.grad.data.zero\_() at line 110?

Regards