# Using gradient\_checkpointing=True in Trainer causes error with LLaMA

**URL:** <https://discuss.huggingface.co/t/using-gradient-checkpointing-true-in-trainer-causes-error-with-llama/46124>\
**Category:** 🤗Transformers\
**Created:** [July 8, 2023, 11:25pm UTC](https://discuss.huggingface.co/t/using-gradient-checkpointing-true-in-trainer-causes-error-with-llama/46124 "2023-07-08T23:25:21Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tsheasha](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tsheasha/32/17449_2.png) [@tsheasha](https://discuss.huggingface.co/u/tsheasha)\
**Post date:** [July 8, 2023, 11:25pm UTC](https://discuss.huggingface.co/t/using-gradient-checkpointing-true-in-trainer-causes-error-with-llama/46124/1 "2023-07-08T23:25:21Z")

</div>

Hey,

I am trying to fine tune llama using the transformers library. I noticed when I set gradient\_checkpointing=True in the trainer args i get the following error:

Expects `BACKWARD_PRE` or `BACKWARD_POST` state but got HandleTrainingState.FORWARD

Has anyone come across this before?

@sgugger I have seen you respond to gradient\_checkpointing questions before, so thought I would tag you

 ![Screenshot 2023-07-09 at 00.25.04](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/6/56d914d6c31dd6e5945d4175aadb4dfab715045e.jpeg)

---

<div class="post-metadata">

**Author:** ![tsheasha](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tsheasha/32/17449_2.png) [@tsheasha](https://discuss.huggingface.co/u/tsheasha)\
**Post date:** [July 8, 2023, 11:58pm UTC](https://discuss.huggingface.co/t/using-gradient-checkpointing-true-in-trainer-causes-error-with-llama/46124/2 "2023-07-08T23:58:48Z")

</div>

> [@tsheasha](#):
>
> gradient\_checkpointing

This happens when i have set use\_reentrant=False in the call to torch.checkpoint, when i set use\_reentrant=True, i get an error when torch.compile() runs on the model