# CUDA out of memory error while predicting (evaluation)

**URL:** <https://discuss.huggingface.co/t/cuda-out-of-memory-error-while-predicting-evaluation/47014>\
**Category:** 🤗Transformers\
**Created:** [July 16, 2023, 8:34pm UTC](https://discuss.huggingface.co/t/cuda-out-of-memory-error-while-predicting-evaluation/47014 "2023-07-16T20:34:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sachmatkris](https://avatars.discourse-cdn.com/v4/letter/s/51bf81/32.png) [@sachmatkris](https://discuss.huggingface.co/u/sachmatkris)\
**Post date:** [July 16, 2023, 8:34pm UTC](https://discuss.huggingface.co/t/cuda-out-of-memory-error-while-predicting-evaluation/47014/1 "2023-07-16T20:34:36Z")

</div>

Hello,

I have experimented with fine-tuning ‘facebook/bart-large-mnli’ model. Since it is very big, I barely fit it in the GPU, had to reduce to batch\_size = 1 through trainingarguments, but I suceeded at training. However, now that I want to simply produce the outputs with that model, I keep on running into CUDA out of memory errors. I tried out multiple steps but nothing helped. I only found out that if I try to do the predictions on dataset of size 1, it works, but the data is still stored in the GPU despite the fact that I have torch.inference\_mode()/torch.no\_grad().

It feels like the problem is in the way information is handled by the model by storing it into GPU even though I explicitly try to avoid that. Any suggestions?

Thanks for help, this issue has been bugging me for over a week and I cant find any solutions.

---

<div class="post-metadata">

**Author:** ![MoritzLaurer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moritzlaurer/32/18932_2.png) [@MoritzLaurer](https://discuss.huggingface.co/u/MoritzLaurer)\
**Post date:** [March 22, 2024, 9:14am UTC](https://discuss.huggingface.co/t/cuda-out-of-memory-error-while-predicting-evaluation/47014/2 "2024-03-22T09:14:21Z")

</div>

had the same problem, found the solution here: [CUDA out of memory when using Trainer with compute\_metrics - #13 by morenolq](https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941/13)