# Trainer leaked memory?

**URL:** <https://discuss.huggingface.co/t/trainer-leaked-memory/38750>\
**Category:** DeepSpeed\
**Created:** [May 5, 2023, 5:29am UTC](https://discuss.huggingface.co/t/trainer-leaked-memory/38750 "2023-05-05T05:29:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sp4912](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sp4912](https://discuss.huggingface.co/u/sp4912)\
**Post date:** [May 5, 2023, 5:29am UTC](https://discuss.huggingface.co/t/trainer-leaked-memory/38750/1 "2023-05-05T05:29:52Z")

</div>

I am trying to train Llama-7B on a batch size of 1 using deepspeed and huggingface trainer. I have 48GB of memory on my GPU. I am able to train, but after training there appears to be 27GB of residual memory that was not there just before trainer.train() (the memory appears to be set there exactly when trainer.train() is called, based on nvidia-smi calls inside my script).

Deleting the model, deleting the trainer, torch.cuda.empty\_cache() all do nothing to remove that memory. How can I edit this memory so that I can continue on in my script?

---

<div class="post-metadata">

**Author:** ![vergilus](https://avatars.discourse-cdn.com/v4/letter/v/0ea827/32.png) [@vergilus](https://discuss.huggingface.co/u/vergilus)\
**Post date:** [October 15, 2024, 11:31am UTC](https://discuss.huggingface.co/t/trainer-leaked-memory/38750/2 "2024-10-15T11:31:16Z")

</div>

did you use deepspeed? try the trainer without deepspeed