# Trainer + Datasets + Pytorch Dataloader Workers - how to manage memory usage?

**URL:** <https://discuss.huggingface.co/t/trainer-datasets-pytorch-dataloader-workers-how-to-manage-memory-usage/152688>\
**Category:** 🤗Transformers\
**Created:** [April 28, 2025, 5:10pm UTC](https://discuss.huggingface.co/t/trainer-datasets-pytorch-dataloader-workers-how-to-manage-memory-usage/152688 "2025-04-28T17:10:50Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![entropy](https://avatars.discourse-cdn.com/v4/letter/e/5f9b8f/32.png) [@entropy](https://discuss.huggingface.co/u/entropy)\
**Post date:** [April 28, 2025, 5:10pm UTC](https://discuss.huggingface.co/t/trainer-datasets-pytorch-dataloader-workers-how-to-manage-memory-usage/152688/1 "2025-04-28T17:10:50Z")

</div>

I’m trying to train a model on a large (600GB on disk) dataset of pre-computed embedding vectors.

Training is i/o bound (loading embeddings from disk). I’m trying to increase the speed of dataloading by using multiple workers, but running into memory issues.

Some benchmarks:

Using 32 workers, I get about 3hr/epoch but OOM about 45 minutes in  
Using 0 workers, I get about 8hr/epoch with no memory issues  
Using 32\>n\>0 workers slows down the rate of memory consumption, but doesn’t stop the core issue.

I believe this issue is caused by pytorch’s dataloader workers replicating memory in-process ([link](https://github.com/pytorch/pytorch/issues/13246)), but I don’t know how to stop that. Is there a fix to this, or a way to periodically restart pytorch dataloader workers to clear memory?

Some notes:  
Datasets are formatted for pytorch before training  
Changing to an iterable dataset slows down the rate of memory accumulation for the same number of workers, but doesn’t stop it

this is using:  
torch==2.2.1  
transformers==4.51.3  
datasets==2.19.1  
accelerate==1.6.0

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 29, 2025, 10:59am UTC](https://discuss.huggingface.co/t/trainer-datasets-pytorch-dataloader-workers-how-to-manage-memory-usage/152688/2 "2025-04-29T10:59:07Z")

</div>

I found a useful post about PyTorch’s DataLoader.

> **[DataLoader memory usage keeps increasing](https://discuss.pytorch.org/t/dataloader-memory-usage-keeps-increasing/200461)**
>
> Hello, i am trying to use pytorchs Dataset and DataLoader to load a large dataset of several 100GB. This is of course too large to be stored in RAM, so parallel, lazy loading is needed. I am trying to load one large HDF file with a combination of...