# A streaming dataset's memory footprint continually grows

**URL:** <https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404>\
**Category:** 🤗Datasets\
**Created:** [June 15, 2025, 6:42pm UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404 "2025-06-15T18:42:00Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 15, 2025, 10:17pm UTC](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404/2 "2025-06-15T22:17:09Z")

</div>

I think it’s similar to this phenomenon.

> <https://github.com/huggingface/datasets/issues/7269>
>
> \### Describe the bug
> 
> I try to use a dataset with streaming=True, the issue I …have is that the RAM usage becomes higher and higher until it is no longer sustainable.
> 
> I understand that huggingface store data in ram during the streaming, and more worker in dataloader there are, more a lot of shard will be stored in ram, but the issue I have is that the ram usage is not constant. So after each new shard loaded, the ram usage will be higher and higher.
> 
> \### Steps to reproduce the bug
> 
> You can run this code and see you ram usage, after each shard of 255 examples, your ram usage will be extended.
> \`\`\`py
> from datasets import load\_dataset
> from torch.utils.data import DataLoader
> 
> dataset = load\_dataset("WaveGenAI/dataset", streaming=True)
> 
> dataloader = DataLoader(dataset\["train"\], num\_workers=3)
> 
> for i, data in enumerate(dataloader):
> print(i, end="\\r")
> \`\`\`
> 
> \### Expected behavior
> 
> The Ram usage should be always the same (just 3 shards loaded in the ram).
> 
> \### Environment info
> 
> \- \`datasets\` version: 3.0.1
> \- Platform: Linux-6.10.5-arch1-1-x86\_64-with-glibc2.40
> \- Python version: 3.12.4
> \- \`huggingface\_hub\` version: 0.26.0
> \- PyArrow version: 17.0.0
> \- Pandas version: 2.2.3
> \- \`fsspec\` version: 2024.6.1

> [@OOM issue with large dataset streaming](https://discuss.huggingface.co/t/oom-issue-with-large-dataset-streaming/144488):
>
> For training, we load a multiple datasets (each with 1000+ 1GB+ each arrow files saved using dataset) with load\_dataset(…streaming=true) and then interleave them together with interleave\_dataset. Even with streaming dataset we are seeing OOM at the interleaving\_datasets step. Note no data has been read from datasets yet. Sharding the initial datasets per gpu before interleaving alleviated the memory pressure. Why should there be such memory issues with streaming dataset?

---

_[View the full topic](https://discuss.huggingface.co/t/a-streaming-datasets-memory-footprint-continually-grows/159404)._