# Prevent iterable dataset from consuming all the rams **URL:** \ **Category:** Beginners\ **Created:** [June 22, 2025, 2:15pm UTC](https://discuss.huggingface.co/t/prevent-iterable-dataset-from-consuming-all-the-rams/160276 "2025-06-22T14:15:12Z")\ **Posts on this page:** 1\ **Showing post:** 2
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 23, 2025, 12:21am UTC](https://discuss.huggingface.co/t/prevent-iterable-dataset-from-consuming-all-the-rams/160276/2 "2025-06-23T00:21:56Z")
That’s strange… Maybe it’s an unknown memory leak or `keep_in_memory` is causing it…? > > > \### System Info > > pytorch 1.11.0 > py 3.8 > cuda 11.3 > transformers 4.26.1 > data…sets 2.9.0 > > \### Who can help? > > @sgugger > > \### Information > > \- \[\] The official example scripts > \- \[X\] My own modified scripts > > \### Tasks > > \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...) > \- \[X\] My own task or dataset (give details below) > > \### Reproduction > > import os > import time > import datetime > import sys > import numpy as np > import random > > import torch > from torch.utils.data import Dataset, DataLoader, random\_split, RandomSampler, SequentialSampler,DistributedSampler,BatchSampler > torch.manual\_seed(42) > > from transformers import GPT2LMHeadModel, GPT2Tokenizer, GPT2Config, GPT2Model,DataCollatorForLanguageModeling,AutoModelForCausalLM > from transformers import AdamW, get\_linear\_schedule\_with\_warmup > > hf\_model\_path ='./Wenzhong-GPT2-110M' > tokenizer = GPT2Tokenizer.from\_pretrained(hf\_model\_path) > tokenizer.add\_special\_tokens({'pad\_token': '\<|pad|\>'}) > > from datasets import load\_dataset > > gpus=8 > max\_len = 576 > batch\_size\_node = 17 > save\_step = 5000 > gradient\_accumulation = 2 > dataloader\_num = 4 > max\_step = 351000\*1000//batch\_size\_node//gradient\_accumulation//gpus > \#max\_step = -1 > print("total\_step:%d"%(max\_step)) > import datasets > > datasets.\_\_version\_\_ > > > dataset = load\_dataset("text", data\_files="./gpt\_data\_v1/\*",split='train',cache\_dir='./dataset\_cache',streaming=True) > > print('load over') > shuffled\_dataset = dataset.shuffle(seed=42) > print('shuffle over') > > def dataset\_tokener(example,max\_lenth=max\_len): > example\['text'\] = list(map(lambda x : x.strip()+'\<|endoftext|\>',example\['text'\] )) > return tokenizer(example\['text'\], truncation=True, max\_length=max\_lenth, padding="longest") > #return tokenizer(example\[0\], truncation=True, max\_length=max\_lenth, padding="max\_length") > > new\_new\_dataset = shuffled\_dataset.map(dataset\_tokener, batched=True, remove\_columns=\["text"\]) > print('map over') > > > configuration = GPT2Config.from\_pretrained(hf\_model\_path, output\_hidden\_states=False) > > model = AutoModelForCausalLM.from\_pretrained(hf\_model\_path) > > model.resize\_token\_embeddings(len(tokenizer)) > > seed\_val = 42 > > random.seed(seed\_val) > np.random.seed(seed\_val) > torch.manual\_seed(seed\_val) > torch.cuda.manual\_seed\_all(seed\_val) > > from transformers import Trainer,TrainingArguments > import os > print("strat train") > training\_args = TrainingArguments(output\_dir="./test\_trainer", > num\_train\_epochs=1.0, > report\_to="none", > do\_train=True, > dataloader\_num\_workers=dataloader\_num, > local\_rank=int(os.environ.get('LOCAL\_RANK', -1)), > overwrite\_output\_dir=True, > logging\_strategy='steps', > logging\_first\_step=True, > logging\_dir="./logs", > log\_on\_each\_node=False, > per\_device\_train\_batch\_size=batch\_size\_node, > warmup\_ratio=0.03, > save\_steps=save\_step, > save\_total\_limit=5, > gradient\_accumulation\_steps=gradient\_accumulation, > max\_steps=max\_step, > disable\_tqdm=False, > data\_seed=42 > ) > > trainer = Trainer( > model=model, > args=training\_args, > train\_dataset=new\_new\_dataset, > eval\_dataset=None, > tokenizer=tokenizer, > # Data collator will default to DataCollatorWithPadding, so we change it. > data\_collator=DataCollatorForLanguageModeling(tokenizer,mlm=False), > #compute\_metrics=compute\_metrics if training\_args.do\_eval and not is\_torch\_tpu\_available() else None, > #preprocess\_logits\_for\_metrics=preprocess\_logits\_for\_metrics > #if training\_args.do\_eval and not is\_torch\_tpu\_available() > #else None, > ) > trainer.train(resume\_from\_checkpoint=True) > > \### Expected behavior > > use the train code uppper > my dataset ./gpt\_data\_v1 have 1000 files, each file size is 120mb > start cmd is : python -m torch.distributed.launch --nproc\_per\_node=8 my\_train.py > here is result: > > !\[image\](https://user-images.githubusercontent.com/15223544/223025177-6f2ec55f-5b2f-42bf-b6cc-dd5f8a8232fe.png) > > here is memory usage monitor in 12 hours > > !\[image\](https://user-images.githubusercontent.com/15223544/223027076-14e32e8b-9608-4282-9a80-f15d0277026d.png) > > every dataloader work allocate over 24gb cpu memory > according to memory usage monitor in 12 hours,sometime small memory releases, but total memory usage is increase. > i think datasets streaming mode should not used so much memery,so maybe somewhere has memory leak. > [@Multiprocessing map taking too much memory footprint](https://discuss.huggingface.co/t/multiprocessing-map-taking-too-much-memory-footprint/27238): > > Hi, I use map to preprocess my super large datasets(about 450G). I run map function on 1 node/1 gpu to finish tokenization, while I got stuck at transferring to multi-node training on 10 nodes/ 8 gpus, num\_proc = 18). I check my system status, and it seems that my memory is out of flow, which didn’t happen at 1node/1 gpu condition. How could I fix it? Thanks! Here’s my 1 node/1 gpu training system status, out of flow happen when transferring to 8gpus total Memory:95… --- _[View the full topic](https://discuss.huggingface.co/t/prevent-iterable-dataset-from-consuming-all-the-rams/160276)._