# Training tokenizer takes too much RAM

**URL:** <https://discuss.huggingface.co/t/training-tokenizer-takes-too-much-ram/14256>\
**Category:** 🤗Tokenizers\
**Created:** [January 31, 2022, 10:09am UTC](https://discuss.huggingface.co/t/training-tokenizer-takes-too-much-ram/14256 "2022-01-31T10:09:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon40200501](https://avatars.discourse-cdn.com/v4/letter/a/b9e5f3/32.png) [@anon40200501](https://discuss.huggingface.co/u/anon40200501)\
**Post date:** [January 31, 2022, 10:09am UTC](https://discuss.huggingface.co/t/training-tokenizer-takes-too-much-ram/14256/1 "2022-01-31T10:09:06Z")

</div>

Hello, I wanted to train my own tokenizer on multi-lingual corpus (115GB of oscar and mc4 data in 15 languages) . My machine has only 16GB RAM so I wrote a generator for this task.  
The problem is it still uses all my RAM. It progressively adds up from using 5GB to 16GB in maybe like 3 hours and then kernel dies. What can be the problem? Thanks in advance.

My code:

```auto
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.normalizer = normalizers.Sequence([Lowercase(), NFKC(), StripAccents()])
tokenizer.pre_tokenizer = pre_tokenizers.Sequence([Metaspace(), ByteLevel()])

def text_generator():
    files = [str(x) for x in Path("data/text/files/").glob("*.txt")]
    for file in tqdm(files): # one file is 10_000 sentences
        with open(file, "r") as f:
            lines = f.read().split('\n')
        yield lines # yields list of 10_000 strings 

trainer = BpeTrainer(
    vocab_size=200261, special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]
)

tokenizer.train_from_iterator(text_generator(), trainer=trainer)

```

---

<div class="post-metadata">

**Author:** ![ahmed2m](https://avatars.discourse-cdn.com/v4/letter/a/53a042/32.png) [@ahmed2m](https://discuss.huggingface.co/u/ahmed2m)\
**Post date:** [February 21, 2022, 10:13am UTC](https://discuss.huggingface.co/t/training-tokenizer-takes-too-much-ram/14256/2 "2022-02-21T10:13:14Z")

</div>

Same problem! Isn’t `train_from_iterator` supposed to be loading the data in batches and never have the complete data on memory at one time? Or is it the internal pair and vocab hashmaps getting bigger and bigger?  
Does anyone know the problem?