# How much memory is needed for training ByteLevelBPETokenizer?

**URL:** <https://discuss.huggingface.co/t/how-much-memory-is-needed-for-training-bytelevelbpetokenizer/1165>\
**Category:** 🤗Tokenizers\
**Created:** [September 16, 2020, 1:10pm UTC](https://discuss.huggingface.co/t/how-much-memory-is-needed-for-training-bytelevelbpetokenizer/1165 "2020-09-16T13:10:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kouohhashi](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kouohhashi](https://discuss.huggingface.co/u/kouohhashi)\
**Post date:** [September 16, 2020, 1:10pm UTC](https://discuss.huggingface.co/t/how-much-memory-is-needed-for-training-bytelevelbpetokenizer/1165/1 "2020-09-16T13:10:07Z")

</div>

Hi, I’m trying to train LM for Japanese from scratch.  
To be honest, I copied almost everything from [https://github.com/huggingface/blog/blob/master/notebooks/01\_how\_to\_train.ipynb](https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb).  
I just changed datasets from Esperanto datasets to Japanese wiki datasets.

But when I tried to train tokenizer, my notebook crashed and restarted probably because of out of memory. My datasets is an entire wikipedia text which is 5.1 G. But my server had 64G memory.

How much memory do I need to train tokenizer from scratch?  
or can I prevent out of memory error with some options?

Thanks in advance.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 16, 2020, 1:26pm UTC](https://discuss.huggingface.co/t/how-much-memory-is-needed-for-training-bytelevelbpetokenizer/1165/2 "2020-09-16T13:26:00Z")

</div>

Pinging @anthony

---

<div class="post-metadata">

**Author:** ![kouohhashi](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kouohhashi](https://discuss.huggingface.co/u/kouohhashi)\
**Post date:** [September 17, 2020, 12:15am UTC](https://discuss.huggingface.co/t/how-much-memory-is-needed-for-training-bytelevelbpetokenizer/1165/3 "2020-09-17T00:15:17Z")

</div>

[UPDATE] I tried subset of entire dataset.

66,101,891 lines, 5.1 G ( entire datasets ) : out of memory  
15,000,000 lines: out of memory  
3,000,000 lines, 231M: success

Can I use subset to train tokenizer and use it to train LM with entire datasets?

---

<div class="post-metadata">

**Author:** ![kouohhashi](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kouohhashi](https://discuss.huggingface.co/u/kouohhashi)\
**Post date:** [September 18, 2020, 5:49am UTC](https://discuss.huggingface.co/t/how-much-memory-is-needed-for-training-bytelevelbpetokenizer/1165/4 "2020-09-18T05:49:14Z")

</div>

I could train a tokenizer with [train\_bert\_wordpiece.py](https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/train_bert_wordpiece.py) without errors.  
So it could be jupyter notebooks or something else which caused out of memory errors.