# Always getting RuntimeError: CUDA out of memory with Trainer

**URL:** <https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948>\
**Category:** 🤗Transformers\
**Created:** [December 18, 2021, 4:08pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948 "2021-12-18T16:08:19Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [December 18, 2021, 4:08pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/1 "2021-12-18T16:08:19Z")

</div>

Hello,

I am using `huggingface` on my google colab pro+ instance, and I keep getting errors like

> RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 15.78 GiB total capacity; 13.92 GiB already allocated; 206.75 MiB free; 13.94 GiB reserved in total by PyTorch) If reserved memory is \>\> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF

I dont understand why? My dataset is microscopic (40K sentences), and all I am doing is loading `bert-large-uncased` and follow along the `text classification` notebook

```auto
from transformers import AutoTokenizer
from transformers import AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('bert-large-cased')
from datasets import load_dataset, load_metric

metric = load_metric('glue', 'sst2')
model = AutoModelForSequenceClassification.from_pretrained("bert-large-cased", num_labels=2)

```

my trainer args are super standard

```auto

from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

batch_size = 16

args = TrainingArguments(
    evaluation_strategy = "epoch",
    save_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=5,
    report_to="none",
    weight_decay=0.01,
     output_dir='/content/drive/MyDrive/kaggle/',
    metric_for_best_model='accuracy')

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['test'],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics

)

```

Am I missing something? Should I change some of the options?  
Thanks!!

---

<div class="post-metadata">

**Author:** ![Jonesy](https://avatars.discourse-cdn.com/v4/letter/j/f04885/32.png) [@Jonesy](https://discuss.huggingface.co/u/Jonesy)\
**Post date:** [December 18, 2021, 5:05pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/2 "2021-12-18T17:05:58Z")

</div>

(Just posting this in case someone smarter doesn’t post a better idea)

Colab’s performance varies a lot. I ran the same script (dataset in question had 1200 sentences) and sometimes I get out of memory error and sometimes not. My latest project has 270 sentences and ran fine on the first try.

---

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [December 18, 2021, 5:27pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/3 "2021-12-18T17:27:02Z")

</div>

thanks. but colab pro+ gives you about 50GB of RAM and a Tesla P100… so I should have enough RAM here…

---

<div class="post-metadata">

**Author:** ![Jonesy](https://avatars.discourse-cdn.com/v4/letter/j/f04885/32.png) [@Jonesy](https://discuss.huggingface.co/u/Jonesy)\
**Post date:** [December 18, 2021, 6:14pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/4 "2021-12-18T18:14:10Z")

</div>

Make sure you’re not running out of GPU ram though? I think the GPU is capped at 16Gigs. PyTorch takes up 14Gigs.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [December 18, 2021, 8:20pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/5 "2021-12-18T20:20:15Z")

</div>

> [@Jonesy](#):
>
> PyTorch takes up 14Gigs.

What do you mean by this? The amount of GPU memory used generally depends on the model, batch size, and sequence length.

---

<div class="post-metadata">

**Author:** ![Jonesy](https://avatars.discourse-cdn.com/v4/letter/j/f04885/32.png) [@Jonesy](https://discuss.huggingface.co/u/Jonesy)\
**Post date:** [December 18, 2021, 8:40pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/6 "2021-12-18T20:40:49Z")

</div>

This is where things go over my head. If anyone smarter can interject I’d appreciate it 😅

---

<div class="post-metadata">

**Author:** ![Jonesy](https://avatars.discourse-cdn.com/v4/letter/j/f04885/32.png) [@Jonesy](https://discuss.huggingface.co/u/Jonesy)\
**Post date:** [December 18, 2021, 8:41pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/7 "2021-12-18T20:41:56Z")

</div>

But if you look at the error message OP posted, it appears that his GPU memory is being hogged by PyTorch?

---

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [December 19, 2021, 2:00am UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/8 "2021-12-19T02:00:05Z")

</div>

@BramVanroy thanks for your input. Is there a rough back-of-the-envelope to know how much memory I need to run a model? It seems the `base-large-cased` is quite big, but how big?

Thanks

---

<div class="post-metadata">

**Author:** ![Jonesy](https://avatars.discourse-cdn.com/v4/letter/j/f04885/32.png) [@Jonesy](https://discuss.huggingface.co/u/Jonesy)\
**Post date:** [December 20, 2021, 3:09pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/9 "2021-12-20T15:09:31Z")

</div>

The question belongs in an FAQ.

---

<div class="post-metadata">

**Author:** ![caitriggs](https://avatars.discourse-cdn.com/v4/letter/c/f1d935/32.png) [@caitriggs](https://discuss.huggingface.co/u/caitriggs)\
**Post date:** [November 28, 2023, 5:29am UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/10 "2023-11-28T05:29:11Z")

</div>

You can use this model memory usage calculator for a general idea: [Model Memory Utility - a Hugging Face Space by hf-accelerate](https://huggingface.co/spaces/hf-accelerate/model-memory-usage)

If it’s failing right at the beginning of calling .train() then I don’t think it’s the optimizer RAM as the culprit.

---

<div class="post-metadata">

**Author:** ![akshat-kumar-akight](https://avatars.discourse-cdn.com/v4/letter/a/858c86/32.png) [@akshat-kumar-akight](https://discuss.huggingface.co/u/akshat-kumar-akight)\
**Post date:** [April 4, 2024, 5:18pm UTC](https://discuss.huggingface.co/t/always-getting-runtimeerror-cuda-out-of-memory-with-trainer/12948/11 "2024-04-04T17:18:30Z")

</div>

If you are getting OOM error inbetween training, most likely you are storing variables tensors on GPU. Push these values to cpu and you should be through with OOM