# Is Transformers using GPU by default?

**URL:** <https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500>\
**Category:** Beginners\
**Created:** [July 19, 2021, 5:39pm UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500 "2021-07-19T17:39:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![OlivierCR](https://avatars.discourse-cdn.com/v4/letter/o/b19c9b/32.png) [@OlivierCR](https://discuss.huggingface.co/u/OlivierCR)\
**Post date:** [July 19, 2021, 5:39pm UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/1 "2021-07-19T17:39:49Z")

</div>

I’m instantiating a model with this

```auto
tokenizer = AutoTokenizer.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")
model = AutoModelForSequenceClassification.from_pretrained("nlptown/bert-base-multilingual-uncased-sentiment")

```

Then running a for loop to get prediction over 10k sentences on a G4 instance (T4 GPU). GPU usage (averaged by minute) is a flat 0.0%. What is wrong? How to use GPU with Transformers?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [July 19, 2021, 5:56pm UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/2 "2021-07-19T17:56:07Z")

</div>

This like with every PyTorch model, you need to put it on the GPU, as well as your batches of inputs.

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [July 20, 2021, 7:22am UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/3 "2021-07-20T07:22:55Z")

</div>

You can take a look at this issue [How to make transformers examples use GPU? · Issue #2704 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/2704) It includes an example for how to put your model on GPU.

```python
device = "cuda:0" if torch.cuda.is_available() else "cpu"
sentence = 'Hello World!'
tokenizer = AutoTokenizer.from_pretrained('bert-large-uncased')
model = BertModel.from_pretrained('bert-large-uncased')

inputs = tokenizer(sentence, return_tensors="pt").to(device)
model = model.to(device)
outputs = model(**inputs)

```

---

<div class="post-metadata">

**Author:** ![AshutoshDongare](https://avatars.discourse-cdn.com/v4/letter/a/ce73a5/32.png) [@AshutoshDongare](https://discuss.huggingface.co/u/AshutoshDongare)\
**Post date:** [March 23, 2022, 11:50am UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/4 "2022-03-23T11:50:02Z")

</div>

@sgugger

Are there any samples of how Huggingface Transformer finetuning should be done using GPU please?

---

<div class="post-metadata">

**Author:** ![Radz](https://avatars.discourse-cdn.com/v4/letter/r/8e8cbc/32.png) [@Radz](https://discuss.huggingface.co/u/Radz)\
**Post date:** [May 12, 2022, 11:57pm UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/5 "2022-05-12T23:57:01Z")

</div>

Hi @sgugger were you able to figure this out?

---

<div class="post-metadata">

**Author:** ![folterj](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/folterj/32/13283_2.png) [@folterj](https://discuss.huggingface.co/u/folterj)\
**Post date:** [December 31, 2022, 12:33pm UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/6 "2022-12-31T12:33:32Z")

</div>

I had the same issue - to answer this question, if pytorch + cuda is installed, an e.g. transformers.Trainer class using pytorch will automatically use the cuda (GPU) version without any additional specification.

(You can check if pytorch + cuda is installed by checking if the `pytorch-cuda` package is installed.)

---

<div class="post-metadata">

**Author:** ![ar78](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ar78/32/22027_2.png) [@ar78](https://discuss.huggingface.co/u/ar78)\
**Post date:** [December 11, 2023, 1:53am UTC](https://discuss.huggingface.co/t/is-transformers-using-gpu-by-default/8500/7 "2023-12-11T01:53:34Z")

</div>

Is there a way to explicitly disable the trainer from using the GPU? I see something about `place_model_on_device` on [Trainer](https://huggingface.co/docs/transformers/main_classes/trainer) but it is unclear how to set it to `False`.

EDIT: Oh, I see I can set `use_cpu` in [TrainingArguments](https://huggingface.co/docs/transformers/v4.35.2/en/main_classes/trainer#transformers.TrainingArguments) to False.