# Model's evaluation in DDP training is using only one GPU

**URL:** <https://discuss.huggingface.co/t/models-evaluation-in-ddp-training-is-using-only-one-gpu/29295>\
**Category:** Beginners\
**Created:** [January 10, 2023, 12:49pm UTC](https://discuss.huggingface.co/t/models-evaluation-in-ddp-training-is-using-only-one-gpu/29295 "2023-01-10T12:49:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sararb](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@sararb](https://discuss.huggingface.co/u/sararb)\
**Post date:** [January 10, 2023, 12:49pm UTC](https://discuss.huggingface.co/t/models-evaluation-in-ddp-training-is-using-only-one-gpu/29295/1 "2023-01-10T12:49:31Z")

</div>

Hello! 😃  
I am training an HF model with torch DDP using the following command line:

`python -m torch.distributed.launch --nproc_per_node 2 my_script.py --{arguments}`

I noticed that while training was using the two available GPUs, the evaluation step was done only on a single GPU. After checking the source code, it seems that [\> here](https://github.com/huggingface/transformers/blob/a3c37825cc1e305dde63455b5f321586e6d29e07/src/transformers/trainer.py#L1360) the model is not wrapped inside the DDP when `training==False`.

Is it expected that only one GPU will be used during the evaluation step? If yes, could you explain why the DDP cannot be used for the evaluation as well?

---

<div class="post-metadata">

**Author:** ![zuzannad1](https://avatars.discourse-cdn.com/v4/letter/z/dc4da7/32.png) [@zuzannad1](https://discuss.huggingface.co/u/zuzannad1)\
**Post date:** [September 14, 2023, 9:02am UTC](https://discuss.huggingface.co/t/models-evaluation-in-ddp-training-is-using-only-one-gpu/29295/2 "2023-09-14T09:02:14Z")

</div>

Have you figured out how to use multiple GPUs for the eval loop during training? Am facing the same issue.