# More processes than GPUs with DeepSpeed launcher

**URL:** <https://discuss.huggingface.co/t/more-processes-than-gpus-with-deepspeed-launcher/70633>\
**Category:** DeepSpeed\
**Created:** [January 25, 2024, 8:07pm UTC](https://discuss.huggingface.co/t/more-processes-than-gpus-with-deepspeed-launcher/70633 "2024-01-25T20:07:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lucius42](https://avatars.discourse-cdn.com/v4/letter/l/8e8cbc/32.png) [@Lucius42](https://discuss.huggingface.co/u/Lucius42)\
**Post date:** [January 25, 2024, 8:07pm UTC](https://discuss.huggingface.co/t/more-processes-than-gpus-with-deepspeed-launcher/70633/1 "2024-01-25T20:07:17Z")

</div>

Hello,

I am training LoRA adaptation of a T5 model in a one-machine multiple GPU setup.  
I am using Transformers 4.26.1 and DeepSpeed 0.9.2 and launching my script with deepspeed (thus the parallelization setup is Distributed Data Parallel).

Just before training, I first get four processes in my first GPU (probably the four models loaded in right?)

When the training starts, I have four processes running in my first GPU and then one process for each other GPU. Although I am using regular PyTorch I have the same issue as in this repo : [extra process when running ddp across multiple GPUs · Lightning-AI/pytorch-lightning · Discussion #9864 · GitHub](https://github.com/Lightning-AI/pytorch-lightning/discussions/9864).

See image

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/3/a33c2eeca37cd7b9b738b7e07712eb2aa7f252df.png)

Is it normal? I have only seen examples with one process per GPU. I would be greatly interested in an explanation.

Thanks !