# Questions about deepspeed multi-node training with sharding parameters inside a single 8-gpu machine

**URL:** <https://discuss.huggingface.co/t/questions-about-deepspeed-multi-node-training-with-sharding-parameters-inside-a-single-8-gpu-machine/24755>\
**Category:** DeepSpeed\
**Created:** [October 21, 2022, 4:49am UTC](https://discuss.huggingface.co/t/questions-about-deepspeed-multi-node-training-with-sharding-parameters-inside-a-single-8-gpu-machine/24755 "2022-10-21T04:49:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyk1337](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cyk1337/32/24460_2.png) [@cyk1337](https://discuss.huggingface.co/u/cyk1337)\
**Post date:** [October 21, 2022, 4:49am UTC](https://discuss.huggingface.co/t/questions-about-deepspeed-multi-node-training-with-sharding-parameters-inside-a-single-8-gpu-machine/24755/1 "2022-10-21T04:49:55Z")

</div>

The [HF documentation](https://huggingface.co/docs/transformers/main_classes/deepspeed) provides detailed guide to ZeRO usage. ZeRO2 partitions gradient states across all the gpu nodes (world size), which greatly slows down the training speed for multi-node training (the more nodes we use, the slower the speed is). Is there a way to controlling the sharding degree so that I can only partitions the parameters and gradients only inside a single 8-gpu machine?