# The same hyperparameters with deepspeed is worse than without deepseepd

**URL:** <https://discuss.huggingface.co/t/the-same-hyperparameters-with-deepspeed-is-worse-than-without-deepseepd/20509>\
**Category:** DeepSpeed\
**Created:** [July 18, 2022, 9:35am UTC](https://discuss.huggingface.co/t/the-same-hyperparameters-with-deepspeed-is-worse-than-without-deepseepd/20509 "2022-07-18T09:35:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dodge](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dodge/32/5406_2.png) [@dodge](https://discuss.huggingface.co/u/dodge)\
**Post date:** [July 18, 2022, 9:35am UTC](https://discuss.huggingface.co/t/the-same-hyperparameters-with-deepspeed-is-worse-than-without-deepseepd/20509/1 "2022-07-18T09:35:17Z")

</div>

I’m training a model (dpcnn) with batch size 512 with out deepspeed, the actual batch size is equal to (512/8)=64 (I have 8 gpus) with deepspeed, but the deepspeed’s loss and accuracy is far worse than without deepspeed. The two experiments have same code, same hyperparameters except batch size. Any one have some idea to explain this situation?

---

<div class="post-metadata">

**Author:** ![smangrul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smangrul/32/3791_2.png) [@smangrul](https://discuss.huggingface.co/u/smangrul)\
**Post date:** [July 18, 2022, 12:55pm UTC](https://discuss.huggingface.co/t/the-same-hyperparameters-with-deepspeed-is-worse-than-without-deepseepd/20509/2 "2022-07-18T12:55:33Z")

</div>

Hello,

We need a small script that we can run to reproduce the behaviour. As per our experiments, it works as expected. For your reference, please go through this blog: [Accelerate Large Model Training using DeepSpeed (huggingface.co)](https://huggingface.co/blog/accelerate-deepspeed) for the GLUE task wherein we see at par results with and without DeepSpeed.

---

<div class="post-metadata">

**Author:** ![mintuos](https://avatars.discourse-cdn.com/v4/letter/m/7ab992/32.png) [@mintuos](https://discuss.huggingface.co/u/mintuos)\
**Post date:** [November 13, 2023, 1:03pm UTC](https://discuss.huggingface.co/t/the-same-hyperparameters-with-deepspeed-is-worse-than-without-deepseepd/20509/3 "2023-11-13T13:03:56Z")

</div>

i have the same quesion with you. in my code, the acc is same with the code without using deepspeed,but the loss is very high.  
I wonder if u have solved this question, could u give me some suggest.thanku