# LoRA training with accelerate / deepspeed

**URL:** <https://discuss.huggingface.co/t/lora-training-with-accelerate-deepspeed/70080>\
**Category:** DeepSpeed\
**Created:** [January 22, 2024, 11:54am UTC](https://discuss.huggingface.co/t/lora-training-with-accelerate-deepspeed/70080 "2024-01-22T11:54:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![iarbel](https://avatars.discourse-cdn.com/v4/letter/i/3da27b/32.png) [@iarbel](https://discuss.huggingface.co/u/iarbel)\
**Post date:** [January 22, 2024, 11:54am UTC](https://discuss.huggingface.co/t/lora-training-with-accelerate-deepspeed/70080/1 "2024-01-22T11:54:50Z")

</div>

I’m training LoRA adapters large models using Accelerate + DeepSpeed, with ZeRO-3. I’m using alignment-handbook implementation for that. However, when saving checkpoints, the full model is being saved, while I need only the adapter (and possibly the optimizer states). Is this possible to configure the training such that save only the adapters are saved?

Or should I used a MULTI\_GPU setup instead of DeepSpeed?

Model to be trained is Llama-70B, on 8\*A100 40GB, so it’s really not possible to fit the model on a single GPU.

---

<div class="post-metadata">

**Author:** ![blazejm](https://avatars.discourse-cdn.com/v4/letter/b/d9b06d/32.png) [@blazejm](https://discuss.huggingface.co/u/blazejm)\
**Post date:** [August 8, 2024, 7:50am UTC](https://discuss.huggingface.co/t/lora-training-with-accelerate-deepspeed/70080/2 "2024-08-08T07:50:27Z")

</div>

hey, did you figure it out?

---

<div class="post-metadata">

**Author:** ![iarbel](https://avatars.discourse-cdn.com/v4/letter/i/3da27b/32.png) [@iarbel](https://discuss.huggingface.co/u/iarbel)\
**Post date:** [August 8, 2024, 9:20am UTC](https://discuss.huggingface.co/t/lora-training-with-accelerate-deepspeed/70080/3 "2024-08-08T09:20:02Z")

</div>

Use multi-GPU and not DeepSpeed

---

<div class="post-metadata">

**Author:** ![MauroExtrac](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mauroextrac/32/38514_2.png) [@MauroExtrac](https://discuss.huggingface.co/u/MauroExtrac)\
**Post date:** [May 28, 2025, 12:30pm UTC](https://discuss.huggingface.co/t/lora-training-with-accelerate-deepspeed/70080/4 "2025-05-28T12:30:56Z")

</div>

@iarbel What do you mean by using multi GPU rather than DeepSpeed?