# Llama 3 peft ddp

**URL:** <https://discuss.huggingface.co/t/llama-3-peft-ddp/89473>\
**Category:** Beginners\
**Created:** [May 30, 2024, 6:08pm UTC](https://discuss.huggingface.co/t/llama-3-peft-ddp/89473 "2024-05-30T18:08:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ameljelidi](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@ameljelidi](https://discuss.huggingface.co/u/ameljelidi)\
**Post date:** [May 30, 2024, 6:08pm UTC](https://discuss.huggingface.co/t/llama-3-peft-ddp/89473/1 "2024-05-30T18:08:43Z")

</div>

Hi everyone (noob, here)! I am currently trying to finetune llama 3 using QLORA and wish to do that on two GPUs in parallel. I have tried in vain to do this using torchrun and seem to always get the following error:

ValueError: You can’t train a model that has been loaded in 8-bit precision on a different device than the one you’re training on. Make sure you loaded the model on the correct device using for example `device\_map={‘’:torch.cuda.current\_device() or device\_map={‘’:torch.xpu.current\_device()}

I have, of course, changed the device mapping to the current cuda device but it was not really helpful as I still got the exact same error.

Any help or resources on the matter would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [June 2, 2024, 12:59pm UTC](https://discuss.huggingface.co/t/llama-3-peft-ddp/89473/2 "2024-06-02T12:59:21Z")

</div>

Hi,

I’d recommend this guide: [Efficiently fine-tune Llama 3 with PyTorch FSDP and Q-Lora](https://www.philschmid.de/fsdp-qlora-llama3). It goes over fine-tuning Llama-3 using Q-LoRa using FSDP, which is PyTorch’s successor of DDP.

There’s also the Alignment Handbook, which includes scripts for both full fine-tuning and Q-LoRa on both single and multi-GPU environments: [alignment-handbook/scripts at main · huggingface/alignment-handbook · GitHub](https://github.com/huggingface/alignment-handbook/tree/main/scripts).

---

<div class="post-metadata">

**Author:** ![conacts](https://avatars.discourse-cdn.com/v4/letter/c/f14d63/32.png) [@conacts](https://discuss.huggingface.co/u/conacts)\
**Post date:** [August 20, 2024, 4:06pm UTC](https://discuss.huggingface.co/t/llama-3-peft-ddp/89473/3 "2024-08-20T16:06:26Z")

</div>

I wrote an article describing how to PEFT finetune llama using [llama recipes](https://github.com/meta-llama/llama-recipes), which is basically a super simple library for finetuning, infrence, etc. made by [FAIR](https://ai.meta.com/research/).