# Saving unique weights while training on multiple GPU - Trainer

**URL:** <https://discuss.huggingface.co/t/saving-unique-weights-while-training-on-multiple-gpu-trainer/70632>\
**Category:** 🤗Transformers\
**Created:** [January 25, 2024, 7:56pm UTC](https://discuss.huggingface.co/t/saving-unique-weights-while-training-on-multiple-gpu-trainer/70632 "2024-01-25T19:56:02Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lucius42](https://avatars.discourse-cdn.com/v4/letter/l/8e8cbc/32.png) [@Lucius42](https://discuss.huggingface.co/u/Lucius42)\
**Post date:** [January 25, 2024, 7:56pm UTC](https://discuss.huggingface.co/t/saving-unique-weights-while-training-on-multiple-gpu-trainer/70632/1 "2024-01-25T19:56:02Z")

</div>

Hello,

I am training LoRA adaptation of a T5 model in a one-machine multiple GPU setup.  
I am using Transformers 4.26.1 and DeepSpeed 0.9.2 and launching my script with deepspeed (thus the parallelization setup is Distributed Data Parallel).

I am using a customized callback in the Trainer to save only the LoRA weights at each epoch. Unfortunately, as I am using multiple GPUs, the script is ran on parallel four times (the numbers of GPU I use). Making it that at the end of each epoch, the weights are saved four times. Is there anything I can do to only save the weights once?

Thanks for your help,

Lucius