# Distributed fine-tuning with frozen embedding layers

**URL:** <https://discuss.huggingface.co/t/distributed-fine-tuning-with-frozen-embedding-layers/21663>\
**Category:** Beginners\
**Created:** [August 16, 2022, 7:14am UTC](https://discuss.huggingface.co/t/distributed-fine-tuning-with-frozen-embedding-layers/21663 "2022-08-16T07:14:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![adrienchaton](https://avatars.discourse-cdn.com/v4/letter/a/41988e/32.png) [@adrienchaton](https://discuss.huggingface.co/u/adrienchaton)\
**Post date:** [August 16, 2022, 7:14am UTC](https://discuss.huggingface.co/t/distributed-fine-tuning-with-frozen-embedding-layers/21663/1 "2022-08-16T07:14:57Z")

</div>

Hi HF community,

I want to fine-tune a HF checkpoint while freezing the transformer’s embedding layers.  
By doing so (description below), I am getting the following when using DDP and torchrun:  
AssertionError: expects all parameters to have same requires\_grad

The same script while not freezing any weights works fine.  
The same script while freezing weights but only using one GPU (no DDP) works too.

This is how I instantiate the model before passing it to the trainer:

```auto
model = AlbertForMaskedLM.from_pretrained("Rostlab/prot_albert")
if freeze_embeddings:
   for param in model.albert.embeddings.parameters():
      param.requires_grad = False

```

Main hyper-parameters for the fine-tuning CLI:  
CUDA\_VISIBLE\_DEVICES=0,1,2,3,4,5,6,7 torchrun --standalone --nnodes=1 --nproc\_per\_node=8 finetune\_protbert.py --fp16 --fp16\_opt\_level O1 --sharded\_ddp zero\_dp\_2

Is this issue known ? Any hints for solving it please ?

Thanks !