# How to load pre-trained model parameters only in specific layers?

**URL:** <https://discuss.huggingface.co/t/how-to-load-pre-trained-model-parameters-only-in-specific-layers/1814>\
**Category:** Beginners\
**Created:** [October 30, 2020, 7:27am UTC](https://discuss.huggingface.co/t/how-to-load-pre-trained-model-parameters-only-in-specific-layers/1814 "2020-10-30T07:27:20Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![guoziyuan](https://avatars.discourse-cdn.com/v4/letter/g/7ab992/32.png) [@guoziyuan](https://discuss.huggingface.co/u/guoziyuan)\
**Post date:** [October 30, 2020, 7:27am UTC](https://discuss.huggingface.co/t/how-to-load-pre-trained-model-parameters-only-in-specific-layers/1814/1 "2020-10-30T07:27:20Z")

</div>

Hey , I want to know how to load pre-trained model parameters only in specific layers ? For example, I use EncoderDecoderModel class (bert-base-uncased2bert-base-uncased model) . And I only want to load parameters in specific layers like 2 or 10 of the pretrained model . How can I do this ?

---

<div class="post-metadata">

**Author:** ![allanjie](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@allanjie](https://discuss.huggingface.co/u/allanjie)\
**Post date:** [October 30, 2020, 9:24am UTC](https://discuss.huggingface.co/t/how-to-load-pre-trained-model-parameters-only-in-specific-layers/1814/2 "2020-10-30T09:24:13Z")

</div>

> [@guoziyuan](#):
>
> or 10 of the pretrained

This is usually what I did:

```python
original_model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
my_model = BartForConditionalGeneration.from_pretrained('facebook/bart-base', encoder_layers=2)
## for example, I want my model's last encoder layer's weight = original model's last encoder's weight
model.model.encoder.layers[1].load_state_dict(original_model.model.encoder.layers[5].state_dict())

```

---

<div class="post-metadata">

**Author:** ![stomir](https://avatars.discourse-cdn.com/v4/letter/s/85f322/32.png) [@stomir](https://discuss.huggingface.co/u/stomir)\
**Post date:** [December 20, 2022, 10:21am UTC](https://discuss.huggingface.co/t/how-to-load-pre-trained-model-parameters-only-in-specific-layers/1814/3 "2022-12-20T10:21:14Z")

</div>

So I found this answer but it’s not very helpful, since this method requires loading the entire model into memory, which is something I specifically want to avoid.  
Anyone know how to only load a specific submodule?