# Pre-trained Model Enlargement

**URL:** <https://discuss.huggingface.co/t/pre-trained-model-enlargement/52557>\
**Category:** Models\
**Created:** [August 28, 2023, 10:36pm UTC](https://discuss.huggingface.co/t/pre-trained-model-enlargement/52557 "2023-08-28T22:36:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![meliksahturker](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meliksahturker/32/28497_2.png) [@meliksahturker](https://discuss.huggingface.co/u/meliksahturker)\
**Post date:** [August 28, 2023, 10:36pm UTC](https://discuss.huggingface.co/t/pre-trained-model-enlargement/52557/1 "2023-08-28T22:36:10Z")

</div>

I have a pre-trained Encoder-Decoder MLM (BART) and have a bit of spare budget. I decided to enlarge and further pre-train it by doubling the number of both encoder and decoder layers. Therefore d\_model, d\_ff, d\_kv, and so on will not change. Hence, the weights of the base model will be compatible.

I plan to use existing embedding, encoder and decoder weights to initialize the bigger model.  
There are 3 scenarios to do that:

1. I can use the weights of the base model to set the weights of the new, bigger model for the first N Encoder and Decoder layers and initailize the rest from scratch.
2. I can use the weights of the base model to set the weights of the new, bigger model for odd layers, such as 1, 3, 5 and so on, and initialize the even numbered layers from scratch such as 2, 4, 6 and so on.
3. Instead of initializing the weights from scratch in the 2nd scenario, I can use the same weights that I used to set the weights of odd layers on the even layers as well.

Since this further pre-training will be a one-shot procedure with a limited budget, I will not have the opportunity to test different scenarios.  
Hence, which scenario makes more sense, what do you think potential upsides and downsides are?

---

<div class="post-metadata">

**Author:** ![meliksahturker](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/meliksahturker/32/28497_2.png) [@meliksahturker](https://discuss.huggingface.co/u/meliksahturker)\
**Post date:** [August 31, 2023, 10:07am UTC](https://discuss.huggingface.co/t/pre-trained-model-enlargement/52557/2 "2023-08-31T10:07:00Z")

</div>

I have decided to go with the 2nd scenario, and it worked pretty well. Cross-entropy loss after the 1st epoch (5000 steps per epoch) is 1.32, compared to +7 in training from scratch.