# DebertaForMaskedLM cannot load the parameters in the MLM head from microsoft/deberta-base

**URL:** <https://discuss.huggingface.co/t/debertaformaskedlm-cannot-load-the-parameters-in-the-mlm-head-from-microsoft-deberta-base/14587>\
**Category:** Models\
**Created:** [February 10, 2022, 3:49pm UTC](https://discuss.huggingface.co/t/debertaformaskedlm-cannot-load-the-parameters-in-the-mlm-head-from-microsoft-deberta-base/14587 "2022-02-10T15:49:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ecoding](https://avatars.discourse-cdn.com/v4/letter/e/e95f7d/32.png) [@Ecoding](https://discuss.huggingface.co/u/Ecoding)\
**Post date:** [February 10, 2022, 3:49pm UTC](https://discuss.huggingface.co/t/debertaformaskedlm-cannot-load-the-parameters-in-the-mlm-head-from-microsoft-deberta-base/14587/1 "2022-02-10T15:49:27Z")

</div>

Hello, I’m trying to run this code:

> tokenizer = DebertaTokenizer.from\_pretrained(‘microsoft/deberta-base’)  
> model = AutoModelWithLMHead.from\_pretrained(‘microsoft/deberta-base’)

and get this warning:

> Some weights of the model checkpoint at microsoft/deberta-base were not used when initializing DebertaForMaskedLM: [‘lm\_predictions.lm\_head.dense.bias’, ‘lm\_predictions.lm\_head.LayerNorm.weight’, ‘lm\_predictions.lm\_head.dense.weight’, ‘lm\_predictions.lm\_head.bias’, ‘deberta.embeddings.position\_embeddings.weight’, ‘lm\_predictions.lm\_head.LayerNorm.bias’]
> 
> - This IS expected if you are initializing DebertaForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
> - This IS NOT expected if you are initializing DebertaForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).  
> Some weights of DebertaForMaskedLM were not initialized from the model checkpoint at microsoft/deberta-base and are newly initialized: [‘cls.predictions.transform.LayerNorm.bias’, ‘cls.predictions.transform.dense.weight’, ‘cls.predictions.transform.dense.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.bias’, ‘cls.predictions.decoder.weight’]  
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

It seems that the checkpoints provided by microsoft/deberta-base doesn’t possess the weights needed for the MLM head, so that DebertaForMaskedLM cannot be directly used for masked token prediction.

Is this a bug of the DebertaForMaskedLM class? Or the checkpoints provided by Microsoft? Or I do not understand something?

Thank you!

---

<div class="post-metadata">

**Author:** ![Yarden](https://avatars.discourse-cdn.com/v4/letter/y/f6c823/32.png) [@Yarden](https://discuss.huggingface.co/u/Yarden)\
**Post date:** [February 27, 2022, 4:39pm UTC](https://discuss.huggingface.co/t/debertaformaskedlm-cannot-load-the-parameters-in-the-mlm-head-from-microsoft-deberta-base/14587/2 "2022-02-27T16:39:54Z")

</div>

Same issue for me.  
help please?

Thanks!

---

<div class="post-metadata">

**Author:** ![kotalaraghav](https://avatars.discourse-cdn.com/v4/letter/k/f05b48/32.png) [@kotalaraghav](https://discuss.huggingface.co/u/kotalaraghav)\
**Post date:** [April 29, 2022, 11:40am UTC](https://discuss.huggingface.co/t/debertaformaskedlm-cannot-load-the-parameters-in-the-mlm-head-from-microsoft-deberta-base/14587/3 "2022-04-29T11:40:00Z")

</div>

I wouldn’t say it complete solution but it does initialize weights starting with “lm\_predictions” checkout the code [here](https://gist.github.com/nbroad1881/3247571d54a4070227a6674614ce0761)

Thanks to @nbroad for his generous contribution.

> from new\_deberta import NewDebertaForMaskedLM  
> model = NewDebertaForMaskedLM.from\_pretrained(model\_name)

after this.

> Some weights of the model checkpoint at microsoft/deberta-base were not used when initializing NewDebertaForMaskedLM: [‘deberta.embeddings.position\_embeddings.weight’]
> 
> - This IS expected if you are initializing NewDebertaForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
> - This IS NOT expected if you are initializing NewDebertaForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).  
> Some weights of NewDebertaForMaskedLM were not initialized from the model checkpoint at microsoft/deberta-base and are newly initialized: [‘lm\_predictions.lm\_head.decoder.weight’, ‘lm\_predictions.lm\_head.decoder.bias’]  
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

I think we need some more hacks to initialize position\_embedding weights too 😃

Hope this helps!

---

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](https://discuss.huggingface.co/u/nbroad)\
**Post date:** [April 29, 2022, 12:46pm UTC](https://discuss.huggingface.co/t/debertaformaskedlm-cannot-load-the-parameters-in-the-mlm-head-from-microsoft-deberta-base/14587/4 "2022-04-29T12:46:38Z")

</div>

There will be more changes coming soon to the main branch of `transformers`. The new changes will allow for the MLM head to produce reasonable suggestions for [MASK] tokens. Thanks for your patience!
