# How to prevent Transformers from deleting task-head?

**URL:** <https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759>\
**Category:** Beginners\
**Created:** [February 15, 2022, 3:15pm UTC](https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759 "2022-02-15T15:15:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![MoritzLaurer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moritzlaurer/32/18932_2.png) [@MoritzLaurer](https://discuss.huggingface.co/u/MoritzLaurer)\
**Post date:** [February 15, 2022, 3:15pm UTC](https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759/1 "2022-02-15T15:15:56Z")

</div>

When I train a Transformer, I mostly get a warning similar to the one below:

```auto
# resulting warning:
Some weights of the model checkpoint at microsoft/deberta-v3-base were not used when initializing DebertaV2ForMaskedLM: ['lm_predictions.lm_head.dense.bias', 'mask_predictions.dense.weight', 'lm_predictions.lm_head.dense.weight', 'lm_predictions.lm_head.LayerNorm.bias', 'mask_predictions.LayerNorm.weight', 'mask_predictions.classifier.bias', 'lm_predictions.lm_head.bias', 'mask_predictions.LayerNorm.bias', 'deberta.embeddings.position_embeddings.weight', 'mask_predictions.classifier.weight', 'mask_predictions.dense.bias', 'lm_predictions.lm_head.LayerNorm.weight']
- This IS expected if you are initializing DebertaV2ForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing DebertaV2ForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of DebertaV2ForMaskedLM were not initialized from the model checkpoint at microsoft/deberta-v3-base and are newly initialized: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.

```

I know that this is normally not a problem because for fine-tuning on a new task with a new classification head, we need to delete the old classification head. **But what if I actually want to keep the task head from pre-training? How do I prevent Transformers from deleting the task head?**

I know that for established tasks like MLM I can use AutoModelForMaskedLM etc. But What if the task head is not MLM (in my case an ELECTRA-style replaced-token-detection head, see here: [DeBERTa-v3: How to keep ELECTRA-style task-head?](https://discuss.huggingface.co/t/deberta-v3-how-to-keep-electra-style-task-head/14622))

---

<div class="post-metadata">

**Author:** ![MoritzLaurer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moritzlaurer/32/18932_2.png) [@MoritzLaurer](https://discuss.huggingface.co/u/MoritzLaurer)\
**Post date:** [March 6, 2022, 3:50pm UTC](https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759/2 "2022-03-06T15:50:57Z")

</div>

Is there no solution for this issue?

---

<div class="post-metadata">

**Author:** ![MoritzLaurer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moritzlaurer/32/18932_2.png) [@MoritzLaurer](https://discuss.huggingface.co/u/MoritzLaurer)\
**Post date:** [July 29, 2022, 11:10am UTC](https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759/3 "2022-07-29T11:10:05Z")

</div>

Is there no way to prevent Transformers from deleting the task head when loading a model like DeBERTa-v3? I would really love to use the replaced-token-detection task head from DeBERTa-v3 (same as ELECTRA), but it always gets deleted when I load it via HuggingFace Transformers.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [July 29, 2022, 11:59am UTC](https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759/4 "2022-07-29T11:59:09Z")

</div>

Hi,

If you’re instantiating a `DebertaV2ForMaskedLM` architecture with the weights of [microsoft/deberta-v3-base](https://huggingface.co/microsoft/deberta-v3-base), it will simply discard the weights that aren’t compatible with the model architecture. So in this case, `DebertaV2ForMaskedLM` only includes the language modeling head on top, as can be seen [here](https://github.com/huggingface/transformers/blob/a64bcb564dbc2a6329235016613a888ca21d513b/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L1094). And the head is defined as an attribute, namely `cls.predictions`.

However, the language modeling head parameters at `microsoft/deberta-v3-base` are defined as ‘lm\_predictions.lm\_head.dense.bias’, ‘lm\_predictions.lm\_head.dense.weight’, ‘lm\_predictions.lm\_head.LayerNorm.bias’, ‘lm\_predictions.lm\_head.bias’, ‘lm\_predictions.lm\_head.LayerNorm.weight’. Hence the authors of Debertav3 defined the head as an `lm_predictions.lm_head` attribute on top of the base model, causing a mismatch.

Moreover, `DebertaV2ForMaskedLM` doesn’t include the mask predictions head.

So, to properly instantiate all the weights from `microsoft/deberta-v3-base`, you need to define the exact same architecture to make sure all weights can be loaded. Showcasing a quick draft here, based on the [original definition](https://github.com/microsoft/DeBERTa/blob/a3f1d7280bbbc5dd715ea85b8269cddfd567696d/DeBERTa/apps/models/replaced_token_detection_model.py#L63):

```auto
from transformers import DebertaV2Model, DebertaV2PreTrainedModel

class DebertaV3ForPreTraining(DebertaV2PreTrainedModel):
  def __init__ (self, config):
    super(). __init__ (config)
    
    # base Transformer
    self.deberta = DebertaV2Model(config)

    # language modeling head
    self.lm_predictions = DebertaV3LMHead(config)

    # mask predictions head
    self.mask_predictions = DebertaV3MaskPredictionHead(config)

  def forward(self, input_ids, attention_mask=None, labels=None, position_ids=None):
    outputs = self.deberta(input_ids, attention_mask)

    sequence_output = outputs[0]
       
    # apply heads
    lm_logits = self.lm_predictions(sequence_output)
    mask_logits = self.mask_predictions(sequence_output, input_ids, attention_mask)

    return lm_logits, mask_logits

```

Here, `DebertaV3LMHead` should be defined as [this class](https://github.com/microsoft/DeBERTa/blob/a3f1d7280bbbc5dd715ea85b8269cddfd567696d/DeBERTa/apps/models/masked_language_model.py#L30) and `DebertaV3MaskPredictionHead` should be defined as [this class](https://github.com/microsoft/DeBERTa/blob/a3f1d7280bbbc5dd715ea85b8269cddfd567696d/DeBERTa/apps/models/replaced_token_detection_model.py#L30).

This way, you’ll be able to do:

```auto
model = DebertaV3ForPreTraining.from_pretrained("microsoft/deberta-v3-base")

```

and all weights will be loaded. Also, feel free to contribute this head model to the library 😃 this way people could use it directly from the library.

---

<div class="post-metadata">

**Author:** ![MoritzLaurer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moritzlaurer/32/18932_2.png) [@MoritzLaurer](https://discuss.huggingface.co/u/MoritzLaurer)\
**Post date:** [July 29, 2022, 2:04pm UTC](https://discuss.huggingface.co/t/how-to-prevent-transformers-from-deleting-task-head/14759/5 "2022-07-29T14:04:29Z")

</div>

Hi @nielsr, thanks a lot for your response!

I tried to follow your instructions, see the colab here: [Google Colab](https://colab.research.google.com/drive/12SFt6j0Xdug3K4SCyJMFksleT3-Xnl_1?usp=sharing)

Now it’s throwing the error: `TypeError: forward() missing 7 required positional arguments: 'ebd_weight', 'target_ids', 'input_ids', 'input_mask', 'z_states', 'attention_mask', and 'encoder'`

Unfortunately, I don’t know PyTorch / Transformers well enough to see all the adaptations that have to be made to make it compatible with Transformers 😕

(There are more and more people who are using the replaced-token-detection / ELECTRA objective for few-shot learning, but they all seem to rely on the old ELECTRA as opposed to the improved DeBERTa-v3, presumably because there is currently no easy way to use DeBERTa-v3’s RTD head. see research from Facebook [GitHub - facebookresearch/ELECTRA-Fewshot-Learning: This repository contains the code for paper Prompting ELECTRA Few-Shot Learning with Discriminative Pre-Trained Models.](https://github.com/facebookresearch/ELECTRA-Fewshot-Learning) or ACL: [https://aclanthology.org/2022.findings-acl.273.pdf](https://aclanthology.org/2022.findings-acl.273.pdf) )