# Separate LM fine tuning and classification head training

**URL:** <https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404>\
**Category:** Beginners\
**Created:** [October 7, 2020, 2:21pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404 "2020-10-07T14:21:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![adamh](https://avatars.discourse-cdn.com/v4/letter/a/71e660/32.png) [@adamh](https://discuss.huggingface.co/u/adamh)\
**Post date:** [October 7, 2020, 2:21pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404/1 "2020-10-07T14:21:57Z")

</div>

I have a large text corpus, and a small subset of it that is labelled for a mutli-label text classification task. I’ve seen many (excellent!) examples of fine-tuning different models for sequence classification, but I couldn’t find one in which the training is separated to two distinct stages:

1. Fine-tune a specific language model on a specific corpus (unsupervised).
2. Train a sequence classification model on top of it on a labelled subset of the original dataset (supervised).

I assume this is a pretty common (and simple) scenario, but I couldn’t find any relevant docs. If anyone can provide any pointers, it would be much appreciated!

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 7, 2020, 4:10pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404/2 "2020-10-07T16:10:09Z")

</div>

Hi @adamh

You can use the `run_language_modeling` script [here](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) to finetune the pre-trained model for ex `BertForMaksedLM`.

Then you should be able to load the model using `BertForSequenceClassification` model which will take the base model and add a classification head on top, which you can then fine-tune for classification.

---

<div class="post-metadata">

**Author:** ![laurb](https://avatars.discourse-cdn.com/v4/letter/l/e47c2d/32.png) [@laurb](https://discuss.huggingface.co/u/laurb)\
**Post date:** [October 15, 2020, 5:20pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404/3 "2020-10-15T17:20:37Z")

</div>

Would love to see a combined script that shows this working.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 16, 2020, 5:30pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404/4 "2020-10-16T17:30:18Z")

</div>

Pre-training usually takes longer time and there are multiple downstream task you can use the pre-trained model for so IMO combined script won’t be useful. But feel free to create one if you feel like it 😉

---

<div class="post-metadata">

**Author:** ![laurb](https://avatars.discourse-cdn.com/v4/letter/l/e47c2d/32.png) [@laurb](https://discuss.huggingface.co/u/laurb)\
**Post date:** [October 16, 2020, 7:22pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404/5 "2020-10-16T19:22:40Z")

</div>

if I could get it to work, I would. But I can’t, which is why I’d like to see a working example.

---

<div class="post-metadata">

**Author:** ![Sedigh](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@Sedigh](https://discuss.huggingface.co/u/Sedigh)\
**Post date:** [July 1, 2021, 9:42pm UTC](https://discuss.huggingface.co/t/separate-lm-fine-tuning-and-classification-head-training/1404/6 "2021-07-01T21:42:12Z")

</div>

Hi @adamh, did you find out how to do this? I also want to leverage a fine-tuned language model for classification. As far as I can tell when we choose a classification head then we must discard the previous head - i.e. in this case our beautifully fine-tuned LM head! So, can we put a classification head on top of this rather than discard it, or - do we need to unlock the next layers of our LM and propagate the weights down into the model so that when discarding our LM head we still have an underlying model that has inherited our fine-tuning?