# Keep NSP head after BertForPretraining

**URL:** <https://discuss.huggingface.co/t/keep-nsp-head-after-bertforpretraining/14307>\
**Category:** 🤗Transformers\
**Created:** [February 1, 2022, 6:14pm UTC](https://discuss.huggingface.co/t/keep-nsp-head-after-bertforpretraining/14307 "2022-02-01T18:14:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tzuhsial](https://avatars.discourse-cdn.com/v4/letter/t/b5a626/32.png) [@tzuhsial](https://discuss.huggingface.co/u/tzuhsial)\
**Post date:** [February 1, 2022, 6:14pm UTC](https://discuss.huggingface.co/t/keep-nsp-head-after-bertforpretraining/14307/1 "2022-02-01T18:14:31Z")

</div>

I am pretraining with MLM + NSP (`BertForPretraining`), and finetuning with NSP (`BertForNextSentencePrediction`).

Is there an elegant way which I can keep the NSP head from the pretrained model?

Thanks!

---

<div class="post-metadata">

**Author:** ![tzuhsial](https://avatars.discourse-cdn.com/v4/letter/t/b5a626/32.png) [@tzuhsial](https://discuss.huggingface.co/u/tzuhsial)\
**Post date:** [February 1, 2022, 6:30pm UTC](https://discuss.huggingface.co/t/keep-nsp-head-after-bertforpretraining/14307/2 "2022-02-01T18:30:58Z")

</div>

Validated that `using BertForNextSentencePrediction` does indeed keep the weights of NSP head of `BertForPreTraining`