# Classification Heads in BERT and DistilBERT for Sequence Classification

**URL:** <https://discuss.huggingface.co/t/classification-heads-in-bert-and-distilbert-for-sequence-classification/6146>\
**Category:** Research\
**Created:** [May 12, 2021, 4:16pm UTC](https://discuss.huggingface.co/t/classification-heads-in-bert-and-distilbert-for-sequence-classification/6146 "2021-05-12T16:16:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![valkyrie](https://avatars.discourse-cdn.com/v4/letter/v/5e9695/32.png) [@valkyrie](https://discuss.huggingface.co/u/valkyrie)\
**Post date:** [May 12, 2021, 4:16pm UTC](https://discuss.huggingface.co/t/classification-heads-in-bert-and-distilbert-for-sequence-classification/6146/1 "2021-05-12T16:16:52Z")

</div>

Hi,

I have been using BertForSequenceClassification and DistilBertForSequenceClassification recently and I have noticed that they have different classification heads.

BertForSequenceClassification has a dropout layer and a linear layer, whereas DistilBertForSequenceClassification has two linear layers and a dropout layer.

Is there a particular reason for this?

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [May 12, 2021, 6:22pm UTC](https://discuss.huggingface.co/t/classification-heads-in-bert-and-distilbert-for-sequence-classification/6146/2 "2021-05-12T18:22:47Z")

</div>

All in all, they have the same head: BertForSequenceClassification has a dropout layer and a linear layer but uses the pooler output, which went through a linear layer inside the BertModel.

DistilBertModel has no pooler output however, so the first linear layer is there to replicate that.

---

<div class="post-metadata">

**Author:** ![valkyrie](https://avatars.discourse-cdn.com/v4/letter/v/5e9695/32.png) [@valkyrie](https://discuss.huggingface.co/u/valkyrie)\
**Post date:** [May 13, 2021, 9:28am UTC](https://discuss.huggingface.co/t/classification-heads-in-bert-and-distilbert-for-sequence-classification/6146/3 "2021-05-13T09:28:34Z")

</div>

Thank you that makes sense!