# Question about GPT2LMHeadModel, GPT2ForSequenceClassification

**URL:** <https://discuss.huggingface.co/t/question-about-gpt2lmheadmodel-gpt2forsequenceclassification/16266>\
**Category:** Models\
**Created:** [March 30, 2022, 4:30pm UTC](https://discuss.huggingface.co/t/question-about-gpt2lmheadmodel-gpt2forsequenceclassification/16266 "2022-03-30T16:30:32Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon3699016](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anon3699016](https://discuss.huggingface.co/u/anon3699016)\
**Post date:** [March 30, 2022, 4:30pm UTC](https://discuss.huggingface.co/t/question-about-gpt2lmheadmodel-gpt2forsequenceclassification/16266/1 "2022-03-30T16:30:32Z")

</div>

Hi, I have some questions about GPT2 models of Huggingface.

1. In order to generate a sentence autoregressively, should I use GPT2LMHeadModel, not GPT2Model? (maybe because there is no LM head on GPT2Model?)

2. Does GPT2ForSequenceClassification also have LM head? or just classification at the end?  
if also has LM head? Can I also use GPT2ForSequenceClassification for generating a sentence?  
How GPT2ForSequenceClassification is pretrained? Were there 2 learning objective, one is LM loss  
and the other is classfication loss?

Thank for your answer in advance!

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [March 31, 2022, 8:33am UTC](https://discuss.huggingface.co/t/question-about-gpt2lmheadmodel-gpt2forsequenceclassification/16266/2 "2022-03-31T08:33:59Z")

</div>

> [@lhg912](#):
>
> In order to generate a sentence autoregressively, should I use GPT2LMHeadModel, not GPT2Model? (maybe because there is no LM head on GPT2Model?)

=\> you should use `GPT2LMHeadModel`, since this adds the language modeling head on top of `GPT2Model`. To do language modeling, you need a language modeling head on top of the base model.

> [@lhg912](#):
>
> Does GPT2ForSequenceClassification also have LM head? or just classification at the end?  
> if also has LM head?

`GPT2ForSequenceClassification` doesn’t have a language modeling head. Instead, it just uses a classification head. It will use the last token in order to do the classification, as other causal models  
(e.g. GPT-1) do.

So basically, let’s say you have the sentence “hello world”, and it gets tokenized (using `GPT2Tokenizer`) into [“he”, “llo”, “world”], then you first forward the tokens through the base model to get final hidden states for each of the tokens, and then you place your classifier head on top of the final hidden state of the last token (in this case, the token “world”). This final hidden state is a vector (for instance of size 768, assuming you’re using a GPT-2 base-sized model), and the classifier turns it into a vector of size 2 (assuming you have 2 labels).

> [@lhg912](#):
>
> Can I also use GPT2ForSequenceClassification for generating a sentence?

No, for generating language you’ll have to use `GPT2LMHeadModel`.

> [@lhg912](#):
>
> How GPT2ForSequenceClassification is pretrained?

GPT2 was pre-trained to predict the next token on a very large corpus of text. So basically `GPT2LMHeadModel` was used for pre-training the model. If you then want to use GPT-2 for sequence classification, you can throw away the language modeling head and place a (randomly initialized) sequence classifier head on top instead: that’s what `GPT2ForSequenceClassification` is doing.

Alternatively, you can treat sequence classification as a language modeling problem, by “prompting” the model (that’s how GPT-3 and friends also treat sequence classification nowadays). You can for instance train the model to generate the class after your prompt, like:

“classify: I really liked this movie”

=\> and then `GPT2LMHeadModel` should generate “positive”.

---

<div class="post-metadata">

**Author:** ![anon3699016](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@anon3699016](https://discuss.huggingface.co/u/anon3699016)\
**Post date:** [April 1, 2022, 4:35am UTC](https://discuss.huggingface.co/t/question-about-gpt2lmheadmodel-gpt2forsequenceclassification/16266/3 "2022-04-01T04:35:00Z")

</div>

@nielsr , Thank you very much for your detailed answer!

Now I know what to do. Since my model has to generate sentence and also sequence classification, I will use GPT2LMHeadModel by attaching classification head on top of a special token that I’m going to append after the last token of a sentence.