# How to get \[CLS\] embeddings from BertForTokenClassification model

**URL:** <https://discuss.huggingface.co/t/how-to-get-cls-embeddings-from-bertfortokenclassification-model/9276>\
**Category:** Beginners\
**Created:** [August 16, 2021, 5:49pm UTC](https://discuss.huggingface.co/t/how-to-get-cls-embeddings-from-bertfortokenclassification-model/9276 "2021-08-16T17:49:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![slecraphi](https://avatars.discourse-cdn.com/v4/letter/s/d26b3c/32.png) [@slecraphi](https://discuss.huggingface.co/u/slecraphi)\
**Post date:** [August 16, 2021, 5:49pm UTC](https://discuss.huggingface.co/t/how-to-get-cls-embeddings-from-bertfortokenclassification-model/9276/1 "2021-08-16T17:49:54Z")

</div>

Sorry for the issue, I don’t really write any code but only use the example code as a tool.

I trained with my own NER dataset with the transformers example code.

I want to get sentence embedding from the model I trained with the token classification example code [here](https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py) (this is the older version of example code by the way.)

I want to get the sentence embedding from the trained model, which I think the [CLS] token embedding output should be one way.

[This github issue answer](https://github.com/huggingface/transformers/issues/1950#issuecomment-558679189) answers exactly how to get an embedding from a BertModel (I can also get [CLS] token as the first token in sentence)  
The answer code is copy paste below:

```auto
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0) # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple  

```

So here comes my problem: How to get embeddings from BertForTokenClassification instead of BertModel? Can I simply replace the BertModel with BertForTokenClassification in the code and the expected output is what I wanted?

---

<div class="post-metadata">

**Author:** ![ehalit](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ehalit/32/28854_2.png) [@ehalit](https://discuss.huggingface.co/u/ehalit)\
**Post date:** [August 17, 2021, 4:47am UTC](https://discuss.huggingface.co/t/how-to-get-cls-embeddings-from-bertfortokenclassification-model/9276/2 "2021-08-17T04:47:55Z")

</div>

I suggest having a look at the `hidden_states` [output](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertForTokenClassification.forward) of the model returned when `output_hidden_states=True` is passed to the forward call, or to the `from_pretrained` method while loading the model. The [CLS] embedding is the 0th slice of the output tensor of the last layer.

---

<div class="post-metadata">

**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\
**Post date:** [August 17, 2021, 8:58am UTC](https://discuss.huggingface.co/t/how-to-get-cls-embeddings-from-bertfortokenclassification-model/9276/3 "2021-08-17T08:58:27Z")

</div>

Hi @slecraphi

Just to elaborate on @ehalit’s correct approach here your example adapted for token classification:

```auto
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForTokenClassification.from_pretrained('bert-base-uncased')
inputs = tokenizer("Hello, my dog is cute", return_tensors='pt')
outputs = model(**inputs, output_hidden_states=True)
last_hidden_states = outputs.hidden_states[-1]

```

The shape of `last_hidden_states` will be `[batch_size, tokens, hidden_dim]` so if you want to get the embedding of the first element in the batch and the `[CLS]` token you can get it with `last_hidden_states[0,0,:]`.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![aralena7](https://avatars.discourse-cdn.com/v4/letter/a/c2a13f/32.png) [@aralena7](https://discuss.huggingface.co/u/aralena7)\
**Post date:** [November 27, 2023, 5:23pm UTC](https://discuss.huggingface.co/t/how-to-get-cls-embeddings-from-bertfortokenclassification-model/9276/4 "2023-11-27T17:23:20Z")

</div>

Is this the same for BertForSequenceClassification model? The documentation mentions the similar method to obtain hidden layer embeddings for both the models. I just want to confirm.  
Also, it should be a vector of 768 size having float values between -1 and 1 , correct?  
Or, is it normalized ( between 0 and 1) . I get it between -1 and 1. is it how it should look like?