# How to get token-embeddings of input with decoder-only models?

**URL:** <https://discuss.huggingface.co/t/how-to-get-token-embeddings-of-input-with-decoder-only-models/60753>\
**Category:** Beginners\
**Created:** [November 2, 2023, 2:13am UTC](https://discuss.huggingface.co/t/how-to-get-token-embeddings-of-input-with-decoder-only-models/60753 "2023-11-02T02:13:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kabrav](https://avatars.discourse-cdn.com/v4/letter/k/bbe5ce/32.png) [@kabrav](https://discuss.huggingface.co/u/kabrav)\
**Post date:** [November 2, 2023, 2:13am UTC](https://discuss.huggingface.co/t/how-to-get-token-embeddings-of-input-with-decoder-only-models/60753/1 "2023-11-02T02:13:48Z")

</div>

How to get token-embeddings of input with decoder-only models? Similar to last layer hidden states of the encoder-part of encoder-only or encoder-decoder models.

---

<div class="post-metadata">

**Author:** ![manojkumar427](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/manojkumar427/32/31842_2.png) [@manojkumar427](https://discuss.huggingface.co/u/manojkumar427)\
**Post date:** [September 20, 2024, 1:13pm UTC](https://discuss.huggingface.co/t/how-to-get-token-embeddings-of-input-with-decoder-only-models/60753/2 "2024-09-20T13:13:09Z")

</div>

I tried using last hidden layers with append EOS as well as pooling techniques, but on cosine similarity of these created vectors still doesn’t performing.  
Really want to know how to access the embeddings through decoder only, to compare different inputs.  
Let me know if you found something