# Unsup fine tuning embeddings

**URL:** <https://discuss.huggingface.co/t/unsup-fine-tuning-embeddings/44015>\
**Category:** 🤗Transformers\
**Created:** [June 21, 2023, 8:50am UTC](https://discuss.huggingface.co/t/unsup-fine-tuning-embeddings/44015 "2023-06-21T08:50:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![yugen2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yugen2/32/13883_2.png) [@yugen2](https://discuss.huggingface.co/u/yugen2)\
**Post date:** [June 21, 2023, 8:50am UTC](https://discuss.huggingface.co/t/unsup-fine-tuning-embeddings/44015/1 "2023-06-21T08:50:49Z")

</div>

Hello,

I am trying to fine tune model embeddings without supervision. I am referring to the output layer of the attention model, the one which is used for generating predictions. In [this tutorial](https://huggingface.co/course/chapter7/3), they show how to generate labels from unsup text using masking, but they do it using a masked LM model, which doesn’t give access to its attention output layer, only word level logits, which is one abstraction level higher. In [this tutorial](https://huggingface.co/blog/how-to-train-sentence-transformers), on the other hand, they do fine tune the output layer embeddings but they do so with labeled data.

I know I can implement it myself by simply generating labels by masking. But I would like to know if there is already a package that does this. And more importantly, how is this not addressed in either of these very well thought-out tutorials? It seems to me like a no brainer.

Thanks!