# Trying to understand the task-specific head for diff. models + Transformers AutoModel

**URL:** <https://discuss.huggingface.co/t/trying-to-understand-the-task-specific-head-for-diff-models-transformers-automodel/37217>\
**Category:** 🤗Transformers\
**Created:** [April 20, 2023, 11:21am UTC](https://discuss.huggingface.co/t/trying-to-understand-the-task-specific-head-for-diff-models-transformers-automodel/37217 "2023-04-20T11:21:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![DanRKan](https://avatars.discourse-cdn.com/v4/letter/d/e47774/32.png) [@DanRKan](https://discuss.huggingface.co/u/DanRKan)\
**Post date:** [April 20, 2023, 11:21am UTC](https://discuss.huggingface.co/t/trying-to-understand-the-task-specific-head-for-diff-models-transformers-automodel/37217/1 "2023-04-20T11:21:24Z")

</div>

Hi! i’m new to HuggingFace and I have to use the models to get embeddings from text. When I was exploring the code and model checkpoints to do so, I came across this code and was confused. Please let me know if I’m thinking in the right direction.

1. **Understanding the difference between CLM heads of different models in the given docs:**  
In the code mentioned in the example of [XLM-R docs](https://huggingface.co/docs/transformers/model_doc/xlm-roberta#transformers.XLMRobertaForCausalLM), `"roberta-base"` checkpoint is of monolingual Roberta. The Roberta model is getting loaded from that with a CLM head on top, but the understanding of the base model is in only 1 single language.

Is this model the same as that loaded by `RobertaForCausalLM.from_pretrained("roberta-base", config=config)` ? Further in this, is the CLM head for `RobertaForCausalLM` same as `XLMRobertaForCausalLM`?

1. Why is there a need for two different classes such as `RobertaForCausalLM()` and `XLMRobertaForCausalLM()` if we can do that using `AutoModelForCausalLM()` and load the models(both pretrained and from their configs)?