# Difference between CausalLMWithValueHead vs ModelForCausalLM

**URL:** <https://discuss.huggingface.co/t/difference-between-causallmwithvaluehead-vs-modelforcausallm/58549>\
**Category:** 🤗Transformers\
**Created:** [October 14, 2023, 11:35am UTC](https://discuss.huggingface.co/t/difference-between-causallmwithvaluehead-vs-modelforcausallm/58549 "2023-10-14T11:35:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![deshwalmahesh](https://avatars.discourse-cdn.com/v4/letter/d/6a8cbe/32.png) [@deshwalmahesh](https://discuss.huggingface.co/u/deshwalmahesh)\
**Post date:** [October 14, 2023, 11:35am UTC](https://discuss.huggingface.co/t/difference-between-causallmwithvaluehead-vs-modelforcausallm/58549/1 "2023-10-14T11:35:26Z")

</div>

I know what is `AutoModelForCausalLM`. The thing I’m asking is that in the [`peft` LoRA Fine tuning tutorial](https://huggingface.co/docs/trl/v0.7.1/lora_tuning_peft), the autors have used `AutoModelForCausalLMWithValueHead` while you pick any code or notebook on Fine-tuning of any LLM with `PEFT` style, you’ll find `AutoModelForCausalLM` being used.

I went to lean on the [official documentation of `AutoModelForCausalLMWithValueHead`](https://huggingface.co/docs/trl/models) and found:

> An autoregressive model with a value head in addition to the language model head

What I want to ask is that **How, where and more importantly, _WHY_ this extra `ValueHead` is used**

---

<div class="post-metadata">

**Author:** ![fasterinnerlooper](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fasterinnerlooper/32/28824_2.png) [@fasterinnerlooper](https://discuss.huggingface.co/u/fasterinnerlooper)\
**Post date:** [January 26, 2024, 9:27pm UTC](https://discuss.huggingface.co/t/difference-between-causallmwithvaluehead-vs-modelforcausallm/58549/2 "2024-01-26T21:27:18Z")

</div>

While looking for an answer to this question, I came across this discussion. Is it helpful? [what is AutoModelForCausalLMWithValueHead? · Issue #180 · huggingface/trl · GitHub](https://github.com/huggingface/trl/issues/180)

---

<div class="post-metadata">

**Author:** ![shahzebnaveed](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shahzebnaveed/32/23733_2.png) [@shahzebnaveed](https://discuss.huggingface.co/u/shahzebnaveed)\
**Post date:** [February 15, 2024, 9:23am UTC](https://discuss.huggingface.co/t/difference-between-causallmwithvaluehead-vs-modelforcausallm/58549/3 "2024-02-15T09:23:47Z")

</div>

First things first, this additional ValueHead has nothing to do with PEFT.

Mainly, PPO optimization (a RLHF technique) relies on computing “advantages” associated with taking a particular action (in this case, selecting a token) in a particular state. The computation relies on value of (state,action) pair minus, the value of being in the state. You can review the exact calculation in this function: [trl/trl/trainer/ppo\_trainer.py at main · huggingface/trl · GitHub](https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1148)

The additional ValueHead simply projects the last hidden states onto a scalar to estimate the value of a state. Check the ValueHead class implementation here: [trl/trl/models/modeling\_value\_head.py at main · huggingface/trl · GitHub](https://github.com/huggingface/trl/blob/main/trl/models/modeling_value_head.py#L21)

Note: The ValueHead class is only needed if you plan to perform training/re-training.