# \[Announcement\] Generation: Get probabilities for generated output

**URL:** <https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075>\
**Category:** 🤗Transformers\
**Created:** [January 20, 2023, 1:12pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075 "2023-01-20T13:12:46Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [January 20, 2023, 1:12pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/1 "2023-01-20T13:12:46Z")

</div>

Hey everyone 👋

We have just merged a [PR](https://github.com/huggingface/transformers/pull/21191) that exposes a new function related to `.generate()`, `compute_transition_scores`. With this function, you can quickly solve any problem that requires the probabilities of generated tokens, for any generation strategy. It is also nicely documented – see [here](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.GenerationMixin.compute_transition_scores).

How can this function help you? Let me give you two simple examples!

> **Example 1 -- print the probabilities for the output generated by Greedy Search**
>
> ```py
> from transformers import GPT2Tokenizer, AutoModelForCausalLM
> import numpy as np
> 
> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
> model = AutoModelForCausalLM.from_pretrained("gpt2")
> tokenizer.pad_token_id = tokenizer.eos_token_id
> inputs = tokenizer(["Today is"], return_tensors="pt")
> 
> outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
> transition_scores = model.compute_transition_scores(
> outputs.sequences, outputs.scores, normalize_logits=True
> )
> 
> input_length = inputs.input_ids.shape[1]
> generated_tokens = outputs.sequences[:, input_length:]
> for tok, score in zip(generated_tokens[0], transition_scores[0]):
> # | token | token string | logits | probability
> print(f"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.4f} | {np.exp(score.numpy()):.2%}")
> # Expected output:
> #| 262 | the | -1.4136 | 24.33%
> #| 1110 | day | -2.6089 | 7.36%
> #| 618 | when | -2.0096 | 13.40%
> #| 356 | we | -1.8593 | 15.58%
> #| 460 | can | -2.5083 | 8.14%
> 
> ```

> **Example 2 -- recompute the sequence scores from Beam Search**
>
> ```py
> from transformers import GPT2Tokenizer, AutoModelForCausalLM
> import numpy as np
> 
> tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
> model = AutoModelForCausalLM.from_pretrained("gpt2")
> tokenizer.pad_token_id = tokenizer.eos_token_id
> inputs = tokenizer(["Today is"], return_tensors="pt")
> 
> outputs = model.generate(
> **inputs,
> max_new_tokens=5,
> num_beams=4,
> num_return_sequences=4,
> return_dict_in_generate=True,
> output_scores=True,
> )
> transition_scores = model.compute_transition_scores(
> outputs.sequences, outputs.scores, outputs.beam_indices, normalize_logits=False
> )
> # If you sum the generated tokens' scores and apply the length penalty, you'll get the sequence scores.
> # Tip: set `normalize_logits=True` to recompute the scores from the normalized logits.
> output_length = np.sum(transition_scores.numpy() < 0, axis=1)
> length_penalty = model.generation_config.length_penalty
> reconstructed_scores = transition_scores.sum(axis=1) / (output_length**length_penalty)
> print(np.allclose(outputs.sequences_scores, reconstructed_scores))
> # Expected output:
> #True
> 
> ```

🔥 There is also an interactive demo [here](https://huggingface.co/spaces/joaogante/color-coded-text-generation) that makes use of these functionalities to color-code generated text according to the probabilities 🔥

Let me know if you have comments, questions, and/or suggestions! 🤗

(P.S.: This new post is also meant as a replacement for an [older one](https://discuss.huggingface.co/t/generation-probabilities-how-to-compute-probabilities-of-output-scores-for-gpt2/3175), which contains stale examples. Let’s keep all further discussion around generated token probabilities here!)

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [January 20, 2023, 9:53pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/2 "2023-01-20T21:53:44Z")

</div>

@joaogante coming from the log probabilities thread here as per your recommendation. This is a great contribution! While at it - do you plan to add utility functions to calculate log probabilities for each token if no scores are given but just token ids? More generally - given a sentence, calculate log probabilities for each token in that sentence.

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [January 21, 2023, 10:47am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/3 "2023-01-21T10:47:57Z")

</div>

Hi @vblagoje 👋

That was not on my plans, but you’re not the first person asking for them… so I’ll include it there! In summary, I think `.generate()` can be modified to also return the scores of the input, for decoder-only models like the GPT family or OPT.

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [January 21, 2023, 11:32am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/4 "2023-01-21T11:32:09Z")

</div>

Hey @joaogante,

Thank you for the prompt response. However, I meant more like a general utility method to calculate log probs for the given text regardless of `generate`. Of course, one can do this “by hand” by applying softmax on logits, but then again, we have to check the docs, do we need to shift inputs, need to invoke gather to get logits on individual tokens, whether calculations are different for this architecture from that architecture, etc etc. Why not have some utility method that, for a given model and text, returns tokens and their logprobs?

---

<div class="post-metadata">

**Author:** ![rsaha](https://avatars.discourse-cdn.com/v4/letter/r/b38774/32.png) [@rsaha](https://discuss.huggingface.co/u/rsaha)\
**Post date:** [January 23, 2023, 4:43pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/5 "2023-01-23T16:43:51Z")

</div>

Getting an ` AttributeError: 'GPT2LMHeadModel' object has no attribute 'compute_transition_scores'` error while trying to run. Installed transformers library using `pip install git+https://github.com/huggingface/transformers`

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [January 24, 2023, 3:11pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/6 "2023-01-24T15:11:12Z")

</div>

Hey @rsaha 👋 perhaps you need to add the `--upgrade` flag (i.e. `pip install --upgrade git+https://github.com/huggingface/transformers.git`)

If the problem persists, please open an issue in transformers 🙏

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [January 24, 2023, 3:13pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/7 "2023-01-24T15:13:40Z")

</div>

@vblagoje I see. That is easy to build, added that to my to do list 🙂

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [January 24, 2023, 3:58pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/8 "2023-01-24T15:58:09Z")

</div>

Awesome, @joaogante I have my own implementation that works well for the model we currently use, but it would undoubtedly be better to use an official API one day. I think others will find it helpful as well especially now that providers like OpenAI and Cohere offer token logprobs as part of the standard API. All the best

---

<div class="post-metadata">

**Author:** ![rsaha](https://avatars.discourse-cdn.com/v4/letter/r/b38774/32.png) [@rsaha](https://discuss.huggingface.co/u/rsaha)\
**Post date:** [January 26, 2023, 4:16pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/9 "2023-01-26T16:16:47Z")

</div>

Worked with the upgrade flag!

---

<div class="post-metadata">

**Author:** ![Seohyeong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seohyeong/32/2273_2.png) [@Seohyeong](https://discuss.huggingface.co/u/Seohyeong)\
**Post date:** [January 27, 2023, 7:02am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/10 "2023-01-27T07:02:20Z")

</div>

@joaogante Thanks for your contribution. I’ve been looking for this and it works well with T5 with the latest tag (v 4.26.0). Can you double check if my understanding is correct?

transition\_scores\_{i} = log(P(y\_i|y\_1, …, y\_{i-1}, x)),  
sequence\_scores (from `.generate()`) = (1/t)\* sum\_{i=1}^{t} log(P(y\_i|y\_1, …, y\_{i-1}, x))

Therefore, as stated in the documentation, the length normalized sum of `transition_scores` equals the `sequence_scores`? (but the documentation states that it’s only guaranteed when `normalize_logits=False`. Does this mean the original implementation of `sequence_scores` does not normalize logits across the vocab dimension?)

Also, my understanding of `scores` from the `.generate()` is still shaky. Could anyone share their understanding on how `scores` from the `.generate()` method is different from `transition_scores` from `.compute_transition_scores()`?

---

<div class="post-metadata">

**Author:** ![Seohyeong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seohyeong/32/2273_2.png) [@Seohyeong](https://discuss.huggingface.co/u/Seohyeong)\
**Post date:** [January 27, 2023, 12:49pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/11 "2023-01-27T12:49:36Z")

</div>

Hi @vblagoje. I’ve been looking for such implementation. Would you be willing to share yours?

---

<div class="post-metadata">

**Author:** ![jas-ho](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jas-ho/32/12786_2.png) [@jas-ho](https://discuss.huggingface.co/u/jas-ho)\
**Post date:** [January 27, 2023, 4:49pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/12 "2023-01-27T16:49:33Z")

</div>

Nice! This is not exposed in the hosted inference API, right?

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [January 28, 2023, 11:35am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/13 "2023-01-28T11:35:12Z")

</div>

@jas-ho let me answer in two parts:

1. We favor inference endpoints ([Inference Endpoints - Hugging Face](https://huggingface.co/inference-endpoints)) to inference API, as the former are much more flexible.
2. Inference endpoints are fully flexible, so it is surely doable. The question is how complex it is to do it 🙂 I have no idea, so I will build a demo and share it here!

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [January 28, 2023, 11:51am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/14 "2023-01-28T11:51:46Z")

</div>

@Seohyeong I agree that formalizing the definitions for those three things (`scores`, `transition_scores`, and `sequence_scores`) can be tricky. Especially because we, in `transformers`, have to ensure that what we build is backward compatible, which means that our naming of variables can become slightly inaccurate.

1. `scores` are the UNNORMALIZED log probabilities (they will only be normalized if you pass `renormalize_logits=True` to `.generate()`). Sadly, we are stuck with unnormalized log probabilities to ensure backwards compatibility ☹ It is a tuple containing one entry for each generated token. Each tuple member is a tensor containing the log probabilities from the model, for all words in the vocabulary. These log probabilities are gathered AFTER manipulating them with our logits processors (e.g. after setting the probability of certain words to 0, after applying top k, …);
2. `sequence_scores` correspond to the sum of the log probabilities in `scores`. If `length_penalty` is not 0, `sequence_scores` will be divided by `sequence_length**length_penalty`;
3. `transition_scores` contains `scores` for the tokens that were selected at generation time. You can set `normalize_logits=True` to ensure they are normalized at a token level (i.e. to ensure the sum of probabilities for all vocabulary at a given generation step is 1).

⚠ An important detail: because `scores` is unnormalized by default, it also means that `sequence_scores` is the sum of unnormalized `scores`, which can be undesirable. A workaround is to create `transition_scores` from `compute_transition_scores` with `normalize_logits=True`, from which you can recompute the sequence scores using normalized scores.

Hopefully we can correct this unintuitive behavior when we release the next major version (`transformers` v5.0) 🙏

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [January 31, 2023, 8:22am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/15 "2023-01-31T08:22:54Z")

</div>

Hey @Seohyeong here it is

```auto
def to_tokens_and_logprobs(model, tokenizer, input_texts):
  encoded_input_texts = tokenizer(input_texts, padding=True, return_tensors="pt")
  outputs = model(encoded_input_texts.input_ids, labels=encoded_input_texts.input_ids)
  probs = torch.log(outputs.logits.softmax(dim=-1)/100).detach()

  # collect the probability of the generated token
  # we need to add a dummy dim in the end to make gather work
  gen_probs = torch.gather(probs, 2, input_ids[:, :, None]).squeeze(-1)

  batch = []
  for input_sentence, input_probs in zip(input_ids, gen_probs):
    text_sequence = []
    for token, p in zip(input_sentence, input_probs):  
      if token not in tokenizer.all_special_ids:
        text_sequence.append((tokenizer.decode(token),p.item()))
    batch.append(text_sequence)    
  return batch

```

Then you would call this method with:

```auto
to_tokens_and_logprobs(model, tokenizer, ["Give me something", "Good morning", "Hello, how are you?"])

```

And get the tokens and probs pairs:

```auto
[[('Give', -13.964306831359863),
  ('me', -5.264768123626709),
  ('something', -6.42432165145874)],
 [('Good', -20.744667053222656), ('morning', -4.624837398529053)],
 [('Hello', -8.46367073059082),
  (',', -5.478875637054443),
  ('how', -6.255255222320557),
  ('are', -4.785802841186523),
  ('you', -4.666200160980225),
  ('?', -4.6729416847229)]]

```

It would be great if @joaogante could review this code snippet 🙂

---

<div class="post-metadata">

**Author:** ![Seohyeong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seohyeong/32/2273_2.png) [@Seohyeong](https://discuss.huggingface.co/u/Seohyeong)\
**Post date:** [February 2, 2023, 3:03am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/16 "2023-02-02T03:03:33Z")

</div>

Thanks for the explanation! I understand why certain things are implemented in such ways. I personally had a difficult time trying to find definitions of scores, sequence\_scores, and transition\_scores and how they’re calculated. It’d be great if this chuck of your explanation is added to the official documentation.

Also, I do run into RuntimeError by tensor size mismatch by running `compute_transition_scores()` with a batch size \> 1. Does this method only support input with batch size 1 for now?

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [February 2, 2023, 11:07am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/17 "2023-02-02T11:07:33Z")

</div>

Hey @vblagoje 👋 I believe you forgot one tiny detail, other than that looks like a solid implementation! In a nutshell, these models return the probabilities for the next token, which means that `logits[batch_idx, seq_idx, vocab_idx]` actually contains the logits corresponding to `input_ids[batch_idx, seq_idx + 1]`. This implies that the pairing between the probabilities and the tokens has to be shifted by one 🙂

Here is a modified script:

```plaintext
from pprint import pprint
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

def to_tokens_and_logprobs(model, tokenizer, input_texts):
    input_ids = tokenizer(input_texts, padding=True, return_tensors="pt").input_ids
    outputs = model(input_ids)
    probs = torch.log_softmax(outputs.logits, dim=-1).detach()

    # collect the probability of the generated token -- probability at index 0 corresponds to the token at index 1
    probs = probs[:, :-1, :]
    input_ids = input_ids[:, 1:]
    gen_probs = torch.gather(probs, 2, input_ids[:, :, None]).squeeze(-1)

    batch = []
    for input_sentence, input_probs in zip(input_ids, gen_probs):
        text_sequence = []
        for token, p in zip(input_sentence, input_probs):
            if token not in tokenizer.all_special_ids:
                text_sequence.append((tokenizer.decode(token), p.item()))
        batch.append(text_sequence)
    return batch

tokenizer = AutoTokenizer.from_pretrained("gpt2", padding_side="left")
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained("gpt2")
model.config.pad_token_id = model.config.eos_token_id

input_texts = ["One plus one is two", "Good morning", "Hello, how are you?"]

batch = to_tokens_and_logprobs(model, tokenizer, input_texts)
pprint(batch)

```

which yields

```auto
[[('One', -5.882715702056885),
  (' plus', -9.785109519958496),
  (' one', -0.7229145169258118),
  (' is', -2.494063377380371),
  (' two', -6.137458324432373)],
 [('Good', -7.5790300369262695), (' morning', -1.826707124710083)],
 [(',', -2.343151807785034),
  (' how', -4.339702606201172),
  (' are', -2.6824729442596436),
  (' you', -0.4109247326850891),
  ('?', -1.8950778245925903)]]

```

Notice how high the logits for certain obvious tokens are, like `morning`, `you`, or `?`! Checking these tokens is always a good sanity check 😃 Also, look at the last sentence: there are no logits for the first token. If you want the logits for that token, you need to add extra padding on the left, so that the first text token is not the actual first token fed to the model.

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\
**Post date:** [February 2, 2023, 11:49am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/18 "2023-02-02T11:49:18Z")

</div>

@Seohyeong yeah, we have a very big backlog and improving the docstrings is part of it. Our team is actually quite small (I’m the only one working on `generate` at the moment), so PRs with improvements are very welcome 🙏

Can you open an issue on GitHub with what you are experiencing? ( `compute_transition_scores()` with a batch size \> 1)

---

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\
**Post date:** [February 2, 2023, 3:28pm UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/19 "2023-02-02T15:28:27Z")

</div>

Excellent, you are right, thank you @joaogante

---

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fzyzcjy/32/25229_2.png) [@fzyzcjy](https://discuss.huggingface.co/u/fzyzcjy)\
**Post date:** [February 4, 2023, 6:45am UTC](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075/20 "2023-02-04T06:45:05Z")

</div>

Hi, thanks for the PR and explanations! I am still a bit confused 😕 I want to get the probability of each output under (for example) a beam search. In other words, P\_\theta(output|input) where \theta is language model parameters, output is the output of the LM, and input is the input. Since it is nothing but a conditional probability, I guess this is clearly defined. Then, how should I get this value?

My current guess:

```plaintext
# modified from "example 1" and "example 2"
outputs = model.generate(**inputs, max_new_tokens=5, return_dict_in_generate=True, output_scores=True)
transition_scores = model.compute_transition_scores(
    outputs.sequences, outputs.scores, normalize_logits=True # NOTE normalize SHOULD be true?
)
output_length = inputs.input_ids.shape[1] + np.sum(transition_scores.numpy() < 0, axis=1)
length_penalty = model.generation_config.length_penalty
probabilities = torch.exp(transition_scores.sum(axis=1) / (output_length**length_penalty))
# what I want is the `probabilities` (correct?)

```

Looks it is wrong, because I can get probabilities that _sum up to larger than 1_

[Next page](https://discuss.huggingface.co/t/announcement-generation-get-probabilities-for-generated-output/30075.md?page=2)
