# Adding prompt / context to Whisper with Huggingface Transformers

**URL:** <https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070>\
**Category:** Models\
**Created:** [February 3, 2023, 7:35pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070 "2023-02-03T19:35:05Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![SamuelAzran](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@SamuelAzran](https://discuss.huggingface.co/u/SamuelAzran)\
**Post date:** [February 3, 2023, 7:35pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/1 "2023-02-03T19:35:05Z")

</div>

The Whisper model, has the possibility of a prompt or adding the previous text to the current transcription task. This helps in case of transcribing long file chunk after chunk.

During training it should “mask out the training loss over the previous context text, and train the  
model to predict all other tokens”.

> **[2212.04356.pdf](https://arxiv.org/pdf/2212.04356.pdf)**
>
> 985.00 KB

I’m wondering if HF has implemented that and how well does it helps with the accuracy.

There is also this issue in Huggingface Transformers:

> <https://github.com/huggingface/transformers/issues/20624>
>
> \### System Info
> 
> v4.25.1 on M1 Mac with python 3.8
> 
> \### Who can help?
> 
> @sanchit-…gandhi @patrickvonplaten @anton-l 
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[x\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> When we want to run Whisper generation for a batch of samples with different prompt lengths (prefix tokens given to the decoder), positional embeddings for the decoder are improperly computed. It assumes all sequences have the same \`past\_key\_values\_length\`, but this is not true in general.
> 
> Scenario:
> \`decoder\_input\_ids = \[50361, 45431, 2584, 28682, 13, 50258, 50257, 50257\]\`
> (\`"\<|startofprev|\>Something completely irrelevant.\<|startoftranscript|\>\<|pad|\>\<|pad|\>"\`)
> 
> \`model.generate(input\_features, decoder\_input\_ids=decoder\_input\_ids, decoder\_attention\_mask=decoder\_attention\_mask)\` will not give the correct output because at the beginning of decoding, the pad tokens won't be taken into account that the positional embedding will be off.
> 
> \### Expected behavior
> 
> Instead of tracking \`past\_key\_values\_length\`, it should use the attention mask to compute position ids. The current implementation is more based off of encoder-decoder architectures that would never do decoder prompting, but it should take more inspiration from decoder-only models to handle prompting. This is done for the Flax implementation in #20479

I’m wondering if it is fixed and stable.  
There it seems that the way of doing it with HF would be is something like so:

```auto
prev_before = '<|startofprev|>'
current_before = "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>"
current_after = "<|endoftext|>"

prev_text = 'Hello and'
current_text = 'welcome. My name is...'

prompt_and_text = prev_before + prev_text + current_before + current_text + current_after
prompt_and_text

```

\<|startofprev|\>Hello and\<|startoftranscript|\>\<|en|\>\<|transcribe|\>\<|notimestamps|\>welcome. My name is…\<|endoftext|\>

```auto
prompt_and_text_tokens = _tokenizer.encode(prompt_and_text, add_special_tokens=False)
_tokenizer.decode(prompt_and_text_tokens)

```

\<|startofprev|\>Hello and\<|startoftranscript|\>\<|en|\>\<|transcribe|\>\<|notimestamps|\>welcome. My name is…\<|endoftext|\>

---

<div class="post-metadata">

**Author:** ![radurevutchi](https://avatars.discourse-cdn.com/v4/letter/r/a698b9/32.png) [@radurevutchi](https://discuss.huggingface.co/u/radurevutchi)\
**Post date:** [March 22, 2023, 8:54pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/2 "2023-03-22T20:54:05Z")

</div>

@SamuelAzran did you manage to get this working? I’d like to have the exact same functionality.

---

<div class="post-metadata">

**Author:** ![andercorral](https://avatars.discourse-cdn.com/v4/letter/a/a9adbd/32.png) [@andercorral](https://discuss.huggingface.co/u/andercorral)\
**Post date:** [March 24, 2023, 1:35pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/3 "2023-03-24T13:35:20Z")

</div>

Any news on this?

---

<div class="post-metadata">

**Author:** ![SamuelAzran](https://avatars.discourse-cdn.com/v4/letter/s/8797f3/32.png) [@SamuelAzran](https://discuss.huggingface.co/u/SamuelAzran)\
**Post date:** [March 24, 2023, 3:28pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/4 "2023-03-24T15:28:17Z")

</div>

I did not see Huggingface supporting it or providing an easy way to do it, like in the open\_ai library, but you can hack around or use lower-level components in HF to give context or prompt in inference time. In training time, training on prompts is much more complicated. I hope that someone from HF can address this issue.

The benefits of using prompts are beyond just giving the text of the previous speech segments. You can also provide prompts to control the style of the transcript and add relevant domain-specific or industry-specific terms to the context so the transcription process will be more likely to use those terms in the final transcript.

---

<div class="post-metadata">

**Author:** ![andercorral](https://avatars.discourse-cdn.com/v4/letter/a/a9adbd/32.png) [@andercorral](https://discuss.huggingface.co/u/andercorral)\
**Post date:** [March 24, 2023, 7:29pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/5 "2023-03-24T19:29:55Z")

</div>

Great! Thanks! That’s what I will end up doing

---

<div class="post-metadata">

**Author:** ![wsfung2008](https://avatars.discourse-cdn.com/v4/letter/w/c57346/32.png) [@wsfung2008](https://discuss.huggingface.co/u/wsfung2008)\
**Post date:** [July 11, 2023, 9:50am UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/6 "2023-07-11T09:50:46Z")

</div>

Related discussion: [Is it possible to do model adaptation? · openai/whisper · Discussion #66 · GitHub](https://github.com/openai/whisper/discussions/66)

You can also use logit biasing to increase the probability of certain domain specific words being predicted.

---

<div class="post-metadata">

**Author:** ![IntraphoneMarlar](https://avatars.discourse-cdn.com/v4/letter/i/b9e5f3/32.png) [@IntraphoneMarlar](https://discuss.huggingface.co/u/IntraphoneMarlar)\
**Post date:** [April 30, 2024, 8:10am UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/7 "2024-04-30T08:10:19Z")

</div>

Any update on support for this in hf libraries?

---

<div class="post-metadata">

**Author:** ![bmialet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bmialet/32/39481_2.png) [@bmialet](https://discuss.huggingface.co/u/bmialet)\
**Post date:** [January 20, 2025, 2:10pm UTC](https://discuss.huggingface.co/t/adding-prompt-context-to-whisper-with-huggingface-transformers/31070/8 "2025-01-20T14:10:34Z")

</div>

For anyone who is still looking for a solution.  
I think I manged to make it work. Here is an example with whisper medium in French.

First extract the prompt ids with `get_prompt_ids()` method:

```auto
import torch
prev_text = "est ce qu'ils sont choqués ?"
prompt_ids = torch.tensor(whisper_processor.get_prompt_ids(prev_text), device=device)
print(whisper_processor.decode(prompt_ids, skip_special_tokens=False))

```

_“\<|startofprev|\> est ce qu’ils sont choqués ?”_

Then inject this prompt in inference. this inference uses no sampling to remain deterministic in order to show the real effect of the prompt:

```auto
audio_features = extract_audio_input_features(
    audio,
    whisper_processor,
    device,
    torch_dtype
)
whisper_model.generation_config.forced_decoder_ids = None
predicted_ids = whisper_model.generate(
    audio_features,
    language="fr",
    task="transcribe",
    eos_token_id=whisper_model.generation_config.eos_token_id,
    pad_token_id=whisper_model.generation_config.pad_token_id,
    max_time=5,
    do_sample=False,
    return_dict_in_generate=False,
    prompt_ids=prompt_ids
)

print(whisper_processor.decode(predicted_ids[0]))

```

_“Non, ça va, pour l’instant ça va, ils sont **choqués** mais ça va.”_

Here is the original transcription, without the prompt (also without any random sampling):  
_“Non, ça va. Pour l’instant, ça va. Ils sont **pour chocrer** , mais ça va.”_  
(Error on word “choqués”)

I’m open to remarks or ways to improve this.