# How to set generation parameters for transformers.pipeline?

**URL:** <https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837>\
**Category:** 🤗Transformers\
**Created:** [July 31, 2023, 2:42pm UTC](https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837 "2023-07-31T14:42:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![vivi0](https://avatars.discourse-cdn.com/v4/letter/v/8c91f0/32.png) [@vivi0](https://discuss.huggingface.co/u/vivi0)\
**Post date:** [July 31, 2023, 2:42pm UTC](https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837/1 "2023-07-31T14:42:08Z")

</div>

I can’t figure out the correct way to update the config/ generation config parameters for transformers.pipeline (temperature etc, max\_new\_tokens, torch\_dtype and device\_map)

```auto
from transformers import pipeline
pipe = pipeline(
   'text-generation',
    model = hf_model_id,
    temperature = 0.1,
    max_new_tokens=30,
    torch_dtype='auto',
    device_map="auto")
pipe(prompt)

```

So if I just use the arguments in pipeline, I get `UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)`

But I try:

```auto
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, pipeline
hf_model_id = "eachadea/vicuna-7b-1.1"
model = AutoModelForCausalLM.from_pretrained(hf_model_id)
tokenizer = AutoTokenizer.from_pretrained(hf_model_id, legacy=False)
generation_config, unused_kwargs = GenerationConfig.from_pretrained(
    hf_model_id, max_new_tokens=200, temperature=0.1, return_unused_kwargs=True
)
model.generation_config = generation_config
pipe = pipeline(
    "text-generation", 
    model=model, 
    tokenizer=tokenizer,

)
pipe(prompt)

```

And it doesn’t usually throw the warning, but it uses the default config instead of whatever I put in model.generation\_config (for example, instead of 200 max\_new\_tokens, it just uses 20 max\_tokens)

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [July 31, 2023, 2:59pm UTC](https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837/2 "2023-07-31T14:59:04Z")

</div>

cc @joaogante

---

<div class="post-metadata">

**Author:** ![Ryan30](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@Ryan30](https://discuss.huggingface.co/u/Ryan30)\
**Post date:** [August 1, 2023, 8:36am UTC](https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837/3 "2023-08-01T08:36:22Z")

</div>

I have the same question

---

<div class="post-metadata">

**Author:** ![Ryan30](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@Ryan30](https://discuss.huggingface.co/u/Ryan30)\
**Post date:** [August 1, 2023, 8:37am UTC](https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837/4 "2023-08-01T08:37:00Z")

</div>

I have the same question.

---

<div class="post-metadata">

**Author:** ![Borell](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@Borell](https://discuss.huggingface.co/u/Borell)\
**Post date:** [August 4, 2023, 8:37pm UTC](https://discuss.huggingface.co/t/how-to-set-generation-parameters-for-transformers-pipeline/48837/5 "2023-08-04T20:37:03Z")

</div>

Hi vivio,

Try passing config parameters directly to pipe as kwargs or as:  
pipe(prompt, max\_new\_tokens=200, temperature=0.1)

I believe it should work, look at the example for TextGenerationPipeline at  
“[Pipelines](https://huggingface.co/docs/transformers/main/en/main_classes/pipelines#transformers.pipeline)”  
where “do\_samples=True” is being passed to the pipeline instance.

The GenerationConfig.from\_pretrained works with a model instance created with AutoModelForCausalLM. Then you can:  
model.generate(\*\*inputs, generation\_config=generation\_config)  
With inputs being the prompt but being tokenized first.

Let me know if it worked.

Best,  
Borell