# Meta-Llama-3-8B-Instruct: Validation Error "Max\_new\_tokens"

**URL:** <https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721>\
**Category:** Models\
**Created:** [June 6, 2024, 3:59pm UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721 "2024-06-06T15:59:19Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![SyedSubahani](https://avatars.discourse-cdn.com/v4/letter/s/b487fb/32.png) [@SyedSubahani](https://discuss.huggingface.co/u/SyedSubahani)\
**Post date:** [June 6, 2024, 3:59pm UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/1 "2024-06-06T15:59:19Z")

</div>

Hi,  
I tried to build the QA RAG chatbot using Llama3 but it failed to give the response with the below error message. It fails at “RetrievalQA” method.

The same code works fine with the model “Mistral-7B-Instruct-v0.2”.

Any Help here is greatly appreciated.

Code:  
llm = HuggingFaceEndpoint(  
repo\_id=llm\_model,  
huggingfacehub\_api\_token=HF\_API\_TOKEN,  
temperature=temperature,  
max\_new\_tokens=max\_tokens,  
top\_k=top\_k,  
)

```
retriever = vector_db.as_retriever(search_kwargs={"k": 3})

compressor = FlashrankRerank(model="ms-marco-MiniLM-L-12-v2")

compression_retriever = ContextualCompressionRetriever(
                        base_compressor=compressor, base_retriever=retriever
                    )

qachain = RetrievalQA.from_chain_type(
                llm=llm,
                chain_type="stuff",
                retriever=compression_retriever,
                return_source_documents=True,
                chain_type_kwargs={"prompt": prompt, "verbose": False},
            )

return qachain

```

Error Message:  
HfHubHTTPError: 422 Client Error: Unprocessable Entity for url: [https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3-8B-Instruct](https://api-inference.huggingface.co/models/meta-llama/Meta-Llama-3-8B-Instruct) Input validation error: `inputs` tokens + `max_new_tokens` must be \<= 8192. Given: 14087 `inputs` tokens and 256 `max_new_tokens`

---

<div class="post-metadata">

**Author:** ![arieltoledo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/arieltoledo/32/28586_2.png) [@arieltoledo](https://discuss.huggingface.co/u/arieltoledo)\
**Post date:** [June 24, 2024, 12:17pm UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/2 "2024-06-24T12:17:16Z")

</div>

Hi, im experience the same error. Did you find any solution yet ?

---

<div class="post-metadata">

**Author:** ![fabioceruti](https://avatars.discourse-cdn.com/v4/letter/f/90ced4/32.png) [@fabioceruti](https://discuss.huggingface.co/u/fabioceruti)\
**Post date:** [October 2, 2024, 7:53am UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/3 "2024-10-02T07:53:58Z")

</div>

Same issue for me here, are there any workarounds?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 2, 2024, 8:16am UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/4 "2024-10-02T08:16:19Z")

</div>

> Input validation error: `inputs` tokens + `max_new_tokens` must be \<= 8192. Given: 14087 `inputs` tokens and 256 `max_new_tokens`

If this error message is correct, then the token you are giving is too long. Why don’t you try giving shorter words?  
It says max\_new\_tokens + tokens must be less than 8192 characters combined.  
But it says that actually given was over 12,000 characters in tokens alone…

---

<div class="post-metadata">

**Author:** ![fabioceruti](https://avatars.discourse-cdn.com/v4/letter/f/90ced4/32.png) [@fabioceruti](https://discuss.huggingface.co/u/fabioceruti)\
**Post date:** [October 2, 2024, 8:37am UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/5 "2024-10-02T08:37:54Z")

</div>

In my case, I am getting the error regardless of the token size.  
If I work in a .ipynb file, I am able to correctly get outputs from the model. Yet, when I plug the model in streamlit to have a nicer visual interface, I get the error regardless of the token size. I could just input “hello” in a “fresh” session and the error would still be there.  
Switching to another model fixed the issue for me

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 2, 2024, 8:41am UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/6 "2024-10-02T08:41:13Z")

</div>

> Switching to another model fixed the issue for me

I wonder if this is it?

> [@Finetuning Meta-Llama-3.1-8B using PEFT](https://discuss.huggingface.co/t/finetuning-meta-llama-3-1-8b-using-peft/108319):
>
> I’ve been working on code to fine-tune (locally) the meta-llama/Meta-Llama-3.1-8B or meta-llama/Meta-Llama-3.1-8B-Instruct model that I downloaded in local server, using the example from the Hugging Face repository at [huggingface/huggingface-llama-recipes](https://github.com/huggingface/huggingface-llama-recipes/blob/main/peft_finetuning.py). I made some adjustments to the code to fit my custom dataset. While the training process seems to complete successfully, I’m encountering an issue during inference: the model generates gibberish responses, even for general questions. Sample …

---

<div class="post-metadata">

**Author:** ![samchain](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/samchain/32/30999_2.png) [@samchain](https://discuss.huggingface.co/u/samchain)\
**Post date:** [October 2, 2024, 8:55am UTC](https://discuss.huggingface.co/t/meta-llama-3-8b-instruct-validation-error-max-new-tokens/90721/7 "2024-10-02T08:55:41Z")

</div>

Hey, the problem comes from a too long context provided to the model.

The error indicates that the length of your context + prompt is of length 14087. It is strange as you select top 3 docs to answer. How long are your documents when tokenized ?

In my opinion:  
Create shorter chunks of your documents in the db for retrieval. Hence, you’ll provide shorter and more relevant context to the LLM to answer the question.