# How to implement chatbot streaming in gradio with a function

**URL:** <https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863>\
**Category:** 🔒 Gradio\
**Created:** [June 20, 2023, 5:07am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863 "2023-06-20T05:07:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![kamiyan89](https://avatars.discourse-cdn.com/v4/letter/k/d78d45/32.png) [@kamiyan89](https://discuss.huggingface.co/u/kamiyan89)\
**Post date:** [June 20, 2023, 5:07am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863/1 "2023-06-20T05:07:48Z")

</div>

Hi everyone.

I am trying to build a basicchatbot with streaming.  
from the documentation we have the following code.

```auto
import gradio as gr
import random
import time

with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.Button("Clear")

    def respond(message, chat_history):
        bot_message = random.choice(["How are you?", "I love you", "I'm very hungry"])
        chat_history.append((message, bot_message))
        time.sleep(2)
        return "", chat_history

    msg.submit(respond, [msg, chatbot], [msg, chatbot])
    clear.click(lambda: None, None, chatbot, queue=False)

if __name__ == " __main__":
    demo.launch()

```

and I wrote this function

```auto
import os
os.environ["OPENAI_API_KEY"] = ' dsdsds'

from pathlib import Path
from llama_index import LLMPredictor, GPTVectorStoreIndex, PromptHelper, ServiceContext
from llama_index import StorageContext, load_index_from_storage
from langchain.chat_models import ChatOpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

llm_predictor = LLMPredictor(
    llm=ChatOpenAI(
        streaming=True,
        callbacks=[StreamingStdOutCallbackHandler()],
        temperature=0.7,
        model_name="gpt-3.5-turbo",
        max_tokens=1000
    )
)

max_input_size = 4096
max_chunk_overlap = 0.2
chunk_size_limit = 60
num_outputs = 1000
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)

service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
storage_context = StorageContext.from_defaults(persist_dir="./storage")

index = load_index_from_storage(storage_context, service_context=service_context)

question = 'how many times has my son watched the curious george movie?'
query_engine = index.as_query_engine()
response = query_engine.query(question)
#print(response)

```

How can I combine them together to make a chatbot with Streaming?

I tried the following code:

```auto
import os
os.environ["OPENAI_API_KEY"] = 'dsdsdsddsdsd'

from pathlib import Path
from llama_index import LLMPredictor, GPTVectorStoreIndex, PromptHelper, ServiceContext
from llama_index import StorageContext, load_index_from_storage
from langchain.chat_models import ChatOpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
import gradio as gr
import random
import time

llm_predictor = LLMPredictor(
    llm=ChatOpenAI(
        streaming=True,
        callbacks=[StreamingStdOutCallbackHandler()],
        temperature=0.7,
        model_name="gpt-3.5-turbo",
        max_tokens=2000
    ))
    

max_input_size = 4096
max_chunk_overlap = 0.2
chunk_size_limit = 600
num_outputs = 2000
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor)
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context, service_context=service_context)
query_engine = index.as_query_engine()

with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    msg = gr.Textbox()
    clear = gr.Button("Clear")

    def user(user_message, history):
        return gr.update(value="", interactive=False), history + [[user_message, None]]

    def bot(history):
        bot_message = query_engine.query(history[len(history)-1][0])
        history[-1][1] = ""
        for character in bot_message:
            history[-1][1] += character
            time.sleep(0.05)
            yield history

    response = msg.submit(user, [msg, chatbot], [msg, chatbot], queue=False).then(
        bot, chatbot, chatbot
    )
    response.then(lambda: gr.update(interactive=True), None, [msg], queue=False)
    clear.click(lambda: None, None, chatbot, queue=False)

demo.queue()
demo.launch()

```

but it says  
TypeError: ‘Response’ object is not iterable

Any kind or hint is greatly appreciated

---

<div class="post-metadata">

**Author:** ![kamiyan89](https://avatars.discourse-cdn.com/v4/letter/k/d78d45/32.png) [@kamiyan89](https://discuss.huggingface.co/u/kamiyan89)\
**Post date:** [June 21, 2023, 12:48am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863/2 "2023-06-21T00:48:03Z")

</div>

Please allow me to bump this topic.

---

<div class="post-metadata">

**Author:** ![kamiyan89](https://avatars.discourse-cdn.com/v4/letter/k/d78d45/32.png) [@kamiyan89](https://discuss.huggingface.co/u/kamiyan89)\
**Post date:** [June 24, 2023, 2:31am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863/3 "2023-06-24T02:31:36Z")

</div>

kind bump.  
any hint even if small is greatly appreciated 🙂

---

<div class="post-metadata">

**Author:** ![radames](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/radames/32/28246_2.png) [@radames](https://discuss.huggingface.co/u/radames)\
**Post date:** [June 25, 2023, 1:00am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863/4 "2023-06-25T01:00:24Z")

</div>

hi @kamiyan89 ,

can you please take a look at this example? It’s a chatbot streaming with Gradio and OpenAI api

> **[app.py · ysharma/ChatGPT4 at main](https://huggingface.co/spaces/ysharma/ChatGPT4/blob/main/app.py)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

More updated Guide using the news ChatBot Component

> **[Creating A Chatbot Fast](https://www.gradio.app/guides/creating-a-chatbot-fast#streaming-chatbots)**
>
> A Step-by-Step Gradio Tutorial

---

<div class="post-metadata">

**Author:** ![kamiyan89](https://avatars.discourse-cdn.com/v4/letter/k/d78d45/32.png) [@kamiyan89](https://discuss.huggingface.co/u/kamiyan89)\
**Post date:** [June 26, 2023, 7:03am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863/5 "2023-06-26T07:03:27Z")

</div>

Hi Radames,

Thank you very much for the suggestion.  
I am a total beginner so there are a lot of things I don’t understand from the code but I will do my best to understand it.  
If questions arise could you allow me to follow up with additional questions?

In the meanwhile if there are other suggestions they are greatly appreciated.

---

<div class="post-metadata">

**Author:** ![sqy-harshil](https://avatars.discourse-cdn.com/v4/letter/s/a9adbd/32.png) [@sqy-harshil](https://discuss.huggingface.co/u/sqy-harshil)\
**Post date:** [August 12, 2023, 8:11am UTC](https://discuss.huggingface.co/t/how-to-implement-chatbot-streaming-in-gradio-with-a-function/43863/6 "2023-08-12T08:11:28Z")

</div>

@kamiyan89 Hi, I too am a beginner and I noticed that I’m facing a similar issue where I am using AzureChatOpenAI with streaming set to True. More specifically, I am unable to stream the chunks on gradio as they’re being generated by the LLM. In case you’ve resolved this issue kindly share the details, it would be really helpful!