# Why does using \`TextIteratorStreamer\` result in so many empty outputs?

**URL:** <https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167>\
**Category:** 🤗Transformers\
**Created:** [November 10, 2025, 5:15am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167 "2025-11-10T05:15:05Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 10, 2025, 10:01am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/4 "2025-11-10T10:01:30Z")

</div>

The simplest case is that you forgot to call `apply_chat_template`?

* * *

You’re printing many blanks because most early “tokens” decode to nothing after your filters. You’re feeding raw chat markup and tags. The streamer removes the prompt and special tokens during decode, so several iterations yield `""`. Your loop still prints the markers for each `""`, so you see `###$$$` many times.

# Why this happens

1. Chat markup and special tokens get stripped  
Your `text` contains ChatML-style control tokens and tags:

```auto
<|im_start|>user
hello
<|im_end|>
<|im_start|>assistant
<think>

</think>

```

With `skip_special_tokens=True`, the tokenizer drops control tokens on decode. `skip_prompt=True` makes the streamer ignore the prompt part. Early decode steps often become empty strings. You still print them. That renders as repeated `###$$$`. This is expected when feeding chat models raw markup while also skipping specials. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

1. The streamer buffers until it has “displayable words”  
`TextIteratorStreamer` accumulates token pieces and only emits text when decoding forms complete, displayable spans. This can delay or suppress output for subword fragments. Combined with your skip filters, several iterations produce `""`. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

2. Subword and whitespace tokens don’t always produce visible characters  
BPE/SentencePiece commonly produce leading spaces or fragments. Until a boundary is closed, decode can be empty or only whitespace. Your print makes empties visible as `###$$$`. The streamer’s documented behavior is to emit when words materialize, not at every raw token. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

3. Multiple EOS and chat tags at the boundary  
Modern chat models often use more than one stop token (for example, `<|end_of_text|>` and `<|im_end|>` or `<|eot_id|>`). If you don’t stop at all relevant EOS tokens, the model may output extra headers or newlines that get stripped to `""`. Transformers supports a **list** for `eos_token_id`. Use it so generation ends cleanly at the first relevant stop. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))

4. You hand-wrote the conversation instead of using the chat template  
Most chat models expect a template. Hand-rolled markup can make the model emit scaffolding tokens first, which your decode then strips. `apply_chat_template(..., add_generation_prompt=True)` produces the exact format expected by that model’s tokenizer and cleanly marks where assistant output should begin. That reduces spurious blanks. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

# What to change

## A. Use chat templates instead of manual `<|im_start|>` strings

```python
# Docs:
# - https://huggingface.co/docs/transformers/en/chat_templating
# - https://huggingface.co/docs/transformers/en/main_classes/text_generation

messages = [{"role": "user", "content": "hello"}]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True, # assistant starts here
    return_tensors="pt"
).to("cuda")

from transformers import TextIteratorStreamer
streamer = TextIteratorStreamer(
    tokenizer,
    skip_prompt=True,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

# Provide multiple EOS to end cleanly on any of them.
stop_tokens = ["<|im_end|>", "<|eot_id|>", "<|end_of_text|>"]
eos_list = []
for tok in stop_tokens:
    tok_id = tokenizer.convert_tokens_to_ids(tok)
    if tok_id is not None and tok_id != tokenizer.pad_token_id:
        eos_list.append(tok_id)

gen_kwargs = dict(
    input_ids=input_ids,
    max_new_tokens=256,
    do_sample=True, temperature=0.7, top_p=0.8, top_k=20,
    eos_token_id=eos_list or tokenizer.eos_token_id,
    streamer=streamer,
)

from threading import Thread
Thread(target=model.generate, kwargs=gen_kwargs).start()

for chunk in streamer:
    if not chunk: # ignore empty deltas
        continue
    print("###" + chunk + "$$$", end="") # no extra newline

```

Rationale: the template matches training-time formatting and sets the assistant start boundary. That avoids leading control tokens and reduces empty emissions. EOS as a list handles multi-stop models. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

## B. Don’t print empty or whitespace-only chunks

Minimal and effective:

```python
for chunk in streamer:
    if not chunk or chunk.isspace():
        continue
    print("###" + chunk + "$$$", end="")

```

This keeps your markers readable and removes `###$$$` lines created by `""` and pure `"\n"` deltas. Behavior aligns with the streamer’s “emit words” design. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

## C. Debug once with specials enabled

To confirm the root cause in your environment, run a single test with:

```python
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)
for chunk in streamer:
    print(repr(chunk), end="") # show \n and specials explicitly

```

You’ll see leading control tokens and newlines that were previously erased by `skip_special_tokens=True`. This verifies why `""` showed up. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

## D. If your model emits tags like `<think>...</think>`

Add stop strings or stop IDs for those tags. Many users combine `eos_token_id=[...]` with substring stoppers to prevent internal tags reaching the UI:

```python
# Example: custom stopper for substrings
from transformers import StoppingCriteria, StoppingCriteriaList
class StopOnStrings(StoppingCriteria):
    def __init__ (self, stop_strings): self.stop_strings = stop_strings
    def __call__ (self, input_ids, scores, **kwargs):
        text = tokenizer.decode(input_ids[0], skip_special_tokens=True)
        return any(s in text for s in self.stop_strings)

stops = StoppingCriteriaList([StopOnStrings(["<think>", "</think>"])])
model.generate(..., stopping_criteria=stops, streamer=streamer)

```

Reason: some chat variants print scaffolding before content. Stopping criteria ensure your loop never receives them. (Patterns vary by model; templates help most.) ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

# Mental model

- Streamer receives token IDs.
- It decodes incrementally and emits only when there is displayable text.
- You asked it to skip prompt and specials, which turn many early iterations into `""`.
- Your print shows each `""` as an empty payload between markers.
- Fix by providing proper chat formatting, giving all stop IDs, and skipping `""` in your UI loop. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

# Quick checklist

- Use `apply_chat_template(..., add_generation_prompt=True)`. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))
- Keep `skip_special_tokens=True`, but ignore `""` or whitespace chunks. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))
- Provide **all** relevant EOS IDs. Many chat models need more than one. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))
- Optional: `clean_up_tokenization_spaces=False` for exact spacing during streaming. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))
- If needed, add substring stoppers for tags like `<think>`. Use minimally.

# Curated references and similar cases

**Official docs**

- Transformers streaming utilities. Describes word-boundary emission and streamer parameters like `skip_prompt` and `skip_special_tokens`. Useful to understand why empties happen. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))
- Generation API. Confirms `eos_token_id` accepts a list for multiple stops. Key for chat models. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))
- Chat templates guide. Shows `apply_chat_template(..., add_generation_prompt=True)` and why templates prevent formatting mismatches. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))
- HF blog on chat templates. Explains training-time formats and why hand-rolled prompts degrade behavior. ([Hugging Face](https://huggingface.co/blog/chat-templates "Chat Templates: An End to the Silent Performance Killer"))
- Tokenizer docs on `skip_special_tokens`. Confirms that specials are omitted from decode. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

**Issues and threads with analogous symptoms**

- Beginners thread: delays and chunking behavior when streaming. Reinforces word-level emission and the need to run `generate` in a background thread with a streamer. ([Hugging Face Forums](https://discuss.huggingface.co/t/cant-stream-response-token-by-token/106520 "Can't stream response token by token - Beginners"))
- Discussion on multi-EOS not stopping cleanly for some chat models. Motivation for passing all stop IDs. ([GitHub](https://github.com/huggingface/transformers/issues/31362 "LLama3 won't stop when provide eos\_token\_id #31362"))
- FastAPI discussion showing iterator streaming patterns and why to skip empty chunks in the loop. Useful for server implementations. ([GitHub](https://github.com/fastapi/fastapi/discussions/9407 "Making TextIteratorStreamer in HuggingFace work with ..."))

**Model-specific chat template notes**

- Qwen chat docs show `apply_chat_template` usage and streaming patterns consistent with the above. Good cross-check if you test Qwen-style templates. ([Qwen](https://qwen.readthedocs.io/en/v1.5/inference/chat.html "Using Transformers to Chat - Qwen docs"))

---

_[View the full topic](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167)._