# Why does using \`TextIteratorStreamer\` result in so many empty outputs?

**URL:** <https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167>\
**Category:** 🤗Transformers\
**Created:** [November 10, 2025, 5:15am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167 "2025-11-10T05:15:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![septemberlemon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/septemberlemon/32/55301_2.png) [@septemberlemon](https://discuss.huggingface.co/u/septemberlemon)\
**Post date:** [November 10, 2025, 5:15am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/1 "2025-11-10T05:15:05Z")

</div>

```python
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
parameters = dict(
    **tokenizer(text, return_tensors="pt").to("cuda"),
    max_new_tokens=256, # Increase for longer outputs!
    temperature=0.7, top_p=0.8, top_k=20, # For non thinking
    streamer=streamer
)
background_thread = Thread(target=model.generate, kwargs=parameters)
background_thread.start()

for new_token in streamer:
    print("###" + new_token + "$$$")

```

I use the code above, and got the result:

```plaintext
###$$$
###$$$
###Hello! $$$
###How $$$
###can $$$
###I $$$
###assist $$$
###you $$$
###$$$
###today? $$$
###$$$
###$$$
###😊$$$

```

the text is `'<|im_start|>user\nhello<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [November 10, 2025, 5:15am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/2 "2025-11-10T05:15:06Z")

</div>

Unlisted on November 10, 2025, 5:15am UTC



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [November 10, 2025, 9:20am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/3 "2025-11-10T09:20:10Z")

</div>

Listed on November 10, 2025, 9:20am UTC



---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 10, 2025, 10:01am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/4 "2025-11-10T10:01:30Z")

</div>

The simplest case is that you forgot to call `apply_chat_template`?

* * *

You’re printing many blanks because most early “tokens” decode to nothing after your filters. You’re feeding raw chat markup and tags. The streamer removes the prompt and special tokens during decode, so several iterations yield `""`. Your loop still prints the markers for each `""`, so you see `###$$$` many times.

# Why this happens

1. Chat markup and special tokens get stripped  
Your `text` contains ChatML-style control tokens and tags:

```auto
<|im_start|>user
hello
<|im_end|>
<|im_start|>assistant
<think>

</think>

```

With `skip_special_tokens=True`, the tokenizer drops control tokens on decode. `skip_prompt=True` makes the streamer ignore the prompt part. Early decode steps often become empty strings. You still print them. That renders as repeated `###$$$`. This is expected when feeding chat models raw markup while also skipping specials. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

1. The streamer buffers until it has “displayable words”  
`TextIteratorStreamer` accumulates token pieces and only emits text when decoding forms complete, displayable spans. This can delay or suppress output for subword fragments. Combined with your skip filters, several iterations produce `""`. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

2. Subword and whitespace tokens don’t always produce visible characters  
BPE/SentencePiece commonly produce leading spaces or fragments. Until a boundary is closed, decode can be empty or only whitespace. Your print makes empties visible as `###$$$`. The streamer’s documented behavior is to emit when words materialize, not at every raw token. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

3. Multiple EOS and chat tags at the boundary  
Modern chat models often use more than one stop token (for example, `<|end_of_text|>` and `<|im_end|>` or `<|eot_id|>`). If you don’t stop at all relevant EOS tokens, the model may output extra headers or newlines that get stripped to `""`. Transformers supports a **list** for `eos_token_id`. Use it so generation ends cleanly at the first relevant stop. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))

4. You hand-wrote the conversation instead of using the chat template  
Most chat models expect a template. Hand-rolled markup can make the model emit scaffolding tokens first, which your decode then strips. `apply_chat_template(..., add_generation_prompt=True)` produces the exact format expected by that model’s tokenizer and cleanly marks where assistant output should begin. That reduces spurious blanks. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

# What to change

## A. Use chat templates instead of manual `<|im_start|>` strings

```python
# Docs:
# - https://huggingface.co/docs/transformers/en/chat_templating
# - https://huggingface.co/docs/transformers/en/main_classes/text_generation

messages = [{"role": "user", "content": "hello"}]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True, # assistant starts here
    return_tensors="pt"
).to("cuda")

from transformers import TextIteratorStreamer
streamer = TextIteratorStreamer(
    tokenizer,
    skip_prompt=True,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False
)

# Provide multiple EOS to end cleanly on any of them.
stop_tokens = ["<|im_end|>", "<|eot_id|>", "<|end_of_text|>"]
eos_list = []
for tok in stop_tokens:
    tok_id = tokenizer.convert_tokens_to_ids(tok)
    if tok_id is not None and tok_id != tokenizer.pad_token_id:
        eos_list.append(tok_id)

gen_kwargs = dict(
    input_ids=input_ids,
    max_new_tokens=256,
    do_sample=True, temperature=0.7, top_p=0.8, top_k=20,
    eos_token_id=eos_list or tokenizer.eos_token_id,
    streamer=streamer,
)

from threading import Thread
Thread(target=model.generate, kwargs=gen_kwargs).start()

for chunk in streamer:
    if not chunk: # ignore empty deltas
        continue
    print("###" + chunk + "$$$", end="") # no extra newline

```

Rationale: the template matches training-time formatting and sets the assistant start boundary. That avoids leading control tokens and reduces empty emissions. EOS as a list handles multi-stop models. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

## B. Don’t print empty or whitespace-only chunks

Minimal and effective:

```python
for chunk in streamer:
    if not chunk or chunk.isspace():
        continue
    print("###" + chunk + "$$$", end="")

```

This keeps your markers readable and removes `###$$$` lines created by `""` and pure `"\n"` deltas. Behavior aligns with the streamer’s “emit words” design. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

## C. Debug once with specials enabled

To confirm the root cause in your environment, run a single test with:

```python
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)
for chunk in streamer:
    print(repr(chunk), end="") # show \n and specials explicitly

```

You’ll see leading control tokens and newlines that were previously erased by `skip_special_tokens=True`. This verifies why `""` showed up. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

## D. If your model emits tags like `<think>...</think>`

Add stop strings or stop IDs for those tags. Many users combine `eos_token_id=[...]` with substring stoppers to prevent internal tags reaching the UI:

```python
# Example: custom stopper for substrings
from transformers import StoppingCriteria, StoppingCriteriaList
class StopOnStrings(StoppingCriteria):
    def __init__ (self, stop_strings): self.stop_strings = stop_strings
    def __call__ (self, input_ids, scores, **kwargs):
        text = tokenizer.decode(input_ids[0], skip_special_tokens=True)
        return any(s in text for s in self.stop_strings)

stops = StoppingCriteriaList([StopOnStrings(["<think>", "</think>"])])
model.generate(..., stopping_criteria=stops, streamer=streamer)

```

Reason: some chat variants print scaffolding before content. Stopping criteria ensure your loop never receives them. (Patterns vary by model; templates help most.) ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

# Mental model

- Streamer receives token IDs.
- It decodes incrementally and emits only when there is displayable text.
- You asked it to skip prompt and specials, which turn many early iterations into `""`.
- Your print shows each `""` as an empty payload between markers.
- Fix by providing proper chat formatting, giving all stop IDs, and skipping `""` in your UI loop. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

# Quick checklist

- Use `apply_chat_template(..., add_generation_prompt=True)`. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))
- Keep `skip_special_tokens=True`, but ignore `""` or whitespace chunks. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))
- Provide **all** relevant EOS IDs. Many chat models need more than one. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))
- Optional: `clean_up_tokenization_spaces=False` for exact spacing during streaming. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))
- If needed, add substring stoppers for tags like `<think>`. Use minimally.

# Curated references and similar cases

**Official docs**

- Transformers streaming utilities. Describes word-boundary emission and streamer parameters like `skip_prompt` and `skip_special_tokens`. Useful to understand why empties happen. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))
- Generation API. Confirms `eos_token_id` accepts a list for multiple stops. Key for chat models. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))
- Chat templates guide. Shows `apply_chat_template(..., add_generation_prompt=True)` and why templates prevent formatting mismatches. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))
- HF blog on chat templates. Explains training-time formats and why hand-rolled prompts degrade behavior. ([Hugging Face](https://huggingface.co/blog/chat-templates "Chat Templates: An End to the Silent Performance Killer"))
- Tokenizer docs on `skip_special_tokens`. Confirms that specials are omitted from decode. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

**Issues and threads with analogous symptoms**

- Beginners thread: delays and chunking behavior when streaming. Reinforces word-level emission and the need to run `generate` in a background thread with a streamer. ([Hugging Face Forums](https://discuss.huggingface.co/t/cant-stream-response-token-by-token/106520 "Can't stream response token by token - Beginners"))
- Discussion on multi-EOS not stopping cleanly for some chat models. Motivation for passing all stop IDs. ([GitHub](https://github.com/huggingface/transformers/issues/31362 "LLama3 won't stop when provide eos\_token\_id #31362"))
- FastAPI discussion showing iterator streaming patterns and why to skip empty chunks in the loop. Useful for server implementations. ([GitHub](https://github.com/fastapi/fastapi/discussions/9407 "Making TextIteratorStreamer in HuggingFace work with ..."))

**Model-specific chat template notes**

- Qwen chat docs show `apply_chat_template` usage and streaming patterns consistent with the above. Good cross-check if you test Qwen-style templates. ([Qwen](https://qwen.readthedocs.io/en/v1.5/inference/chat.html "Using Transformers to Chat - Qwen docs"))

---

<div class="post-metadata">

**Author:** ![septemberlemon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/septemberlemon/32/55301_2.png) [@septemberlemon](https://discuss.huggingface.co/u/septemberlemon)\
**Post date:** [November 11, 2025, 3:06am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/5 "2025-11-11T03:06:29Z")

</div>

I used the chat\_template instead of hand-write, I did not say it

 ![微信图片_20251111110536_40_2](https://us1.discourse-cdn.com/hellohellohello/original/3X/1/3/1323c51ef76832c3b56292929a279a1282a3ac9f.png)  
I still got so many empty output even with `skip_prompt=False` and `skip_special_tokens=False`

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 11, 2025, 4:05am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/6 "2025-11-11T04:05:19Z")

</div>

This time, regardless of the specific cause, it’s likely an avoidable issue.

But generally speaking, whether using Transformers’ Streamer or not, **LLM raw outputs often retain leftover template elements and other garbage**. Manual post-processing is frequently necessary. Therefore, I recommend processing with the expectation that you will perform post-processing yourself.

* * *

Root cause: you are printing newline-only and zero-delta chunks. `TextIteratorStreamer` decodes tokens incrementally and only yields user-displayable text when it decides a “word” or span is formed. Early generation from chat models usually includes headers, control tokens, and several `\n`. Even with `skip_prompt=False` and `skip_special_tokens=False`, many yielded chunks are just `"\n"` or spaces, which your `print("###" + new + "$$$")` renders as apparently empty lines. This is expected behavior for the streamer and for chat-templated inputs. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

# What is happening under the hood

- **Streamer buffering policy.** The streamer processes incoming token IDs and emits text “as soon as [entire] words are formed.” That implies internal buffering and non-character granularity. Subword tokens and whitespace often accumulate without adding visible glyphs, so intermediate deltas are `""` or `"\n"`. ([Hugging Face](https://huggingface.co/docs/transformers.js/en/api/generation/streamers "generation/streamers - .TextStreamer"))

- **Chat templates add structure and newlines.** `apply_chat_template(..., add_generation_prompt=True)` builds a prompt with role headers and separators. Many built-in templates put a newline before the assistant turn, and models often start their reply with one or more newlines. Those tokens stream out first and show up as blank prints. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

- **Decode cleanup can hide deltas.** Decoding may “clean up tokenization spaces,” collapsing artifacts and yielding the same visible string after a step. The streamer still iterates, but the delta you print is `""`. Disable cleanup to reduce zero-delta iterations while debugging. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

- **Multiple stop tokens.** Modern chat models use several EOS markers (e.g., `<|eot_id|>`, `<|im_end|>`, `<|end_of_text|>`). If you only stop on one, the model often appends headers or trailing newlines that become extra whitespace-only chunks at the end. Transformers allows a **list** for `eos_token_id`. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))

- **Your print adds more line breaks.** `print("###" + new + "$$$")` writes a newline _after_ the chunk. If `new` already equals `"\n"`, the terminal sees two line breaks. Use `end=""` and filter whitespace-only spans. Community patterns and Q&A about “word-by-word vs sentence-by-sentence” streaming describe the same effect. ([Stack Overflow](https://stackoverflow.com/questions/79726988/fastapi-endpoint-stream-llm-output-word-for-word "FastAPI endpoint stream LLM output word for word"))

# Concrete fixes

1. **Filter whitespace-only and zero-delta chunks.**  
Minimal change that removes the visual noise.

```python
for part in streamer:
    if not part or part.isspace():
        continue
    print(part, end="", flush=True) # no extra newline each step

```

This matches recommended streaming loops that skip empty strings from the iterator. ([Stack Overflow](https://stackoverflow.com/questions/79726988/fastapi-endpoint-stream-llm-output-word-for-word "FastAPI endpoint stream LLM output word for word"))

1. **Disable decode cleanup during streaming.**

```python
streamer = TextIteratorStreamer(
    tokenizer,
    skip_prompt=False,
    skip_special_tokens=False,
    clean_up_tokenization_spaces=False, # key for fewer zero-delta steps
)

```

The decode API documents these kwargs; turning cleanup off makes emitted text match raw decoding and reduces “no-op” emits. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))

1. **Stop cleanly on all relevant EOS tokens.**

```python
stop_tokens = ["<|eot_id|>", "<|im_end|>", "<|end_of_text|>"]
eos_list = [tid for tok in stop_tokens
            if (tid := tokenizer.convert_tokens_to_ids(tok)) is not None]

gen_kwargs = dict(
    **tokenizer(text, return_tensors="pt").to("cuda"),
    max_new_tokens=256,
    do_sample=True, temperature=0.7, top_p=0.8, top_k=20,
    eos_token_id=eos_list or tokenizer.eos_token_id,
    streamer=streamer,
)

```

Passing a list for `eos_token_id` prevents trailing header/newline runs that decode to whitespace-only chunks. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))

1. **Optional: add substring stoppers for tags like `<think>` or double newlines.**  
For models that emit scaffolding, add a `StoppingCriteria` that checks decoded text for `"</think>"` or `"\n\n\n"`, etc. Documentation and examples show how to implement custom stoppers. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))

2. **Inspect the template and the actual stream once.**

- Print `repr(part)` for one run to confirm that most early chunks are `"\n"` or spaces.
- Verify your template’s separators; some add blank lines by design. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

# Why you still saw empties even with `skip_prompt=False` and `skip_special_tokens=False`

- Those flags only control _whether_ the prompt and specials are removed on decode. They don’t alter the model’s tendency to start with `\n` or the streamer’s buffering. You’ll still receive many `"\n"` chunks and partials. The streamer itself is documented to emit text at word boundaries, not at every token, so you will observe iterations that carry no new visible glyphs. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

# Quick diagnostic checklist

- Turn cleanup off: `clean_up_tokenization_spaces=False`. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))
- Log `repr(part)` for a short run to see `\n` explicitly.
- Filter `if not part or part.isspace(): continue` in the loop. ([Stack Overflow](https://stackoverflow.com/questions/79726988/fastapi-endpoint-stream-llm-output-word-for-word "FastAPI endpoint stream LLM output word for word"))
- Provide **all** EOS IDs. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))
- Keep `apply_chat_template(..., add_generation_prompt=True)` so the assistant boundary is set. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))

# Curated references

**Core docs**

- Hugging Face chat templates guide: how `apply_chat_template` structures turns and why `add_generation_prompt` matters. Useful to reason about leading/trailing newlines. ([Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating "Chat templates"))
- Generation API: `eos_token_id` can be a list. Prevents trailing whitespace bursts. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/text_generation "Generation"))
- Tokenizer decode options: `skip_special_tokens` and `clean_up_tokenization_spaces`. ([Hugging Face](https://huggingface.co/docs/transformers/en/main_classes/tokenizer "Tokenizer"))
- Streamers: emit chunks when words are formed; iterator semantics. ([Hugging Face](https://huggingface.co/docs/transformers/main/internal/generation_utils "Utilities for Generation"))

**Threads and examples**

- FastAPI “word-by-word vs sentence” streaming with `TextIteratorStreamer`. Confirms buffering and the need to adjust your loop. ([Stack Overflow](https://stackoverflow.com/questions/79726988/fastapi-endpoint-stream-llm-output-word-for-word "FastAPI endpoint stream LLM output word for word"))

---

<div class="post-metadata">

**Author:** ![septemberlemon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/septemberlemon/32/55301_2.png) [@septemberlemon](https://discuss.huggingface.co/u/septemberlemon)\
**Post date:** [November 11, 2025, 10:40am UTC](https://discuss.huggingface.co/t/why-does-using-textiteratorstreamer-result-in-so-many-empty-outputs/170167/7 "2025-11-11T10:40:22Z")

</div>

I added `clean_up_tokenization_spaces=False` and `eos_token_id=eos_list or tokenizer.eos_token_id`, but the empty output does not change at all for the same test  
emmm…  
if model generate **token\_id** for `Hello`、`!`、`Ġ` one by one (I find them three in vocab.json), the three can be decode into visible character easily, why it put two **empty string** into streamer?