s23deepak/streaming-vlm-qwen3-rocm

StreamingVLM — Qwen3-VL (2B & 4B) for CUDA + ROCm

PitchAI: Your AI Broadcast Companion — Real-time AI-powered football commentary with live tactical vision.

This is an updated port of StreamingVLM, adapted for Qwen3-VL-2B/4B-Instruct. StreamingVLM is designed for real-time, stable understanding of infinite video streams by maintaining a compact KV cache.

• Paper: StreamingVLM: Real-Time Understanding for Infinite Video Streams
• Project Page: https://streamingvlm.hanlab.ai
• Original Code: mit-han-lab/streaming-vlm

This port is specifically optimized for:

• Qwen3-VL-2B-Instruct ⚡ (lightweight, runs on consumer GPUs)
• Qwen3-VL-4B-Instruct 🏋️ (higher quality commentary)
• NVIDIA CUDA with flash_attention_2 (maximum throughput)
• AMD ROCm with sdpa (no flash-attn build needed)
• Latest Transformers (4.57.0+ from source)

Supported Configurations

Model	GPU	Attention	VRAM	Speed	Use Case
Qwen3-VL-2B	RTX 3090/4090	flash_attention_2	~6 GB	~12 FPS	Real-time on consumer HW
Qwen3-VL-2B	MI250X/MI300X	sdpa	~6 GB	~8 FPS	ROCm deployment
Qwen3-VL-4B	A100/H100	flash_attention_2	~12 GB	~8 FPS	Production quality
Qwen3-VL-4B	MI300X	sdpa	~12 GB	~6 FPS	ROCm high quality

Key Changes from Original StreamingVLM

Component	Original	This Port
Model	Qwen2.5-VL-7B-Instruct	Qwen3-VL-2B/4B-Instruct
Attention	flash_attention_2 (CUDA only)	SDPA + flash_attention_2 (both!)
Transformers	4.50.0–4.52.4	4.57.0+ (from source)
Vision Attn	flash_attn_varlen_func	chunked SDPA
ViT Patch Size	14×14	16×16
RoPE	mrope (standard)	mrope_interleaved
DeepStack	❌	✅ (layers 5, 11, 17)
Max Context	128K	256K

Installation

For CUDA (NVIDIA GPUs)

conda create -n streaming_qwen3 python=3.11 -y
conda activate streaming_qwen3

# PyTorch with CUDA
pip install torch torchvision

# Transformers from source (Qwen3-VL requires 4.57.0+)
pip install git+https://github.com/huggingface/transformers

# Flash Attention 2 (recommended for CUDA)
pip install flash-attn --no-build-isolation

# Other deps
pip install -r requirements.txt

For ROCm (AMD GPUs)

conda create -n streaming_qwen3 python=3.11 -y
conda activate streaming_qwen3

# PyTorch with ROCm
pip install torch torchvision --index-url https://download.pytorch.org/whl/rocm6.2

# Transformers from source
pip install git+https://github.com/huggingface/transformers

# No flash-attn needed! SDPA works out of the box.
pip install -r requirements.txt

Quick Start

🚀 2B Model on CUDA

python -m streaming_vlm.inference.inference \
    --model_path Qwen/Qwen3-VL-2B-Instruct \
    --video_path /path/to/football_match.mp4 \
    --output_path ./commentary.vtt \
    --fps 4 \
    --attn_implementation flash_attention_2 \
    --window_size 24 \
    --text_sliding_window 768

💬 Interactive Q&A Mode

python -m streaming_vlm.inference.inference \
    --model_path Qwen/Qwen3-VL-2B-Instruct \
    --video_path /path/to/match.mp4 \
    --interactive \
    --attn_implementation flash_attention_2

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "s23deepak/streaming-vlm-qwen3-rocm"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

Architecture Overview

StreamingVLM maintains a compact KV cache for infinite video streams:

• Attention Sinks (512 tokens): Stabilize attention over long sequences.
• Visual Window (16–24s): Recent video frames kept in KV cache.
• Text Window (512–768 tokens): Recent text context maintained.
• Contiguous RoPE: Keeps position indices bounded for infinite streams.

Citation

@misc{xu2025streamingvlmrealtimeunderstandinginfinite,
      title={StreamingVLM: Real-Time Understanding for Infinite Video Streams}, 
      author={Ruyi Xu and Guangxuan Xiao and Yukang Chen and Liuning He and Kelly Peng and Yao Lu and Song Han},
      year={2025},
      eprint={2510.09608},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2510.09608}, 
}

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://huggingface.co/proxy/smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern