docs: add bilingual model cards and streamline inference guidance

README.md

@@ -22,405 +22,87 @@ tags:
 - sglang
 ---
 
-<p align="center">
-  <img src="assets/logo.png" width="300" alt="MOSS-VL"/>
-</p>
-
 # MOSS-VL-Realtime-SGLANG
 
-This repository packages **MOSS-VL-Realtime for Transformers 5.12.1 and the MOSS-VL SGLang-Omni realtime backend**. It is a compatibility release of the original streaming checkpoint, not a newly trained or quantized model.
-
-| Resource | Purpose |
-| --- | --- |
-| [OpenMOSS-Team/MOSS-VL-Realtime-SGLANG](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime-SGLANG) | This Transformers 5.12.1-compatible checkpoint and custom code |
-| [OpenMOSS-Team/MOSS-VL-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime) | Original checkpoint and Transformers 4.57-series reference code |
-| [fnlp-vision/sglang-omni-realtime](https://github.com/fnlp-vision/sglang-omni-realtime) | Specialized realtime serving backend, developed on [SGLang-Omni](https://github.com/sgl-project/sglang-omni) |
-| [fnlp-vision/MOSS-VL-Realtime_Demo](https://github.com/fnlp-vision/MOSS-VL-Realtime_Demo) | Separate Demo and gateway integration |
-
-The custom Python files and configuration in this repository must be used together. Do not replace them with the original repository's 4.57-series files or assume that installing the official `sglang-omni` PyPI package includes the specialized backend.
-
-## Compatibility and RoPE Fixes
-
-The validated stack uses Transformers **5.12.1**, SGLang **0.5.16**, and PyTorch **2.11.0** on NVIDIA CUDA. The package keeps the 5.12.1 adaptations for configuration/RoPE APIs, output recording, chat-template return values, and attention-mask/cache interfaces.
-
-Two targeted fixes are included:
-
-- **Cross-attention Query RoPE:** every newly computed text query is rotated, including steps that reuse cached visual KV without new visual input. Visual keys are rotated only when newly computed; cached keys are not rotated again. The equivalent narrow fix is also published in the [original reference repository](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime/commit/1e6a45b292eeaf02aa733bd3aa7b6c85214ddc86).
-- **Vision rotary frequencies under Transformers 5.12.1:** non-persistent frequency buffers may be rematerialized during model loading. This implementation reconstructs the canonical FP32 frequencies on the active device rather than trusting potentially invalid buffer contents. This is a 5.12.1 compatibility fix, not a claim that the original 4.57 environment has the same loading problem.
+English | [简体中文](./README_zh.md)
 
-The five BF16 weight shards, tokenizer and vocabulary are unchanged from the original checkpoint. The compatibility changes are in code and configuration, not model training. Pin the model revision and backend commit together when reproducing results. CUDA validation does not imply NPU support, nor should the latest 4.57-series source changes be assumed to be present in this separate compatibility branch.
+<p align="center"><img src="assets/logo.png" width="300" alt="MOSS-VL"/></p>
 
-MOSS-VL is an open vision-language model family from OpenMOSS, supporting image understanding, long-video understanding, and realtime streaming interaction.
+MOSS-VL-Realtime-SGLANG provides the MOSS-VL realtime checkpoint with Transformers 5.12.1-compatible model and processor code for the specialized SGLang-Omni backend. This is a compatibility package, not a retrained or quantized model.
 
-**Technical Report**: [https://arxiv.org/pdf/2608.15045](https://arxiv.org/pdf/2608.15045)
+## Features
 
-## Overview
+- Continuous frame-by-frame video understanding and incremental text generation.
+- Questions at any point in a stream, with response interruption.
+- Proactive silence when no response is needed or evidence is insufficient.
+- Updated responses as new visual information arrives.
+- Timestamp-aware reasoning about event order and duration.
 
-MOSS-VL-Realtime is the realtime streaming checkpoint of the MOSS-VL release, part of the OpenMOSS ecosystem for open visual understanding.
+## Quick Start
 
-Unlike offline video-language models that first read a complete video and then answer, MOSS-VL-Realtime is designed for continuous video streams. It perceives incoming frames and generates text in parallel, supports questions at arbitrary moments in the stream, and can decide whether to respond or keep observing when the visual evidence is insufficient.
-
-This release keeps the MOSS-VL cross-attention design and a 256K text context window while adding realtime streaming data and an inference interface for timestamped frame-by-frame input.
-
-## Key Features
-
-- Realtime streaming understanding: processes incoming frames continuously instead of waiting for a complete video.
-- Interruptible interaction: users can ask questions at any timestamp in a running stream, and the model answers based on the frames observed so far.
-- Proactive silence: the model can emit `<|silence|>` and continue observing when there is no meaningful visual update or the context is not sufficient.
-- Dynamic correction: as new frames arrive, the model can revise earlier responses instead of being locked to an initial interpretation.
-- Timestamp-aware frames: each streamed frame is associated with an absolute timestamp, helping the model reason about event order, duration, pacing, and fine-grained temporal localization.
-- Unified MOSS-VL family: released together with MOSS-VL-Instruct and MOSS-VL-Base for offline use, continued pretraining, fine-tuning, and applied research.
-
-## Model Design
-
-### Architecture
-
-MOSS-VL-Realtime adopts a cross-attention-based vision-language architecture that decouples visual encoding from language reasoning. This design is important for realtime usage because incoming visual content can be integrated into the running generation context without forcing the model into a strictly offline "load all frames, then answer" workflow.
+| Goal | Entry |
+| --- | --- |
+| Browser video/voice interaction and optional memory | [MOSS-VL-Realtime Demo](https://github.com/fnlp-vision/MOSS-VL-Realtime_Demo#quick-start) |
+| Standalone streaming inference service | [sglang-omni-realtime](https://github.com/fnlp-vision/sglang-omni-realtime#installation) |
+| Direct Transformers inference | [Python examples](./inference.md) |
 
-<p align="center">
-  <img src="assets/architecture.png" alt="MOSS-VL Architecture" width="100%"/>
-</p>
+This repository contains weights, configuration, tokenizer, processor, and custom code. It does not include a running service. The model is public; download the complete repository rather than individual weight files.
 
-### Timestamp-aware Video Encoding
+ASR, TTS, and memory are provided by the Demo. The model's offline Python APIs do not mean that the Demo's realtime deployment enables offline chat.
 
-For video and realtime frame inputs, MOSS-VL injects absolute timestamps alongside sampled frames. This helps the model reason about when an event happens, how long it lasts, and how the scene changes over time instead of relying only on frame order.
+## Architecture and Configuration
 
-MOSS-VL also uses Cross-attention Rotary Position Embedding (XRoPE), which maps text tokens and visual patches into a unified three-dimensional coordinate space defined by Time (t), Height (h), and Width (w). This gives the model a consistent positional representation for image, offline video, and realtime streaming video reasoning.
+MOSS-VL separates visual encoding from language reasoning with cross-attention. Timestamped frames and Cross-attention Rotary Position Embedding (XRoPE) align text and visual patches across time, height, and width.
 
-### Configuration
+<p align="center"><img src="assets/architecture.png" alt="MOSS-VL Architecture" width="100%"/></p>
 
 | Item | Value |
 | --- | --- |
 | Parameters | 11B |
-| Tensor type | BF16 |
-| Context length | 256K |
-| Vision patch size | 16 |
-| Temporal patch size | 1 |
-| Default video FPS | 1.0 |
-| Default max video frames | 256 |
-| Realtime frame format | PIL-compatible image plus timestamp |
-| Direct Transformers session scope | One active realtime session per model instance |
-| SGLang-Omni session scope | Configurable with `--max-running-requests`; subject to KV capacity |
-
-## Performance
-
-MOSS-VL-Realtime is designed for streaming video understanding benchmarks where questions can arrive before a full video has been observed and correct answers may change as the scene evolves. It targets realtime interaction quality, proactive silence, and dynamic response updates in addition to standard video understanding accuracy.
-
-<p align="center">
-  <img src="assets/benchmark-streaming.png" alt="MOSS-VL Streaming Benchmark" width="100%"/>
-</p>
-
-Detailed benchmark tables and comparisons for this release will be maintained in the MOSS-VL project resources.
-
-## Quickstart
-
-### Installation
-
-For the specialized backend, install its source and pinned dependencies in a fresh, compatible CUDA environment. Install `uv` first if it is not already available:
-
-```bash
-git clone https://github.com/fnlp-vision/sglang-omni-realtime.git
-cd sglang-omni-realtime
-uv venv .venv -p 3.12
-source .venv/bin/activate
-uv pip install -e .
-```
-
-See the [backend README](https://github.com/fnlp-vision/sglang-omni-realtime#readme) for CUDA/toolchain prerequisites and deployment details. This is not a universal installation recipe for arbitrary CUDA versions. For direct Transformers inference, use the same compatible Transformers 5.12.1 environment and the example below; an SGLang server does not need to be running.
-
-### Load the Model
-
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoProcessor
-
-checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime-SGLANG"
-
-processor = AutoProcessor.from_pretrained(
-    checkpoint,
-    trust_remote_code=True,
-    frame_extract_num_threads=1,
-)
-model = AutoModelForCausalLM.from_pretrained(
-    checkpoint,
-    trust_remote_code=True,
-    device_map="auto",
-    dtype=torch.bfloat16,
-    attn_implementation="eager",
-)
-model.eval()
-```
-
-This direct-Transformers example uses eager attention. The SGLang-Omni backend has its own attention configuration; do not confuse the two execution paths.
-
-### Start the SGLang-Omni Backend
+| Weights | BF16 |
+| Model context | 256K |
+| Vision patch size / temporal patch size | 16 / 1 |
+| Default video FPS / maximum sampled frames | 1.0 / 256 |
+| Direct Transformers runtime | One active realtime session per model instance |
+| SGLang-Omni runtime | Configurable session capacity, subject to GPU memory |
 
-Download this checkpoint to a directory outside the backend source tree, then launch from the backend repository root:
+The model context is not a per-session memory allocation guarantee. The recommended service configuration uses 131072 context; adjust context and concurrency for the available GPU memory.
 
-```bash
-hf download OpenMOSS-Team/MOSS-VL-Realtime-SGLANG \
-  --local-dir /path/to/moss-vl-realtime-sglang
-
-python examples/run_moss_vl_realtime_server.py \
-  --model-path /path/to/moss-vl-realtime-sglang \
-  --gpu 0 --host 127.0.0.1 --port 8000 \
-  --context-length 131072 \
-  --mem-fraction-static 0.60 \
-  --max-running-requests 1
-```
-
-Access requires authorization while the repository is private. The command uses a 128K context as an explicit deployment example; the model's 256K capacity and launcher defaults do not guarantee sufficient KV memory on every device. Tune context, memory fraction and concurrency for your hardware. The backend endpoint is `/v1/video/realtime`; its WebSocket protocol and TP options are documented in the [Realtime Cookbook](https://github.com/fnlp-vision/sglang-omni-realtime/blob/main/docs/cookbook/moss_vl_realtime.md).
-
-This model repository contains weights and custom model/processor code, not the backend scheduler, Demo frontend, customer authentication or billing service.
-
-## Inference Examples
-
-### Online Inference
-
-<details>
-<summary><b>Session-style Online Inference</b></summary>
-
-The recommended direct API is `create_realtime_session(...)`. A service or application owns the video capture pipeline, converts camera, screen, or video-file input into PIL-compatible frames, and pushes each frame with a non-decreasing timestamp.
-
-Common session operations:
-
-- `session.push_frame(image, timestamp=...)` appends one visual frame.
-- `session.push_prompt("...")` appends a user question while the stream is running.
-- `session.push_prompt_frame(prompt, image, timestamp=...)` aligns a prompt with a specific frame.
-- `session.poll_output(...)` or `session.stream_outputs(...)` returns incremental text chunks.
-
-`system_prompt` and `initial_prompt` are tokenized as the initial system/user turns before the first frame arrives. Subsequent user turns can be appended with `push_prompt(...)` while the same session continues observing frames.
-
-For complete real-time inference usage, including local-video replay and service deployment, see [`realtime_inference`](https://github.com/OpenMOSS/MOSS-VL/tree/main/realtime_inference) in the MOSS-VL GitHub repository.
-
-```python
-import time
-from PIL import Image
-
-session = model.create_realtime_session(
-    processor,
-    initial_prompt=(
-        "As the video streams frame by frame, describe important changes as they happen. "
-        "Stay silent when there is no relevant update."
-    ),
-    frame_queue_size=256,
-    max_tokens_per_turn=12,
-    max_new_tokens=4096,
-    do_sample=False,
-)
-
-frame_paths = [
-    "data/frame_0001.jpg",
-    "data/frame_0002.jpg",
-    "data/frame_0003.jpg",
-]
-
-try:
-    session.start()
-
-    for index, frame_path in enumerate(frame_paths):
-        image = Image.open(frame_path).convert("RGB")
-        session.push_frame(image, timestamp=index / 1.0)
-
-        while True:
-            chunk = session.poll_output(timeout=0.0)
-            if chunk is None:
-                break
-            print(chunk, end="", flush=True)
-
-        time.sleep(1.0)
-
-    session.push_prompt("What changed in the latest frames?")
-
-    # Realtime sessions stay alive waiting for future input, so use a bounded
-    # drain window and close the session explicitly when the producer is done.
-    drain_deadline = time.monotonic() + 5.0
-    while time.monotonic() < drain_deadline:
-        chunk = session.poll_output(timeout=0.1)
-        if chunk is not None:
-            print(chunk, end="", flush=True)
-finally:
-    session.close()
-```
+## Performance
 
-Frame timestamps are measured in seconds and must be non-decreasing within a session. The input producer can be a camera, screen capture, decoded video file, browser frame sampler, or any other source that yields images with timestamps.
-
-</details>
-
-<details>
-<summary><b>Queue-style Online Inference</b></summary>
-
-`online_generate(...)` is useful for backend systems that separate frame production and model inference through queues. It accepts dictionaries containing frames, prompts, events, reset controls, and stop controls.
-
-```python
-import queue
-import threading
-from PIL import Image
-
-input_queue = queue.Queue()
-output_queue = queue.Queue()
-
-worker = threading.Thread(
-    target=model.online_generate,
-    args=(processor, input_queue, output_queue),
-    kwargs={
-        "frame_queue_size": 256,
-        "max_tokens_per_turn": 12,
-        "max_new_tokens": 4096,
-        "do_sample": False,
-    },
-    daemon=True,
-)
-worker.start()
-
-input_queue.put({
-    "initial_prompt": "Answer only when the streamed video provides enough evidence.",
-})
-
-input_queue.put({"frame": Image.open("data/frame_0001.jpg").convert("RGB"), "timestamp": 0.0})
-input_queue.put({"frame": Image.open("data/frame_0002.jpg").convert("RGB"), "timestamp": 1.0})
-input_queue.put({"prompt": "What is happening now?"})
-
-try:
-    while True:
-        chunk = output_queue.get(timeout=0.5)
-        print(chunk, end="", flush=True)
-except queue.Empty:
-    pass
-
-input_queue.put({"stop_online_generate": True})
-worker.join()
-```
+MOSS-VL-Realtime targets streaming understanding, proactive silence, and dynamic response updates. Benchmark results are summarized below; see the [technical report](https://arxiv.org/abs/2608.15045) for model research.
 
-Each queue item can contain `frame` or `image`, `timestamp`, `prompt`, `frames`, `event`, `events`, `initial_prompt`, `system_prompt`, `generate_kwargs`, `reset_session`, or stop controls such as `stop_online_generate`.
-
-</details>
-
-### Offline Inference
-
-MOSS-VL-Realtime also keeps the offline helper APIs for image and video prompts. For purely offline use, MOSS-VL-Instruct is usually the preferred checkpoint, but the realtime checkpoint can still process complete image and video inputs.
-
-<details>
-<summary><b>Single-video Offline Inference</b></summary>
-
-```python
-video_path = "data/example_video.mp4"
-prompt = "Describe this video."
-
-text = model.offline_video_generate(
-    processor,
-    prompt=prompt,
-    video=video_path,
-    shortest_edge=4096,
-    longest_edge=16777216,
-    video_max_pixels=201326592,
-    patch_size=16,
-    temporal_patch_size=1,
-    merge_size=2,
-    video_fps=1.0,
-    min_frames=1,
-    max_frames=256,
-    num_extract_threads=4,
-    image_mean=[0.5, 0.5, 0.5],
-    image_std=[0.5, 0.5, 0.5],
-    max_new_tokens=256,
-    temperature=1.0,
-    top_k=50,
-    top_p=1.0,
-    repetition_penalty=1.0,
-    do_sample=False,
-    vision_chunked_length=64,
-)
-
-print(text)
-```
+<p align="center"><img src="assets/benchmark-streaming.png" alt="MOSS-VL Streaming Benchmark" width="100%"/></p>
 
-</details>
-
-<details>
-<summary><b>Batched Offline Inference</b></summary>
-
-`offline_batch_generate` accepts independent image/video/text queries. Queries in the same batch should share the same `media_kwargs` and `generate_kwargs`.
-
-```python
-queries = [
-    {
-        "prompt": "Describe sample A.",
-        "images": [],
-        "videos": ["data/sample_a.mp4"],
-        "media_kwargs": {
-            "video_fps": 1.0,
-            "min_frames": 8,
-            "max_frames": 256,
-        },
-        "generate_kwargs": {
-            "temperature": 1.0,
-            "top_k": 50,
-            "top_p": 1.0,
-            "max_new_tokens": 256,
-            "repetition_penalty": 1.0,
-            "do_sample": False,
-        },
-    },
-    {
-        "prompt": "Describe sample B.",
-        "images": [],
-        "videos": ["data/sample_b.mp4"],
-        "media_kwargs": {
-            "video_fps": 1.0,
-            "min_frames": 8,
-            "max_frames": 256,
-        },
-        "generate_kwargs": {
-            "temperature": 1.0,
-            "top_k": 50,
-            "top_p": 1.0,
-            "max_new_tokens": 256,
-            "repetition_penalty": 1.0,
-            "do_sample": False,
-        },
-    },
-]
-
-with torch.no_grad():
-    result = model.offline_batch_generate(
-        processor,
-        queries,
-        vision_chunked_length=64,
-    )
-
-texts = [item["text"] for item in result["results"]]
-print(texts)
-```
-
-</details>
+## Compatibility
 
-## Related Checkpoints
+Use this package's custom code and configuration together with the [specialized backend](https://github.com/fnlp-vision/sglang-omni-realtime). Installing a generic `sglang-omni` package does not supply the MOSS-VL integration.
 
-| Model | Parameters | Context | Usage | Hugging Face |
-| --- | ---: | ---: | --- | --- |
-| MOSS-VL-Realtime-SGLANG | 11B | 256K | Transformers 5.12.1 / specialized SGLang-Omni compatibility package | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime-SGLANG |
-| MOSS-VL-Realtime | 11B | 256K | Realtime streaming video interaction | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime |
-| MOSS-VL-Instruct | 11B | 256K | Offline multimodal instruction following | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct |
-| MOSS-VL-Base | 11B | 256K | Continued pretraining and fine-tuning | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base |
-| MOSS-VL-Instruct-0408 | 11B | 256K | Previous instruction-tuned checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct-0408 |
-| MOSS-VL-Base-0408 | 11B | 256K | Previous base checkpoint | https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base-0408 |
+The backend uses Transformers 5.12.1, SGLang 0.5.16, and PyTorch 2.11.0. This package adapts configuration, RoPE, and generation interfaces to Transformers 5.12.1 while keeping the original five BF16 weight shards, tokenizer, and vocabulary unchanged.
 
-## Validation of This Compatibility Package
+- **Cross-attention Query RoPE:** rotate each newly computed text query, including when visual KV is reused; cached visual keys are not rotated again. The corresponding fix is also available in the [original reference implementation](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime/commit/1e6a45b292eeaf02aa733bd3aa7b6c85214ddc86).
+- **Vision rotary frequencies:** reconstruct canonical FP32 frequencies on the active device to handle buffer rematerialization in Transformers 5.12.1. This does not imply the same loading issue exists in the original 4.57 environment.
 
-On 2026-09-06, the Query RoPE and vision-frequency regression suite passed **17 tests**, including CUDA dtype/autocast and non-persistent-buffer rematerialization cases. An isolated H200 smoke test with Transformers 5.12.1 / PyTorch 2.11.0 loaded the full checkpoint, completed an image forward pass and generated a finite natural-language answer with eager attention.
+The original [MOSS-VL-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime) uses separate Transformers 4.57-series code. Do not mix its custom files into this package. Select matching model and backend versions; CUDA compatibility does not imply NPU support.
 
-These checks cover the targeted fixes and basic model execution. They do not certify every live-stream schedule, full NPU compatibility, or a production latency/concurrency SLA. The current Transformers runtime still emits deprecation warnings for processor aliases and `cache_position`; future Transformers releases require separate adaptation rather than an unqualified dependency upgrade.
+## Limitations
 
-## Limitations and Roadmap
+- Response latency depends on hardware, frame rate, transport, and decoding settings.
+- The direct Python runtime and the SGLang-Omni service have different session and backpressure behavior.
+- Frame dropping and visual KV windows limit the visible history. Continuing beyond the context limit requires application-level memory handling.
+- Applications should handle control tokens such as `<|silence|>`, `<|round_start|>`, and `<|round_end|>`.
 
-MOSS-VL-Realtime is optimized for timestamped frame-by-frame streaming, but production latency depends on GPU hardware, frame sampling rate, transport overhead, and decoding speed. The direct Transformers API supports one active realtime session per model instance, and its bounded frame queue may drop older pending frames when overloaded. The specialized SGLang-Omni backend supports configured multi-session serving and uses protocol backpressure; these are different runtime contracts.
+## Related Models
 
-The 1 FPS workflow is the semantic validation baseline. Real-time input arrival, sampling, session stop rules and optional vision-KV eviction can change outputs even when fixed-boundary comparisons agree. Neither the RoPE fixes nor a successful smoke test imply bitwise equality for every live HF/SGLang session or a production SLA. Optional frame pooling is not required for this checkpoint and remains disabled by default in the specialized backend.
+| Model | Purpose |
+| --- | --- |
+| [MOSS-VL-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime) | Original realtime checkpoint and reference implementation |
+| [MOSS-VL-Instruct](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct) | Offline multimodal instruction following |
+| [MOSS-VL-Base](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base) | Continued pretraining and fine-tuning |
 
-The model may emit realtime control tokens such as `<|silence|>`, `<|round_start|>`, and `<|round_end|>` depending on the application protocol. Downstream services should filter or render these tokens according to their UI needs.
+## License
 
-We are continuing to improve realtime response timing, dynamic correction, broader streaming evaluations, RL post-training, and task-specific deployment recipes for future MOSS-VL releases.
+Apache-2.0. See [OpenMOSS/MOSS-VL](https://github.com/OpenMOSS/MOSS-VL) for the model family.
 
 ## Citation
 

README_zh.md

@@ -0,0 +1,105 @@
+# MOSS-VL-Realtime-SGLANG
+
+[English](./README.md) | 简体中文
+
+<p align="center"><img src="assets/logo.png" width="300" alt="MOSS-VL"/></p>
+
+MOSS-VL-Realtime-SGLANG 提供 MOSS-VL 实时模型权重，以及面向特化 SGLang-Omni 后端的 Transformers 5.12.1 兼容模型与 processor 代码。这是兼容版本，不是重新训练或量化的模型。
+
+## 功能
+
+- 连续逐帧视频理解与增量文本生成。
+- 在视频流任意时刻提问，并打断当前回答。
+- 无需回答或证据不足时主动保持静默。
+- 随新增视觉信息更新回答。
+- 基于时间戳理解事件顺序和持续时间。
+
+## 快速开始
+
+| 目标 | 入口 |
+| --- | --- |
+| 浏览器视频/语音交互与可选 memory | [MOSS-VL-Realtime Demo](https://github.com/fnlp-vision/MOSS-VL-Realtime_Demo/blob/main/README_zh.md#快速开始) |
+| 独立流式推理服务 | [sglang-omni-realtime](https://github.com/fnlp-vision/sglang-omni-realtime/blob/main/README_zh.md#安装) |
+| 直接使用 Transformers 推理 | [Python 示例](./inference.md) |
+
+本仓库包含权重、配置、tokenizer、processor 和自定义代码，不包含正在运行的服务。模型公开可访问，请下载完整仓库，而不是只下载个别权重文件。
+
+ASR、TTS 和 memory 由 Demo 提供。模型保留离线 Python API，不代表 Demo 的实时部署已经启用离线聊天。
+
+## 架构与配置
+
+MOSS-VL 使用交叉注意力分离视觉编码与语言推理。带时间戳的帧及 Cross-attention Rotary Position Embedding（XRoPE）在时间、高度和宽度维度上对齐文本与视觉 patch。
+
+<p align="center"><img src="assets/architecture.png" alt="MOSS-VL 架构" width="100%"/></p>
+
+| 项目 | 值 |
+| --- | --- |
+| 参数量 | 11B |
+| 权重类型 | BF16 |
+| 模型上下文 | 256K |
+| 视觉 patch 大小 / 时间 patch 大小 | 16 / 1 |
+| 默认视频 FPS / 最大采样帧数 | 1.0 / 256 |
+| 直接 Transformers 推理 | 每个模型实例一个活跃实时会话 |
+| SGLang-Omni 推理 | 会话容量可配置，受 GPU 显存限制 |
+
+模型上下文不等于部署时每路保证分配的容量。推荐服务配置使用 131072 context，请按可用显存调整上下文和并发数。
+
+## 性能
+
+MOSS-VL-Realtime 面向流式理解、主动静默及动态回答更新。下图汇总相关基准结果，模型研究详见[技术报告](https://arxiv.org/abs/2608.15045)。
+
+<p align="center"><img src="assets/benchmark-streaming.png" alt="MOSS-VL 流式基准" width="100%"/></p>
+
+## 兼容性
+
+本仓库的自定义代码和配置应配套使用，并连接[特化后端](https://github.com/fnlp-vision/sglang-omni-realtime)。安装通用 `sglang-omni` 包不等于获得 MOSS-VL 的接入实现。
+
+后端使用 Transformers 5.12.1、SGLang 0.5.16 和 PyTorch 2.11.0。本版本适配 Transformers 5.12.1 的配置、RoPE 和生成接口，原有五个 BF16 权重分片、tokenizer 和词表保持不变。
+
+- **交叉注意力 Query RoPE：** 每次新计算的文本 query 都应用旋转，包括复用视觉 KV 的情况；已缓存的视觉 key 不重复旋转。对应修复也已应用于[原版参考实现](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime/commit/1e6a45b292eeaf02aa733bd3aa7b6c85214ddc86)。
+- **视觉旋转频率：** 在当前设备重建标准 FP32 频率，以处理 Transformers 5.12.1 加载时重新创建 buffer 的情况，不代表原版 4.57 环境存在同样的加载问题。
+
+原版 [MOSS-VL-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime) 使用独立的 Transformers 4.57 系列代码，不要将其自定义文件混入此版本。模型与后端应配套选择版本；CUDA 兼容不等于支持 NPU.
+
+## 限制
+
+- 响应延迟受硬件、帧率、传输和解码配置影响。
+- 直接 Python 推理与 SGLang-Omni 服务具有不同的会话及背压行为。
+- 丢帧和视觉 KV 窗口会限制可访问的视觉历史；跨上下文延续需要应用层记忆管理。
+- 应用需处理 `<|silence|>`、`<|round_start|>`、`<|round_end|>` 等控制 token。
+
+## 相关模型
+
+| 模型 | 用途 |
+| --- | --- |
+| [MOSS-VL-Realtime](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime) | 原版实时模型与参考实现 |
+| [MOSS-VL-Instruct](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Instruct) | 离线多模态指令理解 |
+| [MOSS-VL-Base](https://huggingface.co/OpenMOSS-Team/MOSS-VL-Base) | 继续预训练与微调 |
+
+## 许可证
+
+Apache-2.0。模型系列见 [OpenMOSS/MOSS-VL](https://github.com/OpenMOSS/MOSS-VL)。
+
+## 引用
+
+```bibtex
+@misc{mossvl,
+  title         = {MOSS-VL Technical Report},
+  author        = {Wang, Pengyu and Tan, Chenkun and Zhou, Shaojun and Zhou, Qirui and Chen, Yanxin and He, Xingyang and Zeng, Huazheng and Cheng, Jijun and Wang, Chenghao and Qian, Xiaomeng and Wang, Pengfei and Huang, Zhan and Gao, Shanqing and Huang, Wei and Cao, Longjun and Ran, Wu and Liu, Jie and Zhu, Changtai and Wang, Hongkai and Tian, Yixian and Liu, Chenghao and Ye, Zhen and Wang, Xinghao and Jiang, Botian and Feng, Guoguo and Fei, Zhaoye and Li, Ruixiao and Chen, Mingshu and Gao, Yang and Cheng, Qinyuan and Li, Shimin and Qiu, Xipeng},
+  year          = {2026},
+  eprint        = {2608.15045},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CV},
+  url           = {https://arxiv.org/abs/2608.15045}
+}
+
+@misc{mossvideopreview,
+  title         = {{MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention}},
+  author        = {Pengyu Wang and Chenkun Tan and Shaojun Zhou and Wei Huang and Qirui Zhou and Zhan Huang and Zhen Ye and Jijun Cheng and Xiaomeng Qian and Yanxin Chen and Xingyang He and Huazheng Zeng and Chenghao Wang and Pengfei Wang and Hongkai Wang and Shanqing Gao and Yixian Tian and Chenghao Liu and Xinghao Wang and Botian Jiang and Xipeng Qiu},
+  year          = {2026},
+  eprint        = {2606.07639},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.CV},
+  url           = {https://arxiv.org/abs/2606.07639}
+}
+```

inference.md

@@ -0,0 +1,256 @@
+# Direct Transformers Inference
+
+These examples use this repository's Transformers 5.12.1-compatible implementation, not the browser Demo or its WebSocket protocol. Prepare the [compatible backend environment](https://github.com/fnlp-vision/sglang-omni-realtime/blob/main/docs/get_started/installation.md); a running SGLang server is not required for direct Python inference. Do not use the Demo CPU environment to load this checkpoint.
+
+Pin the model revision selected by the [compatibility manifest](https://github.com/fnlp-vision/MOSS-VL-Realtime_Demo/blob/main/deployment/repro/manifest.json). Sample paths below refer to media supplied by the caller.
+
+## Load the Model
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+checkpoint = "OpenMOSS-Team/MOSS-VL-Realtime-SGLANG"
+revision = "bcfd9ccf1e9db2896ad852301cc8dde4a6349c78"
+
+processor = AutoProcessor.from_pretrained(
+    checkpoint,
+    revision=revision,
+    trust_remote_code=True,
+    frame_extract_num_threads=1,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    checkpoint,
+    revision=revision,
+    trust_remote_code=True,
+    device_map="auto",
+    dtype=torch.bfloat16,
+    attn_implementation="eager",
+)
+model.eval()
+```
+
+This direct-Transformers example uses eager attention. The SGLang-Omni backend has its own attention configuration; do not confuse the two execution paths.
+
+
+## Inference Examples
+
+### Online Inference
+
+<details>
+<summary><b>Session-style Online Inference</b></summary>
+
+The recommended direct API is `create_realtime_session(...)`. A service or application owns the video capture pipeline, converts camera, screen, or video-file input into PIL-compatible frames, and pushes each frame with a non-decreasing timestamp.
+
+Common session operations:
+
+- `session.push_frame(image, timestamp=...)` appends one visual frame.
+- `session.push_prompt("...")` appends a user question while the stream is running.
+- `session.push_prompt_frame(prompt, image, timestamp=...)` aligns a prompt with a specific frame.
+- `session.poll_output(...)` or `session.stream_outputs(...)` returns incremental text chunks.
+
+`system_prompt` and `initial_prompt` are tokenized as the initial system/user turns before the first frame arrives. Subsequent user turns can be appended with `push_prompt(...)` while the same session continues observing frames.
+
+For complete real-time inference usage, including local-video replay and service deployment, see [`realtime_inference`](https://github.com/OpenMOSS/MOSS-VL/tree/main/realtime_inference) in the MOSS-VL GitHub repository.
+
+```python
+import time
+from PIL import Image
+
+session = model.create_realtime_session(
+    processor,
+    initial_prompt=(
+        "As the video streams frame by frame, describe important changes as they happen. "
+        "Stay silent when there is no relevant update."
+    ),
+    frame_queue_size=256,
+    max_tokens_per_turn=12,
+    max_new_tokens=4096,
+    do_sample=False,
+)
+
+frame_paths = [
+    "data/frame_0001.jpg",
+    "data/frame_0002.jpg",
+    "data/frame_0003.jpg",
+]
+
+try:
+    session.start()
+
+    for index, frame_path in enumerate(frame_paths):
+        image = Image.open(frame_path).convert("RGB")
+        session.push_frame(image, timestamp=index / 1.0)
+
+        while True:
+            chunk = session.poll_output(timeout=0.0)
+            if chunk is None:
+                break
+            print(chunk, end="", flush=True)
+
+        time.sleep(1.0)
+
+    session.push_prompt("What changed in the latest frames?")
+
+    # Realtime sessions stay alive waiting for future input, so use a bounded
+    # drain window and close the session explicitly when the producer is done.
+    drain_deadline = time.monotonic() + 5.0
+    while time.monotonic() < drain_deadline:
+        chunk = session.poll_output(timeout=0.1)
+        if chunk is not None:
+            print(chunk, end="", flush=True)
+finally:
+    session.close()
+```
+
+Frame timestamps are measured in seconds and must be non-decreasing within a session. The input producer can be a camera, screen capture, decoded video file, browser frame sampler, or any other source that yields images with timestamps.
+
+</details>
+
+<details>
+<summary><b>Queue-style Online Inference</b></summary>
+
+`online_generate(...)` is useful for backend systems that separate frame production and model inference through queues. It accepts dictionaries containing frames, prompts, events, reset controls, and stop controls.
+
+```python
+import queue
+import threading
+from PIL import Image
+
+input_queue = queue.Queue()
+output_queue = queue.Queue()
+
+worker = threading.Thread(
+    target=model.online_generate,
+    args=(processor, input_queue, output_queue),
+    kwargs={
+        "frame_queue_size": 256,
+        "max_tokens_per_turn": 12,
+        "max_new_tokens": 4096,
+        "do_sample": False,
+    },
+    daemon=True,
+)
+worker.start()
+
+input_queue.put({
+    "initial_prompt": "Answer only when the streamed video provides enough evidence.",
+})
+
+input_queue.put({"frame": Image.open("data/frame_0001.jpg").convert("RGB"), "timestamp": 0.0})
+input_queue.put({"frame": Image.open("data/frame_0002.jpg").convert("RGB"), "timestamp": 1.0})
+input_queue.put({"prompt": "What is happening now?"})
+
+try:
+    while True:
+        chunk = output_queue.get(timeout=0.5)
+        print(chunk, end="", flush=True)
+except queue.Empty:
+    pass
+
+input_queue.put({"stop_online_generate": True})
+worker.join()
+```
+
+Each queue item can contain `frame` or `image`, `timestamp`, `prompt`, `frames`, `event`, `events`, `initial_prompt`, `system_prompt`, `generate_kwargs`, `reset_session`, or stop controls such as `stop_online_generate`.
+
+</details>
+
+### Offline Inference
+
+MOSS-VL-Realtime also keeps the offline helper APIs for image and video prompts. For purely offline use, MOSS-VL-Instruct is usually the preferred checkpoint, but the realtime checkpoint can still process complete image and video inputs.
+
+<details>
+<summary><b>Single-video Offline Inference</b></summary>
+
+```python
+video_path = "data/example_video.mp4"
+prompt = "Describe this video."
+
+text = model.offline_video_generate(
+    processor,
+    prompt=prompt,
+    video=video_path,
+    shortest_edge=4096,
+    longest_edge=16777216,
+    video_max_pixels=201326592,
+    patch_size=16,
+    temporal_patch_size=1,
+    merge_size=2,
+    video_fps=1.0,
+    min_frames=1,
+    max_frames=256,
+    num_extract_threads=4,
+    image_mean=[0.5, 0.5, 0.5],
+    image_std=[0.5, 0.5, 0.5],
+    max_new_tokens=256,
+    temperature=1.0,
+    top_k=50,
+    top_p=1.0,
+    repetition_penalty=1.0,
+    do_sample=False,
+    vision_chunked_length=64,
+)
+
+print(text)
+```
+
+</details>
+
+<details>
+<summary><b>Batched Offline Inference</b></summary>
+
+`offline_batch_generate` accepts independent image/video/text queries. Queries in the same batch should share the same `media_kwargs` and `generate_kwargs`.
+
+```python
+queries = [
+    {
+        "prompt": "Describe sample A.",
+        "images": [],
+        "videos": ["data/sample_a.mp4"],
+        "media_kwargs": {
+            "video_fps": 1.0,
+            "min_frames": 8,
+            "max_frames": 256,
+        },
+        "generate_kwargs": {
+            "temperature": 1.0,
+            "top_k": 50,
+            "top_p": 1.0,
+            "max_new_tokens": 256,
+            "repetition_penalty": 1.0,
+            "do_sample": False,
+        },
+    },
+    {
+        "prompt": "Describe sample B.",
+        "images": [],
+        "videos": ["data/sample_b.mp4"],
+        "media_kwargs": {
+            "video_fps": 1.0,
+            "min_frames": 8,
+            "max_frames": 256,
+        },
+        "generate_kwargs": {
+            "temperature": 1.0,
+            "top_k": 50,
+            "top_p": 1.0,
+            "max_new_tokens": 256,
+            "repetition_penalty": 1.0,
+            "do_sample": False,
+        },
+    },
+]
+
+with torch.no_grad():
+    result = model.offline_batch_generate(
+        processor,
+        queries,
+        vision_chunked_length=64,
+    )
+
+texts = [item["text"] for item in result["results"]]
+print(texts)
+```
+
+</details>