--- library_name: transformers pipeline_tag: image-text-to-text license: apache-2.0 tags: - multimodal - vision-language - image-text-to-text - video-text-to-text - llava - llava-onevision-2 - qwen3 language: - en - zh --- # LLaVA-OneVision-2-8B-Instruct > [!IMPORTANT] > 🌟 **The core work on this project is led by [Glint Lab](https://www.glint-mvt.com).** > For an interactive online experience, visit **[ve2s.ai](https://ve2s.ai/)**. A multimodal vision-language model that handles **single images, multi-image, and video** inputs, built on a Qwen3-8B language backbone with a OneVision-style vision encoder. The model is distributed as a HuggingFace `transformers` checkpoint with custom code (`trust_remote_code=True`). ## Requirements ### Base (image + frame-sampling video) ```bash pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord ``` ### Optional: codec video backend The model ships a second video backend (`video_backend="codec"`) that replaces uniform frame sampling with codec-aware **canvas packing** driven by motion vectors and bit-cost — typically yielding stronger long-video accuracy at the same token budget. To enable it you need two extra pieces: ```bash # 1. The cv-preinfer CLI (PyPI: codec-video-prep) drives canvas extraction. pip install codec-video-prep opencv-python # 2. A working `ffmpeg` binary must be on PATH. # Verify with: ffmpeg -version ``` **ffmpeg version:** ffmpeg **4.4.x – 7.x** is recommended. The codec backend additionally needs **POSIX `flock`** (already present on Linux/macOS) for the on-disk result cache, and roughly **2 GB free disk** under `$ONLINE_CODEC_CACHE_DIR` (defaults to `$HF_HOME/online_codec`) per processed video. ## Quick start The repository ships a ready-to-run `demo_inference.py` that covers both image and video paths. ```bash # Image (default sample image; no auth required) python demo_inference.py # Image, custom file + prompt python demo_inference.py --mode image --media /path/to/cat.jpg \ --prompt "What is the cat doing?" # Video (16 uniformly-sampled frames; max-pixels caps per-frame resolution for memory) python demo_inference.py --mode video --media /path/to/clip.mp4 \ --num-frames 16 --max-pixels 200704 \ --prompt "Describe what happens in this video." ``` ## Programmatic use ```python import torch from transformers import AutoProcessor, AutoModelForImageTextToText from PIL import Image MODEL_ID = "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True) model = AutoModelForImageTextToText.from_pretrained( MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda", ).eval() # ----- Image ----- image = Image.open("cat.jpg").convert("RGB") messages = [{"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Describe this image in detail."}, ]}] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True) inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} out = model.generate(**inputs, max_new_tokens=256, do_sample=False) print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) # ----- Video ----- # Lower max_pixels if you hit OOM on long videos. processor.video_processor.max_pixels = 200704 messages = [{"role": "user", "content": [ {"type": "video"}, {"type": "text", "text": "Describe what happens in this video."}, ]}] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = processor( text=[text], videos=["clip.mp4"], return_tensors="pt", padding=True, num_frames=16, # exact frame count; or use target_fps / max_frames ) inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} out = model.generate(**inputs, max_new_tokens=256, do_sample=False) print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` ### Video — codec backend (recommended for long videos) The codec backend is exposed as a single processor kwarg (`video_backend="codec"`). Everything else — canvas extraction via `cv-preinfer`, on-disk caching, patch-position bookkeeping, chat-template rewriting — happens inside `processor(...)`: ```python # Make sure: `pip install codec-video-prep opencv-python` and ffmpeg on PATH. messages = [{"role": "user", "content": [ {"type": "video"}, {"type": "text", "text": "Describe what happens in this long video."}, ]}] text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = processor( text=[text], videos=["long_clip.mp4"], video_backend="codec", max_pixels=150000, # per-canvas pixel budget; lower if OOM return_tensors="pt", padding=True, # Optional: override codec defaults from preprocessor_config.json # codec_config={"target_canvas": 32, "group_size": 32, "images_per_group": 4}, ) inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} out = model.generate(**inputs, max_new_tokens=256, do_sample=False) print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` Defaults for the codec pipeline live in `preprocessor_config.json` under the `"codec"` key (`target_canvas=32`, `group_size=32`, `images_per_group=4`, `patch=14`, `min_group_frames=8`, `max_group_frames=64`); they can be overridden per call via `codec_config={...}`. **Short-video behaviour:** if the input video has fewer frames than `target_canvas` requires (or fewer than `min_group_frames`), a `UserWarning` is emitted and inference proceeds with however many canvases `cv-preinfer` can actually form. For very short clips, falling back to the frame-sampling backend is usually a better choice. ## Notes - The vision tower is a OneVision-style encoder; the language backbone is **Qwen3-8B**. - `chat_template.jinja` follows the Qwen3 chat format and emits `<|vision_start|>...<|vision_end|>` placeholders; the processor expands them per-frame (frames backend) or per-canvas-patch-run (codec backend). - Two video backends are available via `processor(..., video_backend=...)`: **`"frames"`** (default, uniform sampling) and **`"codec"`** (canvas packing via `cv-preinfer`, requires `codec-video-prep` + `ffmpeg`). - Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends. ## Citation If you find **LLaVA-OneVision-2** useful in your research, please cite the following paper: ```bibtex @misc{an2026llavaonevision2nextgenerationperceptualintelligence, title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence}, author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng}, year = {2026}, eprint = {2605.25979}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2605.25979} } ``` ```bibtex @misc{an2025llavaonevision15fullyopenframework, title = {LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}, author = {Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Didi Zhu and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng}, year = {2025}, eprint = {2509.23661}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2509.23661} } ``` ```bibtex @misc{tang2026onevisionencodercodecalignedsparsityfoundational, title = {OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence}, author = {Feilong Tang and Xiang An and Yunyao Yan and Yin Xie and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Chunyuan Li and Shikun Feng and Changrui Chen and Huajie Tan and Ming Hu and Manyuan Zhang and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng}, year = {2026}, eprint = {2602.08683}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = {https://arxiv.org/abs/2602.08683} } ``` ## License Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B).