neroued/Qwen3.8-27B-NInfer

Qwen3.8-27B for NInfer

This model card is the version-controlled source for neroued/Qwen3.8-27B-NInfer.

The repository contains Qwen3.8-27B converted to the native NInfer .ninfer artifact format. The artifact is intended only for NInfer; it is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Artifact

Field	Value
Filename	qwen3_8_27b.ninfer
Size	20,437,336,576 bytes (19.03 GiB)
SHA-256	0634abb07024221de141456cf04a42ab74b18bc38e1b781c6eb2e062a467eec3
Container version	2
NInfer model ID	qwen3.8-27b
NInfer weights ID	groupwise-int
NInfer target key	qwen3_8_27b
Stored objects	1,190 (1,184 tensors and 6 resources)

The Text body uses the registered Q4/Q5/Q6 groupwise allocation, while the token embedding and full output head use W8G32_F16S. The file also contains the registered Vision, MTP, DFlash2, proposal-head, tokenizer, chat-template, generation, and media-processor objects required by NInfer.

Verify a downloaded file with:

printf '%s  %s\n' \
  '0634abb07024221de141456cf04a42ab74b18bc38e1b781c6eb2e062a467eec3' \
  'qwen3_8_27b.ninfer' | sha256sum --check

This release includes the complete DFlash2 companion weights from z-lab/Qwen3.8-27B-DFlash2 at revision 50307d4c4cde6860d4eee73e2547cd786fe8e8a4. Select --spec dflash2 --draft-tokens 7 --lm-head-draft; draft counts 1..15 are supported. DFlash2 requires the runtime revision listed below. Existing performance and evaluation tables retain their stated MTP configurations and revisions.

Requirements

• NInfer revision 5232055 or later, built from source;
• 64-bit Linux;
• NVIDIA GeForce RTX 5090 (sm_120a);
• CUDA Toolkit 13.1 or newer.

NInfer does not provide an install target or packaged binary. See the repository README for source-build dependencies.

Download and run a CLI example

hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

./build/apps/ninfer models/qwen3_8_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

For images, videos, and structured chat history, see the CLI guide.

Start a local server

./build/apps/ninfer-serve models/qwen3_8_27b.ninfer \
  --host 127.0.0.1 \
  --port 8080 \
  --max-context 240000 \
  --kv-capacity 240000 \
  --max-concurrency 2 \
  --kv-dtype fp8 \
  --device-state-slots 2 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft \
  --preserve-thinking

Each request has a 240,000-token logical ceiling. The shared 240,000-token Device KV pool admits two active requests when their combined completion reservations fit; either request may use the full pool while running alone. Two extra Device checkpoint slots, eight pinned Host State slots, and 8 GiB of pinned Host KV retain reusable continuations under resource pressure.

See the HTTP serving guide for the API surface and the resource scheduling reference for cache and admission semantics.

Supported use

The artifact supports:

• text generation in thinking and non-thinking modes;
• image, multi-image, video, and mixed multimodal messages;
• MTP speculative decoding with draft windows from one to five;
• DFlash2 with draft windows from one to fifteen using the included DFlash2 companion weights (--spec dflash2 --draft-tokens 7, optionally --lm-head-draft);
• BF16, INT8, FP8, NVFP4, and K8V4 KV cache;
• CUDA Graph decode and compatible-prefix reuse;
• startup-bounded small-scale concurrent serving with true batched decode;
• the NInfer CLI;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages serving.

Evaluation

The artifact was evaluated through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and INT8 group-64 KV. EvalScope 1.9.0 used 0-shot prompts, rule-based scoring, and one sample per problem with temperature 1.0, top-p 0.95, top-k 20, presence penalty 0.0, and seed 42. The text suite ran at a 262,144-token context limit; the multimodal suite ran with --vision at a 81,920-token limit.

Benchmark	NInfer groupwise-int	Correct / total	Official Qwen3.8-27B BF16
IFBench (prompt-level strict)	77.67%	233 / 300	79.5
AIME 2025	96.67%	29 / 30	—
AIME 2026	96.67%	29 / 30	—
GPQA-Diamond	87.37%	173 / 198	89.2
ERQA	66.25%	265 / 400	65.5
RealWorldQA	82.22%	629 / 765	85.9

All 1,723 configured samples completed and were scored. IFBench additionally reports 81.00% instruction-level strict, 80.67% prompt-level loose, and 83.67% instruction-level loose. These are single-sample results, not pass@k.

The official Qwen3.8-27B BF16 figures come from the upstream model card; its sampling settings and IFBench metric level are not stated there, so the last column is not a same-protocol comparison. The groupwise-int deltas stay within ±3.1 points on the four overlapping benchmarks, and the upstream card reports no AIME results.

Limits

• The artifact is accepted only by NInfer revision 5232055 or later and the matching registered target.
• NInfer executes on one RTX 5090 and one CUDA device, with a startup-fixed capacity of 1–8 active requests per Engine.
• It does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU/GPU offload, or distributed serving.
• Context allocation is subject to GPU memory and the selected KV-cache type.
• NInfer does not execute generated tool calls.

Provenance

Field	Value
Source repository	Qwen/Qwen3.8-27B
Source revision	1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Download source	modelscope.cn/models/Qwen/Qwen3.8-27B
Conversion recipe	qwen3_8_27b-v2
Converter repository	https://github.com/Neroued/ninfer
Converter revision	863aa8a5f1e866db74f29f8999b83b4021398dee
Minimum runtime revision	385b30ce1757bafe5a82680e9b5aeb940b14eec1
Ranking input SHA-256	c692dc76388132c910547589b4fb4a0503fbd6ad50aaac6a509bbcb192a8afa5

The local source configuration, tensor index, frontend resources, and published CRC32 inventory were checked against the canonical Hugging Face source revision above before conversion publication.

The artifact identity, summarized object inventory, and conversion provenance are published in artifact-manifest.json. The exact storage contract is maintained in the Qwen3.8-27B artifact reference.

License

This NInfer artifact is distributed under the Apache License 2.0. The source Qwen3.8-27B repository is also licensed under Apache-2.0. Users remain responsible for complying with the license and applicable laws.