Strata-Sage-A1.5-GGUF

InternScience/Sage-A1.5 (a Qwen3.6-35B-A3B fine-tune, 256 experts / top-8, ~3B active, MTP layer included) as a Q4_K_M GGUF for Strata, running natively on the optimized Qwen3.6 engine path in the q36-r2 branch. It fits the full 262K context and the whole model on a 16 GB card + 64 GB RAM.

file size what
Sage-A1.5-Q4_K_M-ssmbf16.gguf 21.7 GB the model (sha256 582d8e11127585b17e2674f5a814c58974beb3865927ac5a08b2766c8902de07)
chat_template.jinja 9 KB the corrected embedded chat template, also available separately for template overrides
expert-profile-agent-v2.prof 84 KB which experts to keep in VRAM from the start, learned from Sage's own agent coding sessions

The 60 small DeltaNet gate tensors (ssm_alpha / ssm_beta, 2.2 MB) carry Sage-A1.5's original BF16 weights, the form Strata's engine reads. Every other tensor is byte-identical to the official Sage-A1.5-Q4_K_M-GGUF. The only metadata change is the chat-template correction described below. It also runs in llama.cpp as any qwen35moe GGUF.

Chat-template update (2026-11-11)

Fixed discussion #1: the embedded tokenizer.chat_template now preserves historical assistant thinking blocks when preserve_thinking is omitted, as well as when it is explicitly true. Explicit false still strips thinking from earlier turns. This keeps the rendered history consistent with previous generation for prompt-prefix caching when the client sends the previous reasoning back. Clients that discard reasoning cannot recover those missing tokens through this change alone.

This deliberately changes the default inherited from the current upstream Sage template. No model weights, tensor descriptors, tensor offsets, or other metadata changed in this update. The Jinja condition was replaced with an equal-length equivalent (padding spaces inside the tag), leaving every byte outside that condition intact. The new model SHA256 is shown in the file table above; previous revisions remain available in the repo history.

Existing downloads: download the updated GGUF and regenerate any extracted tokenizer with the command below; otherwise an already-extracted tokenizer can keep using the old template. Alternatively, use the supplied chat_template.jinja with a backend that supports a custom template, or pass preserve_thinking: true through your backend's chat-template kwargs while using the old file. The standalone template is byte-identical to the one embedded in the updated GGUF.

What the branch adds

q36-r2 is upstream Strata 0.2.3 plus two rounds of work on the Qwen3.6 engine path. The weights did not change between rounds; only the engine did. The later chat-template update above changes metadata only.

Round 1 (q36-optimized, now proposed upstream with round 2 as #3699 and #3903-#3910): opt-in flags, set in the config below.

flag what it does
(always) loads Q4_K_M files whose down-projection experts are Q4_K
--kv int8 q8_0 K / V cache, 3.5x smaller than fp32: the full 262K context fits next to the experts
--gqa-attn grouped-query decode attention: each cache row read once per KV head, not once per query head
--expert-profile F start with the profile's experts in VRAM; --decode-swaps 64 keeps adapting while decoding
--hybrid-zero-copy, --expert-prefetch 2 the CPU pool reads expert rows in place; the next layer's likely experts are seen ahead
--mtp 2 --spec-min-p 0.5 --mtp-sampled q MTP drafts for sampled (temperature > 0) requests too, only when confident; the output distribution stays exact
--snapshots 2, "splice_turns": "parsed" an agent's next turn continues from the cached state instead of re-reading the whole conversation

Round 2 (this update): on by default wherever the card supports it, so the config needs no new flags. Each one has a --no-<flag> escape. With all of them off, the engine is bit-identical to round 1. Measured on the RTX 5080 below.

what effect needs
decode attention on tensor cores (--attn-mma) an MTP verify window over 262K cells: 1.55 ms -> 0.43 ms per layer; MTP keeps paying at depth sm_80+, int8 cache
fused prompt attention from the int8 cache (--fused-prompt-attn), chunked DeltaNet prompt pass (--chunked-gdn) faster prompt reads, most at depth; frees 0.9 GB for experts sm_80+, int8 cache
prompt-side expert stream: split copies on their own stream, one quantization per token, wide dequantization prompt read +9-26% (most on short prompts) always (bit-identical)
decode chain: register reductions, a side stream for the gate projections decode +3-5% always (bit-identical)
more experts in VRAM: prompt staging lent to decode (--lend-staging), snapshots in host memory (--snapshot-host), headroom sized from measurement 78 -> 97 cached experts per layer, plus the lent staging while decoding hybrid mode
CPU pool warms the next experts' rows in its caches (--pool-warm), slot swaps off the main stream (--async-swaps) decode +4-8% hybrid mode
sampled MTP rows sent to the host as GPU shortlists (--sample-gpu) sampled MTP +9% sm_70+

Opt-in only: --draft-vocab 16384 --draft-ctx 4096 (MTP drafts over the common and in-context tokens: +7-9% MTP on English and code, slower on other scripts) and --random-seed (a fresh seed for unseeded sampled requests).

Running it

git clone -b q36-r2 https://github.com/xyehya/Strata && cd Strata
./setup.sh          # builds the engine (Linux; Windows: START-HERE.bat)
hf download ykarout/Strata-Sage-A1.5-GGUF --local-dir models/sage
python tools/strata_tokenizer.py --gguf models/sage/Sage-A1.5-Q4_K_M-ssmbf16.gguf --out models/sage/tokenizer

strata-sage.json (the importer only takes Flash-Next files, so this config is written by hand; it is the same as round 1's, because round 2 is on by default):

{
 "exe": "engine/strata",
 "args": ["--native", "models/sage/Sage-A1.5-Q4_K_M-ssmbf16.gguf", "--max-context", "262144", "--prefill", "auto",
          "--kv", "int8", "--pool-workers", "16", "--gqa-attn", "--hybrid-zero-copy", "--expert-prefetch", "2",
          "--expert-profile", "models/sage/expert-profile-agent-v2.prof", "--decode-swaps", "64", "--snapshots", "2",
          "--mtp", "2", "--spec-min-p", "0.5", "--mtp-sampled", "q"],
 "tokenizer": "models/sage/tokenizer",
 "model_name": "sage-a1.5",
 "arch": "qwen35moe",
 "splice_turns": "parsed",
 "sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5}
}
.venv/bin/python serve/server.py --engine strata --config strata-sage.json --port 8090

OpenAI- and Anthropic-compatible on http://127.0.0.1:8090/v1. "sampling" is InternScience's recommendation (temp 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5) as the server's default for requests that send none: many agent clients (opencode, for one) send no temperature, which otherwise means greedy decoding. A request's own values always win. --pool-workers is the CPU threads for the experts not in VRAM (16 on a 24-core CPU). The profile was learned with 16 GB of VRAM; with less, the engine says so and starts from its default order. The engine prints which round-2 defaults are on and, for any that are off, why (for example "attn-mma off: sm_75").

Measured

RTX 5080 16 GB, Core Ultra 9 285K, 64 GB DDR5. Through the server: fresh code prompts, 384 tokens generated, temperature 0.7, thinking off, MTP 2 as configured, average of two runs per side, run back to back (round 1, round 2, round 1, round 2), tok/s. llama.cpp b11834 tuned for this card (round 1's measurement): --cpu-moe --moe-cache-mib 10300, -b 4096 -ub 2048, MTP draft 2, 8 threads on the P-cores.

decode 4K 16K 40K 90K 160K 240K
Strata q36-r2 (262K context) 231 210 213 187 179 162
Strata q36-optimized (round 1) 173 154 151 124 105 86
llama.cpp (131K context) 166 171 146
prompt read 4K 16K 40K 90K 160K 240K
Strata q36-r2 6,425 6,903 6,115 4,561 3,351 2,477
Strata q36-optimized 4,685 5,111 4,350 2,706 1,511 806
llama.cpp 3,185 3,830 3,730

The gain grows with context: at 240K, round 2 decodes faster than round 1 did at 40K (162 vs 151 tok/s) and reads prompts 3.1x faster. An agent's next turn on a cached conversation (server, prefix reuse, MTP off): +2,000 tokens on a reused 198K context in 0.96 s (round 1: 5.1 s), +300 tokens in 0.32 s (0.95 s); a cold 198K read takes 67 s (180 s). MTP acceptance is the same as round 1's at every depth (65-73%).

Engine only, no server, forced text, MTP off (the clean kernel number): decode 193 -> 249 (+29%), 170 -> 219 (+29%), 183 -> 229 (+25%) at 4K / 16K / 40K; prompt read +38% / +35% / +42%.

Real agent workload (opencode --auto, one coding task of 52 requests up to 182K context, decode per request grouped by the context it ran at, tok/s). Measured on round 1 only; round 2 has not had an agent run yet:

decode, agent run (round 1) < 16K 40-90K 90-160K 160K+ peak
Strata q36-optimized 156 137 116 101 206
MTP acceptance 73% 72% 73% 71%

Close to round 1's sweep row above at each depth (Sage's MTP acceptance is about the same on agent work as on prose), so round 2's sweep gains should carry over. The base model, Qwen3.6-35B-A3B (Strata-Qwen3.6-35B-A3B), accepts 88-95% of its drafts on the same task and decodes 273 / 254 / 234 / 206 tok/s on round 2 in this layout.

Quality. Round 2 changes numerics, so it was checked against a reference rather than against round 1 alone. With the fp32 cache (no KV quantization) as the reference, every round-2 path is at least as close as round 1: mean KL 6.6-8.5e-3 against round 1's 1.55e-2 (full-vocabulary logits, 195 teacher-forced steps at 14K / 57K / 76K), and top-1 agreement 184-189 of 195 against 185. At 200K (65 forced steps), round 2 against round 1 (mean KL 6.8e-3, top-1 63/65) moves the output no more than changing round 1's own prompt chunk size does (6.9e-3, 63/65). Needle retrieval (10 planted codes at 5-95% depth, ~100K and ~200K, with and without MTP): 40/40, as round 1.

License

Apache-2.0, as the original model. All credit for the model to InternScience / Shanghai AI Laboratory (arXiv 2610.28578); this repository only re-packages their weights.

Downloads last month
5,620
GGUF
Model size
36B params
Architecture
qwen35moe
Hardware compatibility
Log In to add your hardware

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ykarout/Strata-Sage-A1.5-GGUF

Quantized
(46)
this model

Paper for ykarout/Strata-Sage-A1.5-GGUF