Instructions to use ykarout/Strata-Sage-A1.5-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ykarout/Strata-Sage-A1.5-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ykarout/Strata-Sage-A1.5-GGUF:BF16 # Run inference directly in the terminal: llama cli -hf ykarout/Strata-Sage-A1.5-GGUF:BF16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ykarout/Strata-Sage-A1.5-GGUF:BF16 # Run inference directly in the terminal: llama cli -hf ykarout/Strata-Sage-A1.5-GGUF:BF16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ykarout/Strata-Sage-A1.5-GGUF:BF16 # Run inference directly in the terminal: ./build/bin/llama cli -hf ykarout/Strata-Sage-A1.5-GGUF:BF16
Use Docker
docker model run hf.co/ykarout/Strata-Sage-A1.5-GGUF:BF16
- Llama new
- LM Studio
- Jan
- Ollama
How to use ykarout/Strata-Sage-A1.5-GGUF with Ollama:
ollama run hf.co/ykarout/Strata-Sage-A1.5-GGUF:BF16
- Unsloth Desktop
- Pi
How to use ykarout/Strata-Sage-A1.5-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ykarout/Strata-Sage-A1.5-GGUF:BF16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ykarout/Strata-Sage-A1.5-GGUF:BF16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use ykarout/Strata-Sage-A1.5-GGUF with Docker Model Runner:
docker model run hf.co/ykarout/Strata-Sage-A1.5-GGUF:BF16
- Lemonade
How to use ykarout/Strata-Sage-A1.5-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ykarout/Strata-Sage-A1.5-GGUF:BF16
Run and chat with the model
lemonade run user.Strata-Sage-A1.5-GGUF-BF16
List all available models
lemonade list
- Hermes Agent
How to use ykarout/Strata-Sage-A1.5-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ykarout/Strata-Sage-A1.5-GGUF:BF16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default ykarout/Strata-Sage-A1.5-GGUF:BF16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ykarout/Strata-Sage-A1.5-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ykarout/Strata-Sage-A1.5-GGUF:BF16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "ykarout/Strata-Sage-A1.5-GGUF:BF16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Strata-Sage-A1.5-GGUF
InternScience/Sage-A1.5 (a Qwen3.6-35B-A3B fine-tune, 256 experts / top-8,
~3B active, MTP layer included) as a Q4_K_M GGUF for Strata, running natively on
the optimized Qwen3.6 engine path in the q36-r2 branch.
It fits the full 262K context and the whole model on a 16 GB card + 64 GB RAM.
| file | size | what |
|---|---|---|
Sage-A1.5-Q4_K_M-ssmbf16.gguf |
21.7 GB | the model (sha256 582d8e11127585b17e2674f5a814c58974beb3865927ac5a08b2766c8902de07) |
chat_template.jinja |
9 KB | the corrected embedded chat template, also available separately for template overrides |
expert-profile-agent-v2.prof |
84 KB | which experts to keep in VRAM from the start, learned from Sage's own agent coding sessions |
The 60 small DeltaNet gate tensors (ssm_alpha / ssm_beta, 2.2 MB) carry Sage-A1.5's original BF16 weights, the form
Strata's engine reads. Every other tensor is byte-identical to the official
Sage-A1.5-Q4_K_M-GGUF. The only metadata change is the
chat-template correction described below. It also runs in llama.cpp as any qwen35moe GGUF.
Chat-template update (2026-11-11)
Fixed discussion #1: the embedded
tokenizer.chat_template now preserves historical assistant thinking blocks when preserve_thinking is omitted,
as well as when it is explicitly true. Explicit false still strips thinking from earlier turns. This keeps the
rendered history consistent with previous generation for prompt-prefix caching when the client sends the previous
reasoning back. Clients that discard reasoning cannot recover those missing tokens through this change alone.
This deliberately changes the default inherited from the current upstream Sage template. No model weights, tensor descriptors, tensor offsets, or other metadata changed in this update. The Jinja condition was replaced with an equal-length equivalent (padding spaces inside the tag), leaving every byte outside that condition intact. The new model SHA256 is shown in the file table above; previous revisions remain available in the repo history.
Existing downloads: download the updated GGUF and regenerate any extracted tokenizer with the command below;
otherwise an already-extracted tokenizer can keep using the old template. Alternatively, use the supplied
chat_template.jinja with a backend that supports a custom template, or pass
preserve_thinking: true through your backend's chat-template kwargs while using the old file.
The standalone template is byte-identical to the one embedded in the updated GGUF.
What the branch adds
q36-r2 is upstream Strata 0.2.3 plus two rounds of work on the Qwen3.6
engine path. The weights did not change between rounds; only the engine did. The later chat-template update above changes metadata only.
Round 1 (q36-optimized, now proposed upstream with round 2 as #3699 and #3903-#3910):
opt-in flags, set in the config below.
| flag | what it does |
|---|---|
| (always) | loads Q4_K_M files whose down-projection experts are Q4_K |
--kv int8 |
q8_0 K / V cache, 3.5x smaller than fp32: the full 262K context fits next to the experts |
--gqa-attn |
grouped-query decode attention: each cache row read once per KV head, not once per query head |
--expert-profile F |
start with the profile's experts in VRAM; --decode-swaps 64 keeps adapting while decoding |
--hybrid-zero-copy, --expert-prefetch 2 |
the CPU pool reads expert rows in place; the next layer's likely experts are seen ahead |
--mtp 2 --spec-min-p 0.5 --mtp-sampled q |
MTP drafts for sampled (temperature > 0) requests too, only when confident; the output distribution stays exact |
--snapshots 2, "splice_turns": "parsed" |
an agent's next turn continues from the cached state instead of re-reading the whole conversation |
Round 2 (this update): on by default wherever the card supports it, so the config needs no new flags. Each one has
a --no-<flag> escape. With all of them off, the engine is bit-identical to round 1. Measured on the RTX 5080 below.
| what | effect | needs |
|---|---|---|
decode attention on tensor cores (--attn-mma) |
an MTP verify window over 262K cells: 1.55 ms -> 0.43 ms per layer; MTP keeps paying at depth | sm_80+, int8 cache |
fused prompt attention from the int8 cache (--fused-prompt-attn), chunked DeltaNet prompt pass (--chunked-gdn) |
faster prompt reads, most at depth; frees 0.9 GB for experts | sm_80+, int8 cache |
| prompt-side expert stream: split copies on their own stream, one quantization per token, wide dequantization | prompt read +9-26% (most on short prompts) | always (bit-identical) |
| decode chain: register reductions, a side stream for the gate projections | decode +3-5% | always (bit-identical) |
more experts in VRAM: prompt staging lent to decode (--lend-staging), snapshots in host memory (--snapshot-host), headroom sized from measurement |
78 -> 97 cached experts per layer, plus the lent staging while decoding | hybrid mode |
CPU pool warms the next experts' rows in its caches (--pool-warm), slot swaps off the main stream (--async-swaps) |
decode +4-8% | hybrid mode |
sampled MTP rows sent to the host as GPU shortlists (--sample-gpu) |
sampled MTP +9% | sm_70+ |
Opt-in only: --draft-vocab 16384 --draft-ctx 4096 (MTP drafts over the common and in-context tokens: +7-9% MTP on English
and code, slower on other scripts) and --random-seed (a fresh seed for unseeded sampled requests).
Running it
git clone -b q36-r2 https://github.com/xyehya/Strata && cd Strata
./setup.sh # builds the engine (Linux; Windows: START-HERE.bat)
hf download ykarout/Strata-Sage-A1.5-GGUF --local-dir models/sage
python tools/strata_tokenizer.py --gguf models/sage/Sage-A1.5-Q4_K_M-ssmbf16.gguf --out models/sage/tokenizer
strata-sage.json (the importer only takes Flash-Next files, so this config is written by hand; it is the same as round 1's,
because round 2 is on by default):
{
"exe": "engine/strata",
"args": ["--native", "models/sage/Sage-A1.5-Q4_K_M-ssmbf16.gguf", "--max-context", "262144", "--prefill", "auto",
"--kv", "int8", "--pool-workers", "16", "--gqa-attn", "--hybrid-zero-copy", "--expert-prefetch", "2",
"--expert-profile", "models/sage/expert-profile-agent-v2.prof", "--decode-swaps", "64", "--snapshots", "2",
"--mtp", "2", "--spec-min-p", "0.5", "--mtp-sampled", "q"],
"tokenizer": "models/sage/tokenizer",
"model_name": "sage-a1.5",
"arch": "qwen35moe",
"splice_turns": "parsed",
"sampling": {"temperature": 1.0, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 1.5}
}
.venv/bin/python serve/server.py --engine strata --config strata-sage.json --port 8090
OpenAI- and Anthropic-compatible on http://127.0.0.1:8090/v1. "sampling" is InternScience's recommendation (temp 1.0,
top_p 0.95, top_k 20, min_p 0, presence_penalty 1.5) as the server's default for requests that send none: many agent
clients (opencode, for one) send no temperature, which otherwise means greedy decoding. A request's own values always win.
--pool-workers is the CPU threads for the experts not in VRAM (16 on a 24-core
CPU). The profile was learned with 16 GB of VRAM; with less, the engine says so and starts from its default order. The
engine prints which round-2 defaults are on and, for any that are off, why (for example "attn-mma off: sm_75").
Measured
RTX 5080 16 GB, Core Ultra 9 285K, 64 GB DDR5. Through the server: fresh code prompts, 384 tokens generated, temperature
0.7, thinking off, MTP 2 as configured, average of two runs per side, run back to back (round 1, round 2, round 1, round 2),
tok/s. llama.cpp b11834 tuned for this card (round 1's measurement): --cpu-moe --moe-cache-mib 10300, -b 4096 -ub 2048,
MTP draft 2, 8 threads on the P-cores.
| decode | 4K | 16K | 40K | 90K | 160K | 240K |
|---|---|---|---|---|---|---|
Strata q36-r2 (262K context) |
231 | 210 | 213 | 187 | 179 | 162 |
Strata q36-optimized (round 1) |
173 | 154 | 151 | 124 | 105 | 86 |
| llama.cpp (131K context) | 166 | 171 | 146 |
| prompt read | 4K | 16K | 40K | 90K | 160K | 240K |
|---|---|---|---|---|---|---|
Strata q36-r2 |
6,425 | 6,903 | 6,115 | 4,561 | 3,351 | 2,477 |
Strata q36-optimized |
4,685 | 5,111 | 4,350 | 2,706 | 1,511 | 806 |
| llama.cpp | 3,185 | 3,830 | 3,730 |
The gain grows with context: at 240K, round 2 decodes faster than round 1 did at 40K (162 vs 151 tok/s) and reads prompts 3.1x faster. An agent's next turn on a cached conversation (server, prefix reuse, MTP off): +2,000 tokens on a reused 198K context in 0.96 s (round 1: 5.1 s), +300 tokens in 0.32 s (0.95 s); a cold 198K read takes 67 s (180 s). MTP acceptance is the same as round 1's at every depth (65-73%).
Engine only, no server, forced text, MTP off (the clean kernel number): decode 193 -> 249 (+29%), 170 -> 219 (+29%), 183 -> 229 (+25%) at 4K / 16K / 40K; prompt read +38% / +35% / +42%.
Real agent workload (opencode --auto, one coding task of 52 requests up to 182K context, decode per request grouped
by the context it ran at, tok/s). Measured on round 1 only; round 2 has not had an agent run yet:
| decode, agent run (round 1) | < 16K | 40-90K | 90-160K | 160K+ | peak |
|---|---|---|---|---|---|
Strata q36-optimized |
156 | 137 | 116 | 101 | 206 |
| MTP acceptance | 73% | 72% | 73% | 71% |
Close to round 1's sweep row above at each depth (Sage's MTP acceptance is about the same on agent work as on prose), so round 2's sweep gains should carry over. The base model, Qwen3.6-35B-A3B (Strata-Qwen3.6-35B-A3B), accepts 88-95% of its drafts on the same task and decodes 273 / 254 / 234 / 206 tok/s on round 2 in this layout.
Quality. Round 2 changes numerics, so it was checked against a reference rather than against round 1 alone. With the fp32 cache (no KV quantization) as the reference, every round-2 path is at least as close as round 1: mean KL 6.6-8.5e-3 against round 1's 1.55e-2 (full-vocabulary logits, 195 teacher-forced steps at 14K / 57K / 76K), and top-1 agreement 184-189 of 195 against 185. At 200K (65 forced steps), round 2 against round 1 (mean KL 6.8e-3, top-1 63/65) moves the output no more than changing round 1's own prompt chunk size does (6.9e-3, 63/65). Needle retrieval (10 planted codes at 5-95% depth, ~100K and ~200K, with and without MTP): 40/40, as round 1.
License
Apache-2.0, as the original model. All credit for the model to InternScience / Shanghai AI Laboratory (arXiv 2610.28578); this repository only re-packages their weights.
- Downloads last month
- 5,620
16-bit
Model tree for ykarout/Strata-Sage-A1.5-GGUF
Base model
InternScience/Sage-A1.5