pragmaticcs/Triumvirate-Qwopus-MiMo-Ornith-9B-Coder

Most sub-10B coding models crumble the moment they enter real-world agentic workflows: they either produce clean code but loop endlessly when a shell command fails, or handle tool calls reasonably well while hallucinating obscure API syntax.

Triumvirate is a merge designed to solve that dilemma. It combines three of the most capable specialized fine-tunes of Qwen 3.5 9B and fuses their task vectors directly into the base backbone:

• Algorithmic & Syntax Precision from Qwopus
• SWE-bench Problem Decomposition & Tool Calling from MiMo-V2.6
• Loop-Termination & Error-Recovery Discipline from Ornith-1.5

The result is a lean, blisteringly fast 9B pure-text causal engine with a native 256k context window that runs comfortably on consumer GPUs.

---

Contents

• Architectural Specifications
• Composition & Donor Weighting
• Merge Methodology & Mathematical Formulation
• Layer-Stratified Component Policies
• Agentic Chat Template
• Recommended Generation Parameters
• How to Use
• Citation & References

---

Architectural Specifications

Parameter	Specification
Total Parameters	8.8B (Text Backbone)
Architecture Type	Dense Causal Language Model (qwen3_5_text)
Hidden Dimension (dmodel)	4096
Intermediate Dimension (dmlp)	12288 (SwiGLU)
Decoder Layers	32
Attention Mechanism	Hybrid Gated DeltaNet (3 Linear Attention : 1 Full Attention)
Full Attention Layers	Layers 3, 7, 11, 15, 19, 23, 27, 31
Linear Attention Heads	16 Key Heads / 32 Value Heads (dk = dv = 128)
Full Attention Heads	16 Query / 4 Key-Value (GQA, dh = 256)
Rotary Position Embedding (RoPE)	1D Partial RoPE (θ = 10⁷, Factor = 0.25)
Maximum Sequence Length	262,144 tokens (256k)
Native Precision	bfloat16

---

Composition & Donor Weighting

The foundation checkpoint serves as the structural base (W₀). Three donor models contribute directional task vectors weighted continuously across network depth:

Model	Role	Specialization Focus	Depth Target
Qwen/Qwen3.5-9B	Base Anchor (W₀)	Structural anchor & GDN linear attention state	Global
Jackrong/Qwopus3.5-9B-Coder	Donor 1 (D₁)	Claude 3.5 Opus distillation; typing, syntax, algorithms	Lower Layers (x ≤ 0.35)
ornith-ai/Ornith-1.5-9B	Donor 2 (D₂)	Agentic RL; loop-termination & error-pivot discipline	Mid Layers (0.35 < x < 0.70)
XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B	Donor 3 (D₃)	77.4B tokens SFT; SWE-bench Pro, multi-turn tool logic	Top Layers (x ≥ 0.70)

---

Merge Methodology & Mathematical Formulation

The merge combines TIES-DELLA saliency trimming, consensus sign election, Gated DeltaNet norm stabilization, and continuous sinusoidal depth modulation.

1. Task Vector Formulation

For each donor checkpoint k ∈ {1, 2, 3}, the parameter update delta is isolated relative to the base anchor W₀:

$${\tau }_k=D_k-W_0,k\in \{\text{MiMo},\text{Ornith},\text{Qwopus}\}\backslash tau\_k\; =\; D\_k\; -\; W\_0,\; \backslash quad\; k\; \backslash in\; \backslash \{\backslash text\{MiMo\},\; \backslash text\{Ornith\},\; \backslash text\{Qwopus\}\backslash \}$$

2. Asymmetric Sinusoidal Depth Modulation

Task vector mixing coefficients are continuously parameterized over normalized network depth x = l / (L - 1), where l ∈ {0, 1, ..., 31} and L = 32:

$$u_{\text{qwopus}}(x)=0.45{\cos }^2\left(\frac{\pi }{2}x\right)+0.25u\_\{\backslash text\{qwopus\}\}(x)\; =\; 0.45\; \backslash cos^2\backslash left(\backslash frac\{\backslash pi\}\{2\}\; x\backslash right)\; +\; 0.25$$

$$u_{\text{ornith}}(x)=0.35{\sin }^2\left(\pi x^{0.85}\right)+0.15u\_\{\backslash text\{ornith\}\}(x)\; =\; 0.35\; \backslash sin^2\backslash left(\backslash pi\; x^\{0.85\}\backslash right)\; +\; 0.15$$

$$u_{\text{mimo}}(x)=0.55{\sin }^2\left(\frac{\pi }{2}{x}^{1.20}\right)+0.20u\_\{\backslash text\{mimo\}\}(x)\; =\; 0.55\; \backslash sin^2\backslash left(\backslash frac\{\backslash pi\}\{2\}\; x^\{1.20\}\backslash right)\; +\; 0.20$$

The donor weights α_k(l) are normalized to form a partition of unity across all layers:

$${\alpha }_k(l)=\frac{u_k(x)}{\sum _{j=1}^3{u}_j(x)},\sum _{k=1}^3{\alpha }_k(l)=1.0\backslash alpha\_k(l)\; =\; \backslash frac\{u\_k(x)\}\{\backslash sum\_\{j=1\}^3\; u\_j(x)\},\; \backslash quad\; \backslash sum\_\{k=1\}^3\; \backslash alpha\_k(l)\; =\; 1.0$$

• Lower Layers (x → 0): Qwopus dominates with α₁(0) ≈ 0.67, ensuring foundational language representations and syntax heads are grounded in Claude 3.5 Opus traces.
• Middle Layers (x ≈ 0.5): The sub-linear exponent (x^0.85) accelerates Ornith's activation to peak across middle transformer blocks with α₂(16) ≈ 0.354, reinforcing state-space continuity and execution discipline.
• Top Layers (x → 1): The super-linear exponent (x^1.20) concentrates MiMo's task vector with α₃(31) ≈ 0.652 into the upper decoders, governing semantic reasoning, multi-turn planning, and final token synthesis.

3. Saliency Trimming (TIES-DELLA Pruning)

To eliminate parameter interference and cross-talk, task vectors are pruned based on parameter energy. Given density parameter ρ = 0.70, an update threshold γ_k is computed per tensor:

$${\gamma }_k={\text{Quantile}}_{1-\rho }(\{\mathrm{\mid }{\tau }_{k,ij}\mathrm{\mid }\})\backslash gamma\_k\; =\; \backslash text\{Quantile\}\_\{1\; -\; \backslash rho\}\backslash left(\backslash \{|\backslash tau\_\{k,\; ij\}|\backslash \}\backslash right)$$

Updates below the top 70% magnitude are zeroed out via a saliency mask:

$$M_k=\mathbb{I}(\mathrm{\mid }{\tau }_k\mathrm{\mid }\ge {\gamma }_k)M\_k\; =\; \backslash mathbb\{I\}\backslash left(|\backslash tau\_k|\; \backslash ge\; \backslash gamma\_k\backslash right)$$

$${\widehat{\tau }}_k={\tau }_k\odot M_k\backslash hat\{\backslash tau\}\_k\; =\; \backslash tau\_k\; \backslash odot\; M\_k$$

4. Consensus Sign Election & Disjoint Averaging

Surviving task vectors often conflict in directional signs, causing mutual cancellation when averaged naively. A directional consensus sign vector Γ is elected:

$$\mathrm{\Gamma }=\mathrm{sgn}(\sum _{k=1}^3{\alpha }_k(l){\widehat{\tau }}_k)\backslash Gamma\; =\; \backslash operatorname\{sgn\}\backslash left(\backslash sum\_\{k=1\}^3\; \backslash alpha\_k(l)\; \backslash hat\{\backslash tau\}\_k\backslash right)$$

A binary agreement mask A_k discards parameter updates that oppose the elected consensus sign:

$$A_k=\mathbb{I}(\mathrm{sgn}({\widehat{\tau }}_k)=\mathrm{\Gamma })\odot \mathbb{I}({\widehat{\tau }}_k\mathrm{\ne }0)A\_k\; =\; \backslash mathbb\{I\}\backslash left(\backslash operatorname\{sgn\}(\backslash hat\{\backslash tau\}\_k)\; =\; \backslash Gamma\backslash right)\; \backslash odot\; \backslash mathbb\{I\}\backslash left(\backslash hat\{\backslash tau\}\_k\; \backslash neq\; 0\backslash right)$$

The merged task delta is reconstructed using only parameters aligned with the majority direction:

The dense layer weights are restored onto the base foundation:

$$W_{\text{dense}}=W_0+{\mathrm{\Delta }}_{\text{TIES}}W\_\{\backslash text\{dense\}\}\; =\; W\_0\; +\; \backslash Delta\_\{\backslash text\{TIES\}\}$$

5. Gated DeltaNet (GDN) Gate Norm Stabilization

In linear attention layers, gate matrices control state retention and output gating via non-linear sigmoid activations. Direct delta merging shifts the operator norm, causing activation saturation or exploding outputs. To guarantee numerical stability, the merged gate weight W_{gate, unscaled} = W₀ + ∑_k α_k τ_k is projected onto the base tensor's Frobenius norm:

$$W_{\text{gate}}=W_{\text{gate, unscaled}}\cdot \frac{\mathrm{\parallel }{W}_0{\mathrm{\parallel }}_F}{\mathrm{\parallel }{W}_{\text{gate, unscaled}}{\mathrm{\parallel }}_F}W\_\{\backslash text\{gate\}\}\; =\; W\_\{\backslash text\{gate,\; unscaled\}\}\; \backslash cdot\; \backslash frac\{\backslash |W\_0\backslash |\_F\}\{\backslash |W\_\{\backslash text\{gate,\; unscaled\}\}\backslash |\_F\}$$

6. Log-Decay and Normalization Parameter Convexity

For state-space logarithmic decay tensors (A_log ∈ (-∞, 0]), biases, and layer normalization parameters, delta blending can violate mathematical boundary constraints. These tensors are merged strictly via convex interpolation:

$$W_{\text{convex}}=\sum _{k=1}^3{\alpha }_k(l)D_{k}W\_\{\backslash text\{convex\}\}\; =\; \backslash sum\_\{k=1\}^3\; \backslash alpha\_k(l)\; D\_k$$

Because ∑_k α_k(l) = 1.0, α_k(l) ≥ 0, and D_{k, ij} ≤ 0 for all decay parameters:

$$\sum _{k=1}^3{\alpha }_k(l)D_{k,ij}\le \underset{k}{\max }(D_{k,ij})\le 0⟹\exp \left(W_{\text{convex},ij}\right)\in (0,1]\backslash sum\_\{k=1\}^3\; \backslash alpha\_k(l)\; D\_\{k,\; ij\}\; \backslash le\; \backslash max\_k(D\_\{k,\; ij\})\; \backslash le\; 0\; \backslash implies\; \backslash exp\backslash left(W\_\{\backslash text\{convex\},\; ij\}\backslash right)\; \backslash in\; (0,\; 1]$$

This guarantees Bounded-Input Bounded-Output (BIBO) stability and prevents exponential divergence in recurrent linear attention states.

---

Layer-Stratified Component Policies

Parameter Group	Target Identifiers	Applied Policy	Density (ρ)	Mathematical Invariant
Embeddings & LM Head	embed_tokens, lm_head	Convex Blend	—	Fixed weights: 50% Qwopus, 30% MiMo, 20% Ornith.
Dense MLPs & Self-Attention	self_attn, mlp.gate_proj, up_proj, down_proj	TIES-DELLA	0.70	Saliency pruning + consensus sign election.
Recurrent Linear Attention	linear_attn.in_proj_*, out_proj, conv1d	Recurrent Delta	—	Unpruned linear delta accumulation.
DeltaNet Attention Gates	attn_output_gate	Norm-Stabilized	—	Projected onto base Frobenius norm ||W₀||F.
Decay Rates & Normalizations	A_log, norm, bias	Convex Blend	—	Enforces Alog ≤ 0 to preserve recurrent stability.

---

Agentic Chat Template

This model uses the Improved Chat Template for Qwen 3.x by Olivia Rossi to support multi-tier Chain-of-Thought (CoT) reasoning, dual-format agentic tool execution, automatic error-recovery heuristics, and strict token-waste elimination.

---

Recommended Generation Parameters

The following parameters are optimal for code synthesis, terminal agent execution, and complex reasoning:

Parameter	Recommended Setting	Operational Function
Temperature	0.6	Balances deterministic syntax structure with creative algorithmic pathing.
Top-P	0.95	Nucleus sampling cutoff to discard degenerate token tails.
Top-K	20	Restricts sampling pool to top candidates, preventing syntactic drift.
Min-P	0.0 (Off)	Disables relative thresholding in favor of Top-K / Top-P governance.
Repetition Penalty	Off (1.0)	Disabled to prevent penalty distortion on repeated syntax (braces, boilerplate).
Presence Penalty	Off (0.0)	Preserves deterministic variable and function naming across long contexts.

---

How to Use

Serving via vLLM

vllm serve pragmaticcs/Triumvirate \
  --dtype bfloat16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --enable-reasoning \
  --reasoning-parser qwen3

Inference via Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "pragmaticcs/Triumvirate"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are a principal software engineer. Think carefully before outputting production-grade code."
    },
    {
        "role": "user",
        "content": "Implement an asynchronous token-bucket rate limiter in Python supporting burst handling and thread-safe redis synchronization."
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=2048,
    temperature=0.6,
    top_p=0.95,
    top_k=20,
    do_sample=True,
)

response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(response)

---

Citation & References

• Qwen/Qwen3.5-9B
• XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B
• ornith-ai/Ornith-1.5-9B
• Jackrong/Qwopus3.5-9B-Coder
• Improved Chat Template for Qwen 3.x

@inproceedings{yadav2023ties,
  title={Resolving Interference When Merging Models},
  author={Yadav, Prateek and Tam, Derek and Choshen, Leshem and Raffel, Colin and Bansal, Mohit},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  volume={36},
  pages={7093--7115},
  year={2023}
}

@article{deep2024della,
  title={DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling},
  author={Deep, Pala Tej and Bhardwaj, Rishabh and Poria, Soujanya},
  journal={arXiv preprint arXiv:2406.11617},
  year={2024}
}

@inproceedings{yu2024dare,
  title={Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch},
  author={Yu, Le and Yu, Bowen and Yu, Haiyang and Huang, Fei and Li, Yongbin},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}