ramgpt/pplx-decider-v1-27b-EXL3

pplx-decider-v1-27b EXL3 4.00 bpw

EXL3 4.00 bpw conversion of perplexity-ai/pplx-decider-v1-27b.

• Source revision: 5117a6c7fe73b19308dc1a6b0fb529a40c2ecad4
• ExLlamaV3: 1.5.2+cu128.torch2.10.0
• Target bitrate: 4.00 bpw
• Architecture: Qwen3_5Model
• Safetensors size: 15.35 GB / 14.29 GiB

Important: this is a decision model, not a normal chat model

pplx-decider-v1-27b does not use the normal LM-head token-generation path for its final answer. It produces a hidden state and applies the model's custom readout.safetensors decision head to obtain option probabilities.

The repository therefore includes pplx_decider_exl3.py, which registers the model architecture and performs the custom decision readout.

TabbyAPI compatibility

This model is not recommended for standard TabbyAPI usage.

A local smoke test with TabbyAPI and ExLlamaV3 1.5.2 failed during startup with:

AssertionError: Unknown architecture Qwen3_5Model

More importantly, simply registering the architecture is not sufficient for normal /v1/chat/completions behavior: TabbyAPI's standard generation path expects an LM head and token generation, while this model requires the custom decision readout described above. A dedicated TabbyAPI backend/endpoint adapter would be needed for correct decision-model semantics.

Use the included EXL3 decision wrapper instead of treating this model as a conventional text-generation model.

RTX 4090 benchmark

Measured locally on an NVIDIA GeForce RTX 4090 24 GB with Torch 2.10.0+cu128, ExLlamaV3 1.5.2, batch size 1, FP16 cache, and direct EXL3 decision inference.

The model was kept resident in VRAM. Each context-length row below is based on 10 sequential measured requests after warm-up. Timing includes tokenization, synchronized GPU inference, decision readout, and transfer of the result back to CPU. It excludes model load time, HTTP/server overhead, and queueing.

Input tokens	Median latency	Mean latency	Sequential decisions/s	Peak PyTorch allocated VRAM
128	135 ms	136 ms	7.34	12.67 GiB
512	278 ms	278 ms	3.59	12.89 GiB
1,024	430 ms	431 ms	2.32	12.98 GiB
2,048	725 ms	725 ms	1.38	13.08 GiB
4,096	1.400 s	1.400 s	0.71	13.35 GiB
8,192	2.777 s	2.778 s	0.36	13.89 GiB

Additional measurements:

• Model setup/load: 3.09 s with a warm OS file cache; this is not a cold-disk benchmark.
• Resident PyTorch allocation immediately after load: 12.57 GiB.
• First 120-token inference after load: 1.46 s; subsequent warm requests were much faster.
• At the end of the 8K run, nvidia-smi reported about 15.0 GiB total GPU memory use for GPU 0, including the pre-existing GPU baseline and allocator-reserved memory.

This is a classifier/decision workload, so conventional autoregressive "decode tok/s" is not the useful metric. Latency per decision and prefill/context scaling are more representative.

Validation

The EXL3 artifact passed direct local decision inference using its custom readout.

A small hand-authored sanity suite produced:

• Routing: 9/9
• Sentiment: 6/6
• Entailment: 9/9
• Arithmetic: 6/6
• Yes/no urgency: 8/8
• Explicit severity-level selection: 5/5
• Total: 43/43
• Reversing multiple-choice option order: 30/30, with the same semantic choice
• Repeating a request after intervening requests: 5/5 identical decisions, maximum probability delta 0.0

These are basic synthetic sanity checks, not an official accuracy benchmark. No BF16-vs-EXL3 accuracy/parity study has been performed, and the long-context timing prompts use synthetic filler rather than a long-context reasoning benchmark.

Notes

The original source model is a custom decision model fine-tuned from Qwen3.8-27B. Do not infer parameter count from Hugging Face's automatic architecture display alone; the source checkpoint contains about 48.6 GiB of BF16 weights, while this 4.00 bpw EXL3 artifact contains about 15.35 GB of safetensors.