Image-Text-to-Text
GGUF
English
quant
target_bpw
experimental
conversational

Experimental global target bits‑per‑weight quantization of Qwen/Qwen3.6-35B-A3B

Using non-standard (forked) LLaMA C++ release b8990 for quantization.

Original model: Qwen/Qwen3.6-35B-A3B

From the original model creators:

Qwen3.6-35B-A3B

Qwen Chat

This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format.

These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransformers, etc.

Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding experience.

Qwen3.6 Highlights

This release delivers substantial upgrades, particularly in

  • Agentic Coding: the model now handles frontend workflows and repository-level reasoning with greater fluency and precision.
  • Thinking Preservation: we've introduced a new option to retain reasoning context from historical messages, streamlining iterative development and reducing overhead.

Benchmark Results

For more details, please refer to our blog post Qwen3.6-35B-A3B.


⚠️ PLEASE READ THIS BEFORE USING THESE EXPERIMENTAL VERSIONS! ⚠️

An area of personal interest is finding ways to optimize the inference performance of LLMs when deployed in resource-constrained environments like commodity hardware, desktops, laptops, mobiles, edge devices, etc. There are many approaches to accomplish this, including architecture simplification and knowledge distillation, but my focus has been primarily on quantization and pruning.

The method to produce these experimental versions involves using a custom version of llama-imatrix to generate an imatrix that includes tensor statistics, and a custom version of llama-quantize, which computes a per-tensor quantization error, to automatically select the lowest error quantization recipe that achieves a global target bits‑per‑weight (bpw). More details on the implementation and test results here

There are two pull requests (#14891 & #15550) to merge these changes back into the core llama.cpp project. This may or may not ever happen so, until then, the modified versions will be available on GitHub.

For testing and comparison, I use models produced by Bartowski (see credits below) and Unsloth (Daniel and Michael Han do some really interesting stuff!) but when they don't provide versions of the required model, tests and comparisons are against standard quantization obtained by simply running llama-quantize with no further optimizations.

All experimental versions were generated using an appropriate imatrix created from datasets available at eaddario/imatrix-calibration. In llama.cpp, an imatrix is a calibration file derived from running representative text through the model and collecting activation statistics. It is used to weight quantization error so that error in more “important” directions (as estimated from activations) is penalized more heavily.

The process to generate these models is roughly as follows:

  1. Convert the original model's safetensors to GGUF F16
  2. Estimate the Perplexity score for the F16 model (baseline) using the wikitext-2-raw-v1 dataset, and save the logits
  3. Generate an imatrix from the most appropriate calibration dataset
  4. Quantize the baseline model targeting a bpw average (e.g. llama-quantize --target-bpw 4.5678 --state-file --imatrix imatrix.gguf baseline-model-F16.gguf 12)
  5. Calculate Perplexity, KL Divergence, ARC (Easy+Challenge), GPQA-Diamond, HellaSwag, MMLU-Redux, Truthful QA and WinoGrande scores for each quantized model
  6. Keep version with the best 𝜌PPL and μKLD scores
  7. Repeat until all desired quants are created

Misconceptions about BF16 to F16 Conversion

A common concern when converting BFloat16 (BF16) models to Float16 (F16) is the potential for accuracy loss. Specifically:

  • Weight Clipping (Overflow): Clipping, or overflow, is often feared but only occurs if a model's weights exceed the range of ±65,503. This is a relatively rare issue in practice.
  • Subnormal Zeroing (Underflow): A more frequent occurrence is underflow, where weights smaller than approximately 5.96x10⁻⁸ are converted to zero.

Crucially, when the F16 model is subsequently used for quantization, the resulting degradation in metrics like Perplexity (PPL) or Kullback–Leibler Divergence (KLD) is minimal. Any variations are typically restricted to the hundreds or thousandths decimal places compared to the BF16 model.

However, considering that weight clipping presents a more substantial risk to model integrity, every BF16 base model undergoes validation prior to the conversion process. Consequently, no models hosted in this repository exhibit performance degradation due to overflow clipping.

While BF16 offers precision benefits, performance remains a key factor.

  • Conversion Speed: Tests, such as timing convert_hf_to_gguf.py, show a notable performance difference, with conversion to BF16 being 15–30% slower than to F16.
  • Inference Speed: A less pronounced but still present difference (3–6%) is observed during inference. Although native BF support has been introduced by many chip manufacturers, the slower performance may stem from the entire software and hardware stack (firmware, libraries, etc.) not being fully optimized yet.

The choice to prioritize F16 over BF16 is driven by a focus on maximizing performance in specific deployment environments. My primary objective is not large-scale quantization production, a domain where others like Bartowski and Unsloth excel at, but rather optimizing inference performance for resource-constrained environments. Since BF16 support is not yet widespread in areas like mobile, edge, and embedded devices, using F16 ensures broader compatibility and easier optimization for these use cases.

Advantages and disadvantages of the global target bits‑per‑weight quantization process

Advantages

  1. Target arbitrary size models

    • When specifying --target-bpw 4.5678 for instance, the algorithm will produce a model (nearly) exactly of that size, which is very useful for maximizing VRAM usage. In a system with 24GB VRAM and a 70B model, standard quants might produce a 16.8GB file (too small, quality left on table) or a 24.1GB file (won't fit). This approach can generate a 23.85GB file to utilize the hardware fully.
  2. Data-driven mixed precision often can improve quality at fixed size

    • Instead of using hardcoded heuristics (e.g. make attn_v Q5_K for a 70B model), that may be sub‑optimal for a given architecture or size, the quantization mix is determined by the actual error sensitivity of the specific model's weights. This, in practice, often yields a better quality/size trade-off, especially in aggressive quantization scenarios (1.5 to 3.5 bpw), or for unusual architectures.

    • Please note: llama.cpp’s heuristics have been tuned across many models and are highly optimized; although the target bpw method produces better quality often (>75% based on tests with 130 models from 11 different families), it can also lose in surprising cases.

  3. Allows better like-for-like comparisons between models and families

    • Standard llama.cpp quantization uses hardcoded rules like: "use Q4_K_M, except bump some tensors up/down, except fall back if incompatible, except keep some tensors unquantized..." and for that reason, two different models quantized with the same Q4_K_M type can end up with very different bpw (e.g. 4.75 and 4.30).

    • All things being equal, the performance of a model is usually proportional to its overall bpw size; models with a higher bpw tend to perform better than lower bpw models. Since model A has simply been given more bits, it will typically perform better (lower perplexity, better eval scores, etc.) even if the underlying quantization method is identical. That makes comparing the performance not a controlled experiment, because the comparison is between models with different effective compression ratios.

    • --target-bpw tries to address that by making the experiment more controlled: each model gets quantized to land on (approximately) the same global byte budget, so that the models' performance differences are more attributable to architecture/training differences, quantization error behaviour at the same compression ratio, optimizer’s allocation decisions, etc.

Disadvantages

  1. Quantization process is significantly slower than standard

    • This approach can take 5x-10x longer as it quantizes a sample of most tensors into 15 different formats, dequantizes them back to floats, computes error diffs, and selects the best size/error option that fits the global bpw budget.

    • However, the --state-file option will save/use the above-mentioned computations so that future quantizations, for the same model, can be generated at normal speed. It also allows to interrupt the computation process and resume it at a later time.

  2. The optimization target is only a proxy for the model's performance quality

    • The process minimizes a per-tensor estimated error computed from sampled rows, not actual perplexity or divergence of output distributions (a future version may address this). Since errors interact nonlinearly across layers, there are no guarantees it will select the best possible quantization recipe subject to the bpw size constraint.
  3. An imatrix with activations data is required for best results

    • Activation data is required to compute the bias factor (i.e. the systematic error projected onto activation directions). If the imatrix file does not contain activation data, the --target-bpw option will refuse to run.

Models

Bits per weight, size, perplexity and KL Divergence scores

Model BPW Size (GB) μPPL 𝜌PPL μKLD Same Top-P
Qwen3.6-35B-A3B-F16 16.0103 65.0 6.270267 ±0.039764 100% N/A N/A
Qwen3.6-35B-A3B-Q2_K 1.7500 7.1 19.727704 ±0.154377 77.72% 1.223315 ±0.003379 54.832 ±0.129
Qwen3.6-35B-A3B-Q2_K 2.5000 10.0 9.699612 ±0.066197 89.27% 0.514580 ±0.001852 69.313 ±0.120
Qwen3.6-35B-A3B-Q3_K 3.5000 14.0 7.265556 ±0.047435 96.15% 0.180509 ±0.000844 80.474 ±0.103
Qwen3.6-35B-A3B-Q4_K 4.5000 18.0 6.517143 ±0.041405 98.64% 0.061493 ±0.000473 89.092 ±0.081
Qwen3.6-35B-A3B-Q5_K 5.5000 22.0 6.341921 ±0.040406 99.38% 0.026679 ±0.000332 92.981 ±0.066
Qwen3.6-35B-A3B-Q6_K 6.5000 26.0 6.297517 ±0.039973 99.68% 0.013373 ±0.000264 95.214 ±0.055
Qwen3.6-35B-A3B-Q7_K 7.5000 30.0 6.261129 ±0.039683 99.82% 0.007794 ±0.000211 96.421 ±0.048
Qwen3.6-35B-A3B-Q8_0 8.5000 34.0 6.268898 ±0.039745 99.86% 0.005965 ±0.000249 97.057 ±0.044

ARC, GPQA-Diamond, HellaSwag, MMLU-Redux, Truthful QA, and WinoGrande scores

Scores generated using llama-perplexity with 750 tasks per test, and a context size of 1024 tokens.

For the test data used in the generation of these scores, follow the appropriate links: ARC Challenge, Truthful QA, GPQA-Diamond, HellaSwag, MMLU-Redux, WinoGrande

Model ARC Challenge GPQA-Diamond HellaSwag MMLU-Redox Truthful QA WinoGrande Avg Score
Qwen3.6-35B-A3B-Q1_L 53.0667 ±1.8235 21.2121 ±2.9127 82.67 55.3333 ±1.8165 29.0667 ±1.6591 60.6667 ±1.7849 50.34
Qwen3.6-35B-A3B-Q2_K 58.6667 ±1.7993 25.7576 ±3.1156 72.40 70.4000 ±1.6680 33.6000 ±1.7259 67.8667 ±1.7063 54.78
Qwen3.6-35B-A3B-Q3_K 65.2000 ±1.7405 28.2828 ±3.2088 81.07 79.0667 ±1.4865 36.5333 ±1.7594 72.2667 ±1.6358 60.40
Qwen3.6-35B-A3B-Q4_K 67.3333 ±1.7137 28.7879 ±3.2259 82.00 83.6000 ±1.3530 38.9333 ±1.7816 74.0000 ±1.6027 62.44
Qwen3.6-35B-A3B-Q5_K 66.2667 ±1.7276 26.2626 ±3.1353 82.67 84.6667 ±1.3165 38.9333 ±1.7816 74.8000 ±1.5864 62.27
Qwen3.6-35B-A3B-Q6_K 66.1333 ±1.7292 29.7980 ±3.2586 82.53 84.1333 ±1.3350 39.0667 ±1.7827 74.2667 ±1.5974 62.65
Qwen3.6-35B-A3B-Q7_K 66.6667 ±1.7225 26.7677 ±3.1544 82.27 84.1333 ±1.3350 38.8000 ±1.7805 73.0667 ±1.6209 61.95
Qwen3.6-35B-A3B-Q8_0 66.8000 ±1.7207 27.7778 ±3.1912 82.00 83.8667 ±1.3441 39.7333 ±1.7880 74.2667 ±1.5974 62.41

Tokens per second benchmarks

Scores generated using llama-bench. Standard (llama-quantize with no optimization) Q4_K_M quantization included for comparison.

model size params backend threads test t/s
Qwen3.6-35B-A3B-Q1_L 7.06 GiB 34.66 B BLAS,MTL 12 pp512 1437.34 ±10.43
Qwen3.6-35B-A3B-Q1_L 7.06 GiB 34.66 B BLAS,MTL 12 tg128 83.93 ±0.41
Qwen3.6-35B-A3B-Q1_L 7.06 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 151.24 ±1.92
Qwen3.6-35B-A3B-Q2_K 10.09 GiB 34.66 B BLAS,MTL 12 pp512 1407.65 ±37.52
Qwen3.6-35B-A3B-Q2_K 10.09 GiB 34.66 B BLAS,MTL 12 tg128 79.43 ±0.31
Qwen3.6-35B-A3B-Q2_K 10.09 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 146.66 ±1.65
Qwen3.6-35B-A3B-Q3_K 14.12 GiB 34.66 B BLAS,MTL 12 pp512 1428.03 ±8.09
Qwen3.6-35B-A3B-Q3_K 14.12 GiB 34.66 B BLAS,MTL 12 tg128 81.50 ±0.86
Qwen3.6-35B-A3B-Q3_K 14.12 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 146.92 ±0.84
Qwen3.6-35B-A3B-Q4_K 18.16 GiB 34.66 B BLAS,MTL 12 pp512 1414.53 ±5.33
Qwen3.6-35B-A3B-Q4_K 18.16 GiB 34.66 B BLAS,MTL 12 tg128 77.78 ±1.23
Qwen3.6-35B-A3B-Q4_K 18.16 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 141.46 ±0.39
Qwen3.6-35B-A3B-Q5_K 22.19 GiB 34.66 B BLAS,MTL 12 pp512 1390.54 ±12.70
Qwen3.6-35B-A3B-Q5_K 22.19 GiB 34.66 B BLAS,MTL 12 tg128 78.46 ±0.82
Qwen3.6-35B-A3B-Q5_K 22.19 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 143.86 ±0.47
Qwen3.6-35B-A3B-Q6_K 26.23 GiB 34.66 B BLAS,MTL 12 pp512 1352.81 ±2.20
Qwen3.6-35B-A3B-Q6_K 26.23 GiB 34.66 B BLAS,MTL 12 tg128 73.21 ±0.38
Qwen3.6-35B-A3B-Q6_K 26.23 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 137.95 ±4.46
Qwen3.6-35B-A3B-Q7_K 30.26 GiB 34.66 B BLAS,MTL 12 pp512 1413.76 ±3.87
Qwen3.6-35B-A3B-Q7_K 30.26 GiB 34.66 B BLAS,MTL 12 tg128 69.63 ±0.13
Qwen3.6-35B-A3B-Q7_K 30.26 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 129.21 ±0.42
Qwen3.6-35B-A3B-Q8_0 34.30 GiB 34.66 B BLAS,MTL 12 pp512 1463.18 ±5.01
Qwen3.6-35B-A3B-Q8_0 34.30 GiB 34.66 B BLAS,MTL 12 tg128 67.55 ±0.40
Qwen3.6-35B-A3B-Q8_0 34.30 GiB 34.66 B BLAS,MTL 12 pp1024+tg1024 124.55 ±0.18

Metrics used

Perplexity: one of the key metrics used in NLP evaluation. It measures the quality of a language model by evaluating how well it predicts the next token given a particular sequence of words. A PPL of 1 indicates an exact match between predicted and actual, whereas values greater than one indicate a degree of "surprise" the generated token differs from the expected.

Kullback–Leibler (KL) Divergence: a statistical measure of how much a probability distribution differs from another. When quantizing models (or altering the original tensors in any way for that matter), the closest we can preserve the weights' probability distribution to the original model the better, thus the closest to 0 the better.

AI2 Reasoning Challenge (ARC): a benchmark to evaluate the ability of AI models to answer complex science questions that require logical reasoning beyond pattern matching.

GPQA-Diamond: a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.

HellaSwag: the Harder Endings, Longer contexts, and Low-shot Activities for Situations With Adversarial Generations (bit of a mouthful!) is a benchmark designed to test commonsense natural language inference. It requires the model to predict the most likely ending of a sentence.

MMLU: the Massive Multitask Language Understanding evaluates LLMs’ general knowledge and problem-solving abilities across 57 subjects, including elementary mathematics, US history, computer science, and law.

Truthful QA: evaluates how well LLMs generate truthful responses to questions. It identifies whether AI models can avoid generating false or misleading information, particularly in areas where human knowledge is prone to misconceptions.

Winogrande: based on the Winograd Schema Challenge, is a natural language understanding task requiring models to resolve ambiguities in sentences involving pronoun references.

Credits

LLaMa C++ has a large and vibrant community of contributors (~1,600 last time I checked) that actively maintain and extend its functionality, adding new models and architectures almost as fast as they appear. Considering the breakneck speed at which the AI/ML field is advancing, this alone is a remarkable feat!

While I'm grateful to all contributors, I want to recognise three in particular:

  • Colin Kealty (Bartowski), for the many contributions and for being one of the best sources of high quality quantized models available on Hugging Face
  • Georgi Gerganov for his amazing work with llama.cpp and the ggml/gguf libraries
  • Iwan Kawrakow for being one of the key authors behind the many quantization algorithms and the imatrix functionality.
Downloads last month
777
GGUF
Model size
35B params
Architecture
qwen35moe
Hardware compatibility
Log In to add your hardware

2-bit

6-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for eaddario/Qwen3.6-35B-A3B-GGUF

Quantized
(430)
this model

Dataset used to train eaddario/Qwen3.6-35B-A3B-GGUF

Paper for eaddario/Qwen3.6-35B-A3B-GGUF