Instructions to use willfalco/GLM-5.2-EXL3-TR3-3.42bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use willfalco/GLM-5.2-EXL3-TR3-3.42bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="willfalco/GLM-5.2-EXL3-TR3-3.42bpw")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("willfalco/GLM-5.2-EXL3-TR3-3.42bpw")
model = AutoModelForCausalLM.from_pretrained("willfalco/GLM-5.2-EXL3-TR3-3.42bpw", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Trellis

How to use willfalco/GLM-5.2-EXL3-TR3-3.42bpw with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use willfalco/GLM-5.2-EXL3-TR3-3.42bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "willfalco/GLM-5.2-EXL3-TR3-3.42bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.42bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/willfalco/GLM-5.2-EXL3-TR3-3.42bpw

SGLang

How to use willfalco/GLM-5.2-EXL3-TR3-3.42bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "willfalco/GLM-5.2-EXL3-TR3-3.42bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.42bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "willfalco/GLM-5.2-EXL3-TR3-3.42bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "willfalco/GLM-5.2-EXL3-TR3-3.42bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use willfalco/GLM-5.2-EXL3-TR3-3.42bpw with Docker Model Runner:
```
docker model run hf.co/willfalco/GLM-5.2-EXL3-TR3-3.42bpw
```

GLM-5.2 EXL3 TR3 3.42 bpw Coder

with Coding expert allignments from 3.25bpw/NF3

This is a TP4, rank-sliced EXL3 Trellis build of zai-org/GLM-5.2, optimized for four NVIDIA Blackwell workstation GPUs. Routed MoE experts in layers 3-78 use EXL3 Trellis weights targeting 3.0/4.0 bits per weight, including the MTP (layer 78) routed experts using malaiwah's calibration-capture. Accuracy-sensitive and dense components remain in BF16 but can be used in mxfp8 or EXL3 Trellis 6bpw format (see below). The repository payload is 327 GiB. This format requires the custom vLLM + Sparkinfer runtime below; it is not a drop-in Transformers model. The routed weights are EXL3 Trellis and the required launch flag is --quantization exl3. NVFP4 in the supplied runtime refers to the KV cache, not the routed-expert weight format.

Weights        | KV format                 | KLD
───────────────────────────────────────────────────────────────────────
NF3            | Dynamic NVFP4 + RoPE8     | 0.139036 ± 0.002010
NF3            | Standard FP8 + BF16 RoPE  | 0.1263†
EXL3 3.0-bpw   | Dynamic NVFP4 + RoPE8     | 0.119525
EXL3 3.0-bpw   | Standard FP8 + BF16 RoPE  | 0.102508
EXL3 3.25-bpw  | Dynamic NVFP4 + RoPE8     | 0.095971
EXL3 3.25-bpw  | Standard FP8 + BF16 RoPE  | 0.087711
EXL3 3.36-bpw  | Dynamic NVFP4 + RoPE8     | 0.077767
EXL3 3.36-bpw  | Standard FP8 + BF16 RoPE  | 0.068458
EXL3 3.40-bpw  | Dynamic NVFP4 + RoPE8     | ...
EXL3 3.40-bpw  | Standard FP8 + BF16 RoPE  | ...
EXL3 3.40-bpw  | FP8 + Dynamic EXL3 6bpw   | ...
EXL3 3.42-bpw  | Dynamic NVFP4 + RoPE8     | ...
EXL3 3.42-bpw  | Standard FP8 + BF16 RoPE  | ...
EXL3 3.42-bpw  | FP8 + Dynamic EXL3 6bpw   | ...

GPQA Diamond benchmark

A controlled paired run on 2026-08-30 scored this checkpoint at 178/198 (89.90%), with a Wilson 95% confidence interval of 84.91%–93.37%. The same serving stack and benchmark settings scored davidsyoung/GLM-5.3-EXL3-TR3-3.42bpw at 169/198 (85.35%).

Metric	GLM-5.2 3.42 bpw	GLM-5.3 3.42 bpw	GLM-5.2 − GLM-5.3
GPQA Diamond accuracy	178/198 (89.90%)	169/198 (85.35%)	+4.55 pp
Wilson 95% CI	84.91%–93.37%	79.76%–89.60%	—
Biology	15/19 (78.95%)	16/19 (84.21%)	−5.26 pp
Chemistry	79/93 (84.95%)	74/93 (79.57%)	+5.38 pp
Physics	84/86 (97.67%)	79/86 (91.86%)	+5.81 pp
Mean completion tokens	13,919	10,969	+26.9%
Median completion tokens	3,136	780	+302.3%
P90 completion tokens	37,083	30,937	+19.9%
Mean request elapsed time	455.5 s	366.9 s	+24.1%
Aggregate generation rate	30.7 tok/s	30.2 tok/s	+1.8%
Hit the 131,072-token cap	9	8	+1
Truncated before an answer	3	2	+1
API errors / unparseable answers	0 / 0	0 / 0	—

All 198 dataset items paired successfully. GLM-5.2 alone answered 18 items correctly; GLM-5.3 alone answered 9. The resulting +4.55 percentage-point point estimate is not statistically significant at α=0.05 (two-sided exact McNemar p=0.1221). This was one temperature-0.6 pass, not repeated same-checkpoint trials, so stochastic self-flip noise was not estimated. Timing is secondary evidence: the GLM-5.3 window received two unrelated external completion requests in its first minute, while the GLM-5.2 window received none. Both windows reached 100% KV utilization and incurred transparent preemption/recompute. This can perturb timing, so the +1.8% generation-rate difference should not be read as a speed win.

Method and provenance

llm-inference-bench 0.4.29, built-in gpqa-diamond profile: all 198 items, deterministic per-item option shuffle, exact option-letter scoring. Dataset SHA-256: a8472c5a82ea2df8f209c17713aba1a6d409120c609ec0582dae0cb940c7e28c.
One pass at temperature 0.6, fixed concurrency 8, and 131,072 maximum completion tokens. No per-request top_p or reasoning_effort override; the server template defaulted to reasoning_effort=high.
Same 4× NVIDIA RTX PRO 6000 Blackwell Server Edition host, immutable image ID sha256:6e2475d0568fd110eeaa1193157c7662747e096b476b05ed71ab247e081e9b82, TP4/DCP4, 393,216-token model limit, dynamic-token NVFP4 MLA KV with FP8 RoPE, native probabilistic MTP3, and online EXL3 K6 for both runs.
Quant revisions: GLM-5.2 a350292cb2038f2c31732569a711a89e5d72fd46; GLM-5.3 6136d7ac2d9df3610987ce3e7f7f481a1eae3b98. Their tokenizer_config.json and generation_config.json files are byte-identical.
Full GLM-5.2 report SHA-256: f53ceb1e1e690264093a627394cd2f0536889991d97623fef766744ebc131218. The measured window contained 198 local completion requests, all HTTP 200, zero external completion requests, and no runtime allocator OOM.

The serving control changed only the mounted checkpoint. This is an A/B of the two published quantized artifacts, not a pure upstream-base-model attribution: this GLM-5.2 artifact declares routed-expert LDLQ calibration, while the GLM-5.3 comparator declares an identity-H/data-free quantization recipe.

FP8 Context 454,656 tok with partial online MXFP8 quant of dense layers, trading more KV for a bit of accuracy:

KLD 0.06862 - '--quantization-config={"linear":{"weight":"mxfp8"},"ignore":["re:.*\\.q_a_proj$$","re:.*kv_a_proj_with_mqa"]}'

KLD 0.06958 - '--quantization-config={"linear":{"weight":"mxfp8"},"shared_experts":{"weight":"mxfp8"},"ignore":["re:.*\\.fused_qkv_a_proj$","re:.*\\.q_a_proj$","re:.*kv_a_proj_with_mqa","re:.*\\.mlp\\.gate$","model.layers.78.eh_proj","lm_head"]}'

KLD ....... ONLINE_QUANT=exl3-b6

Mind that reasoning_effort:high, set to reasoning_effort:max

services:
  g52h:
    image: voipmonitor/vllm:gilded-gnosis-v20-vllme1e9426-si200c1db-fi801d57a-cu132-20260804-r28
    container_name: g52h
    ports:
      - "0.0.0.0:8000:8000"
    gpus: all
    shm_size: "32g"
    ipc: "host"
    ulimits:
      memlock: -1
      nofile: 1048576
    environment:
      - CUDA_VISIBLE_DEVICES=0,1,2,3
      - CUDA_DEVICE_MAX_CONNECTIONS=32
      - CUTE_DSL_ARCH=sm_120a
      - OMP_NUM_THREADS=16
      - PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
      - SAFETENSORS_FAST_GPU=1
      - NCCL_IB_DISABLE=1
      - NCCL_P2P_LEVEL=SYS
      - NCCL_PROTO=LL,LL128,Simple
      - VLLM_USE_FLASHINFER_SAMPLER=1
      - VLLM_USE_B12X_FP8_GEMM=0  # +kld
      - VLLM_USE_B12X_SPARSE_INDEXER=1
      - VLLM_USE_V2_MODEL_RUNNER=1
      - VLLM_ENABLE_PCIE_ALLREDUCE=1
      - VLLM_PCIE_ALLREDUCE_BACKEND=b12x
      - VLLM_PCIE_ONESHOT_ALLREDUCE_MAX_SIZE=64KB
      - VLLM_PCIE_ONESHOT_FUSED_ADD_RMS_NORM_MAX_SIZE=84KB
      - B12X_PCIE_DMA_FP8=0  # +kld
      - B12X_DENSE_SPLITK_TURBO=1
      - B12X_W4A16_TC_DECODE=1
      - B12X_MOE_FORCE_A16=1
      - VLLM_USE_AOT_COMPILE=1
      - VLLM_USE_BREAKABLE_CUDAGRAPH=0
      - VLLM_USE_FUSED_MOE_GROUPED_TOPK=1
      - VLLM_USE_B12X_MHC=1
      - B12X_MHC_MAX_TOKENS=16384
      - VLLM_USE_B12X_WO_PROJECTION=1
      - B12X_MLA_SM120_UNIFIED=1
      - VLLM_CACHE_DIR=/cache/jit/vllm
      - TRITON_CACHE_DIR=/cache/jit/triton
      - TORCH_EXTENSIONS_DIR=/cache/jit/torch_extensions
      - TORCHINDUCTOR_CACHE_DIR=/cache/jit/torchinductor
      - FLASHINFER_WORKSPACE_BASE=/cache/jit/flashinfer
      - XDG_CACHE_HOME=/cache/jit
      - TVM_FFI_CACHE_DIR=/cache/jit/tvm-ffi
      - GLOO_SOCKET_IFNAME=lo
      - NCCL_SOCKET_IFNAME=lo
      - VLLM_WORKER_MULTIPROC_METHOD=spawn
      - VLLM_PCIE_DMA_MIN_BYTES=6MB
      - VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=0  # +pp +kld
      - VLLM_B12X_MLA_SPEC_DECODE_MAX_Q=8
      - VLLM_USE_B12X_DCP_A2A=1
      - VLLM_DCP_A2A_MAX_TOKENS=16
      - VLLM_DCP_A2A_LARGE_BACKEND=ag_rs
      - VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_B12X_MLA_CKV_GATHER_MIN_TOKENS=512  # for VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=16384  # for VLLM_B12X_MLA_CKV_GATHER=1
      - VLLM_DCP_QUERY_SPLIT=1  # r14
      - VLLM_MEMORY_PROFILE_INCLUDE_ATTN=1
      - VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
      - TORCH_CUDA_ARCH_LIST=12.0a
      - FLASHINFER_CUDA_ARCH_LIST=12.0f
      - FLASHINFER_DISABLE_VERSION_CHECK=1
      - VLLM_USE_B12X_MOE=1
      - VLLM_CPP_AR_1STAGE_NCCL_CUTOFF=56KB
      - VLLM_CPP_AR_IGNORE_CUTOFF_MAX_ROWS=0
      - VLLM_RTX6K_FUSED_ALLREDUCE_ADD=0
      - VLLM_RTX6K_FUSED_ALLREDUCE_ADD_END_BARRIER=0
      - VLLM_DISABLE_SHARED_EXPERTS_STREAM=0  # v20 
      - VLLM_DISABLED_KERNELS=MarlinFP8ScaledMMLinearKernel
      - VLLM_DCP_GLOBAL_TOPK=1
      - VLLM_DCP_SHARD_DRAFT=1
      - VLLM_DCP_QUERY_SPLIT=0
      - VLLM_EXL3_TRELLIS_MIN_M=1
      - VLLM_EXL3_TRELLIS_MAX_M=48
      - VLLM_EXL3_TRELLIS_BLOCK_M=8
      - VLLM_EXL3_PREFILL_CHUNK=128
      - KV_FP8_ROPE=0  # +kld
      - VLLM_B12X_ABSORB_BMM=0
      - ONLINE_QUANT=exl3-b6
      - VLLM_EXL3_ONLINE_TRELLIS_BITS=6
      - VLLM_EXL3_ENCODER_SOURCE=/opt/exllamav3-python/exllamav3
      - VLLM_EXL3_ONLINE_CACHE_DIR=/cache/exl3-online
      - VLLM_EXL3_ONLINE_CACHE_MODE=readwrite
    volumes:
      - /data1/GLM-5.2-EXL3-TR3-3.42bpw:/model:ro
      - /data1/GLM-5.2-EXL3-TR3-3.42bpw.cache:/cache:rw
      - /data1/GLM-5.2-EXL3-TR3-3.42bpw.cache:/root/.cache:rw
      - /data1/GLM-5.2-EXL3-TR3-3.42bpw.cache:/container-tmp:rw
    entrypoint:
      - /bin/sh
      - -c
      - "unset NCCL_GRAPH_FILE NCCL_GRAPH_DUMP_FILE VLLM_B12X_MLA_EXTEND_MAX_CHUNKS && exec vllm serve \"$@\""
      - --
    command:
      - /model
      - --served-model-name=g52h
      - --trust-remote-code
      - --tensor-parallel-size=4
      - --decode-context-parallel-size=4
      - --dcp-comm-backend=a2a
      - --dcp-kv-cache-interleave-size=1
      - --quantization=exl3
      - --kv-cache-dtype=fp8
      - --attention-backend=B12X_MLA_SPARSE
      - --moe-backend=b12x
      - --load-format=safetensors
      - '--compilation-config={"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"],"pass_config":{"fuse_allreduce_rms":true}}'
      - --gpu-memory-utilization=0.971
      - '--quantization-config={"linear":{"weight":"mxfp8"},"ignore":["re:.*\\.q_a_proj$$","re:.*kv_a_proj_with_mqa"]}'  # KLD 0.06862
      - --max-model-len=128128
      - --max-num-seqs=16
      - --max-num-batched-tokens=2048
      - --max-cudagraph-capture-size=64
      - --enable-auto-tool-choice
      - --tool-call-parser=glm47
      - --reasoning-parser=glm45
      - --enable-prefix-caching
      - --enable-chunked-prefill
      - --no-async-scheduling
      - --enable-flashinfer-autotune
      - '--default-chat-template-kwargs={"reasoning_effort":"high"}'
      - '--hf-overrides={"use_index_cache":true,"index_topk_pattern":"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS"}'
      - '--speculative-config={"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton","draft_sample_method":"greedy"}'
#      - '--override-generation-config={"top_p":0.95,"repetition_penalty":1.18}'  # for temp=0.1 MMLU-Pro
      - --host=0.0.0.0
      - --port=8000

Source

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

Downloads last month: 340

Safetensors

Model size

176B params

Tensor type

BF16

F32

F16

I16

Model tree for willfalco/GLM-5.2-EXL3-TR3-3.42bpw

Base model

zai-org/GLM-5.2

Quantized

(149)

this model

Paper for willfalco/GLM-5.2-EXL3-TR3-3.42bpw

GLM-5: from Vibe Coding to Agentic Engineering

Paper • 2602.15763 • Published Feb 17 • 220