Instructions to use AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4")
model = AutoModelForCausalLM.from_pretrained("AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4

SGLang

How to use AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4
```

AxionML Nemotron-3-Ultra-550B-A55B-NVFP4

Mirrored by AxionML for open-source serving and deployment use cases. Part of AxionML's effort to provide ready-to-serve quantized models for the community.

Released by NVIDIA. The weights in this repository are an unmodified copy of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 (revision 02462641f13d3af838b904f48195b9bb8a1e4ebc). All credit belongs to NVIDIA.

NVIDIA Nemotron 3 Ultra is a frontier-scale reasoning and agentic model (550B total, 55B active) built on a hybrid LatentMoE architecture — interleaved Mamba-2 and MoE layers with select attention layers and Multi-Token Prediction. This checkpoint is NVIDIA's own release: the model was pre-trained with an NVFP4 recipe and NVIDIA publishes the NVFP4 weights directly.

About NVFP4 quantization: NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity while higher-precision FP32 accumulation protects dot-product accuracy.

Ready for commercial and non-commercial use under OpenMDW-1.1.

Model Summary


Architecture	Hybrid LatentMoE: Mamba-2 + MoE + attention, with MTP
Total Parameters	550B
Activated Parameters	55B
Context Length	Up to 1M tokens
Minimum GPUs	4x B200 / 4x GB200 / 4x B300 / 4x GB300 / 8x H100
Reasoning	On/off via chat template (`enable_thinking`)
Checkpoint Size	~352 GB

Evaluation Results

Benchmark	BF16	NVFP4
Terminal-Bench 2.1	56.4	53.9
SWE-Bench Verified	70.7	69.5
SWE-Bench Multilingual	67.7	69.1
GDPVal	46.7	47.9
TauBench V3 (avg)	70.9	70.3
BrowseComp	44.4	41.4
GPQA (no tools)	87.0	87.9
HLE (no tools)	26.7	26.1
SciCode (subtask)	44.6	43.5
IFBench (prompt)	81.7	82.3
AA-LCR	65.4	65.5
RULER 1M	94.7	94.0

Scores reported by NVIDIA (NeMo Evaluator SDK). See the upstream card for the full table.

Quantization Details

Precision: NVFP4 pre-training recipe; released checkpoint is mixed NVFP4 (MoE linears) + FP8, with latent projections, MTP, attention projections and embeddings in higher precision
KV cache: FP8

Usage

Deploy with SGLang

python3 -m sglang.launch_server \
    --model-path AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 \
    --tp-size 4 \
    --ep-size 4 \
    --context-length 262144 \
    --mem-fraction-static 0.85 \
    --mamba-scheduler-strategy extra_buffer \
    --mamba-backend flashinfer \
    --attention-backend trtllm_mha \
    --reasoning-parser nemotron_3 \
    --tool-call-parser qwen3_coder \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 5 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 6 \
    --trust-remote-code

Deploy with vLLM

vllm serve AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4 \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --enable-expert-parallel \
    --kv-cache-dtype fp8 \
    --max-model-len 262144 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --reasoning-parser nemotron_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --mamba-ssm-cache-dtype float16 \
    --mamba-backend flashinfer \
    --speculative-config '{"method": "nemotron_h_mtp", "num_speculative_tokens": 5}'

Tested upstream on 4x B200 with lmsysorg/sglang:v0.5.13 and vllm/vllm-openai:v0.22.0. For 1M context set SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 / VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. With tools, send "chat_template_kwargs": {"enable_thinking": true, "force_nonempty_content": true}.

Limitations

The base model was trained on data that may contain toxic language and societal biases. The quantized model inherits these limitations. It may generate inaccurate, biased, or offensive content. Please refer to the original model card and the upstream quantized model card for full details.

Credits

Base model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 by NVIDIA
Mirror: AxionML

Downloads last month: 345

Safetensors

Model size

303B params

Tensor type

BF16

F8_E4M3

F32

Model tree for AxionML/Nemotron-3-Ultra-550B-A55B-NVFP4

Base model

nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16

Quantized

(18)

this model