Hmm⌠Even outside Jetson, if the practical memory budget is only around 8 GB, I think using LLMs much beyond the 7Bâ8B class is already pretty roughâŚ
Below is my current read of the situation for Jetson Orin Nano 8GB + GGUF + llama.cpp, especially for trying to run Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf.
Short version
I would not treat this as a âbad GGUF fileâ first.
I would treat it as a memory-budget problem plus a Jetson runtime-contract problem:
- Orin Nano 8GB has 8GB shared LPDDR5 memory, not â8GB VRAM + lots of normal system RAMâ.
- MoE âactive parametersâ help compute cost, but the runtime still has to deal with the full quantized weight file, KV cache, CUDA buffers, OS memory, file cache, and server overhead.
Mellum2-12B-A2.5B-Instruct is a 12B total / 2.5B active MoE model, so âA2.5Bâ does not make it equivalent to a 2.5B dense model for memory residency.- For this board, I would generally start from 2Bâ4B models, then maybe test 8B low-bit / edge-MoE models only after the Jetson stack and llama.cpp settings are known-good.
- I would keep Mellum2 on this board in the âinteresting experimentâ bucket, not the âpractical recommendationâ bucket.
Useful official/semi-official starting points:
Why this is hard on Orin Nano 8GB
The key issue is not just parameter count. It is the whole runtime memory picture.
| Component |
Why it matters on Orin Nano 8GB |
| GGUF model weights |
The quantized file still has to be mapped/loaded and used by the runtime. An 8GB-class GGUF is already too close to the total memory pool. |
| KV cache |
Grows with context length, number of layers, KV heads, cache dtype, and parallel slots. Long context is expensive even if the weights fit. |
| CUDA buffers |
GPU offload needs extra temporary and persistent CUDA allocations. |
| OS + desktop + services |
They consume memory from the same 8GB pool. |
| File cache / mmap behavior |
Can help or hurt depending on pressure; it does not create more physical RAM. |
| Swap |
Can prevent crashes, but if actual inference is paging heavily, performance can become unusable. |
| CPU offload |
On a desktop with 32GB+ RAM, this can be a real escape hatch. On Orin Nano 8GB, CPU and GPU share the same small memory pool. |
This is why a model that is âinteresting on a 32GB RAM PCâ can still be a bad fit for Orin Nano 8GB.
Mellum2-specific issue
The model itself is interesting. It is not a toy model.
JetBrains/Mellum2-12B-A2.5B-Instruct is described as a coding/software-engineering model with:
- 12B total parameters
- 2.5B active parameters per token
- MoE with 64 experts and 8 active experts
- 131,072-token context
- GQA with 32 Q heads and 4 KV heads
- software engineering, code generation/editing, tool use, function calling, and agentic workflows as core target use cases
Sources:
But for Orin Nano 8GB, the important part is:
12B total weights still matter. â2.5B activeâ helps per-token compute, but it does not magically make the full model fit like a 2.5B dense model.
That is also why larger Qwen MoE models can be interesting on a 32GB RAM CPU box, but not necessarily on Orin Nano 8GB. On a PC, llama.cpp can sometimes make good use of system RAM. On Orin Nano 8GB, the âsystem RAMâ is still the same tiny shared pool.
Practical interpretation of NVIDIA/Jetson guidance
NVIDIAâs own forum practical guide says that, using llama.cpp with Q4_K GGUF, Orin Nano 8GB can fit approximately:
| Class |
Approximate upper range from the guide |
| LLMs |
up to around 10B parameters |
| VLMs |
up to around 4B parameters |
Source: AI Models That Run on Jetson Orin Nano Super 8GB â A Practical Guide
I would read that as an upper-bound / best-case sizing guide, not as â10B will be comfortableâ.
A more conservative operational reading is:
| GGUF size / model class |
My expectation on Orin Nano 8GB |
| <= 2GB |
Usually comfortable if the runtime is configured correctly. |
| 2GBâ4GB |
Best practical target zone. |
| 4GBâ5.5GB |
Possible, but context/batch/KV/cache/offload settings matter a lot. |
| 5.5GBâ7GB |
Experimental; expect tuning and instability. |
| 7GB+ |
Usually not a good practical recommendation on this board. |
| 8GB-class GGUF |
The model file itself is too close to the total memory pool. Expect CUDA allocation failures, very short context, or unusable CPU fallback. |
So, for this board, I would not start with a 12B MoE Q4 model. I would start with 2Bâ4B and then test 8B low-bit only if the baseline is stable.
First thing to check: JetPack / L4T version
There is a very relevant recent NVIDIA forum thread where Gemma 4 E4B Q4_K_M failed with CUDA OOM on Orin Nano, then NVIDIA identified a known memory issue in r36.4.7 that was fixed in r36.5.
Source:
So before changing models, I would check the Jetson software stack:
cat /etc/nv_tegra_release
dpkg-query --show nvidia-l4t-core
If this shows an affected JetPack/L4T combination, update the runtime first. Otherwise model-level debugging can be misleading.
Recommended baseline: use the Jetson-oriented llama.cpp container first
Before doing custom builds, I would first try the NVIDIA AI IoT / Jetson AI Lab llama.cpp container.
The NVIDIA forum guide points to this container tag:
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin
Source:
Jetson AI Labâs Gemma 4 E2B page also says the model is configured to run on Jetson with vLLM and llama.cpp, and describes E2B as the edge-first low-memory member of the Gemma 4 family:
A conservative smoke test would be something like:
sudo docker run -it --rm --pull always \
--runtime=nvidia \
--network host \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
llama-server \
-hf unsloth/gemma-4-E2B-it-GGUF:Q4_K_S \
-c 1024 \
-ngl auto \
-fa on \
-ctk q8_0 \
-ctv q8_0 \
-b 64 \
-ub 32 \
--host 0.0.0.0 \
--port 8080
This is not because Gemma 4 E2B is necessarily the best coding model. It is because it is a good known-good Jetson baseline.
If this fails, I would suspect the Jetson stack, container/runtime, CUDA setup, power mode, memory pressure, or JetPack/L4T version before blaming Mellum2.
Runtime tuning checklist
These are the knobs I would try, in order.
1. Reduce context length aggressively
Do not start with 8K, 32K, or 128K context on Orin Nano 8GB.
Start with:
-c 512
Then only increase if stable:
-c 1024
-c 2048
For a very marginal model:
-c 256
In llama.cpp, --ctx-size directly affects KV cache memory. Long context is not free.
Reference:
2. Reduce batch and micro-batch
For memory-constrained Jetson runs, I would not use large defaults.
Try:
-b 64 -ub 32
If it still fails:
-b 32 -ub 16
This may slow prompt processing, but it can avoid allocation failures.
Reference:
3. Quantize the KV cache
Try safer KV cache quantization first:
-ctk q8_0 -ctv q8_0
If the model is still too tight:
-ctk q4_0 -ctv q4_0
I would not start with q4_0 unless memory is really tight. It can be less robust, but it is worth trying for a last-resort fit test.
Reference:
4. Use -fit / fitting mode when available
Recent llama.cpp builds have model-fitting behavior in the server path. The Gemma 4 E4B NVIDIA forum log explicitly shows:
common_init_result: fitting params to device memory
llama_params_fit_impl: projected to use 5533 MiB of device memory vs. 6387 MiB of free device memory
Source:
So I would let llama.cpp fit parameters when using the Jetson container, unless debugging a suspected fitting bug.
5. Do not blindly maximize GPU layers
-ngl 999 or --n-gpu-layers 99 can be fine when memory is enough. On Orin Nano 8GB it can also push the system over the edge.
Try:
-ngl auto
or step manually:
-ngl 8
-ngl 16
-ngl 24
If CPU-only works but GPU mode fails, the model may be too tight for the CUDA path even if the weights can be mapped.
6. Try MoE CPU expert offload only as an experiment
llama.cpp has MoE-related CPU offload options such as --cpu-moe / --n-cpu-moe in recent builds.
This can help on systems with real CPU RAM headroom, for example:
- desktop PC with 32GB/64GB RAM
- small GPU with large system RAM
On Orin Nano 8GB, the CPU and GPU share the same limited memory pool, so this is not a true escape hatch. Still, for MoE models, it may be worth testing:
-ncmoe 20
But I would not expect it to rescue an 8GB-class Mellum2 GGUF on an 8GB Jetson.
Reference:
7. Avoid --mlock on low-memory Jetson
--mlock can be useful on some systems, but on an 8GB Jetson it can make memory pressure worse by preventing paging.
In this case I would avoid:
--mlock
Reference:
8. Try --no-mmap only as a late diagnostic
mmap behavior can interact with page cache and memory pressure. I would keep the default first, then try:
--no-mmap
only if debugging load behavior.
Reference:
System-level memory tuning
Jetson AI Lab has a specific RAM optimization guide:
The jetson-containers setup guide also has practical advice on swap and disabling the desktop GUI:
Disable the desktop GUI
If using the desktop environment, stopping it can free a meaningful amount of memory.
Temporary:
sudo init 3
Restore:
sudo init 5
For Orin Nano 8GB, even a few hundred MB can matter.
Add NVMe swap
Swap is not a performance solution, but it can prevent immediate OOM kills and help identify whether the failure is a hard fit problem.
Example:
sudo systemctl disable nvzramconfig
sudo fallocate -l 16G /mnt/16GB.swap
sudo mkswap /mnt/16GB.swap
sudo swapon /mnt/16GB.swap
Persist in /etc/fstab:
/mnt/16GB.swap none swap sw 0 0
Prefer NVMe over SD card for swap if available.
Clear caches before a load test
For repeatable testing:
sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
This does not fix the model, but it makes memory-pressure tests cleaner.
Use max power / clocks after the model actually fits
Power mode will not make an oversized model fit, but it matters once the model runs.
Check and set power mode:
sudo nvpmodel -q
sudo nvpmodel -m <mode>
sudo jetson_clocks
The exact mode depends on the installed JetPack / board configuration.
If building llama.cpp yourself
If not using the Jetson container, build with CUDA explicitly.
cmake -B build \
-DGGML_CUDA=ON \
-DCMAKE_CUDA_ARCHITECTURES=87
cmake --build build --config Release -j
Orin is compute capability 8.7 / SM87, and NVIDIA forum logs for Orin show ARCHS = 870.
References:
If experimenting with memory-constrained CUDA behavior, these build options may be worth knowing about:
-DGGML_CUDA_FORCE_MMQ=ON
-DGGML_CUDA_FA_ALL_QUANTS=ON
But I would first test the NVIDIA container before going deep into custom builds.
A realistic Mellum2 last-try profile
If you still want to try Mellum2 just to see whether it can load, I would use an extreme low-memory profile.
This is a load experiment, not a practical recommendation:
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
llama-server \
-m ./Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf \
-c 256 \
-ngl auto \
-fa on \
-ctk q4_0 \
-ctv q4_0 \
-b 32 \
-ub 16 \
--host 0.0.0.0 \
--port 8080
If your llama.cpp build supports MoE CPU expert offload:
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
llama-server \
-m ./Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf \
-c 256 \
-ngl auto \
-fa on \
-ctk q4_0 \
-ctv q4_0 \
-b 32 \
-ub 16 \
-ncmoe 20 \
--host 0.0.0.0 \
--port 8080
Again, I would not expect this to become a good daily-use setup. If it loads but runs very slowly, that is still useful information: it confirms that the bottleneck is not simply âunsupported modelâ, but practical memory and bandwidth limits.
Better model targets for this board
For Orin Nano 8GB, I would pick models by actual GGUF size and runtime stability, not leaderboard score alone.
Practical shortlist
| Model family |
Why it is more realistic than Mellum2 Q4 |
Suggested quant range |
| Gemma 4 E2B |
Explicit Jetson AI Lab support; edge-first low-memory target. |
Q4_K_S / Q4_K_M |
| Gemma 4 E4B |
Supported on Jetson, but already near the upper edge on Nano 8GB. |
Q3 / Q4, short context |
| Qwen 4B-class models |
Good instruction/coding/tool-use balance if GGUF support is current. |
Q4 / IQ4 / Q3 |
| Qwen2.5-Coder-3B |
Older but still a useful coding-specific baseline. |
Q4 / Q5 |
| StarCoder2-3B |
Better for FIM/completion-style coding than chat-agent use. |
Q4 / Q5 |
| LFM2 / LFM2.5 8B-A1B |
Edge-oriented MoE; interesting if Q3/IQ4 GGUF fits. |
Q3 / IQ4 |
| Granite Code 3B |
Practical code model, enterprise-friendly posture. |
Q4 / Q5 |
| OpenCoder 1.5B/8B |
Code-specialized candidate; 8B needs low-bit. |
1.5B Q5/Q8, 8B Q3 |
Models I would not recommend first on Orin Nano 8GB
| Model type |
Why I would avoid it first |
| Mellum2 Q4_K_M |
12B total MoE; Q4 file is too close to / above practical memory budget. |
| 12B+ dense models |
Even if they load, context and speed will likely be poor. |
| 30B-A3B / 35B-A3B MoE |
Interesting on 32GB RAM PCs, not on 8GB shared-memory Jetson. |
| Qwen3-Coder-Next 80B-A3B |
Very interesting model, wrong memory class for this board. |
| Devstral-style 20B+ coding agents |
Good benchmark story, wrong memory budget for Orin Nano 8GB. |
| Long-context runs |
Context length will often fail before âmodel intelligenceâ matters. |
Suggested debugging sequence
I would debug in this order.
| Step |
Goal |
Command / action |
| 1 |
Confirm JetPack/L4T |
cat /etc/nv_tegra_release |
| 2 |
Confirm available memory |
free -h, tegrastats |
| 3 |
Stop desktop GUI |
sudo init 3 |
| 4 |
Add NVMe swap |
16GB swap on NVMe if available |
| 5 |
Test known-small GGUF |
Gemma 4 E2B or a 2Bâ3B GGUF |
| 6 |
Test 4B-class model |
Qwen 4B / Gemma E4B / coder 3B |
| 7 |
Tune ctx/KV/batch |
-c 512, -ctk q8_0, -b 64, -ub 32 |
| 8 |
Try 8B low-bit |
only after baseline is stable |
| 9 |
Try Mellum2 |
only as a load experiment |
| 10 |
Decide |
if small models work but Mellum2 fails, the conclusion is memory budget, not setup failure |
Example âknown-good firstâ profile
sudo docker run -it --rm --pull always \
--runtime=nvidia \
--network host \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
llama-server \
-hf unsloth/gemma-4-E2B-it-GGUF:Q4_K_S \
-c 1024 \
-ngl auto \
-fa on \
-ctk q8_0 \
-ctv q8_0 \
-b 64 \
-ub 32 \
--host 0.0.0.0 \
--port 8080
If this works, move to a 3B/4B coding model.
If this fails, Mellum2 is not the right next test. Fix the Jetson runtime first.
Example â4B-classâ profile
sudo docker run -it --rm --pull always \
--runtime=nvidia \
--network host \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
ghcr.io/nvidia-ai-iot/llama_cpp:latest-jetson-orin \
llama-server \
-hf <repo>:<quant> \
-c 512 \
-ngl auto \
-fa on \
-ctk q8_0 \
-ctv q8_0 \
-b 64 \
-ub 32 \
--host 0.0.0.0 \
--port 8080
If it fails:
-c 256
-ctk q4_0
-ctv q4_0
-b 32
-ub 16
If it only works with -c 256, it may technically run but not be useful for coding assistance.
How I would summarize the recommendation
For Orin Nano 8GB, I would frame it this way:
Mellum2 is an interesting coding MoE, but the Orin Nano 8GB memory budget is probably the wrong target for the Q4_K_M GGUF. The board has 8GB shared memory, so CPU offload is not the same escape hatch that it is on a 32GB RAM PC. I would first validate the Jetson stack with NVIDIAâs Jetson-oriented llama.cpp container and a known-small model such as Gemma 4 E2B. Then I would test 3Bâ4B coding/instruct models, or possibly 8B-A1B / 8B low-bit models. I would only try Mellum2 with very short context, quantized KV cache, tiny batch sizes, and possibly unified-memory fallback as a load experiment.
Resource list
Jetson / NVIDIA
Containers / setup
llama.cpp / GGUF
Mellum2
Model alternatives to check
Final practical take
I would not spend too much time trying to force Mellum2-12B-A2.5B-Instruct-Q4_K_M onto an Orin Nano 8GB.
I would do this instead:
- Make sure JetPack/L4T is not on a known-problem release.
- Use the Jetson-oriented llama.cpp container.
- Disable GUI and add NVMe swap if possible.
- Validate with Gemma 4 E2B or another 2Bâ3B GGUF.
- Try 3Bâ4B coding models.
- Try 8B low-bit / 8B-A1B edge-MoE models only after that.
- Treat Mellum2 as an experiment, not the practical target.
For this board, the useful question is probably not:
âCan I run a 12B MoE somehow?â
It is more:
âWhich 2Bâ4B or low-bit 8B model gives the best coding usefulness per GB of actual Jetson memory?â