Gemma 4 26B-A4B at NVFP4: a mixture-of-experts recipe for a DGX Spark

The DGX Spark is an odd and likable machine. It carries 128 GB of unified LPDDR5X memory and feeds it to the GPU at 273 GB/s, which is not much. Decoding a language model is mostly a bandwidth exercise, because every generated token means reading every active weight once. On the Spark that puts a dense 27B model at about 20 tok/s theoretical, and no kernel will change the arithmetic. A mixture-of-experts model does change it. Gemma 4 26B-A4B holds 26 billion parameters but sends each token through 8 of its 128 experts, so a token touches about 4 billion of them, roughly 2.8 GB of reads, for a theoretical ceiling near 97 tok/s on the same memory bus.

That gap is why this model is in my catalog. What follows is how I got it running well on one Spark: a predecessor I archived, a quantization that damaged the expert routers, a drafter whose first published numbers came from a broken path, and the two recipes that came out the other side.

What the model is

The weights are an uncensored fine-tune of Gemma 4 26B-A4B, built on TrevorJS/gemma-4-26B-A4B-it-uncensored. Uncensored here means abliterated: a direct edit of the weights that removes the learned refusal behavior rather than training against it. Gemma 4's architecture: 30 layers, 25 of them sliding-window attention and 5 full attention, 128 experts with top-8 routing, a 27-layer vision tower, native tool calling, and a 262,144-token context window. I quantized the language weights to NVFP4, NVIDIA's 4-bit floating-point format for Blackwell GPUs, in the compressed-tensors format, and kept the vision tower in BF16. The result is 15.3 GB on disk and 16.25 GB of VRAM once loaded.

It suits a local agent gateway or coding assistant on a single Spark: long context, many short-lived subagents, fast single-stream coding output. It is not for maximum raw throughput, and not for anyone chasing dense-model quality; the dense sibling, Gemma 4 31B Deckard Heretic, decodes at about 12 to 14 tok/s on the same box per its card.

The predecessor I archived

Before this model there was supergemma4-26b-abliterated-multimodal-nvfp4, a ModelOpt NVFP4 quantization of a different abliterated 26B mixture of experts, about 16.4 GB with its vision tower in BF16. It stays public, archived, for reproducibility, and every piece of it was a compromise. The FlashInfer MoE kernels could not handle its 704-per-expert intermediate dimension, so it ran on Marlin MoE. It used an FP8 KV cache (the attention state a model keeps for every token in its context), held only 4 sequences at 65,536 context on a patched vLLM 0.19.1 image, and took 4 to 5 minutes to start. The numbers were modest: 52.6 tok/s single-stream, 143.3 tok/s aggregate across 4 streams, and 16K-token prefill at 744 tok/s, a 21.5 s wait for the first token.

The part I still use is the calibration: a naive batch-size-1 pass over the experts projected to about 50 hours, and modelopt-fast-moe did it in 12 minutes. Compressed-tensors NVFP4 with native CUTLASS kernels and a DFlash drafter gives roughly three times the predecessor's single-stream speed and far higher concurrency.

The quantization that ate the routers

My first llmcompressor pass had a bug. The ignore patterns, re:.*visual.* and re:.*gate.*, were meant to keep the vision tower and the expert routers in high precision, but Gemma 4 does not name those tensors that way, so they matched nothing. The vision tower, the vision embedding and every MoE router were quantized to 4 bits with the rest. A router is the small layer that decides which 8 experts a token visits, and in 4-bit its choices turn degenerate. The fix restored 760 vision tensors, 120 router tensors and 4 embedding tensors from BF16, and the Hugging Face release contains the repaired checkpoint. A patched loader, gemma4_patched.py in the same repository, handles the expert-path, suffix and dimension mismatches when vLLM loads the compressed-tensors layout.

A drafter that guesses ten tokens at a time

Speculative decoding gets a bandwidth-bound model past its own ceiling: a small drafter proposes several next tokens and the big model verifies all of them in one forward pass, keeping the ones it would have produced itself. It is lossless, because the target checks every drafted token; only the speed varies. The share of drafted tokens the target keeps is called acceptance.

My drafter is z-lab/gemma-4-26B-A4B-it-DFlash, a 5-layer block-diffusion drafter that proposes a block of tokens at once. Four of its five layers use a 2048-token sliding window. It was trained with a block size of 16, z-lab ships it set to 15 speculative tokens, and I run 10. Whether 10 beats 15 is open: my earlier finding that it did by 20 to 50 percent came from the broken path described below, and a re-bench is pending. An older EAGLE-style path with my Gemma 4 E4B drafter on vLLM 0.19.1 claimed 50 tok/s single-stream and 1,430 tok/s aggregate on the Spark; DFlash replaced it.

Three defects in the drafter path

The card now carries a sentence I did not enjoy writing: all previously published DFlash numbers for this model were measured on a defective drafter path. First, rejected drafted tokens still had their KV entries written into the drafter's cache, and prefix caching, which reuses the cache of a shared prompt prefix, made the corruption persistent; acceptance, which had been 34 to 56 percent, fell to 0.0 percent, and decode fell from 144 to 24 tok/s. Second, the drafter's 2048-token sliding window ran as full attention, so acceptance was about zero beyond 2k tokens of context. Third, the Gemma 4 embedding normalizer and logit softcap were missing, which held the mean acceptance length at 4.4 to 6.6 where z-lab publishes 6.1 to 8.6. The pre-fix image also crashed at 32 or more concurrent requests, fixed by slicing the drafter block table to the unpadded batch.

The historical numbers still matter, as floors. On the pre-fix image :2026-06-04-pr44389, with the flex_attention drafter at n=10 and a BF16 KV cache, a short-context sweep dated 2026-06-09 gave 144.2 tok/s single-stream coding and a peak of 1,724 tok/s aggregate coding at 128 concurrent requests. Those are the "144 single, 1,724 peak" figures people quote back to me. The drafter backend they used depends on the image tag: on the pre-fix tag only flex_attention loads, while on :latest it crashes at the first request and flash_attn is required, so the n=10 flex recipe belongs to the old image only.

What the fixed image does today

On the vLLM 0.23.0 build with the drafter on flash_attn and the body on triton_attn, measured on the Spark with short prompts at one concurrent request (c=1) and n=10, coding decodes at 155.8 tok/s. Extraction and JSON run at 202.4 tok/s with 77.5 percent acceptance, and prose at 49.8 tok/s with 11.1 percent: the drafter thrives on predictable text. For coding, the median time to first token is 83 ms and the median time per output token 6.4 ms. The peak aggregate on the fixed image is 1,937 tok/s of coding at c=64, with no crash.

Long prompts cost acceptance. At about 16k tokens of context and c=1, coding runs at 128.2 tok/s with 58.7 percent acceptance; at about 32k it is 92.8 tok/s and 46.7 percent. The card's headline, up to 4.3x faster single-stream decode and about 2.5x on average over a stock vLLM 0.20.1 baseline of 47 to 49 tok/s, is provisional because the baseline is.

The recipe I run

Pull the container, download the target and the drafter, and serve. The container is the AEON vLLM Ultimate image from its own article; :latest resolves to :2026-07-01-v0.24.0 (vLLM 0.24.0 built from source for sm_121a, with the NVFP4 KV cache and DFlash fixes), and the rollback tag is :2026-06-18-v0.23.0-dflashfix.

Code, shelldocker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest
huggingface-cli download AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 --local-dir ./models/gemma4
huggingface-cli download z-lab/gemma-4-26B-A4B-it-DFlash --local-dir ./models/gemma4-dflash

docker run --gpus all --ipc host --network host \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e TORCH_MATMUL_PRECISION=high \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_TEST_FORCE_FP8_MARLIN=0 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -v "$PWD/models/gemma4:/models/gemma4:ro" \
  -v "$PWD/models/gemma4-dflash:/models/gemma4-dflash:ro" \
  --entrypoint vllm \
  ghcr.io/aeon-7/aeon-vllm-ultimate:latest \
  serve /models/gemma4 \
    --served-model-name gemma4-aeon-uncensored gemma4-fast gemma4-deep \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --dtype auto \
    --quantization compressed-tensors \
    --linear-backend flashinfer_cutlass \
    --moe-backend cutlass \
    --attention-backend triton_attn \
    --max-model-len 262144 \
    --max-num-seqs 64 \
    --max-num-batched-tokens 32768 \
    --gpu-memory-utilization 0.65 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser gemma4 \
    --reasoning-parser gemma4 \
    --speculative-config '{"method":"dflash","model":"/models/gemma4-dflash","num_speculative_tokens":10,"attention_backend":"flash_attn"}'

Why each flag is there

The body runs on triton_attn because Gemma 4 uses a head dimension of 256 on its sliding layers and 512 on its global layers, which breaks the other backends; the drafter runs on flash_attn, which the current image requires. Tensor parallel, which splits each layer across several GPUs, is 1: the Spark has one. There is no --kv-cache-dtype fp8: the DFlash drafter is non-causal and needs a BF16 KV cache, and the NVFP4 KV cache is an opt-in for causal speculators only. --reasoning-parser gemma4 moves the thinking into a separate reasoning_content field (without it the trace leaks and quality drops), and --tool-call-parser gemma4 with --enable-auto-tool-choice gives native tool calling. --max-num-seqs 64 at the full 262,144-token context suits one long chat plus many short subagents; --max-num-batched-tokens 32768 with chunked prefill, which feeds a long prompt in pieces between decode steps, keeps a large paste from stalling decode. Prefix caching is safe on the current image.

The flag to respect is --gpu-memory-utilization 0.65. The Spark's memory pool is shared by the CPU and the GPU, so the card says to stay between 0.6 and 0.7 and go lower with co-located services, high concurrency, fp16 KV or DFlash; above about 0.8 the box page-thrashes and stalls, and even 0.85 stalls. The older GitHub README still shows 0.80; the Hugging Face card is newer.

The bench recipe behind 816 tok/s at c64

Production is tuned for one person and a swarm of subagents; the entry on the Aeon Bench board is a different animal: 816 tok/s aggregate at 64 concurrent streams on the Spark, a composite score of 65.7, DFlash active at n=11. It keeps the production parsers, triton_attn attention, the CUTLASS MoE backend, 64 sequences, prefix caching, chunked prefill and 32,768 batched tokens, but runs --max-model-len 65536, --gpu-memory-utilization .8, --kv-cache-dtype auto and 11 speculative tokens. The .8 sits at the edge of the card's warning, and it is not a production setting.

It did not start there. The seed-era recipe for the same model, with n=11 and 32 concurrent streams, produced about 33 tok/s aggregate; the retuned recipe reached 517, and the board entry 816. The tuning log holds the lesson: a high-n drafter at high concurrency starves batching. Speculative decoding is a workload choice, not a free upgrade. High n buys single-user speed; low n, or no drafter at all, buys throughput. For raw throughput, drop the drafter: the card's stock baseline, vanilla vLLM 0.20.1 at 32K context with --max-num-seqs 256, peaked at 3,674.70 tok/s on extraction at c=256 with 0 errors. I run those sweeps in the Recipe Lab.

Caveats before you run it

This is an uncensored model, and the card's disclaimer stands: the operator assumes full and sole responsibility for outputs, safeguards, filtering and legal compliance. The weights inherit the Gemma license. Use the container rather than Python wheels, because wheels and prebuilt FlashInfer are risky on sm_121a and ARM64, and my tuning notes say FlashInfer attention is broken on GB10; triton_attn and flash_attn work. Gemma 4 needs transformers v5 or newer, so pin it. For neighbors, see the Qwen 3.6 35B-A3B DFlash recipe and the catalog.

Sources

Model: AEON-7/Gemma-4-26B-A4B-it-Uncensored-NVFP4 on GitHub and on Hugging Face; base weights TrevorJS/gemma-4-26B-A4B-it-uncensored; drafter z-lab/gemma-4-26B-A4B-it-DFlash.

Older drafter path: AEON-7/Gemma-4-E4B-it-Uncensored-NVFP4 on GitHub and on Hugging Face.

Archived predecessor: AEON-7/supergemma4-26b-abliterated-multimodal-nvfp4 on GitHub and on Hugging Face, and AEON-7/modelopt-fast-moe.

Container and benchmarks: AEON-7/vllm-ultimate-dgx-spark (the ghcr.io/aeon-7/aeon-vllm-ultimate image); AEON-7/Aeon-Bench-Pod and aeon-bench.com.

Dense sibling and license: AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4; the Gemma terms of use.