AEON vLLM Ultimate: one serving image for the DGX Spark

Every model I publish ends up on the same machine, one NVIDIA DGX Spark in the lab, and all of them need things from the serving layer that stock vLLM does not provide there. ghcr.io/aeon-7/aeon-vllm-ultimate is the answer: vLLM compiled from source for the Spark's GB10 chip and packaged with the speculative-decoding stack, a 4-bit KV cache and the runtime patches that keep the Spark from falling over. One container serves two 27B hybrid models, two mixture-of-experts models and a 31B AWQ checkpoint, and since the 0.27.1 omni build it also hears and speaks.

Nearly everything else in this library runs on this image, which makes this the flagship article and the one with the most scars. Compiling vLLM for a new chip is the easy part; keeping the build correct through release after release, while carrying patches upstream has not merged, is where the effort went, and most of what broke did so silently.

The current tip is vLLM 0.29.0 omni, tag 2026-09-11-v0.29.0-omni, and its release notes say to pin that dated tag until the :latest retag has been verified.

Why vLLM had to be built from source for sm_121a

The Spark is built around the GB10, a Grace Blackwell part with an aarch64 CPU and a GPU of compute capability 12.1, which the CUDA toolchain calls sm_121a. It has 128 GB of unified memory shared between CPU and GPU and runs CUDA 13. Stock vLLM runs on it; what the repository's fix list calls the default behaviour is a server that runs, then crashes under load and loses its drafting on long context, and under that sat a kernel question.

The kernel question was NVFP4, NVIDIA's 4-bit floating-point format in which each small block of values shares one scale. Blackwell tensor cores execute it natively, so a 27B model fits in about 20 GB of weights. Whether vLLM's CUTLASS NVFP4 kernels are compiled in at all depends on the architecture flag at build time, and without them an NVFP4 checkpoint on GB10 quietly falls back to Marlin instead of the tensor-core path. I build with TORCH_CUDA_ARCH_LIST=12.1a and ENABLE_NVFP4_SM100=0, which compiles in the SM120-family CUTLASS NVFP4 and FP8 kernels that GB10 dispatches to and leaves out the dead B200-only ones. The a suffix matters: a bare 12.1 preprocesses the FP4 kernels out, and the fallback to Marlin is quiet.

The payoff showed in the first fleet benchmark: on the v0.23.0 build, Qwen3.6-27B with the MTP-XS body and a DFlash drafter at twelve speculative tokens decoded single-stream at 34.1 to 56.1 tokens per second by prompt category, against about 10.5 from the stock nightly image, a baseline I still mark provisional pending a fresh vanilla rerun.

What broke before it worked

A clean compile was not a clean boot. Two idempotent runtime patches handle that: a lazy load that leaves the SM100-only MXFP4 symbols unresolved until a call that never comes on sm_121a, and capture-size alignment for speculative decoding under CUDA graphs.

Then the early images taught me that "it runs" means very little here: three defects, each silent until load arrived. The drafter's KV block table was padded to the batch while the attention kernel wanted it unpadded, and the engine died once 32 requests were in flight; upstream had fixed that for MTP in PR 43982, never for DFlash, so I ported it. With prefix caching on, the DFlash kernel wrote rejected draft tokens into the drafter's paged cache, shared blocks included, so acceptance decayed from a healthy 34 to 56 percent to zero over minutes to hours of traffic, a sixfold slowdown, 144 to 24 tokens per second, that only a restart healed. And sliding-window drafters ran every layer as full attention, so a request with more than 2,048 tokens of history got almost no accepted drafts. All three fixes are carried today, they are why the fleet scales from one to 64 concurrent requests without a crash, and every one of those defects would have passed a smoke test.

The fleet one container serves

Five bodies and a handful of drafters share the image. Qwen 3.6 27B and Qwen 3.8 27B are hybrids that mix Gated DeltaNet linear-attention layers with full attention. Qwen 3.6 35B-A3B and Gemma 4 26B-A4B are mixture-of-experts models, which activate only a few billion parameters per token. Gemma 4 31B DECKARD is a dense NVFP4 AWQ checkpoint. Ornith 1 35B, an agentic coding MoE on the Qwen 3.6 35B-A3B lineage, runs here too, on the Marlin path because its checkpoint lacks per-input global scales. Four bodies and Ornith have their own articles: Qwen 3.8 27B, Qwen 3.6 35B-A3B, Gemma 4 26B-A4B, Gemma 4 31B DECKARD and Ornith 1 35B.

Nearly every number below has a drafter attached: speculative decoding, in which a small drafter proposes several tokens and the large model verifies them in one forward pass, is the biggest lever on a chip whose decode is bound by memory bandwidth.

On one Spark, Gemma 4 26B-A4B in NVFP4 with DFlash at eleven speculative tokens served 1151.5 tokens per second aggregate at 128 concurrent requests with max-num-seqs set to 128, up from 754.7 at 32 and 981.3 at 64. Qwen3.6-27B on the 0.26 build, with MTP at eleven tokens and an FP8 KV cache, decoded single-stream at 39.6 tokens per second on code, 32.4 on math, 27.3 on reasoning, 23.0 on summaries, 17.4 on dialogue and 16.0 on prose. Since the 0.27.1 omni build the container also hears and speaks: the vllm-omni layer runs the Thinker, Talker and Code2Wav speech stages in one process, for 0.2 GB more image, 18.3 to 18.5 GB.

The upgrade ritual and the two silent killers

Each upstream release is merged onto the previous AEON branch as a three-way merge, aeon-v0.24.0 through aeon-v0.27.1 and then 0.29.0. The v0.27.1 merge brought 577 upstream commits with only seven conflicts, which is exactly the problem: git flags the lines it cannot reconcile and says nothing about the lines it reconciles wrongly.

So every merge is followed by the same audit: an enum that must still equal 5, a validity function that must still exist, the marker comments on the UMA clamp and the alignment patch, no as_strided anywhere in the NVFP4 splitter, a sweep for conflict markers, and py_compile over the tree. Two classes of silent killer wrote that list.

The first is the renumbered enum. When v0.25.0 landed, upstream had renumbered KVQuantMode.NVFP4 from 4 to 5, and the carried kernel hard-coded USE_NVFP4 = KV_QUANT_MODE == 4 in a region git never flagged; a plain merge would have left the NVFP4 comparison pointing at the old value 4. The second appeared at 0.27.1: the auto-merge combined the old +1 layer-id convention with upstream's new aux-1, producing eagle = target+2, so the drafter read the wrong layer and acceptance collapsed while every answer stayed correct. I resolved to upstream's convention after verifying that every fleet drafter is native format. Neither bug is caught by asking whether the server answers.

Carries and cherry-picks

A carry is a patch upstream has not merged that I re-apply at every release; a cherry-pick is an upstream commit that landed after the tag I built from. The Triton NVFP4 KV cache from PR 44389 is the only 4-bit KV path on sm_121a, worth about three times the KV capacity in the PR author's benchmarks, and I rewrote its slicing to be NHD-safe because upstream's as_strided math covers 7760 of 9216 bytes with five overlapping views on GB10, which defaults to the NHD layout: silent KV corruption. PR 40898 gives DFlash sliding-window attention on the V1 runner, PR 41703 is the prefix-cache fix from above, PR 46932 clamps the negative CUDA-graph memory estimates that inflate the KV budget on unified memory, and PR 48053 sets thread_local CUDA-graph capture at every capture site.

At 0.27.1 there were eight cherry-picks: the packed-KV zeroer stride fix in 50276, correctness-critical under the packed NVFP4 KV; the Gated DeltaNet gate alignment in 51812; the hybrid prefix-cache fix in 51843; and the DSpark set, 51602, 50693, 52288, 50910 and 49969. Three more were skipped on purpose, 49718 among them for an accuracy regression, and three old carries were dropped as upstreamed.

The image ends up with five speculation families. DFlash, the fleet default, drafts a block of tokens in parallel with non-causal attention. DFlash 2 landed upstream in 52816 after 0.27.1 and is cherry-picked as a pure-Python layer; it needs both the layer_type fix and the is_causal precedence fix or it silently becomes DFlash 1. DSpark attaches Markov heads to the target and needs --mamba-cache-mode align. MTP and Eagle3 round out the set.

The canary is acceptance length

Every merge bug above kept the output correct while the speed quietly left, so the health check I trust most is one metric, the drafter's mean acceptance length, or MAL, the average number of drafted tokens the target accepts per step. A mis-wired drafter keeps producing correct text while acceptance falls to about 0.3 percent. Healthy acceptance at 0.27.1 on GB10 with Model Runner V2 reads 2.88 to 3.38 tokens at 18.8 to 23.8 percent for Gemma with DFlash and 3.40 to 4.82 tokens at 20.0 to 31.8 percent for Qwen3.6-27B with DFlash.

MAL, not the acceptance percentage, is also how to choose num_speculative_tokens, and reading it needs the vllm:spec_decode_num_drafts_total counter. The drafter's KV footprint caps real concurrency: on Qwen3.8-27B at 16K context with an FP8 KV cache, 64 sequences and 0.75 memory utilization, the KV pool with no drafter was 1,521,371 tokens, which vLLM reports as 92.9 times concurrency, while DFlash at twelve tokens fell to 328,590, or 20.1 times, and that arm ran at 0.53 times the no-drafter baseline at 64 concurrent requests with a 22.6 second time to first token. High draft counts are for interactive single-stream use, low ones for batch.

The slim-image diet, 50.6 to 38.1 to 18.3 GB

The first image weighed 50.6 GB. It had inherited a base carrying a vLLM 0.20.1-era stack buried under Docker layer whiteouts, files deleted by a later layer that still ship in every pull, so its live filesystem was only about 22 GB. Inside /usr/local/cuda-13.0, about 3.8 GB of 4.3 GB was dead: 2.1 GB of static archives and 1.7 GB of shared libraries duplicating the pip nvidia-* wheels torch actually loads, proven via /proc/self/maps. A multistage build stopped shipping 11.5 GB of CMake objects and landed at 38.1 GB. A rebase onto nvidia/cuda:13.0.2-base-ubuntu22.04, a 383 MB base in place of the 2.52 GB runtime image, brought it to 18.3 GB: about 8.2 GB compressed to pull instead of 21.4 GB.

A base that small has to carry the runtime organs by hand, because no CUDA base image ships them and Triton compiles kernels at runtime: ptxas, nvcc and cicc; an ld.so.conf.d entry so the pip CUDA wheels resolve; the unversioned gcc, g++, cc, c++ and make aliases Triton shells out to; rdma-core for the two-Spark RoCE path; and the entrypoint, the pip and python3 symlinks, ninja and the OCI labels. The Dockerfile fails the build if any load-bearing binary is missing. Ubuntu 22.04 stayed because a newer base would have meant revalidating 242 native wheels, and the slim image was validated across all four fleet phases (Gemma DFlash, Qwen 3.6 DFlash, DSpark Markov heads, NVFP4 KV) before it became :latest.

Two Sparks, tensor parallel 2 and cross-node CUDA graphs

Tensor parallel splits each layer's weight matrices across GPUs, so every GPU computes a slice of every layer and the slices are combined with an all-reduce after each one. Two Sparks have no NVLink between them, only RoCE v2 over a direct 200 GbE link, and GB10 has no GPUDirect, so every all-reduce is staged through host memory at about 9 to 10 GB/s.

Upstream issue 46253 says multi-node GB10 has to run with --enforce-eager, which forbids CUDA graphs, the recorded decode steps that replay without launch overhead. PR 48053 changed this by setting the capture error mode to thread_local at every capture site, so one worker's NCCL activity no longer poisons another's capture. With that and NCCL 2.29.7, the pair captured 35 piecewise, 16 full and 15 DSpark graphs in 26 seconds, and both capture and replay held.

The result on the 0.27.1 image: 33.7 against 26.7 tokens per second single-stream with graphs against eager, a 26 percent gain, and 29.6 ms against 38.4 ms per output token, 23 percent lower. Aggregate throughput was about 80 tokens per second at a concurrency of 8, and the pooled KV cache reached 56.4 GiB, which vLLM reports as 24.9 times concurrency at 65k tokens per request. The soak was 64 requests and 18.5k tokens, then a 6-cycle 8-way burn-in, with zero NCCL errors in a 90-minute log sweep.

Two honest notes. I first credited the stability to NCCL 2.30.7; that was wrong, both images run 2.29.7 and the thread_local carry bears the load. And TP=2 is a capacity play, not a latency win: a model that fits one Spark is faster on one Spark. DeepSeek V4 on two DGX Sparks is the model that needs two.

Pull it and run it

You need a DGX Spark, or another consumer Blackwell card such as the RTX 5090, 5080 or RTX PRO 6000 (sm_120), since anything before Blackwell lacks the NVFP4 tensor cores; Docker with the NVIDIA container toolkit; and driver 580 or newer. Budget about 20 GB of disk for a 27B NVFP4 body, 3.3 GB for its drafter and 18.3 GB for the image. The RTX path is in AEON vLLM Ultimate on the RTX 5090.

Code, shelldocker pull ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-11-v0.29.0-omni
docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP /models/Qwen3.6-27B-AEON-MM-MTP
( cd /models/Qwen3.6-27B-AEON-MM-MTP && git lfs pull )
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/z-lab/Qwen3.6-27B-DFlash /models/Qwen3.6-27B-DFlash-drafter
( cd /models/Qwen3.6-27B-DFlash-drafter && git lfs pull )

The daily-driver recipe, one line so it survives copy and paste, is the Qwen3.6-27B multimodal NVFP4 MTP body with the z-lab DFlash drafter and an FP8 KV cache on port 8000.

Code, shelldocker run -d --name aeon-vllm --restart unless-stopped --gpus all --ipc=host --shm-size=16g --net=host -e VLLM_USE_FLASHINFER_SAMPLER=1 -v /models/Qwen3.6-27B-AEON-MM-MTP:/model:ro -v /models/Qwen3.6-27B-DFlash-drafter:/drafter:ro --entrypoint vllm ghcr.io/aeon-7/aeon-vllm-ultimate:latest serve /model --served-model-name aeon aeon-fast aeon-deep aeon-ultimate qwen36-ultimate aeon-ultimate-xs --dtype auto --quantization modelopt --kv-cache-dtype fp8_e4m3 --attention-backend TRITON_ATTN --max-model-len 229376 --max-num-seqs 16 --max-num-batched-tokens 32768 --gpu-memory-utilization 0.60 --enable-chunked-prefill --enable-prefix-caching --generation-config vllm --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --mm-encoder-tp-mode data --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":12,"attention_backend":"TRITON_ATTN"}' --trust-remote-code
curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"aeon","messages":[{"role":"user","content":"Hello."}],"max_tokens":64,"temperature":0.0}' | jq .choices[0].message.content

Four flags carry the weight. --attention-backend TRITON_ATTN appears twice because the drafter does not inherit the target's backend. --kv-cache-dtype fp8_e4m3 is the DFlash pairing: DFlash is non-causal, and NVFP4 KV needs a causal speculator such as MTP, so DFlash takes the FP8 KV cache. --quantization modelopt matches a ModelOpt body, while compressed-tensors bodies take --quantization compressed-tensors. --max-num-batched-tokens is mandatory with a drafter, and --gpu-memory-utilization 0.60 is deliberately low so the Qwen3 ASR and Qwen3 TTS sidecars can share the Spark.

Memory, long context and the second Spark

With the language model alone, 0.70 to 0.75 memory utilization is the safe range and 0.82 the ceiling, less about 0.03 for TP=2.

Qwen3.8-27B also reaches a million tokens of context through YaRN, needle-verified 3 of 3 at 271,904 and at 543,761 tokens, but only without a drafter, whose rope cache has 262,144 rows, and only with the override nested as rope_parameters under text_config; a flat override is a silent no-op, VLLM_ALLOW_LONG_MAX_MODEL_LEN is not the answer, and the exact override is in the 0.27.1 release notes.

For two Sparks on 0.29, FlashInfer's all-reduce is on by default and wrong for RoCE, so the environment below turns it off before --tensor-parallel-size 2 --disable-custom-all-reduce --compilation-config '{"pass_config":{"fuse_allreduce_rms":false}}' goes on the serve command. The HCA variable takes a single equals sign; a doubled == silently breaks the path.

Code, shellexport VLLM_ALLREDUCE_USE_FLASHINFER=0
export VLLM_USE_FLASHINFER_SAMPLER=0
export NCCL_IB_HCA=rocep1s0f0:1

Rollback is always the previous dated tag, because :latest always has a dated twin, and the 0.29.0 omni digest is sha256:2421bb1228a85370c1c50adb31f605c4361acf4d48d65282fcb919e74f34fae7. As the repository puts it, "Pin digests in production; dated tags beat folklore."

Sources

• Repository and release notes: AEON-7/vllm-ultimate-dgx-spark, 0.27.1 omni notes, 0.29.0 omni notes
• Container image: ghcr.io/aeon-7/aeon-vllm-ultimate
• Upstream vLLM: issue 46253, PR 44389, PR 48053, PR 52816
• Related AEON repositories: vllm-dflash (superseded), turboquant and the TurboQuant paper, Model-Optimizer
• Model cards: Qwen3.6-27B Multimodal NVFP4 MTP, Qwen3.6-27B NVFP4, Qwen3.6-27B Multimodal NVFP4 MTP-XS, Gemma-4-26B-A4B NVFP4, Qwen3.6-35B-A3B heretic NVFP4, Step-3.7-Flash AEON Ultimate NVFP4 (not smoke-tested)
• Drafters: z-lab/Qwen3.6-27B-DFlash, incoai/Qwen3.8-27B-DFlash2, AEON-7/DFlash-Qwen3.5-27B-Uncensored