AEON vLLM Ultimate RTX: vLLM for the RTX 5090 and Blackwell sm_120

I build vLLM images for two kinds of Blackwell, and they do not mix. The first is the DGX Spark, a small aarch64 box whose GB10 chip reports compute capability sm_121a and shares one pool of memory between CPU and GPU. The second is the card most people can actually buy: the GeForce RTX 5090 with 32 GB of dedicated GDDR, and its workstation sibling, the RTX PRO 6000 Blackwell with 96 GB. Both of those report sm_120. AEON vLLM Ultimate RTX is the image for the second kind: a slim linux/amd64 container compiled for CUDA 13 with the arch list "12.0;12.0a", so the NVFP4 and FP8 kernels actually land on these cards.

NVFP4 is NVIDIA's 4-bit floating-point format, with a scale stored for every small block of weights, and it is the format my Qwen3.8-27B AEON Ultimate checkpoint ships in. A 27B model in NVFP4 leaves a 32 GB card enough room for a 131k-token window, which is the whole point. The Spark half of this story has its own article. This is the RTX half, scars included.

Two Blackwells that look alike

Compute capability is the version number NVIDIA gives each GPU's instruction set, and CUDA kernels are compiled against it. The GB10 in a Spark is sm_121a. The GB202 in an RTX 5090, and the chip in an RTX PRO 6000 Blackwell, are sm_120. One digit and one letter apart, and a container built for one is useless on the other. The differences stack three deep: architecture (sm_121a against sm_120), instruction set (aarch64 against amd64, so the Spark image would not even start on an RTX host) and memory model (unified on the Spark, dedicated VRAM on the cards). The RTX image carries no GB10 12.1a code and no unified-memory clamp, because a 5090 has nothing to clamp.

So I keep two packages and never let them touch. The Spark image is ghcr.io/aeon-7/aeon-vllm-ultimate; this one is ghcr.io/aeon-7/aeon-vllm-ultimate-rtx. They share the same AEON 0.27.1 merge, the NVFP4 kernels, the FlashInfer trio and the torch 2.13+cu130 pins, but they are different builds from the first layer up. The RTX labels say so: ai.aeon.not_for=DGX Spark GB10 sm_121a, B200 sm_100, B300 sm_103. Two smaller rules follow. Ada 8.9, the 4090 class, is not a target. And the arch list is always spelled out, never "native", which compiles for whatever card the build machine happens to have; a build that works on my bench by accident is not one I can hand to you.

What is in the slim tag

The base is a CUDA 13.0.2 devel stack. On it sits the AEON 0.27.1 branch of vLLM, aeon-v0.27.1 at commit 34aeb2303114bf3d8b2ecd582b7301868691735a, on top of upstream v0.27.1; the version label reads 0.27.1+aeon.sm120.rtx. Vision and audio extras ship in this public tag (vllm-audio-extra, av, soundfile, soxr, librosa, torchaudio).

Slim means I removed what sm_120 never uses: FlashInfer cubins for other architectures, CUDA static libraries, Nsight, and the sm_100 and sm_103 cubins for the B200 and B300. The result is about 25.2 GB locally, against about 49.8 GB for :sm120-fat, which keeps those datacenter cubins. For a B200 or B300, use the fat tag or a later :sm100 or :sm103 tag, neither of which I claim exists yet.

The package is public on GHCR, so no login is needed to pull. The README pins :sm120 and publishes its digest, sha256:6f580a1c2f83aac1dafba88b59859d8e95bee0edde57144d935650b8e7e00e67, while the recipe files pull :latest. I use :sm120 throughout, so what you run matches that digest; :aeon-vllm-rtx-sm120 is the same digest under another name. The repository is Apache-2.0, as is the OCI license label; the AEON patches are MIT, and the image is provided as is.

What changes when it runs

Three serving choices separate the RTX recipes from the Spark ones.

The first is attention. Every recipe here sets --attention-backend TRITON_ATTN, and the reason is written in the source: FlashInfer combined with an fp8 KV cache can produce garbage on sm_120 at long context. The KV cache holds the attention keys and values for every token in flight, and it is where a long window spends its memory, so I keep it in fp8_e4m3, an 8-bit float with four exponent bits and three of mantissa, and set --mamba-ssm-cache-dtype bfloat16 for the state-space layers. One subtlety: with speculative decoding on, the backend must also be written inside --speculative-config, because the top-level flag does not always propagate to the drafter.

The second is speculation itself. Speculative decoding lets a drafter guess several tokens ahead and the main model verify them in one forward pass. On the Spark I use DFlash, a separate drafter model. On a 32 GB card there is no room to be generous, so the RTX recipes use the checkpoint's own multi-token prediction heads, MTP, with three speculative tokens, which cost about 1% of KV on the 5090. The recipe files are blunt: do not add a fat DFlash drafter on 32 GB for the long seat, and never run MTP and DFlash together.

The third is memory housekeeping. VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 lets vLLM accept the 131k and 262k windows. PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True reduces allocator fragmentation. Chunked prefill is on and prefix caching off, so --max-num-batched-tokens becomes the size of each prefill chunk, which will matter in a moment. --quantization stays unset; the checkpoint's compressed-tensors config already says what it is.

The four recipes as a worked example

Recipe one is the 5090 quality seat: 131072 tokens of context, 4 sequences, 8192 batched tokens, GPU memory utilization 0.92. Mount the model snapshot read-only at /model and run:

docker pull ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:sm120
MODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>
IMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:sm120
docker run -d --name aeon-mixed-5090 --gpus all --ipc=host --shm-size=8g -p 8000:8000 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -v "$MODEL:/model:ro" \
  --entrypoint vllm "$IMAGE" serve /model \
  --served-model-name aeon --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 --max-num-seqs 4 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.92 --kv-cache-dtype fp8_e4m3 \
  --mamba-ssm-cache-dtype bfloat16 --attention-backend TRITON_ATTN \
  --enable-chunked-prefill --no-enable-prefix-caching \
  --limit-mm-per-prompt '{"image":2,"video":1}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.05}' \
  --trust-remote-code

The tail sets the parsers, qwen3 for reasoning and qwen3_coder for tool calls with automatic tool choice, and the default sampling parameters; --ipc=host and --shm-size=8g give the engine its shared memory. If KV will not allocate at 131k, try --gpu-memory-utilization 0.95, then --max-num-seqs 2.

Recipe two, the 5090 speed seat, is the same command with --max-model-len 65536 and --gpu-memory-utilization 0.90; VLLM_ALLOW_LONG_MAX_MODEL_LEN may be omitted at that window, and MTP stays at three with TRITON_ATTN inside the speculative config.

Recipes three and four move to the RTX PRO 6000 and its 96 GB. The quality seat runs 262144 tokens of context, 8 sequences, 8192 batched tokens and utilization 0.80, in a container named aeon-mixed-pro6000 with --shm-size=16g and --limit-mm-per-prompt '{"image":4,"video":2}'. The base command in that recipe file carries no speculative flag; native MTP, which I prefer on this seat, is added with the same line as above:

Code, shell--speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'

The speed seat trades window for concurrency: 131072 tokens, 12 sequences, 16384 batched tokens, utilization 0.85, MTP kept at three. If allocation fails, drop to 8 sequences and 8192 batched tokens.

The fault that passed every test

Here is the scar. On 2026-08-26 I verified a long-prompt engine fault on an RTX 5090 running under WSL2 Ubuntu-24.04, headless, driver 610.47, serving Qwen3.8-27B NVFP4. The service had passed a smoke test and a 24-prompt category benchmark. Then the first real conversation of about 24k tokens arrived, and the whole engine died partway through the prompt with RuntimeError: CUDA driver error: device not ready and EngineDeadError. That was not a rejected request but a dead engine, for every client, until a restart.

The traceback lied. The fault site moved between runs, first flashinfer_mm_fp4, then, after I changed the linear backend, an inductor MLP buffer. That pattern means an asynchronous CUDA fault surfacing at the next synchronising call, so the frame in the traceback is merely whoever allocated next. Chasing it was wasted time, and my --linear-backend cutlass hypothesis was wrong, as the last table row shows.

Chunk size was the only variable that mattered. The fix is --max-num-batched-tokens 2048, and it comes with a small gift: the profiler reported 137,240 KV tokens at 8192 batched tokens and 142,637 at 2048, because the activation workspace shrinks. Prompts longer than --max-model-len are still rejected with a clean HTTP 400 while the engine stays up; that is correct behaviour, not this bug.

My suspected mechanism is the Windows display watchdog. WDDM keeps its TDR timeout, TdrLevel 3 and TdrDelay 2 seconds by default, over GPU work in WSL2, and one 8192-token prefill chunk through a 27B model appears to exceed that budget, so the device is reset underneath the process. Raising TdrDelay is a machine-wide registry change that needs a reboot; capping the chunk is per-service and reversible. A native Linux host should not need it, but I have only tested WSL2 and will not claim more. So, one amendment to the recipes above: if you serve from WSL2, cap the chunk at 2048 and accept the slower prefill.

The judge seat

The leanest recipe in the repository is the judge seat, where the model grades rather than chats: Qwen3.8 with the same parsers, no YaRN rope scaling, no MTP, and tight batching of 4 sequences and 2048 batched tokens, the WSL2 fix baked in. With the same model mount as before:

Code, shelldocker run --gpus all --ipc=host --shm-size=16g -p 18081:18081 -v "$MODEL:/model:ro" --entrypoint vllm ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:sm120 serve /model --host 0.0.0.0 --port 18081 --served-model-name judge --dtype auto --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-num-seqs 4 --max-num-batched-tokens 2048

One last habit from the recipes: for Qwen long thinking, prefer a per-request thinking_token_budget over raising max_tokens, otherwise the model can spend the whole budget inside its <think> block. The ROCm cousin of this image is in early access, and the full set of images, models and tools lives in the catalog.

Sources

• AEON-7/vllm-ultimate-rtx on GitHub
• aeon-vllm-ultimate-rtx container package on GHCR
• AEON-7/vllm-ultimate and AEON-7/vllm-ultimate-dgx-spark, the Spark siblings
• Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED model card
• vllm-project/vllm
• vLLM v0.27.1 release