AEON vLLM Ultimate for ROCm: early access on vLLM 0.31

This one is not finished, and I would rather say so in the first paragraph than the last. AEON vLLM Ultimate for ROCm is a container image that serves my Qwen3.8-27B AEON ULTIMATE UNCENSORED checkpoint on AMD GPUs through vLLM 0.31. The image builds and its main patch produces the same numbers as stock vLLM, but nobody has posted a speed log from real Radeon hardware yet, so treat it as unmeasured.

I am publishing it anyway, because only the people who own the cards can finish it. The checkpoint I serve on the Spark and on the RTX box is NVFP4, a four-bit floating-point format whose kernels are CUDA only, so AMD owners have had nothing to run.

What the image is

It is the official vLLM v0.31.0 ROCm image plus two Python patches on top. Nothing was recompiled: no HIP kernels (AMD's equivalent of CUDA code) and no AITER, AMD's inference kernel library, which stays off. ROCm, AMD's compute stack, is 7.2.3 inside the base, which ships kernels for gfx90a, gfx942, gfx950, gfx1100, gfx1101, gfx1150, gfx1151, gfx1200 and gfx1201: Instinct, RDNA3, RDNA4 and Strix Halo class parts.

It is the AMD sibling of the RTX 5090 image and the DGX Spark image, and it serves one checkpoint: the Qwen3.8 27B described in Qwen3.8-27B AEON ULTIMATE UNCENSORED, abliterated, meaning the refusal direction was edited out of its weights. For AMD it was quantized with Quark, AMD's quantization toolkit, into two microscaling formats, where a small block of weights shares one scale: MXFP4, four-bit floats, for the feed-forward (MLP) layers, and MXFP6, six-bit floats, for the Gated DeltaNet layers. Gated DeltaNet is the linear-attention layer Qwen3.8 interleaves with ordinary attention, keeping a fixed-size recurrent state instead of a growing key-value cache. The model file is 23,681,100,408 bytes, about 23.7 GB.

The weight repo is gated and I approve requests by hand; when I checked in November it showed zero downloads.

Why the stock path crawled and what the patches change

Decoding, not loading, was the problem. Stock vLLM handles MXFP6 on this path by emulation: on every forward pass it unpacks the six-bit Gated DeltaNet projections into BF16, runs the ordinary matrix multiply, and discards the unpacked copy. A community tester ran the checkpoint that way on two Radeon AI PRO R9700 cards, 32 GB each, and measured about 1.7 tokens per second on a single stream. The time went to the unpack: about 8.7 milliseconds per layer, against about 0.2 milliseconds for the BF16 matrix multiply (the GEMM) it was feeding. The model spent most of every token converting its own weights.

The first patch, MXFP6 dequant-once, is the least clever fix I could find. With AEON_MXFP6_DEQUANT_AT_LOAD=1, the default, the projections are unpacked once at load, with the same function stock vLLM calls on every forward, and the copy is kept. Activations are handled as before, and the dequantized weights match stock emulation exactly; torch.equal says so. The price is memory for the BF16 copy, which is why 16 GB cards need AEON_MXFP6_DEQUANT_AT_LOAD=0: same output, slower decode, but it fits.

The second patch, vLLM change #55601, is always applied and concerns the drafter. DFlash2 is a speculative-decoding design: a small model proposes several tokens and the big model verifies them in one pass. In a hybrid model the recurrent state has to be indexed by mamba_block_size; otherwise a prefix-cache hit (vLLM reusing the computed state of a shared prompt prefix) can point the drafter at the wrong state. Until that is checked on AMD, the recipe keeps prefix caching off while the drafter is on.

What is unknown

Speed with this image is unmeasured. The 1.7 tok/s figure is from the stock path; unpack-once has not been the default in any published community run on Radeon, and whether it speeds decode on real hardware is the first thing I need someone to find out. Prefix caching with the drafter is untested on AMD. Only the R9700 pair has a documented recipe; the gated weight repo carries serve scripts named for the RX 9070 XT, RX 7900, W7900, MI250, MI300, MI355, Strix Halo and V710, but names are all I can vouch for.

Deliberately absent, because they change numerics or are unvalidated: a packed MXFP4 kernel for gfx12, speech output through vllm-omni, the reasoning-EOS force-end port, and any W4A8 (four-bit weights, eight-bit activations), FP8 KV cache or quantized all-reduce path.

The reference recipe on two R9700s

You need a ROCm host with /dev/kfd and /dev/dri passed into Docker, the video and render groups, and the usual ROCm container flags, which the command below carries. The pair runs tensor parallel 2, which splits each layer's weights across both cards so both work on every token. The drafter on ROCm is the BF16 one from AEON-DFlash2-Qwen3.8-27B, folder bf16/, never the NVFP4 one; the weight repo also ships an earlier dflash2/ folder of 3,848,817,920 bytes, about 3.8 GB.

Pull the image; the tags early-access and x86 point at the same build until a tested tag replaces them, so pin the digest once docker pull prints it.

Code, shelldocker pull ghcr.io/aeon-7/aeon-vllm-ultimate-rocm:2026-10-06-v0.31.0-x86-early-access

Serve with the two R9700s as devices 0 and 1, keeping the integrated GPU out of HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES.

MODEL=$HOME/models/aeon-mxfp4-rocm
DRAFT=$MODEL/dflash2          # or the bf16/ folder of AEON-DFlash2-Qwen3.8-27B
CACHE=$HOME/.cache/aeon-vllm-rocm
mkdir -p "$CACHE"
docker run -d --name aeon-rocm \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
  --shm-size 16g \
  -p 8000:8000 \
  -e HIP_VISIBLE_DEVICES=0,1 \
  -e ROCR_VISIBLE_DEVICES=0,1 \
  -e NCCL_PROTO=Simple \
  -e NCCL_P2P_DISABLE=1 \
  -e VLLM_ROCM_USE_AITER=0 \
  -e AEON_MXFP6_DEQUANT_AT_LOAD=1 \
  -v "$MODEL":/model:ro \
  -v "$DRAFT":/draft:ro \
  -v "$CACHE":/root/.cache/vllm \
  ghcr.io/aeon-7/aeon-vllm-ultimate-rocm:2026-10-06-v0.31.0-x86-early-access \
  /model \
  --served-model-name aeon \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --max-model-len 65536 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype auto \
  --attention-backend TRITON_ATTN \
  --no-enable-prefix-caching \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,8,7],[9,12,6],[13,16,4]],"attention_backend":"TRITON_ATTN","draft_sample_method":"probabilistic","rejection_sample_method":"standard"}'

Plain serve without the drafter: drop --speculative-config and the /draft mount, remove --no-enable-prefix-caching, and raise --max-model-len to 131072. One 32 GB card: --tensor-parallel-size 1, HIP_VISIBLE_DEVICES naming that card alone, and no NCCL_* variables. A 16 GB card: AEON_MXFP6_DEQUANT_AT_LOAD=0, as above.

How to help test

If you own the hardware, run the recipe, or the variant for your card, and keep the complete container log. Report four things: whether the model loaded, single-stream tokens per second, tokens per second under concurrency, and whether prefix caching survives being turned on with the drafter. Pass the temperature you want to measure; the recipe's sampling is only the kit default. Request access to the gated weight repo and read its TESTER_GUIDE.md and scripts folder. Post the log as an issue on the GitHub repository.

The image is Apache-2.0, like the checkpoint, and contains no code from GGZ14/vllm-mxfp4, vllm-radiance or libr4d, which have no redistributable license. DFlash2 is the work of z-lab and Inco AI; credit them when you serve it. When the first logs come in I will update this article and retire the early-access tag. The rest of the family is in the ÆON FORGE catalog.

Sources

• AEON-7/vllm-ultimate-rocm on GitHub
• Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-MXFP4-MXFP6-ROCm on Hugging Face (gated)
• Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED on Hugging Face (CUDA only)
• AEON-DFlash2-Qwen3.8-27B on Hugging Face
• vLLM #55601 in vllm-project/vllm
• Container images: ghcr.io/aeon-7/aeon-vllm-ultimate-rocm (tag 2026-10-06-v0.31.0-x86-early-access), siblings ghcr.io/aeon-7/aeon-vllm-ultimate-rtx and ghcr.io/aeon-7/aeon-vllm-ultimate, base vllm/vllm-openai-rocm:v0.31.0