Qwen 3.8 27B AEON Ultimate: the current flagship and how to serve it

Every family of models I work on eventually produces the one I would hand to a stranger first. For the Qwen family that is now Qwen3.8-27B AEON Ultimate Uncensored: a 27-billion-parameter model from the Qwen team with its refusals removed, quantized by me to fit one DGX Spark, one RTX 5090 or one RTX PRO 6000, and served through vLLM with a speculative decoder chosen for each machine. It reads images and video, thinks before it answers, calls tools, and on one Spark it has peaked at 237.67 tokens per second in aggregate on a coding board at sixteen concurrent streams. For new deployments it supersedes the whole Qwen3.6 line.

The word uncensored carries weight here, so I want to be precise from the first page: this model was not pushed until some number reached zero. It was edited until it stopped refusing, measured until the edit had cost as little capability as my boards could detect, and then frozen.

What the model is

Qwen/Qwen3.8-27B is a hybrid. Part of its layers use Gated DeltaNet, a form of linear attention that carries a fixed-size recurrent state forward instead of looking back over every token, and the rest use ordinary softmax attention. It has sixty-four MLP layers, numbered 0 to 63 below, a vision tower for images and video, and a native MTP head, extra weights trained to predict several tokens ahead. The license is Apache 2.0, inherited from the base.

I publish it in two forms. The BF16 master, the full sixteen-bit weights at about 54 GB, is the source of truth and wants an H200 or several GPUs. The NVFP4-MIXED cut, about 23.8 GB in four shards, is what you serve day to day; NVFP4 is NVIDIA's four-bit floating-point weight format for Blackwell tensor cores, and MIXED signals that not everything in it is four-bit. A gated MXFP4 and MXFP6 cut for AMD's Radeon AI PRO and Instinct cards belongs to the ROCm article.

One mundane check saves an evening: the MIXED weights should occupy 23 to 25 GB on disk, and a folder of a few kilobytes is the Git LFS skeleton.

What uncensored means here

Abliteration is a targeted edit: you find the direction in the model's internal activations that corresponds to refusing and project it out of the weights, so the model can no longer lean on it. Done carelessly it is a lobotomy, and a model that no longer refuses but also no longer finishes a sentence is not uncensored, it is broken. The method behind 3.8, which I call coherent abliteration, grew out of watching that happen.

The pipeline runs in a fixed order. First an outlier repair on the SSM conv1d weights, the small convolutions inside the linear-attention layers, which are fragile and must not be disturbed. Then abliterix 1.12.2 runs a 50-trial search over the strength and scope of the edit using Optuna, a hyperparameter search library, with each trial's thinking scored after the </think> marker so the search judges the answer rather than the deliberation. The vision tower is never touched. Afterwards the MTP head is grafted back and hash-matched against the original. The winner was trial 48 of 50.

Why I did not chase zero

KL divergence, measured here in nats per token between the edited model and the original on harmless prompts, is how you detect collateral damage; zero means nothing changed. Chasing that zero, or treating every answer a judge model marks with an R as a leftover refusal to grind out, ends in over-editing and, eventually, gibberish. The zero-KL trials in my search were no-ops: they had changed nothing, so they had unlocked nothing. I picked the coherent unlock and read the answers myself, with one rule: a disclaimer followed by the actual payload is not a refusal, whatever a keyword scorer thinks.

The released numbers follow from that stance. Smash KL is 0.0991 nats per token across the first three teacher-forced tokens of 100 harmless held-out prompts (median 0.013). On a harmful held-out set a Flash Lite judge marks 36 of 100 answers with an R, against roughly 100 of 100 for stock Qwen, and hard refusals are zero, the number I actually care about. On a sexual-content set the judge flags 5 of 50 against 30 for stock; on harmless prompts, 1 of 100 against 3.

The card is blunt about the cost. The intended uses are research, red-teaming and creative work "without a hall monitor", the duty of care sits with you, and a User Responsibility and Arbitration Clause makes that binding: as-is, sole responsibility, individual arbitration. For high-risk work in front of other people, add input and output filtering and human review; this is not a model that polices itself.

The NVFP4-MIXED lattice

The weight layout, which the card calls the lattice, decides tensor by tensor how much precision each part of a 54 GB master can lose. It has four tiers.

• MLP layers 0 to 55: NVFP4, four-bit weights and activations, calibrated by mean squared error with a sweep over FP8 scales.
• MLP layers 56 to 63: FP8, eight-bit floating point, for the whole block: gate, up and down, 24 projections.
• Softmax attention and the GDN writer projections: FP8.
• Vision tower, MTP head, embeddings, output head and GDN internals: BF16.

The first bake, Mix B, moved only the down projection of the last eight MLPs to FP8, and it looked fine until the coding board. In my local AEON Suite v4 run on the GB10 at a concurrency of 8, Mix B scored 0.694 on coding unit tests, and the leftover KL pointed straight at layers 56 to 63. Promoting the full MLP of those eight layers to FP8 lifted coding to 0.833 in the same run, above the 0.806 the stock Unsloth NVFP4 cut reports on its public card; the text mean moved from 0.759 to 0.782, and the agentic boards (first agent board, OpenCode, OpenClaw) went from about 0.917, 0.793 and 0.733 to 0.918, 0.818 and 0.707.

ModelOpt dropped the MTP head during export, so its 15 tensors were grafted back. The same tree attests about 91.9 overall on AEON Bench with the Perf dial at 100.

Speculative decoding and the dynamic lattice

Speculative decoding gets several tokens out of a model built to produce one per forward pass: a small, cheap drafter proposes a run of tokens, the large model checks the whole run in a single pass, and the tokens it would have produced itself are kept. The output is still the large model's; only the speed changes.

On the Spark the drafter is DFlash, z-lab's block drafter: a separate small model that reads the target's hidden states and proposes a whole block of tokens at once, here z-lab's Qwen3.8-27B-DFlash2. My own fine-tuned drafter, AEON-DFlash2-Qwen3.8-27B, about 1.9 GB at NVFP4 or 3.85 GB in BF16, is in gated early access; on one Spark at training stage lk3 it gained 21 percent over stock DFlash2 at a temperature of 0.6 and 35 percent at 1.0, and ran 3.2 times faster than no drafter at all.

A fixed depth is a compromise: a deep draft pays off when one person is typing and the verify pass is cheap, and at sixteen streams the same depth burns compute on tokens that will be rejected. vLLM's num_speculative_tokens_per_batch_size key maps concurrency to draft depth: the locked map drafts nine tokens at one or two streams, eight at three or four, seven at five to eight, six at nine to twelve and five at thirteen to sixteen, and the card's runtime readout tapers further, from 10 down to 3. Deep when idle, shallow when busy, so verify cost never starves acceptance. The 237.67 tokens per second wave peak, AEON Bench's peak aggregate reading, came from this seat on the coding board at sixteen streams.

The seats

Each seat is a complete configuration, and its speculative method is not a knob to move between seats.

Generation defaults are the same everywhere, baked into the serve command: temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.0. Thinking is a per-request choice, and because Qwen 3.x templates default reasoning_effort to xhigh, I set medium explicitly.

Code, JSON{"chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "medium"}}

The server is ready when /v1/models lists aeon; one request proves the chat path.

Code, shellcurl -s http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"aeon","messages":[{"role":"user","content":"Say hi in one sentence."}],"max_tokens":64}'

With --reasoning-parser qwen3 the response carries both a reasoning field and a content field; a client that reads only content looks empty while the model thinks.

One Spark: the dynamic lattice

The Spark seat runs on ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-18-v0.29.0-omni, vLLM 0.29.0 built for the GB10, digest sha256:cc91c51559d66854718fd9a8db6423e605ba76fb338bb55902caccb62aa9c677, with :2026-09-07-reasoning-eos as the rollback tag; pin the dated tag until :latest matches that digest. The image has its own article. This seat uses vLLM's newer V2 model runner and FULL_AND_PIECEWISE CUDA graphs, and it needs 16384 batched tokens; 8192 leaves too few draft slots for sixteen sequences. Utilization stays at 0.80: the Spark's 128 GB is unified memory shared with the CPU, higher risks an out-of-memory under KV pressure, and 0.85 is only for a box that does nothing else. Prefix caching stays on except for benchmark purity. The attention backend is TRITON_ATTN on every seat, because FlashInfer attention with an FP8 KV cache can emit garbage tokens on sm_120.

Code, shellMODEL=$HOME/.cache/huggingface/hub/models--AEON-7--Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED/snapshots/<rev>
DRAFT=/path/to/z-lab__Qwen3.8-27B-DFlash2
IMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-18-v0.29.0-omni
docker rm -f aeon-mixed-spark 2>/dev/null
docker run -d --gpus all --network host --name aeon-mixed-spark -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_ENABLE_CUDA_COMPATIBILITY=0 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -v "$MODEL:/model:ro" -v "$DRAFT:/draft:ro" --entrypoint vllm "$IMAGE" serve /model --served-model-name aeon --host 0.0.0.0 --port 8000 --gpu-memory-utilization 0.80 --max-model-len 262144 --max-num-seqs 16 --max-num-batched-tokens 16384 --kv-cache-dtype fp8 --enable-chunked-prefill --tool-call-parser qwen3_coder --enable-auto-tool-choice --reasoning-parser qwen3 --limit-mm-per-prompt '{"image":4,"video":2}' --attention-backend TRITON_ATTN --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE"}' --trust-remote-code --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":9,"num_speculative_tokens_per_batch_size":[[1,2,9],[3,4,8],[5,8,7],[9,12,6],[13,16,5]],"attention_backend":"TRITON_ATTN"}' --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

The modelopt.py bind-mount for vLLM PR #54367 is optional on the 2026-09-18 tag and later; older tags still need it. Measure acceptance length, not only correctness: a mis-wired drafter keeps the output correct while acceptance falls to around 0.3 percent, and the server looks healthy at no-drafter speed. The compose route is shorter.

Code, shellhf download AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED --local-dir ./models/aeon-mixed
hf download z-lab/Qwen3.8-27B-DFlash2 --local-dir ./models/dflash2
cp .env.example .env
docker compose up -d
docker compose logs -f vllm

Two Sparks: tensor parallel and a million tokens

Tensor parallelism splits each layer's weight matrices across two GPUs that exchange partial results at every layer, which is why it wants InfiniBand or RoCE. A TP=1 against TP=2 comparison on the new image held quality and TP=1 was slightly faster, so TP=2 buys context and KV pool, not quality. With two Sparks the seat runs a million tokens of context using YaRN, a method of stretching rotary position embeddings beyond the trained length, at factor 4 over the native 262,144. On the older GDNFP8-LLMC tree the expected speed at a million tokens was about 15 tokens per second.

Three things are easy to get wrong. The YaRN override must be nested as rope_parameters under text_config. Rank 1 starts first, with --headless, then rank 0, which carries the API. And the HCA line is not a typo: IB_HCA==rocep1s0f0:1 gives the variable a value beginning with an equals sign, which NCCL reads as an exact match on the device name. The rest of the group the card calls B0 is VLLM_ALLREDUCE_USE_FLASHINFER=0, fuse_allreduce_rms=false, --disable-custom-all-reduce and the V2 runner off. The drafter is a fixed DFlash2 at n=7, not the lattice. Set MASTER_IP, NODE_IP and IFACE for your pair; image, model and drafter are the same on both boxes.

Code, shellIMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate:2026-09-18-v0.29.0-omni
IB_HCA==rocep1s0f0:1
HF_OVERRIDES='{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}'
docker run -d --name tp2-node0 --gpus all --ipc=host --shm-size=16g --net=host -e VLLM_HOST_IP="$NODE_IP" -e NCCL_SOCKET_IFNAME="$IFACE" -e GLOO_SOCKET_IFNAME="$IFACE" -e NCCL_IB_HCA="$IB_HCA" -e NCCL_IB_GID_INDEX=3 -e NCCL_DEBUG=WARN --device /dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1:-1 -e VLLM_USE_FLASHINFER_SAMPLER=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_USE_V2_MODEL_RUNNER=0 -e VLLM_ALLREDUCE_USE_FLASHINFER=0 -v "$MODEL":/model:ro -v "$DRAFT":/draft:ro --entrypoint vllm "$IMAGE" serve /model --served-model-name aeon --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 --nnodes 2 --node-rank 0 --master-addr "$MASTER_IP" --master-port 29501 --kv-cache-dtype fp8_e4m3 --attention-backend TRITON_ATTN --max-model-len 1000000 --max-num-seqs 16 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.70 --disable-custom-all-reduce --enable-chunked-prefill --mamba-cache-mode align --limit-mm-per-prompt '{"image":4,"video":2}' --speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":7}' --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --hf-overrides "$HF_OVERRIDES" --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' --trust-remote-code

Rank 1 is the same command with --headless, --node-rank 1, its own NODE_IP and the name tp2-node1, without --served-model-name, --host, --port or the parsers. Watch docker logs -f tp2-node0 and check curl -s http://127.0.0.1:8000/health. A 64k speed variant drops VLLM_ALLOW_LONG_MAX_MODEL_LEN and --hf-overrides, with --max-model-len 65536 and utilization 0.60. Without InfiniBand, drop --device /dev/infiniband, NCCL_IB_HCA, NCCL_IB_GID_INDEX and --cap-add IPC_LOCK, and keep NCCL_SOCKET_IFNAME on the fast NIC.

The RTX box: MTP n=3

The RTX 5090 with 32 GB and the RTX PRO 6000 with 96 GB are sm_120 parts on amd64 with dedicated VRAM, and they run a different image, ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest, with :2026-09-07-omni-mm as the dated known-good tag; the RTX 5090 article covers it. Here the drafter is MTP, the multi-token prediction head already inside the checkpoint, so nothing competes with the KV cache; a 32 GB card cannot spare what a DFlash drafter takes from that pool. In an H200 smoke test on vLLM 0.27.1 with three draft tokens, MTP acceptance ran between about 40 and 66 percent. The 5090 runs 131,072 tokens of context at four sequences with utilization between 0.92 and 0.95; the PRO 6000 takes --max-model-len 262144, --max-num-seqs 8 and --gpu-memory-utilization 0.80, with MTP optional.

Code, shellIMAGE=ghcr.io/aeon-7/aeon-vllm-ultimate-rtx:latest
docker run -d --name aeon-mixed-5090 --gpus all --ipc=host --shm-size=8g -p 8000:8000 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -v "$MODEL:/model:ro" --entrypoint vllm "$IMAGE" serve /model --served-model-name aeon --host 0.0.0.0 --port 8000 --max-model-len 131072 --max-num-seqs 4 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.92 --kv-cache-dtype fp8_e4m3 --mamba-ssm-cache-dtype bfloat16 --attention-backend TRITON_ATTN --enable-chunked-prefill --limit-mm-per-prompt '{"image":2,"video":1}' --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}' --reasoning-parser qwen3 --tool-call-parser qwen3_coder --enable-auto-tool-choice --override-generation-config '{"temperature":0.6,"top_p":0.95,"top_k":20,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' --trust-remote-code

If an older RTX image from August 21 answers with 'MergedColumnParallelLinear' object has no attribute 'data', that is the ModelOpt bug fixed by vLLM PR #54367; bind the patched file or move to a newer tag.

The hard rules

The repository's agent file exists because these recipes fail in ways that look like something else. Its rules for the current MIXED tree:

The older setup file for the GDNFP8-LLMC tree is superseded in every default but one, which I kept word for word: if you are stuck after one log read and one table match, ask the human.

Lessons from the 3.6 line

The Qwen3.6 27B line is where most of those rules were paid for. Its claim was lossless abliteration: refusals at 0 of 100 with a KL of 0.000492, reached by a four-point method, a precise refusal direction, a norm-preserving projection, a Pareto search over strength and layer scope, and a capability cross-check. It shipped in six variants, from a 51 GB BF16 to 21 and 20 GB XS cuts that quantize the GDN projections but keep the linear-attention conv1d in BF16, across two quantization formats, compressed-tensors and modelopt, which is where the quantization-flag rule comes from. Its keyword refusal scoring taught me that a scorer cannot tell a refusal from word salad.

The serving lessons were sharper. DFlash at n=12 under concurrency crashed with a device-side assert; n=10 was the fix. Utilization 0.70 was safe and 0.88 a ceiling nobody should touch. Prefix caching with DFlash needed vLLM PR #41703. A silent decay in drafter acceptance hung long sessions until a restart, which is where the habit of measuring acceptance came from. DDTree, a tree-shaped drafting experiment, stayed research-only (its notes). None of that line's agent-file flags transfer to 3.8, and the 3.6 body has a second life on Apple silicon.

Caveats and what comes next

The BF16 card is marked as an Early Access Draft, and it is honest about one failure: on very long answers small gaps stack, and a loop or a repeated phrase can appear. A GA cut is promised. On math, reasoning and prose the model still trails stock Qwen's 84.5 composite, and the heal track for that gap will be a later brand rather than a quiet revision of this one. The ROCm cut is still a gated early release. Three smaller traps from the lab notes look like model problems and are not. A bench client whose default max tokens equals the model length draws a 400 on every request. thinking_token_budget is ignored on the V2 runner. And num_speculative_tokens must be at least the drafter checkpoint's block size. Everything I publish is in the catalog. Copy the command for your machine, check the folder size, read both fields of the response, and measure acceptance before you trust the speed.

Sources

Code and containers:

• Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED repository
• vllm-ultimate-dgx-spark and the aeon-vllm-ultimate image
• vllm-ultimate-rtx and the aeon-vllm-ultimate-rtx image
• Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash repository (legacy)
• vLLM PR 54367
• AEON Bench

Model cards, 3.8 line:

• AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-BF16
• AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED
• AEON-7/AEON-DFlash2-Qwen3.8-27B (gated)
• AEON-7/Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-MXFP4-MXFP6-ROCm (gated)
• z-lab/Qwen3.8-27B-DFlash2
• Qwen/Qwen3.8-27B

Model cards, 3.6 line:

• AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-BF16
• AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP-XS