DDTree on vLLM: candid research notes on tree speculation for Qwen3.6

Let me say the awkward part first. DDTree, as it stands in this repository, is not faster than the flat speculative decoding I already ship for Qwen3.6. The README says so plainly, the contributor rules forbid claiming otherwise, and the raw files under bench/results agree. What the track has produced instead is a map: where tree speculation holds inside a real serving engine, where it quietly corrupts output, and what still has to land before anyone should promote it.

So these are lab notes. If you simply want to serve Qwen3.6 on a DGX Spark, use the stable DFlash image described in the production image article; if you are starting fresh, the model card now says to prefer Qwen3.8, which has its own write-up. Everyone else, read on; the failures are the interesting part.

Why a tree instead of a chain

Speculative decoding is the trick where a small drafter guesses several tokens ahead and the large target model checks the whole guess in one forward pass, keeping the longest prefix it agrees with. My production path uses DFlash, a small drafter built for Qwen3.6-27B that proposes a chain of up to 15 tokens per step. A chain has one weakness: if the target rejects the token at position two, positions three through fifteen are discarded, even when the drafter's second choice there was right and everything after it would have matched.

DDTree spends the same verification budget on a tree of alternatives, so when the drafter's top-1 is wrong at some position, a sibling branch can still be accepted. The idea comes from the DDTree paper and fits on a napkin; the engineering is hard for a specific reason.

Why hybrid attention makes it hard

Qwen3.6 is a hybrid. Some layers are ordinary full attention with a KV cache, the store of past keys and values each new token attends to. Others are Gated DeltaNet layers, a Mamba-style recurrent design that carries a fixed-size state forward instead of a cache, with a short convolution state beside it. For a tree, that state becomes a three-surface problem. Tree attention: each candidate row must see the prefix plus its own ancestors and never its siblings. Branch-local recurrent state: each node reads GDN and conv state from its parent and writes to scratch, not to the committed state. The commit: once a path is accepted, only that path may be copied back into vLLM's contiguous KV and recurrent buffers.

Upstream vLLM does not do this for you; its tree-attention issue (18327) was closed as not planned, so the repo carries its own patches over the community nightly base, on vLLM 0.20.2rc1.dev166 at the same commit as the stable DFlash image, plus the DFlash sliding-window fix from PR 40898.

What the numbers say

The reference is the stable v4 DFlash image on the Spark, serving Qwen3.6-27B AEON Ultimate XS, the multimodal NVFP4 build (NVFP4 is NVIDIA's four-bit floating-point weight format for Blackwell GPUs), with DFlash k=15, thinking on, natural prompts. It averages 36.91 tokens per second of decode, with a median time to first token of 223 ms and 27.0 ms per output token.

The M1 bridge routes DDTree through vLLM's method plumbing but still commits only the flat accepted prefix, so it was expected to match v4, and it did; any regression would have meant a plumbing bug. The table compares the two at a single stream (c=1, where c is the number of concurrent requests).

The honest probe is M53, the milestone tag on the public image. In a debug profile, the mean accepted length was about 2.7 tokens and the draft acceptance rate about 21 percent: 50 to 67 percent at position 0, 44 to 50 at position 1, about 33 at position 2, 28 to 33 at position 3, and mostly zero after that. The plumbing is alive; the deep positions are not yet giving the tree anything worth verifying.

The sweep up to c=256 produced large aggregate numbers, 306.75 tokens per second on extraction at c=128 and 270.29 on math at c=256, but they are a saturation map: at c=256 the median first token takes 77 to 139 seconds, and a long sweep heat-soaks the Spark enough that I run cooldown cycles between passes. A real gateway lives between c=1 and c=32.

What broke along the way

The first idea was to take the drafter's top-k at every depth and fan it out under the root. For Qwen3.6 that was toxic: DFlash's depth logits (its raw next-token scores) are not branch-conditioned alternatives, so the siblings were noise. A slow Python replay of the GDN recurrence, meant as a quality oracle, could degrade even flat-chain probes, so it could not referee anything. Full non-flat branch commit through the hybrid state produced plausible text that was quietly wrong, which is why that path is now gated behind DDTREE_UNSAFE_FULL_BRANCH_RESEARCH=1. The Triton GDN replay, a GPU kernel in the Triton language, survived normal text and then hit a CUDA illegal memory access during guided JSON decoding, where the sampler is constrained to a schema. And a tree attention that is safe for CUDA graphs, the recorded launch sequences vLLM replays to cut per-step overhead, is not the same as a fast one; every dynamic fallback erases the speed the tree was meant to buy.

Smaller scars were plain bugs: empty context KV, a zero-context position, query input padding, and a sampler output signature that changed under us in vLLM. Testing only with greedy sampling hid anomalies that moved once native sampling returned. Accepted and emitted tokens also need separate plumbing, because feeding the target's bonus token back to the drafter as an accepted draft state corrupts its context.

Running the research image

Everything runs from one container on a DGX Spark with Docker; the target mounts at /models/aeon-xs and the z-lab DFlash drafter at /models/dflash-drafter.

Code, shelldocker pull ghcr.io/aeon-7/vllm-aeon-ultimate-ddtree:qwen36-v5-m53-experimental

The safe ddtree mode commits only the flat accepted prefix and verifies extra root-leaf branches as bonus candidates, with a 2048-token context and one sequence by default.

Code, shelldocker run --gpus all --ipc host --network host --rm \
  -v /path/to/aeon-xs:/models/aeon-xs:ro \
  -v /path/to/dflash-drafter:/models/dflash-drafter:ro \
  ghcr.io/aeon-7/vllm-aeon-ultimate-ddtree:qwen36-v5-m53-experimental \
  ddtree

The tree tests need no GPU:

Code, shelldocker run --rm ghcr.io/aeon-7/vllm-aeon-ultimate-ddtree:qwen36-v5-m53-experimental ddtree-tree-test

The image also carries a small full-attention proving ground, Qwen2.5-0.5B-Instruct behind the qwen25-m2 entrypoint, where the tree math can be right or wrong without ambiguity. There the one-pass ancestor-mask verifier (qwen25-tree-mask-verifier) matches per-path replay on every node, 8 of 8 top-1 matches with a maximum absolute logit difference of 0.265625 at --budget 8 --depth 6 --top-k 4, and the oracle suite reproduces baseline greedy output in 6 of 6 prompt categories in 6.771 seconds total. That is the parity harness the hybrid work still lacks.

Where the next step is

The blockers, in order: scratch branch-state buffers for GDN and conv that never touch committed state before acceptance; an explicit accepted-branch commit with first-class node and parent ids; fused ancestor-mask branch attention over vLLM's paged KV blocks; DFlash drafter context compaction; and position and RoPE (rotary position embedding) correctness by branch depth. The plan's go/no-go is strict: nothing gets promoted if GDN replay erases the speedup, if tree masks force a slow attention backend, if long context drifts, or if tools and reasoning regress.

Pull requests are welcome for:

• a small-model parity harness, token by token
• a fused branch attention kernel for paged KV
• scratch-buffer GDN and conv replay
• an accepted-branch commit contract
• branch-conditioned proposal logic
• quality canaries for text, reasoning, tools, JSON, vision and long context

Until then, production stays on ghcr.io/aeon-7/vllm-aeon-ultimate-dflash:qwen36-v4. DFlash on a sibling model is in the Qwen3.6-35B-A3B article, and Aeon Bench is where to start for your own category sweeps.

Sources

• Repository: AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DDTree, its integration plan, raw benchmark results and container package.
• Production path: AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash.
• Model cards: Qwen3.6-27B AEON Ultimate Uncensored Multimodal NVFP4-MTP-XS, Qwen3.8-27B AEON Ultimate Uncensored NVFP4 Mixed and its recipe.
• Papers and reference code: DDTree paper, DFlash paper, DDTree reference implementation, Lucebox Hub.
• vLLM: repository (issues 18327 and 41559, PR 40898) and speculative decoding docs.