AEON Bench recipe lab: tuning vLLM flags one at a time on a DGX Spark

A model has no single speed. It has a speed per recipe, meaning the exact set of flags it was served with, and two people with the same weights on the same hardware can post numbers an order of magnitude apart. My seed-era run of a 26B mixture-of-experts model on the DGX Spark came in at about 33 tokens per second aggregate. The same model on the same machine later reached 517, and its entry on the board today does 816. The weights never changed. The recipe did.

The recipe lab is the part of the AEON Bench Pod, my open-source (MIT) benchmark appliance, built around that gap: a way of working inside the pod's Run tab where you restore a prior run from a Template, change one knob, launch, and let the Performance tab say what the change bought. Every tuned run carries its full serve recipe to the public board at aeon-bench.com, which marks the best recipe per hardware class as the champion, applied with one click. The loop: tune, bench, publish, get beaten, steal their recipe, go again. If you have never run the pod, start with the first-run guide; this article picks up where it ends.

What the bench needs

The pod runs in Docker and needs the Docker socket, because it launches the engine and the coding-agent harness containers as siblings. An NVIDIA machine also needs the NVIDIA Container Toolkit so that --gpus all works; without GPU access the pod mislabels the box as CPU-only and the fast engines (aeon-vllm-ultimate, vLLM, SGLang) switch off. Budget about 10 GB of disk for the pod and harness images, plus the weights. The benchmark floor is a 64K context, so --max-model-len never goes below 65536 here. Pull the latest image first.

Code, shelldocker pull ghcr.io/aeon-7/aeon-pod:latest
docker rm -f aeon-pod
docker run -d --name aeon-pod --network host --gpus all \
  -v /var/run/docker.sock:/var/run/docker.sock -v aeon-pod-state:/root/.aeon \
  -v "$HOME/aeon-models:/models" -e AEON_MODELS_HOST_DIR="$HOME/aeon-models" \
  -v "$HOME:/host-home:ro" -e AEON_HOST_HOME_DIR="$HOME" \
  ghcr.io/aeon-7/aeon-pod:latest
curl -s localhost:8091/healthz

On a Mac, drop --gpus all, replace --network host with -p 8091:8091, and expect the pod to recommend MLX on bare metal; a CPU-only box runs GGUF weights through llama.cpp. Then open http://localhost:8091, go to the Run tab, paste a Hugging Face link in the form org/model, and you are in the lab. The engine behind the numbers below is AEON vLLM Ultimate, vLLM built from source for the Spark's GB10 chip.

The iron law: change one thing

The Template picker restores everything about a prior run: model, engine, every flag, drafter, concurrency and test plan. From there you change exactly one knob and relaunch. Two changes teach nothing: if the number moves you cannot say which change moved it, and if it holds still you cannot say whether two effects cancelled. I broke it most often before the picker existed, because retyping a dozen flags invites a second improvement.

While hunting, set the Test plan to "Text suite only". The text suite is 174 cases (aeon-suite-v4, five categories across six difficulty tiers), and at high concurrency a run takes about 2 to 5 minutes, so you can iterate quickly. When a recipe wins, run it Comprehensive: text, the coding-agent harnesses, vision (31 cases), audio (21), video (10) and the full performance sweep. Only comprehensive runs rank, and a few rules travel with that. The pod hash-verifies the weights against Hugging Face before every run; never bypass it. A smoke test is for your eyes only, never presented as validated. A run whose coding-agent part could not execute still ranks, badged "agentic untested". And a slow run is not a broken run, so do not kill it.

The first three knobs: concurrency, memory and the KV cache

The RECIPE TUNING panel lays the serve flags out as cards, each with pros, cons and a live warning when two choices conflict; the order below is the order of effect on aggregate throughput.

--max-num-seqs is the concurrency ceiling and the biggest aggregate lever. The post calls 32 sane at 64K context, and 64 produced the 816 tok/s; the agent playbook is more conservative, putting the GB10 sweet spot at 16 to 24 streams at 64K against a default of 32. One chases peak aggregate, the other a comfortable quality run. Push it until time to first token gets ugly or memory runs out, and set --perf-max-conc, the performance ladder (1, 4, 8, 16, 32; default cap 32), equal to it; otherwise the top rung measures queueing in front of the engine, not the engine.

--gpu-memory-utilization means something different on unified memory, where GB10 and Apple silicon share one pool between CPU and GPU, so the default is 0.7, the recommendation is 0.6 to 0.7, and above about 0.8 the Spark does not report an error, it hangs. Go lower still with a DFlash drafter, whose verify buffers are not counted. Dedicated VRAM can run at 0.90 or higher. The playbook's line: "A model card's --gpu-memory-utilization is for the card's hardware, not yours."

--kv-cache-dtype fp8 halves the KV cache, the attention state a model keeps for every token in its context, so more streams fit at long context. It may touch quality, which the composite score is there to reveal. One hard caveat: fp8 KV crashes Gemma 4's sliding-window layers; keep auto there.

The rest of the catalog: prefill, attention, parsers and experts

--max-num-batched-tokens 32768 with --enable-chunked-prefill keeps a long prompt from stalling everyone else: chunked prefill feeds a large paste through in pieces between decode steps. --attention-backend on GB10 is triton_attn or flash_attn; FlashInfer attention is broken on that chip.

The parsers are where honest numbers go to die. --reasoning-parser moves the model's thinking into its own field; without it the think trace leaks into the answer and quality tanks. --tool-call-parser with --enable-auto-tool-choice turns the model's tool calls into real calls; without them the agentic scores sandbag for reasons unrelated to the model. The pod reads the chat template, picks a parser and tests tool calling before the long suite starts; if the choice is wrong it prints WRONG TOOL-CALL PARSER with the fix flag.

Two more are model-specific. --moe-backend cutlass selects the kernel for mixture-of-experts layers, where each token visits only a few of many expert sub-networks; it helped the 26B A4B on GB10. Audio models need --limit-mm-per-prompt, for example '{"audio":2}', and the bench warns when audio is declared but rejected. A wrong quant_method (compressed-tensors against a ModelOpt checkpoint, or the reverse) fails at boot, which is the kind failure: loud and early.

Speculative decoding and the n knob

Speculative decoding is the biggest lever on a memory-bound chip and the easiest one to misuse. A small drafter proposes several next tokens, the big model verifies all of them in one forward pass and keeps the ones it would have produced itself, so the output is identical to running the big model alone and only the speed varies. DFlash is a family of such drafters, published per model family by z-lab. In the pod you paste the drafter's Hugging Face card the way you paste the model's; it is hash-validated the same way and mounted at /drafter. The one number you set is num_speculative_tokens, called n: 11 or 12 is the single-stream setting, 6 is the concurrent-friendly baseline, and the Gemma and DFlash preset allows 1 to 15. On the 26B at n=11 the mean acceptance length was 4.77, meaning the big model kept about 5 drafted tokens per step.

The trap: 33 versus 517 tok/s

The seed-era run of that 26B used DFlash at n=11 while the performance grid pushed 32 concurrent streams, and the aggregate came in at about 33 tok/s. The retuned recipe reached 517 on the same Spark. The tuning log holds the lesson in one line: a high n at high concurrency starves batching. Each speculative step asks the big model to verify n drafted tokens for every stream in the batch, so a larger n trades single-stream latency against concurrent throughput, and n and concurrency are one experiment, not two.

To be plain about what the sources show, the board entry that does 816 tok/s at c64 still runs n=11, so the trap is not that n=11 is wrong. The clean comparison, n=6 against n=11 against no drafter at c64, is the next experiment named in the post; no result exists yet, and I will not guess at one. Two mechanical caveats are settled. If a checkpoint ships both an MTP head and a DFlash drafter, they are mutually exclusive in --speculative-config. And a mis-wired drafter keeps the output correct while acceptance drops to about 0.3 percent and throughput halves, so read the acceptance length, and keep num_speculative_tokens at or above the checkpoint's block size.

The recipe to steal

This is the serve recipe behind the current 26B A4B board entry on a Spark: 816 tok/s aggregate at c64 with DFlash active and a composite of 65.7. The model has its own article with the production recipe I run day to day; this one is tuned for the board.

--max-model-len 65536
--gpu-memory-utilization .8
--max-num-seqs 64
--kv-cache-dtype auto
--attention-backend triton_attn
--enable-prefix-caching
--enable-chunked-prefill
--reasoning-parser gemma4
--tool-call-parser gemma4
--enable-auto-tool-choice
--moe-backend cutlass
--max-num-batched-tokens 32768
--speculative-config {"method":"dflash","model":"/drafter","num_speculative_tokens":11}

Read it with the catalog in mind: the 64K floor, the aggressive end of the concurrency lever, a KV cache kept at auto because of the Gemma 4 sliding-window crash, the GB10 kernels, prefix caching (which reuses the cache of a shared prompt prefix) and the gemma4 parsers that protect the quality and agentic scores. The flag I would not copy blindly is --gpu-memory-utilization .8. The post calls 0.70 to 0.80 safe on unified memory, but the agent playbook says the shared pool page-thrashes above about 0.8, recommends 0.6 to 0.7 and lower with DFlash, and the pod's skill file agrees. I present .8 as the quoted recipe and would start at 0.7. Know the cost, too: on the quality bench at c64 the time to first token was 23 seconds, fine for batch work and unusable for chat. The cheapest way to steal it is the board: the champion for your hardware class applies in one click, and the Template picker holds it for the next single change.

Reading results like a tuner

The Performance tab rewards slow reading. PEAK AGG TOK/S names its cohort, "coding @ c64" for example, because it is a real measured pool, one category at one concurrency, not a projection. The hero chart carries two lines that multiply: concurrent total divided by streams is the per-stream share, and your operating point is where that share drops below what you tolerate. Time to first token is queue pain and time per output token is per-token feel. The category by concurrency heatmap is the most honest panel on the page: coding batches well and reasoning does not, which is why I benchmark by category.

The COMPOSITE rides on every recipe. A shortcut that costs correctness, an fp8 cache on the wrong model or a mis-wired drafter, shows up as a quality drop beside the speed gain, and the board crowns the best recipe per axis (throughput, single-stream, latency and quality) per hardware class. One habit keeps you honest: quote any board number with its cohort. The repository carries an open note asking whether the aggregate should be tokens over wall time or decode rate times concurrency, with a fix proposed; until that is settled the cohort is the context.

Four traps of benchmarking your own server

These come from the lessons file, and each cost me a run before it cost me a sentence. First, the default --max-tokens equal to --max-model-len makes every request fail with a 400 that looks like a network error; set the token budget well under the context. Second, a case that is cut off retries at twice the token budget and hits the HTTP timeout, so set the timeout explicitly or bound the retries. Third, Qwen 3.x chat templates default reasoning_effort to xhigh, so hard cases think past the budget; copy the template, set the default to medium, mount it and serve with --chat-template, or pass chat_template_kwargs per request. Fourth, --think-budget is ignored by the V2 runner, so with DSpark the template is the only lever.

Two smaller ones hide behind them. The pod image has no ps, so a liveness check that greps ps passes silently forever; read /proc/*/cmdline or the /metrics endpoint instead. And a slow model, meaning thinking turned on or a dense model of 20B or more without speculative decoding, anything under about 25 tok/s single-stream, needs the slow-model settings below, the environment on the pod and the flags on the run. Expect such a run to take 1 to 2.5 days, and let it.

Code, plain textAEON_HTTP_TIMEOUT=36000 GOD_TASK_TIMEOUT_S=21600 AEON_HARNESS_CONC=2
--concurrency 2 --max-tokens <card total> --retry-max-tokens 0

A recipe on the board is not a claim but a reproducible object: weights, engine, flags and drafter, with the cohort attached. Steal mine, beat it by one flag, and publish the recipe that did it. The Qwen 3.6 35B-A3B DFlash recipe is the neighboring mixture-of-experts case; the catalog lists the rest.

Sources

• Repository: AEON-7/Aeon-Bench-Pod (MIT), with AGENTS.md, SKILL.md and the illustrated walkthrough
• Public board: aeon-bench.com
• Images: ghcr.io/aeon-7/aeon-pod and ghcr.io/aeon-7/aeon-vllm-ultimate
• Drafters: z-lab on Hugging Face
• NVIDIA Container Toolkit documentation