Serving Qwen 3.6 35B-A3B heretic at NVFP4 with DFlash on a DGX Spark

This is a release people actually run. In early November 2026 the Hugging Face API showed 91,125 downloads and 74 likes for the NVFP4 weights of Qwen3.6-35B-A3B-heretic, and the repository that explains how to serve them had 150 stars. That is a lot of DGX Sparks, or at least a lot of people curious about what one can do.

The short version: a 35-billion-parameter mixture-of-experts model from the Qwen team, its refusals removed by a third party, quantized to four-bit floating point so it fits in about 22 GB, and served through vLLM with a small drafter that guesses ahead. On one DGX Spark it decodes at about 97 tokens per second on a single stream and around 740 tokens per second in aggregate at 64 concurrent requests, and it reads images, thinks before it answers and calls tools.

A sparse model with its refusals removed

Qwen3.6-35B-A3B is a sparse mixture-of-experts model: its feed-forward blocks are split into many small expert networks, and a router sends each token to only a few of them. About 35 billion parameters exist in total and about 3 billion are active for any one token. Each of the 40 layers has 256 routed experts plus one shared expert, and 8 routed experts fire per token. Thirty of the 40 layers use Gated DeltaNet linear attention and every fourth is ordinary full attention, a 27-block vision transformer handles images and video, and native context is 262,144 tokens, 1,010,000 with YaRN.

The MoE angle is why this suits a Spark. The GB10 has 128 GB of unified memory shared between CPU and GPU, but it is one chip. A model that touches only about 3B parameters per token decodes fast and stays light on memory, which leaves room for many streams at once; in my measurements compute saturates around 64 concurrent requests, so that is where I set the ceiling.

The base is tvall43's Qwen3.6-35B-A3B-heretic, made with Heretic v1.2.0, an open tool for abliteration: it finds the direction in the model's activations that corresponds to refusing and removes it from the weights, so the model answers instead of declining. The heretic card cites a 5/100 refusal rate. I did not do that step; I quantized the result and made it run well. This is an uncensored model, and you take full responsibility for its outputs.

Quantizing it to NVFP4

NVFP4 is NVIDIA's four-bit floating-point format for Blackwell: each weight is an FP4 value in E2M1 form, two exponent bits and one mantissa bit, every block of 16 weights shares an FP8 scale, and each tensor carries an FP32 scale on top. Blackwell tensor cores run it natively, with no dequantization step on every matmul. I quantized with llm-compressor into the compressed-tensors nvfp4-pack-quantized layout, which is why the serving flag is --quantization compressed-tensors.

Not everything went to four bits. The attention projections and all the experts did, 122,880 expert tensors. The output head, the token embeddings, the MoE routers and gates, the norms, the whole vision tower and all 30 linear-attention layers stayed in BF16, because quantizing those, as my card puts it, tanks accuracy. Calibration used 256 open-platypus samples at a sequence length of 2048 and took about three hours on an RTX PRO 6000 Blackwell in the lab. The result is a single model.safetensors of 23.35 GB.

The two mistakes in the weights

That is v2, and v1 deserves a confession. To load the weights through vLLM's text-only model class I stripped the language_model. prefix from every tensor key. It loaded, then crashed in real chats with illegal addresses and NaNs. The lesson was to stop fighting the loader. v2 was re-quantized straight from the heretic base with AutoModelForImageTextToText, keeping the canonical multimodal layout so vLLM serves it through the class it was built for, and the crashes stopped.

One scar remained. A quantization-time ignore pattern nested the 333 vision tensors one level too deep, under model.language_model.visual.* instead of model.visual.*. vLLM silently skipped the vision tower and ran it uninitialized, so text was perfect and any image came back as a string of exclamation points. The fix on 2026-06-18 was a header-only rename, the weight bytes untouched, validated 7 of 7 on an image probe. If you downloaded the weights before that date, pull them again.

The drafter: DFlash speculative decoding

Speculative decoding is the trick that gets several tokens out of a model built to produce one per forward pass: a small, cheap drafter proposes a run of tokens, the big model checks them all in a single pass, and the ones that match what it would have produced are kept. The output is still the big model's; the drafter changes only the speed.

DFlash, from z-lab, is a block-diffusion drafter: it proposes a whole block of tokens in parallel, reading hidden states from the target. The drafter in this recipe is z-lab's Qwen3.6-35B-A3B-DFlash, about 905 MB, eight layers, all full attention, proposing 11 tokens per step. On math the target accepts 48% of drafted tokens at position 0 on a single stream; on open prose, 24%, which is why the figures below vary so much by category. Greedy decoding, temperature 0, gives the best acceptance because the drafter is predicting a deterministic target; on the older image my card measured roughly 10 to 20% acceptance at a temperature of 0.7 against 60 to 78% greedy. So the server exposes three aliases for one backend: qwen36-fast for greedy, agentic work, qwen36-deep for sampled, creative work, and qwen36-35b-heretic as the canonical name.

The z-lab drafter published before 2026-04-19 had a long-context bug that produced cudaErrorIllegalAddress after about 16K tokens, so re-pull if yours is older. And the block is 11 tokens, not the 15 my older docs show; do not set n to 15. My own drafter for this family, AEON-DFlash-Qwen3.6-35B-A3B, was validated on the Ornith model, and its story is in the Ornith article.

The seven fixes in the container

None of this runs on a stock vLLM wheel. The GB10 is a Blackwell chip with compute capability sm_121a, which the stock wheel does not target. So much of this release is the container, ghcr.io/aeon-7/aeon-vllm-ultimate, vLLM 0.23.0 built from source for GB10 with patches on top; it has its own article. The first fix in the table changed everything for this model: the previous image crashed whenever 32 or more requests ran at once under speculative decoding, and with the drafter's block table sliced to the unpadded batch the server climbs to 64 streams with zero errors.

The numbers at one stream

All of this was measured on the Spark with the published recipe. I quote no speedup over stock vLLM because I have not published a vanilla baseline for this model.

The average across the six categories is about 97 tokens per second, with time to first token between 88 and 137 ms at p50 on short prompts.

Under load and at long context

Aggregate throughput on math climbs from 715.1 tokens per second at 32 streams to 739.8 at 64, with 716.2 on extraction and 608.7 on coding at 64, and zero errors. Creative categories run lower, about 429 on prose and 474 on natural language at 64 streams in the model card's table. Re-validating the exact published recipe on 2026-06-19 peaked at about 800 tokens per second on reasoning at 32 streams.

Long prompts hold. At about 16K tokens of context a single stream decodes at 90.8, 95.9 and 106.1 tokens per second on coding, reasoning and extraction, with acceptance up to about 52%. At about 32K the three are 79.3, 73.0 and 94.4, acceptance reaches about 58%, and time to first token is 7.1 to 9.5 seconds because prefill runs at about 4.3 to 5.1 thousand tokens per second. The 83.9 tokens per second and the plateau near 313 quoted in older revisions came from the old vllm-spark-omni-q36:v1.2 image at k=15 and are superseded. Aeon Bench runs the same measurement on any box.

Running it

You need a DGX Spark, about 35 GB of free disk, an NVIDIA driver at 580.x or newer, and Docker 25.x or newer with the nvidia-container-toolkit; Ubuntu 24.04 LTS is confirmed. The published image will not run on Hopper, Ampere, the B200 or desktop Blackwell without a rebuild. Its entrypoint is /bin/bash, so a raw docker run passes --entrypoint vllm.

Code, shelldocker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest
huggingface-cli download AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 --local-dir ./aeon-model
huggingface-cli download z-lab/Qwen3.6-35B-A3B-DFlash --local-dir ./aeon-drafter
docker run --gpus all --rm -p 8000:8000 -v ./aeon-model:/model:ro -v ./aeon-drafter:/drafter:ro --entrypoint vllm ghcr.io/aeon-7/aeon-vllm-ultimate:latest serve /model --served-model-name qwen36-35b-heretic qwen36-fast qwen36-deep --host 0.0.0.0 --port 8000 --quantization compressed-tensors --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 16384 --gpu-memory-utilization 0.85 --enable-chunked-prefill --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt '{"image":4,"video":2}' --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"/drafter","num_speculative_tokens":11}'

The first boot takes 10 to 12 minutes; add -v ./vllm-cache:/root/.cache/vllm and restarts drop to 3 to 5 minutes. --gpu-memory-utilization 0.85 is for the model alone on the box, 0.70 if other services share the box. --reasoning-parser qwen3 puts the thinking in the reasoning field; thinking requests need max_tokens of at least 2048 or content comes back null with a finish reason of length, and {"chat_template_kwargs": {"enable_thinking": false}} turns thinking off.

Then prove it.

Code, shellcurl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "qwen36-fast", "messages": [{"role":"user","content":"What is 17 × 23?"}], "max_tokens": 2048, "temperature": 0}'
curl -s localhost:8000/v1/models
curl -s localhost:8000/metrics | grep -E "spec_decode_num_(draft|accepted)_tokens_total"

The models call lists the three aliases, and the two counters climb with every request. With the compose file from the repo the container is named qwen36-35b, and docker logs qwen36-35b 2>&1 | grep "not found in params_dict" must print nothing. Where the compose example or the model card's quickstart disagree with the command above, trust the command.

What not to undo

AGENTS.md exists because many operators running this are themselves agents, OpenClaw-style gateways that read a repository and configure a server. It carries a DO NOT UNDO list, and every line is a scar.

From the troubleshooting doc: KV at BF16 costs about 5 GB per 256K-token sequence, so about 16 full-length sequences fit at 86 GB; --enforce-eager is the fallback for CUDA illegal-instruction errors, at a cost of about 30% according to my card.

Where this fits

This is for an owner of a DGX Spark who wants a fast, local, uncensored, multimodal agent model with 256K of context, or a fleet of agents making 64 requests at once. The weights travel further than the image: the card lists the RTX PRO 6000 Blackwell and the RTX 5090 as compatible, with reduced context and low concurrency, and the RTX 5090 article covers that path. Gemma 4 26B-A4B is another MoE on the Spark, and the catalog lists everything I publish.

Sources

Code and container:

• Qwen3.6-35B-A3B-heretic-NVFP4-DFlash repository
• vllm-ultimate-dgx-spark container repository
• aeon-vllm-ultimate container image
• DFlash by z-lab
• Heretic
• Example docker-compose.yml

Model cards:

• AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4
• AEON-7/AEON-DFlash-Qwen3.6-35B-A3B
• z-lab/Qwen3.6-35B-A3B-DFlash
• tvall43/Qwen3.6-35B-A3B-heretic
• Qwen/Qwen3.6-35B-A3B