Gemma 4 31B DECKARD HERETIC at NVFP4 with DFlash on a DGX Spark

Gemma 4 31B is the dense model in Google's Gemma 4 family, and DavidAU's DECKARD HERETIC build is an uncensored version of it that keeps the thinking mode. I wanted it running on one DGX Spark, the small GB10 desktop with 128 GB of unified memory, at a speed an agent gateway could live with. Getting there took three repositories: the weights, quantized to NVFP4 (NVIDIA's four-bit floating-point format, with a scale for every block of 16 values) in two variants; an EAGLE-style drafter for speculative decoding, the technique where a small model guesses several tokens ahead and the big model checks them in one pass; and a validated vLLM container built around a different drafter, DFlash, which is the one that worked.

The short version: on one Spark, single-stream decode went from 11 to 38.82 tokens per second, and aggregate throughput at 32 concurrent requests from 162 to 426.66, with reasoning, tool calling, image and video input and structured output all kept. The longer version includes a handful of NaN values that produced empty output and a drafter that made the model slower, both worth knowing before you copy the commands.

Why a dense 31B crawls on unified memory

The Spark has room for this model: the original BF16 checkpoint is about 62 GB, and 128 GB of unified memory holds it with space left for the KV cache, the memory that stores attention state for the tokens already processed. Speed is the problem. Every new token requires a pass through all 31 billion parameters, so the time per token is set by how quickly the GPU can stream the weights out of memory, not by how fast it can multiply. A mixture-of-experts model touches only a slice of its weights per token, which is why the MoE sibling, Gemma 4 26B-A4B, reached 368 tokens per second aggregate at 16 concurrent requests on the same non-DFlash stack, against 161 for the dense 31B, roughly 23 per request against 11.

Quantizing to four bits cuts the weights from about 62 GB to 20.45 GB, but it does not change the arithmetic of a dense decode. The baseline for everything that follows is the AWQ_FULL weights served without speculative decoding, generating 200-token outputs: 11 tokens per second for a single request, 161 aggregate at 16 concurrent requests and 162 at 32, where it saturates, each of those 32 requests receiving about 8 tokens per second and waiting an average of 29.6 seconds for its 200 tokens.

Two NVFP4 variants from ModelOpt

The quantization ran with NVIDIA ModelOpt 0.42.0 on a B200, which handles FP4 natively, calibrated on 2048 samples from CNN DailyMail at 1024 tokens each. The model has 60 layers, 50 of them sliding-window attention with a 1024-token window and 10 full attention, plus a 27-layer vision tower. The attention projections and every MLP layer are quantized; the vision tower, the embedding projection, the norms and the output head stay in BF16.

Each quantized layer stores packed FP4 weights in the E2M1 format (two exponent bits, one mantissa bit) with an FP8 scale per block of 16 elements, plus per-tensor, pre-quantization and input scales. AWQ_FULL uses ModelOpt's NVFP4_AWQ_FULL_CFG, an activation-aware search for the per-channel scaling that loses the least: an exhaustive grid with an alpha step of 0.1 and ten candidates per layer, plus weight clipping. It took 74.4 minutes on the B200, where the lighter AWQ_LITE search took 10.6. SVDQuant uses NVFP4_SVDQUANT_DEFAULT_CFG and takes a different route: the outlier channels that four bits represent poorly go into a rank-32 BF16 residual and everything else is quantized. That took 69.1 minutes and makes the result slightly larger, 20.94 GB against 20.45.

The model cards say AWQ_FULL is the faster of the two and SVDQuant is potentially higher quality and slightly slower. I want to be plain about the word "potentially": no published quality benchmark compares the two, so it is a claim, not a result. Start with AWQ_FULL, which is also the variant the DFlash container was validated with.

The NaN block scales that produced empty output

The first serving attempts returned nothing, or garbage, and the cause was in the scales. ModelOpt 0.42.0 had written FP8 block scales containing NaN: 60 of them, spread across 39 of the 60 layers, among roughly 280 million scale elements. Sixty bad values in 280 million sounds harmless until you remember that each scale governs a block of 16 weights; one NaN poisons its block, and the damage spreads through the whole forward pass.

The fix is a loader patch, patch_modelopt_nvfp4_awq.py in the container, which accepts the quantization algorithm name NVFP4_AWQ, applies the pre-quantization scale and zeroes any NaN block scale at load. If you serve these weights in your own vLLM instead of the prebuilt image, the patch is mandatory:

Code, shellcurl -L -o modelopt_patched.py https://raw.githubusercontent.com/AEON-7/Gemma-4-31B-DECKARD-HERETIC-Uncensored-NVFP4/main/modelopt_patched.py
cp modelopt_patched.py $(python3 -c "import vllm; print(vllm.__path__[0])")/model_executor/layers/quantization/modelopt.py

The EAGLE detour, and why it made things slower

Speculative decoding attacks the memory-bound decode directly: a forward pass costs about the same whether it scores one token or several, so a small drafter proposes a run of tokens, the target verifies the run in one pass, and every accepted guess is a token that did not need its own trip through the weights. EAGLE is one family of drafters, and the E4B repository holds the one I built for this target: a 9.6 GB model with 42 layers and a 131,072-token context, quantized to NVFP4 like the target. The pair needs about 30 GB.

Running it on vLLM 0.19.1rc1 took two patches. eagle_patched.py lets the EAGLE path accept a multimodal Gemma 4 target and its two KV cache groups, needed because the sliding-window layers use a head dimension of 256 and the global layers 512. serving_chat_patched.py keeps the reasoning delimiters intact in non-streaming responses. The pair runs with five speculative tokens in the older container, vllm-spark-gemma4-nvfp4-awq, built from eugr's spark-vllm-docker on PyTorch 2.12.0 and CUDA 13.0.

It worked, and it was slower. With 300-token outputs on one Spark, a single request decoded at 8.9 tokens per second against the baseline's 11, with 39.4 seconds of latency. At two concurrent requests the aggregate reached 21.7 tokens per second, 10.8 per request; at four, 42.7 aggregate and 10.7 per request, with latencies of 27.7 and 28.1 seconds. The reason is simple: the drafter and the target share one GPU and one pool of memory, so the drafter's own passes come out of the budget they were meant to save. The cards now say it plainly: this drafter's gain belongs to multi-GPU setups or smaller targets. The repository stays public because a negative result on the same hardware is useful to the next person.

DFlash, and the numbers that changed

DFlash is a different drafter design from z-lab, and for Gemma 4 31B there is an official drafter, z-lab/gemma-4-31B-it-DFlash. It proposes a flat chain of future tokens, fifteen in the validated configuration, and the 31B verifies the chain in one pass. The container pairs it with the AWQ_FULL weights on a vLLM 0.20.1 build with Gemma 4 DFlash support (the lineage of vLLM pull request 41703), on top of the image I made for the 26B-A4B MoE, with triton_attn for the target, flash_attn for the drafter and FlashInfer's CUTLASS kernels for the NVFP4 matrix multiplications.

The numbers below come from one Spark with the drafter warmed, natural prompts rather than synthetic text, thinking on, 200-token outputs and zero request errors.

The gain depends on the kind of text, because the drafter's guesses are accepted more often when the next tokens are predictable. The per-category run used the short-context profile: 8192 tokens of context, 32 sequences, 32768 batched tokens and 0.80 of memory.

At 32 concurrent requests the median time to first token, the wait before the answer starts arriving, sat between 854 ms for JSON and 1,260 ms for reasoning, and the median time per output token between 36.10 ms for math and 91.04 ms for prose. The same recipe runs my Qwen 3.6 35B-A3B DFlash build.

Running the container

The image is linux/arm64 and the validation data is specific to the GB10, so do not assume it runs on an x86 workstation. Weights are not baked in: download the target and the drafter once and mount them read-only.

sudo mkdir -p /models/deckard /models/gemma-dflash
sudo chown -R "$USER:$USER" /models
huggingface-cli download AEON-7/Gemma-4-31B-it-DECKARD-HERETIC-Uncensored-NVFP4 --local-dir /models/deckard
huggingface-cli download z-lab/gemma-4-31B-it-DFlash --local-dir /models/gemma-dflash
docker pull ghcr.io/aeon-7/gemma-4-31b-uncensored-nvfp4-dflash:latest
docker rm -f gemma31-dflash 2>/dev/null || true
docker run -d --gpus all --ipc=host --network host --name gemma31-dflash --restart unless-stopped -v /models/deckard:/models/deckard:ro -v /models/gemma-dflash:/models/gemma-dflash:ro -e GPU_MEMORY_UTILIZATION=0.82 -e MAX_MODEL_LEN=65536 -e MAX_NUM_SEQS=16 -e MAX_NUM_BATCHED_TOKENS=32768 ghcr.io/aeon-7/gemma-4-31b-uncensored-nvfp4-dflash:latest
docker logs -f gemma31-dflash
curl -fsS http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool

Stop other large GPU containers before the first launch, and expect the first boot to take minutes while CUDA graphs are captured for batch sizes up to 64 and FlashInfer autotunes its FP4 kernels. On a Spark, nvidia-smi may show N/A for memory; trust free -h and the vLLM logs. Once the health check passes, any OpenAI-compatible client can point at http://localhost:8000/v1 and ask for deckard-31b, gemma-4-31b-uncensored or gemma31-dflash. The entrypoint's first argument chooses the mode: serve by default, baseline for a drafter-free comparison, bench, or bash.

Profiles and memory pressure

The four environment variables in the run command are the knobs, in the order context length, sequences, batched tokens and memory fraction. The default is a balanced profile, 65536, 16, 32768 and 0.82. For short-context throughput I use 8192, 32, 32768 and 0.80, the benchmark profile above. Long documents get 131072, 4, 32768 and 0.82, and a Spark shared with other work gets 32768, 8, 32768 and 0.70 to 0.76.

Under memory pressure, lower MAX_MODEL_LEN first, then MAX_NUM_SEQS, before you touch GPU_MEMORY_UTILIZATION; if decode slows under concurrency, context length is again the first cut. One more detail: a plain docker start reuses the old environment, so after any image or flag change, remove the container and run it again.

Reasoning, tools and the token budget

The container keeps vLLM's gemma4 reasoning parser and tool-call parser on, along with chunked prefill, prefix caching, a KV cache in its native dtype, and limits of four images and two videos per prompt. The repository's rule is not to disable the parsers; that would shrink the capability surface the container advertises.

The surprise is the token budget. Thinking happens in the output, so a small max_tokens can be spent entirely on the reasoning trace and leave an empty answer. Give thinking tasks at least 1200 output tokens, and expect part of the response in message.reasoning rather than the content field; that is the parser doing its job. A first smoke test:

Code, shellcurl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "deckard-31b", "messages": [{"role": "user", "content": "Write one concise paragraph about why speculative decoding helps."}], "max_tokens": 600, "temperature": 0.2}'

Benchmarking it yourself

The container's bench mode runs the same category benchmark I used, against a running server, and writes a JSON report into a directory you mount:

Code, shelldocker run --rm --network host -v /path/to/bench:/bench ghcr.io/aeon-7/gemma-4-31b-uncensored-nvfp4-dflash:latest bench --base-url http://127.0.0.1:8000/v1 --model deckard-31b --levels 1,4,8,16,32 --max-tokens 200 --temperature 0.0 --runs-per-point 1 --min-samples-per-point 8 --trim-fraction 0.1 --output /bench/gemma31_dflash_category_c1_c32.json

The rules I hold myself to are in the repository. Benchmark with natural prompts, never random synthetic text, and capture time to first token, time per output token, median and peak decode, aggregate throughput and error counts, not one headline number. Keep NUM_SPECULATIVE_TOKENS at 15 unless a benchmark on this exact model and workload proves another value better. Do not expect DDTree numbers from this repository: DDTree is planned and unbenchmarked here, and my notes live in the DDTree research notes. The harness behind these tables is Aeon Bench.

Before you serve it

A few things in the sources disagree, and I would rather you hear it from me. The GitHub README gives a maximum context of 262,144 tokens while the SVDQuant card says 131,072. The 31B card lists any NVIDIA GPU with at least 24 GB of VRAM as the minimum; with 20.45 GB of weights alone that is a floor, not a recommendation. The tokens-per-second tables on the AWQ card are estimates, not measurements.

For more Spark recipes, see the Aeon vLLM Ultimate article and the catalog.

Sources

Repositories and containers:

• DFlash container repository
• 31B weights repository
• E4B drafter repository
• DFlash container image
• AWQ container image
• eugr/spark-vllm-docker

Model cards and licenses:

• AWQ_FULL model card
• SVDQuant model card
• E4B drafter model card
• z-lab DFlash drafter
• DavidAU source model
• Gemma terms of use