DeepSeek-V4 on two DGX Sparks: an experimental TP=2 image

This image is experimental, and I would rather say so in the first sentence than in a footnote. vllm-ultimate-deepseek-v4-gb10 lets DeepSeek-V4-Flash serve across two NVIDIA DGX Sparks at once, using tensor parallelism (TP=2: each layer's weights split between two GPUs that exchange partial results every layer) plus expert parallelism (the mixture-of-experts layers spread over both nodes too), over a RoCE link (RDMA over Converged Ethernet: direct memory access between machines over an Ethernet cable). It boots, it answers, and it has passed concurrency soaks. It is not a claim that two Sparks now run any model at TP=2, and Qwen or Gemma users should stay on the production image, ghcr.io/aeon-7/aeon-vllm-ultimate:latest.

My production image, AEON vLLM Ultimate on the DGX Spark, already ships vLLM's deepseek_v4 package, so on paper DeepSeek-V4 was supported. On the Spark's GB10, a Blackwell part of compute capability 12.1, it died instead, one missing kernel or mismatched dependency at a time. This layer is the record of fixing that cascade. It lives in its own repository and under its own image name so the daily driver is never put at risk. It is for trusted testers with two Sparks and a RoCE fabric who will reproduce the lane and file a report, and nothing in it is production-ready.

Seven ways GB10 said no

The layer began as PR #4 against the production repository, submitted by an outside contributor. The original patches are archived in the repo's patches/ folder with the contributor as Git author; I packaged them on top of aeon-vllm-ultimate:2026-07-01-v0.24.0, A/B tested them against that base, documented the results and published two tags, experimental-pr4 and the dated pin 2026-07-08-pr4-experimental.

E8M0, which appears twice in that table, is the exponent-only 8-bit scale format behind block-scaled fp8 (8-bit floating point) and MXFP4 (4-bit microscaling) weights. One thing needed no fix at all: NCCL, NVIDIA's library for the all-reduces between GPUs, runs cross-node over RoCE at the base image's version 2.28.9 with no LD_PRELOAD override.

Two Sparks as one machine

Tensor parallelism buys memory: two Sparks hold twice the weights and twice the KV cache, the per-token attention state that grows with context length. On the production image, with a different model family (so a lesson, not a DeepSeek claim), TP=2 gained single-stream speed, 33.7 against 26.7 tokens per second, and gave a 56 GiB KV pool, which is what makes million-token contexts possible on two desktops.

The link is RoCE over the direct 200 GbE ConnectX cable. Fabric addresses are assigned inside a container with NET_ADMIN, rdma-core must be in the image or there is no RoCE path, and Docker needs --device=/dev/infiniband, --network host, --ipc host, --shm-size 16g, IPC_LOCK and unlimited memlock. Only rank 0 listens on port 8000; rank 1 runs --headless, and the two meet on port 29519.

Two lessons cost me evenings. The first boot spends about 20 minutes compiling FlashInfer 0.6.14 kernels just in time, so persist the container's /root/.cache/flashinfer and /root/.triton caches and launch both ranks together only once they are warm, or rank 1's Gloo barrier (the CPU-side rendezvous) times out while rank 0 is still compiling; warm, the engine initializes in about 58 seconds. And start the headless rank first, then the head.

Why PIECEWISE graphs are the rule

A CUDA graph records a sequence of GPU kernel launches once and replays it to cut launch overhead during decode. vLLM's PIECEWISE mode captures the stretches between attention and other splitting operations and runs the rest eagerly; FULL replays the whole decode step as one graph, collectives included. On this path, FULL and FULL_AND_PIECEWISE decode replay desync NCCL between the two ranks. Even capture sizes restricted to [1,2] died under batched long-context decode with a c10::DistBackendError on rank 1. Enable FULL on this image only to reproduce that bug on purpose. The production image later captured cross-node graphs with a different patch set and NCCL 2.30.7, newer than the 2.28.9 here; that result belongs to that image, not to this one.

What the numbers say, and what they do not

The PR layer reports, for a DeepSeek-V4-Flash-DSpark checkpoint in fp8 on two Sparks at TP=2 with PIECEWISE graphs and --max-num-seqs 8: 16.6 tokens per second single-stream decode, 66.4 tokens per second aggregate across 10 concurrent multi-turn conversations, 48 of 48 benchmark turns completed, zero engine errors. At --gpu-memory-utilization 0.80 with fp8 KV the engine reports a 782,838-token GPU KV cache, and the soak passed 4 and 10 concurrent 800-token generations at 9K context in both eager and PIECEWISE modes. The README calls this "not yet a universal claim for all checkpoints, all fabrics, or all graph modes".

A Qwen3.6 non-regression smoke on one Spark, same recipe on both images, put this image slightly ahead of the base: 26.67 against 25.30 tokens per second single-stream, 68.49 against 63.87 at four concurrent requests. A second A/B in the B12X README put PR #4 with PIECEWISE behind the FULL_AND_PIECEWISE default, 45.78 against 53.21 tokens per second median c=4 aggregate, and concluded PR #4 is not a general win. I report both because they disagree, and because one PR #4 warm run hit a 1.6 tokens-per-second outlier on a reasoning prompt. That disagreement is the whole case for keeping DeepSeek in its own experimental repo.

Quick start on two Sparks

You need two Sparks with the same image and model path, a RoCE fabric visible in Docker through /dev/infiniband, fabric IPs that reach each other, and ports 29519 and 8000 free. Stop any running vLLM containers on both machines first.

Code, shelldocker pull ghcr.io/aeon-7/vllm-ultimate-deepseek-v4-gb10:experimental-pr4

On both nodes, fetch the model and the repo, then check the folder size; an LFS skeleton or a zero-byte preprocessor_config.json breaks loading.

Code, shellmkdir -p /models/deepseek-ai
huggingface-cli download deepseek-ai/DeepSeek-V4-Flash --local-dir /models/deepseek-ai/DeepSeek-V4-Flash --local-dir-use-symlinks False
git clone https://github.com/AEON-7/vllm-ultimate-deepseek-v4-gb10.git
cd vllm-ultimate-deepseek-v4-gb10

Start the headless worker first.

Code, shellexport IMAGE=ghcr.io/aeon-7/vllm-ultimate-deepseek-v4-gb10:experimental-pr4
export MODEL_DIR=/models/deepseek-ai/DeepSeek-V4-Flash
export NODE_RANK=1
export MASTER_ADDR=<head-fabric-ip>
export VLLM_HOST_IP=<worker-fabric-ip>
export FABRIC_IF=<fabric-interface>
export NCCL_IB_HCA=<roce-hca>
./scripts/serve-deepseek-v4-gb10-tp2.sh

Then the head.

Code, shellexport IMAGE=ghcr.io/aeon-7/vllm-ultimate-deepseek-v4-gb10:experimental-pr4
export MODEL_DIR=/models/deepseek-ai/DeepSeek-V4-Flash
export NODE_RANK=0
export MASTER_ADDR=<head-fabric-ip>
export VLLM_HOST_IP=<head-fabric-ip>
export FABRIC_IF=<fabric-interface>
export NCCL_IB_HCA=<roce-hca>
./scripts/serve-deepseek-v4-gb10-tp2.sh

The script passes TP=2 with expert parallelism, the marlin MoE backend, the triton linear backend, fp8 KV, a 65,536-token context limit and the PIECEWISE compilation config. Verify on the head.

Code, shellcurl http://127.0.0.1:8000/v1/models | jq
python3 scripts/validate-endpoint.py --base-url http://127.0.0.1:8000/v1
docker logs -f vllm-ds4

Expect /v1/models to list DeepSeek-V4-Flash and a short chat to complete, with no NCCL timeout, no "EngineCore encountered an issue" and no garbled output. To build the layer yourself:

Code, shelldocker build -t aeon-vllm-ultimate:deepseek-v4-gb10 -f Dockerfile.deepseek-v4-gb10-layer .

What comes next

The next recipe is DSpark, DeepSeek's speculative decoding path: a draft module proposes several tokens and the full model verifies them in one larger pass, the same idea DFlash brings to Qwen in the Qwen3.6 DFlash article. The trial keeps the stable baseline and adds method: dspark, num_speculative_tokens: 3 and draft_sample_method: probabilistic. Watch the acceptance metrics, because a mis-wired drafter fails silently: acceptance collapses, the output stays correct, and speed is the only symptom.

Beyond that is a high-risk lane, B12X with nvfp4_ds_mla, which keeps the KV cache of DeepSeek's multi-head latent attention in NVFP4, NVIDIA's 4-bit floating-point format with per-block scales, and targets a million-token context. It is not promoted: it dies with ImportError: cannot import name 'build_flashinfer_mixed_sparse_indices', an overlay mismatch, and has only been validated for build and import on one Spark. Leave it alone unless fixing it is your task.

The roadmap is to fix B12X, validate DSpark with acceptance numbers, run TP=2 and TP=4 on independent rigs, then fold the common fixes into a universal image with optional model-family overlays. If you run the baseline, record the image tag and digest, repo commit, model revision, fabric settings, latency percentiles, throughput, errors and thermals; the README has a template. The catalog lists the rest of the lab's vLLM work.

Sources

• vllm-ultimate-deepseek-v4-gb10 repository
• GHCR package
• vllm-ultimate-dgx-spark repository
• PR #4
• DeepGEMM PR #324
• DeepSeek-V4-Flash model card