Qwen3-TTS server: streaming speech that starts in 0.4 seconds

A voice agent is judged on its first second. When the caller stops talking, the machine has to hear the sentence, choose a reply and begin saying it before the pause turns awkward, and on the Spark my text-to-speech link was the slow one: it finished the whole reply before sending any of it. This is how that server came to start talking about 0.4 seconds after the request arrives.

qwen3-tts-server is my OpenAI-compatible speech endpoint for Qwen3-TTS-12Hz-1.7B, the 1.7-billion-parameter text-to-speech model from the Qwen team. It answers at /v1/audio/speech with the request body OpenAI's speech API takes, so the OpenAI SDK, OpenWebUI and Home Assistant can use it by changing one base URL. The documented version, v0.3.0, runs on the faster-qwen3-tts CUDA-graph engine on an NVIDIA DGX Spark, the GB10 Blackwell machine with 128 GB of unified memory, beside the Qwen3 ASR server (ASR is automatic speech recognition) and the Matrix voice agent that answers the calls.

The first build worked, and still made callers wait

The original server, v0.1.0, is the code in the repository today: a thin FastAPI wrapper around the qwen-tts Python SDK that loads the VoiceDesign model on CUDA in bf16, the 16-bit brain-float format, with flash-attn 2, a fused attention kernel that never writes the full attention matrix to memory. On the Spark the hot path ran at a realtime factor of about 1.30x, meaning 1.3 seconds of audio per second of compute, or 1480 ms for a 1.92 s clip, about 32x faster than the same model on a CPU in fp32. But the wrapper synthesized the whole reply before sending one WAV file, so the silence before the first byte grew with the answer.

Two other warnings came from that build. When the flash-attn wheel failed to compile, the server fell back quietly to SDPA, PyTorch's built-in scaled dot-product attention, and the realtime factor dropped to about 0.7x, slower than the speech it was making. And the runbook forbids pip install vllm[audio] in any derivative image, because the extra downgrades vLLM's core dependencies and silently regresses the hot inference latency by about 5x.

What changed in v0.3.0: CUDA graphs and chunked PCM

v0.3.0 runs on faster-qwen3-tts, an engine by andimarafioti that runs the stock Qwen weights and captures the generation step in CUDA graphs: the GPU kernel launches for one decoding step are recorded once and replayed on every later step, so the GPU stops waiting for the CPU to issue each small kernel. When steps are short, that overhead is a real share of the time.

Qwen3-TTS does not write waveforms directly. It writes codec tokens, 16 codebooks at 12.5 frames per second, which the Qwen3-TTS-Tokenizer-12Hz turns back into sound, so the server can decode a handful of frames into PCM, the plain signed 16-bit samples a sound card plays, and send them while the model generates the next ones. With response_format set to pcm, the chunked HTTP response carries s16le mono at 24 kHz, ready to pipe into a player.

Measured on the Spark, v0.3.0 delivers the first audio chunk in about 0.4 s and then sustains about 2 s of audio per 1.16 s of wall time, roughly 1.65 to 1.75x realtime. Without streaming, the same build makes a caller wait about 1.48 s for a 1.92 s clip and about 12.7 s for a 22 s reply, long enough to hang up. Inside the voice agent, with streaming on, speech starts about 1.0 s into the turn, against around 2.1 s end to end without it.

Two voices in one server: design, clone and the library

The server keeps two models resident, about 4 GB of CUDA memory each for the 1.7B weights, so switching between them is a field in the request rather than a redeploy. qwen3-tts is VoiceDesign: you describe the voice you want in plain language, per request, in the voice field or in instructions, and "an elderly British man, gravelly and warm" is a valid voice. qwen3-tts-clone is the Base model used for zero-shot cloning, zero-shot meaning the model imitates a speaker it was never trained on from a reference clip of about 3 seconds. You pass ref_audio as a path or a data URL along with ref_text, the transcript of that clip, and the reply comes back in that voice; xvec_only, append_silence and parity_mode are the finer clone knobs.

Shipping a reference clip with every call gets old, so there is a voice library. The operator mounts a directory at /voices holding, for each voice, an <id>.wav reference and an <id>.txt transcript, and GET /v1/audio/voices lists what it finds there:

Code, JSON{"id": "einstein", "audio": "/voices/einstein.wav", "has_ref_text": true, "ref_text": "/voices/einstein.txt"}

From then on a client sends model: qwen3-tts-clone with voice: einstein and nothing more. The reference deployment carries 11 voices, most of them named for public figures: dali, dekker, einstein, hemingway, jobs, mcafee, mckenna, musk, neville, tesla, watts. The library is operator-managed on purpose, so its contents are the operator's responsibility; cloning a living person's voice without consent is not something I would do for anything that leaves the lab. Build yours from voices you have the right to use.

The request body and the headers worth reading

Only input is required; a body that fails validation comes back as a 422.

Supported languages are auto, Chinese, English, French, German, Italian, Japanese, Korean, Portuguese, Russian and Spanish; short codes like zh were a v0.1 habit. Sampling defaults are a temperature of 0.9, top_k 50, top_p 1.0, a repetition penalty of 1.05 and max_new_tokens 2048 within a range of 1 to 8192. Read x-audio-sample-rate on the way back rather than hard-coding 24000.

Which build you are talking to

Until then, the deployed-equivalent stack is the upstream packaging of the same engine, the DGX Spark image from mARTin-B78, which serves the stock VoiceDesign and Base weights and reads a voice library from /voices:

Code, shelldocker pull martinb78/faster-qwen3-tts-dgx-spark:latest-streaming
curl http://<host>:8002/health

A v0.3.0 server reports "backend": "faster-qwen3-tts"; a v0.1 server answers only {"status":"ok","model_loaded":true}, and nothing below will stream from it. Streaming to a speaker is one line, and -N matters, because without it curl buffers the whole reply:

Code, shellcurl -N -X POST http://<host>:8002/v1/audio/speech -H 'Content-Type: application/json' -H 'Authorization: Bearer <YOUR_API_KEY>' -d '{"input": "Streaming means playback starts before generation finishes.", "voice": "A bright, expressive female voice with natural pacing.", "response_format": "pcm", "stream": true}' | pw-play --rate 24000 --format s16 --channels 1 -

aplay -r 24000 -f S16_LE -c 1 works in place of pw-play, and swapping the body for model: qwen3-tts-clone with voice: einstein gives the same stream in a library voice. For the OpenAI SDK, set base_url to http://<host>:8002/v1. The voice agent points VOXTRAL_BASE_URL at the same path with VOXTRAL_STREAMING=true, and in checked-in code the host stays a ${SPARK_HOST} substitution, never a literal address. The legacy build is still there for VoiceDesign without streaming: docker compose up -d in the repository brings it up on the aeon-stack network, pulls an image of about 33 GB, wants at least 8 GB of free RAM, and is ready about 10 to 20 s after the first boot.

Sharing one Spark with a 27B model

In my pairing the LLM main is Qwen 3.6 27B AEON Ultimate Uncensored under vLLM, served from the Spark container; the Mac-native builds of that model have their own article. The architecture notes budget the 128 GB as about 96 GB for the LLM at a gpu-memory-utilization of 0.75, 5 to 10 GB for ASR, about 10 GB for the host and about 10 GB of margin; the two TTS models at roughly 4 GB each come out of that margin, so the fit is tight. Push a 27B or larger vLLM main toward the cap and the TTS server fails with an out-of-memory error; the fixes are to lower the LLM to --gpu-memory-utilization 0.70 or to run a 0.6B TTS variant at about 2 GB, a switch QWEN_TTS_MODEL offers on the v0.1 build only, and never to push utilization past 0.88 on a unified-memory Spark.

The last rule is about the network. The API key is operator-configured and enforcement depends on the deployment, so treat port 8002 as unauthenticated until you have verified otherwise, and never bind it to a public interface without an auth proxy in front. That done, the speech endpoint takes its place in the catalog beside the rest of the stack.

Sources

• qwen3-tts-server on GitHub, Apache-2.0; legacy image ghcr.io/aeon-7/qwen3-tts-server:latest
• Engine and packaging: andimarafioti/faster-qwen3-tts, mARTin-B78/dgx-spark-faster-qwen3-tts
• Model cards: Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign, Qwen/Qwen3-TTS-12Hz-1.7B-Base, Qwen/Qwen3-TTS-Tokenizer-12Hz; weights under the Qwen license
• Siblings: qwen3-asr-server, matrix-voip-agent, Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash