Matrix VoIP agent: call your AI, and let it see through your camera

I wanted to phone my AI: pick up Element on my phone, tap the call icon beside a contact, and talk to a model running on my own hardware while I cooked. The pieces existed: speech recognition, language models that call tools, speech synthesis, a chat protocol with calls built in. The thing in the middle that picks up did not.

matrix-voip-agent is that thing: a headless voice and video call agent for Matrix, the open, federated chat protocol behind Element, nheko and FluffyChat. It auto-answers calls, can place its own, and holds a spoken conversation in real time: speech-to-text, a direct request to any OpenAI-compatible language model, then text-to-speech. Any Matrix client becomes a button that means "dial the AI", and an optional video add-on lets the model look through your camera, with nothing it sees ever written to disk. This is how it came together, including the version that needed four seconds to say hello.

Matrix carries the call, not the conversation

Matrix handles only the signaling: the invite, the answer and the hangup. The media travels over WebRTC, the browser standard for real-time audio and video, directly between the caller's client and the agent, and the spoken exchange never passes through a Matrix room; the transcript goes to the language model over plain HTTP with no gateway in the loop.

The agent logs in with a bot account on your homeserver, the server that holds your Matrix accounts: Dendrite, Synapse or Conduit. A homeserver alone cannot carry audio between a phone on cellular data and a box at home; that needs a TURN server, a relay that forwards media when two parties cannot reach each other directly. A turnkey setup-homeserver.sh installs Dendrite, PostgreSQL, Caddy with Let's Encrypt, coturn and a dynamic DNS updater in Docker.

The first version took four seconds to answer

My first working stack was the obvious one: whisper.cpp for recognition on the CPU, so no GPU was required, and ElevenLabs in the cloud for the voice. It worked, and it was slow in a way you feel in your body. A turn cost about 4 s: roughly 1.5 s for the base Whisper model to transcribe, about 1.7 s for the language model, and about 0.4 s per sentence of synthesized speech. Four seconds of silence after every sentence is not a conversation; it is voicemail. Swapping the recognizer for OpenAI Realtime brought a turn to about 3 s, better and still not good, and my voice was now leaving the house for two companies.

All three remain as fallbacks: whisper.cpp for a host without a GPU, where the 142 MB base model is the sensible choice, and the cloud services for anyone who prefers them. The recommended path is now fully local, and it answers in about a second.

Auto-answer and the PipeWire bridge

When a call arrives from an authorized user, the agent answers, no ringing and no human in the loop; the AUTHORIZED_USERS list in .env is the entire access policy. The caller's Opus-encoded audio arrives through werift, the agent's WebRTC library.

The plumbing between a network call and a speech model is PipeWire, the Linux audio server. Decoded audio runs through a few virtual devices: a sink for the caller's voice, a capture source that pw-record reads as 16 kHz PCM for the recognizer, and a TTS sink paired with a source that feeds the synthesized voice back into the call. Replies play through pw-play --rate at the backend's declared rate, 24 kHz for the Qwen3 TTS server, then go back out Opus-encoded.

The agent runs as a systemd user unit with loginctl enable-linger, so nobody needs to stay logged in, and one shared build can serve several personas, each with its own working directory, .env, API port and PipeWire device names, behind one shared backend; create-agentic-personas goes deeper.

A one-second budget on a single Spark

The recommended stack runs three sidecars on one GPU host, validated on the DGX Spark; the docs say any Blackwell consumer GPU works. The language model is Qwen3.6-27B AEON Ultimate MTP-XS, from the AEON Ultimate Qwen family, in NVFP4, a four-bit floating-point format with per-block scales that Blackwell tensor cores run natively, served by the vLLM image with DFlash speculative decoding, in which a small drafter proposes up to 15 tokens and the large model verifies them in one pass. Recognition is Qwen3-ASR-0.6B behind the qwen3-asr-server, the voice comes from the qwen3-tts-server, and all of them sit in the catalog. Together they want about 110 GB of the Spark's unified memory: 96 GB for the LLM at --gpu-memory-utilization 0.75 with a 32768-token context, 10 GB for ASR, 4 GB for TTS, and headroom. Measured hot on that one machine, after the roughly 3 minutes the LLM needs to load on first boot, the budget adds up like this.

Streaming speech, and thinking switched off

Two choices buy most of that second. The first is streaming synthesis: with VOXTRAL_STREAMING=true the agent asks for raw PCM as a stream and pipes it into playback as it arrives, so the first audio lands in about 0.4 s rather than after the roughly 1.48 s a 1.92 s clip takes to synthesize in full, and a long reply is spoken as it is produced, at about 1.7x realtime. The VOXTRAL_ prefix is historical; any OpenAI-compatible speech endpoint works.

The second is switching the model's thinking off for casual talk: VLLM_VOICE_THINKING_MODE takes auto, on or off; reasoning off cuts the LLM leg to about 480 ms, and auto saves it for hard questions. When the model does reach for a tool, filler phrases such as "Let me take a look." cover the wait. One honest caveat: the 480 ms belongs to this model with DFlash; another LLM can push that leg past 2 s.

Letting it see: the VP8 frame ring

A language model handed a camera stream at full frame rate is a model drowning in tokens, so the design is receive-only, lazy and small.

When an incoming offer carries a video track and VIDEO_ENABLED=true, the agent adds a receive-only video transceiver. VP8 packets, the video codec these calls use, pass through a jitter buffer that restores their order and a depacketizer that reassembles whole frames, then into one persistent ffmpeg process per call with the filter fps=1,scale=512:-2, which keeps one frame per second, the rest dropped before decode so CPU load stays flat whatever the camera sends, and scales it to 512 pixels wide. ffmpeg emits MJPEG, and each JPEG lands in an in-memory ring buffer of the last 8 frames, oldest overwritten first. A VP8 decoder cannot start mid-stream, so the sampler sends RTCP PLI messages, picture loss indications that ask for a fresh keyframe, at call start, on the first packet, and every 5 s that the stream stalls.

The per-call look tool

The model never sees that stream directly. A look tool is registered per call, and only when the call has a live video track; a voice-only call presents exactly the tool list it always had, and with video disabled the audio path is byte-identical to before. When called, look returns 1 to 4 of the freshest JPEGs, optionally spread across spread_seconds so the model can judge motion. If you would rather it always have eyes, VIDEO_AUTO_ATTACH=latest attaches the newest frame to every user turn, at a cost in tokens and none in latency.

I tested the vision side with the Gemma 4 26B multimodal model on vLLM; if a model's chat template refuses images inside tool messages, VIDEO_LOOK_IMAGE_ROLE=user delivers them as a user message instead. Two limits: only classic 1:1 video calls from Element Web or Desktop work, not Element Call or MatrixRTC group calls, and every video failure is fail-soft, so audio keeps working.

Privacy by design

A model with a camera deserves suspicion, so the rules are strict and few. Camera frames live only in RAM, in that small per-call ring buffer, and are discarded on hangup. Nothing is ever written to disk, and call transcripts, saved on hangup as Markdown and JSON, are text only. In the offline stack no audio leaves your network; the only packets that cross the internet are Matrix federation traffic, which can be end-to-end encrypted, with E2EE enabled through MATRIX_E2EE_ENABLED and the agent's own crypto store.

The access policy is just as plain. AUTHORIZED_USERS is required, and only the Matrix IDs listed there may call. MAX_CONCURRENT_CALLS defaults to 1, CALL_TIMEOUT_MS hangs up after 30 minutes by default, and the HTTP API that places outbound calls listens on port 8179 behind a bearer token. The repository's AGENTS.md adds hard rules: never commit .env, secrets/, crypto-store/ or transcripts/, and mark any LAN address in the docs as an example.

Getting it running

The turnkey path builds the homeserver and the agent on one Linux machine, Ubuntu 22.04 or newer, with PipeWire, Node.js 22, ffmpeg, libopus and cmake:

Code, shellgit clone https://github.com/AEON-7/matrix-voip-agent.git
cd matrix-voip-agent
bash setup-homeserver.sh
bash setup.sh

With a homeserver already in place, skip the first script. The three sidecars start on the GPU host as Docker containers on one network, the LLM first because it is the heaviest; docs/FULLY-OFFLINE-VOICE.md has the exact commands, and the essentials of the agent's .env then point at them, SPARK_HOST being the GPU host's address and OMNI_ASR_BASE_URL the full /v1 URL:

OMNI_ASR_ENABLED=true
OMNI_ASR_BASE_URL=http://${SPARK_HOST}:8001/v1
OMNI_ASR_MODEL=qwen3-asr
VOXTRAL_ENABLED=true
VOXTRAL_BASE_URL=http://${SPARK_HOST}:8002/v1
VOXTRAL_MODEL=qwen3-tts
VOXTRAL_STREAMING=true
VLLM_BASE_URL=http://${SPARK_HOST}:8000/v1
VLLM_API_KEY=ignored
VLLM_MODEL=qwen36-ultimate-xs
WHISPER_ENABLED=false

Enabling video takes three steps: set VIDEO_ENABLED=true, restart with systemctl --user restart matrix-voip-agent.service, then place a video call from Element and ask it what it sees. The sampler can be checked without a call:

Code, shellnpm run build
node scripts/smoke-video.mjs

It should print PASS with at least 2 JPEG frames, and a healthy call logs Offer contains video - added recvonly video transceiver, Video sampler started (fps=1, width=512, ring=8) and First video frame decoded.

Sources

• matrix-voip-agent on GitHub, MIT license: README, AGENTS.md and docs/FULLY-OFFLINE-VOICE.md
• Sidecars: qwen3-asr-server, qwen3-tts-server, vllm-ultimate-dgx-spark
• Models: aeon-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-MTP-XS, drafter z-lab/Qwen3.6-27B-DFlash, deploy guide Qwen3.6-27B-AEON-Ultimate-Uncensored-DFlash
• Speech engine: faster-qwen3-tts, Spark packaging
• Element, AEON-7 profile