Create agentic personas: a voice, a knowledge base and a phone line

I wanted characters I could talk to the way I talk to people: send one a message in a chat room, or ring it and hear it answer in its own voice, on hardware I own rather than through a cloud voice service. Today the lab runs about 30 distinct voice-and-chat personas on one workstation, sharing one inference backend and one Matrix homeserver, and create-agentic-personas is the generalized, secret-free writeup of how that is built: a guide, scripts, templates and one agent skill.

The idea that made it tractable is that a persona is not a program but a composition of five independent layers: identity, personality, knowledge, voice and a call line. The shared infrastructure is built once; every persona after that is a workspace of markdown files, an .env, a Matrix account and a voice, so the thirtieth costs the same as the second.

What a persona is made of

Identity is an OpenClaw agent plus a Matrix account on the homeserver, the server that holds the accounts and rooms: the name you can message directly or @-mention. Personality is a few markdown files in workspace-<name>/. SOUL.md is the one with leverage, written in the second person, and it sets the voice, the values and the manner; IDENTITY.md holds the facts; USER, TOOLS, SAFETY, HEARTBEAT and AGENTS inherit defaults from the templates.

Knowledge is the optional layer. Gather public-domain sources, normalize them into an Obsidian-style vault of small titled notes, then chunk, embed and index them with any local embedding model into a RAG store, where RAG (retrieval-augmented generation) means the agent reads passages from its own corpus before it answers. The corpus lives outside the agent directory so it stays swappable, the same vault pattern as my second brain.

Voice is a Qwen3-TTS timbre that stays recognizably that character. The call line is an account you can ring from Element, the Matrix client, that picks up, talks and can ring you back.

The three sidecars

The last two layers rest on three smaller projects, sidecars in the sense of small services beside the main one, each with its own article here. qwen3-asr-server puts Qwen3-ASR, the speech recognizer, behind an OpenAI-compatible endpoint on port 8001. qwen3-tts-server does the same for Qwen3-TTS on port 8002, cloning and designing voices from one process. matrix-voip-agent is the Node.js program that answers a WebRTC call in a Matrix room and carries the audio through PipeWire, the Linux audio server, to those two services and back. vLLM on port 8000 serves a Qwen3-class model under two aliases, fast for calls and deep, with thinking, for chat; the docs/01 example docker run uses --max-model-len 32768 and --gpu-memory-utilization 0.80 with a placeholder model path.

The reference shape is two machines, a GPU box for those three services and an agent workstation for the OpenClaw gateway and the voice agents, and it collapses to one. The homeserver is Dendrite with Postgres behind Caddy for TLS. Coturn provides TURN, the relay that WebRTC falls back on when two peers cannot reach each other directly; both can live on a small VPS for calls from outside your network.

The call path, and the golden rule of TURN

The golden rule in docs/03: TURN URIs must point at the server's real IP, never at a domain fronted by a CDN, or you get one-way audio, and the relay UDP range must be open end to end. Matrix carries only the signaling; after pickup the audio never touches the homeserver again: caller, WebRTC, PipeWire, Qwen3-ASR, vLLM, Qwen3-TTS, PipeWire, WebRTC, caller. On a call from Element with the fast alias and thinking off, the gap from the end of your sentence to the first syllable of the reply is about 5 to 8 seconds, of which recognition takes about 1 to 2 and the language model another 1 to 2. That is not yet a human pace, but it is a conversation.

One command per persona

scripts/new-persona.sh is where the sidecars compose. Given a name and a port, it seeds the workspace and an annotated ~/voip-<name>/.env, creates empty secrets/ and crypto-store/ directories at mode 700, drops the two PipeWire loopback scripts and writes three systemd user units: a loopback for speech in, one for speech out, and the voice agent. It creates no account and starts no service, because I wanted a scaffold I could read before anything ran. The shared layer is verified first with curl checks against ports 8000 to 8002 and 8008.

./setup.sh --infer-host INFER_HOST
./scripts/new-persona.sh ada --port 8210
$EDITOR ~/.openclaw/workspace-ada/SOUL.md
$EDITOR ~/.openclaw/workspace-ada/IDENTITY.md
openclaw agents add ada --workspace ~/.openclaw/workspace-ada --model vllm/qwen36-deep
openclaw gateway restart
systemctl --user daemon-reload
systemctl --user enable --now ada-voip-stt ada-voip-tts matrix-voip-agent-ada
curl -s http://127.0.0.1:8210/status -H "Authorization: Bearer $ADA_API_TOKEN"

Each persona owns one API port, 8210 and up, with a localhost API behind a bearer token: POST /call, POST /hangup and GET /status. A /call with a roomId, a userId and a greeting makes the persona ring you, and a script can trigger it. Who may summon it in chat is mention-gating, an allow list in ~/.openclaw/credentials/matrix-<name>-allowFrom.json; who may call it is AUTHORIZED_USERS in the .env. The latency flags live there too, VLLM_VOICE_FAST_MODEL with VLLM_VOICE_THINKING_MODE=off for calls and VOXTRAL_STREAMING=true for sentence-by-sentence synthesis. Defaults are conservative: MAX_CONCURRENT_CALLS is 1, CALL_TIMEOUT_MS is 1800000 (30 minutes) and VOICE_HISTORY_MAX_MESSAGES, the multi-turn memory window, is 24.

A roster is a loop:

Code, shellfor p in ada grace tesla; do ./scripts/new-persona.sh "$p" --port "$((8210 + i++))"; done

It scales on three facts: one shared language model serves everyone, audio is isolated per agent so calls do not bleed into one another, and chat for every agent runs inside one gateway process, so you size for concurrency, not for the count of names.

Clone a voice or design one

One Qwen3-TTS server does both jobs; the persona chooses with VOXTRAL_TTS_MODE in its .env, whose VOXTRAL_ prefix is legacy naming that now drives Qwen3-TTS. voice_clone wants 10 to 30 seconds of clean, single-speaker reference audio and a VOXTRAL_VOICE name. voice_design wants a VOXTRAL_VOICE_DESCRIPTION and imitates no one, so the skill defaults to it when unsure. The guardrail is blunt: clone only "the user's own voice, a rights-cleared voice, or a public-domain recording."

Audition it first:

Code, shellcurl -s http://INFER_HOST:8002/v1/audio/speech -H "Content-Type: application/json" -d '{"model":"qwen3-tts-clone","voice":"ada","input":"Hello, this is a voice test."}' --output /tmp/ada-test.wav

Designed voices can drift between calls, so save a seed or use the tts-voice-designer skill's voice lock in ComfyUI. Stage directions such as "(whispering)" are spoken literally, so the personality files must forbid them.

Secret-free by construction

Every host, domain, token and key in the repository is a placeholder: INFER_HOST, SERVER_IP, matrix.example.com. Real secrets stay in the per-persona .env, secrets/ and credentials/, with secrets/ and crypto-store/ created at mode 700. scripts/gen-secrets.sh mints the random secrets to stdout and never writes a file. SECURITY.md ships a pre-push grep scrub to run before a first public push. The network rule is as plain: inference on 8000 to 8002, the homeserver on 8008 and the per-agent APIs on 8210 and up stay off the public internet; only 443 and the TURN ports face out: 3478 over UDP and TCP, 4443 over TLS and the relay range 10000 to 10100 over UDP.

The skill that builds the next one

skills/create-agentic-persona/SKILL.md hands the whole loop to an existing OpenClaw agent, so an operator persona can scaffold its colleagues. It confirms the name, the voice mode, whether there is a corpus and who may summon the result. It scaffolds, drafts the personality files and waits for approval, picks or designs a voice and auditions it, registers the agent, binds the Matrix account and sets the gate, brings up the call line, then verifies and reports without pasting a token into the chat. The honesty rules travel with it: legal corpora only, no scraping, an AI label wherever a human would be assumed, and a historical persona says it is an interpretation rather than the real person.

What else broke along the way

Qwen3-ASR sometimes transcribes a phantom phrase out of pure silence, so gate on energy or voice activity detection, or filter the known strings. The TTS server can wedge mid-stream under load, so keep a restart one-liner and a healthcheck. A monitoring shim that shells out to the openclaw CLI must kill the whole process group on timeout, or helper processes leak. An encrypted room that says "unable to decrypt" has an unverified crypto store: enable automatic cross-signing, Matrix's device verification, and restore the key backup on start. Do not make the persona announce its tool calls; the filler phrase is automatic. And a quantized, abliterated (refusals removed by editing the weights) or fine-tuned derivative may carry a different license than the Apache-2.0 base model, so read its card.

Sources

• Repository: create-agentic-personas, AGENTS.md runbook, create-agentic-persona skill
• Sidecars: matrix-voip-agent, qwen3-tts-server, qwen3-asr-server
• Model cards: Qwen3-TTS-12Hz-1.7B-VoiceDesign, Qwen3-ASR-0.6B
• Upstream: OpenClaw, Dendrite, Coturn, vLLM