Aeon Magick Orb: a Raspberry Pi that gives an AI agent eyes and hands

Most tools that let an AI agent use a computer live inside the computer they drive, so each needs a booted operating system, a logged-in user and a permission prompt someone has clicked. None of them can see a BIOS menu, type a FileVault disk-encryption password or walk through the Windows out-of-box setup on a machine with nothing on it yet.

The Aeon Magick Orb is my answer, and it is a Raspberry Pi. The target's HDMI output runs through a capture device into the Pi, which gives the agent eyes. The Pi's USB-C port runs in gadget mode, in which the Pi presents itself to a host as a USB device rather than driving one, and appears as a keyboard, a mouse or a trackpad, which gives the agent hands. The target sees a peripheral and a display; nothing is installed on it, and revoking the agent means revoking a token or pulling the cable.

From the other end of the cable the Orb looks ordinary. The target enumerates a USB HID device, HID being the human interface device class every keyboard and mouse belongs to, and firmware trusts a keyboard before any operating system exists, so a BIOS menu and a FileVault prompt take its keystrokes and a machine can be rebooted into its boot menu and reinstalled from the other side of the planet. That is pre-OS control.

Everything the Orb does is exposed as a REST API and as an MCP endpoint, MCP being the Model Context Protocol, the open standard an agent uses to discover and call tools. Everything else in the box serves one loop: see, decide, act, look again.

Eyes in, hands out

The eyes are a capture device: on the Pi 4 an Elgato Cam Link 4K or a cheap MS2109 HDMI-to-USB stick, both UVC devices, the USB video class, so they need no drivers, with up to 1080p60 from the Cam Link. An agent fetches single frames as JPEG snapshots.

The hands are aeon-hid, which builds the USB gadget through ConfigFS, the kernel's filesystem interface for describing one, and ships five hot-swappable personas. generic-absolute is the one for agents: an absolute pointer, so move_abs and click_at take screen fractions from 0 to 1 and land where they aim. Four others cover a relative mouse, a Logitech receiver and two Apple layouts. Switching costs about 1 s of USB re-enumeration on the target.

The input API is deliberately atomic: type, key chord, click and move, with no key_down, because a dropped packet after key_down would leave a key stuck on a machine nobody is sitting at. On a LAN a frame arrives in 40 to 80 ms on either Pi, a snapshot round trip fits in roughly 50 to 150 ms and a HID action takes about 10 to 30 ms. With the admin password in $PW, two calls look at the screen and type on it:

Code, shellcurl -sk -u admin:$PW "https://aeon-magick.local/api/streamer/snapshot" -o frame.jpg
curl -sk -u admin:$PW -X POST -H "Content-Type: application/json" -d '{"text": "Hello, world."}' https://aeon-magick.local/api/hid/type

The MCP endpoint exposes 67 tools, gated by the scope of the calling token, so an agent drops in one URL.

What went wrong on the way to 40 milliseconds

The capture pipeline took the most scars. Packed YUYV frames from the Cam Link pegged the Pi's CPU at 1080p, so the streamer now prefers planar NV12 or YU12.

The latency hunt ended somewhere I did not expect. The dominant factor was not codec or resolution but the Ethernet path between the Pi and the browser, and moving the Pi to WiFi dropped the visible latency to near zero. H.264 needs WebCodecs, the browser API for hardware video decoding, which Chrome, Brave and Edge have and Safari and Firefox do not, so the shipped default stayed MJPEG, a stream of JPEG frames.

The worst scar was not in the code. The image build works on a copy of the pi-gen stage, pi-gen being the Raspberry Pi OS image builder, and forgetting to sync that copy produced a "v74" image missing a session of work. Now I verify the contents of an image, never its inputs.

On-device vision on the Pi 5

On the Pi 5 the Orb can read the screen itself, with no frame leaving the room, through a Hailo HAT, an NPU, a neural processing unit, in the Pi's add-on board format. The AI HAT+ carries a Hailo-8 or 8L that borrows the Pi's system RAM, so the docs ask for an 8 GB Pi 5 at minimum and recommend 16 GB. The AI HAT+ 2 carries a Hailo-10H with 8 GB of dedicated memory, about 5.5 GB of it usable, and is the only one of the three that runs generative models.

screen_text returns all OCR text, optical character recognition, with boxes. screen_find takes a label such as Save and returns a click-ready center, and because it reads the live OCR daemon it needs no generative slot. describe_screen runs Qwen2-VL, a vision language model that takes an image and answers in words. It is 2.2 GB, answers in about 3 s warm and about 13 s cold, and unloads after about 180 s idle. It shares the Orb's single generative slot with the local LLM, so when that LLM is resident, screen_find is the tool to use.

One honest note. The X1301 HDMI-to-CSI bridge, which feeds HDMI in over the Pi's camera ribbon through a Toshiba TC358743, and the Waveshare UPS HAT (E) support were written from documentation and carry a hardware-in-the-loop notice, not verified on physical boards at the time of writing, so run their post-flash checklists. The capture check should read about 59.94 fps; if no tc358743 is found, change cam1 to cam0, and if dma-buf allocation fails, add cma=128M to cmdline.txt.

Pi 4 or Pi 5

The two boards are separate release tracks, aeon-magick-pi4 on Raspberry Pi OS Bookworm and aeon-magick-pi5 on Trixie. The Pi 4 is the tested, lowest-cost path; the Pi 5 trades the hardware H.264 encoder for PCIe, CSI capture, voice and UPS support. Board quirks are detected at boot, so the Pi 4 image boots a Pi 5 in a pinch.

A kernel regression from 6.6.42 onward, tracked as raspberrypi/linux issue 6289, breaks gadget enumeration over a C-to-C cable on the Pi 5, so wire a Pi 5 from a USB-A port on the target. A charge-only cable is the other trap: the Pi powers up, no keyboard appears, and /api/state reports keyboard.online = false.

Three Rust daemons and a supervisor

Behind the SvelteKit web UI the Orb is three Rust daemons. aeon-streamer owns capture and encoding, aeon-hid owns the gadget, the personas and the atomic input API, and aeon-supervisor is the front door: HTTPS on rustls, a TLS library written in Rust, authentication with argon2, a memory-hard password hash, and the REST, MCP and dashboard routes. They are separate processes for failure isolation, for sandboxing and so that each restarts on its own. Premade images baked with pi-gen go to members of my supporter program, BUILDING.md covers building from source, and a Docker image runs the hardware-free parts on any server, the subject of Magick Orb Docker.

The design descends from a dongle built on an ESP32-S2, a small microcontroller with native USB, and borrows from PiKVM, the open-source KVM-over-IP project, a KVM being a keyboard, video and mouse switch reached over the network, as a sibling rather than a fork. Its one hard rule, printed in the README and the agent manual, is that it drives your own computers, accounts and services, never other people's, and never to get around fraud controls or device attestation.

The jump box and the lights-out KVM

With a trusted SSH key the box can do more than type. The Agent Dashboard adds systems over SSH or from a Tailscale tailnet, Tailscale being the WireGuard-based mesh that gives each machine a stable address wherever it is, and shows GPU, CPU, RAM and temperature per host, containers and compose projects, web terminals and per-agent token telemetry. One-click deploy pushes a model to a Spark running the AEON vLLM Ultimate image, with --gpu-memory-utilization, --max-model-len, --max-num-batched-tokens and --max-num-seqs beside the button, and the same console deploys the Aeon Bench pod onto a GPU host.

Upload an ISO and the Pi exposes it as a read-only USB CD-ROM; call target_reboot, send the chord for the boot menu and install an operating system without touching the machine, provided the USB link carries passthrough power. Tokens come as admin, full, macros or read, agent provisioning issues only read or full, and minting tokens, granting SSH, enrolling systems and controlling power are human-admin actions hidden from agent tokens. An audit log records actor, method, path and outcome.

The Aether: sharing models over IPFS

Open models are only as available as the servers that host them, so every Orb runs kubo, the reference IPFS node, IPFS being the peer-to-peer file network in which a file is named by the hash of its bytes. Orbs gossip their catalogs on a pubsub topic and find each other through a rendezvous CID, a content identifier that exists only to be looked up, on the Amino DHT, the public distributed hash table of IPFS.

Importers verify what they bring in from Hugging Face, Ollama and Civitai against the hashes those sources publish, and a download from a peer passes through quarantine: a disk check for roughly 2x the model size, the download, file-type validation by magic bytes, a ClamAV antivirus scan, and only then a pin. Executables, scripts, pickle or zip weights, which can run code when loaded, and fake extensions are rejected.

On provenance I want to be precise. What ships today is content addressing plus hash verification on import: the CID is the integrity check. The publisher identity, an ed25519 keypair recoverable from a 24-word BIP39 seed, has its signing primitives in place but is not yet wired into the publish path, so there is no signed badge yet and nothing on the Aether should be called signed.

The privacy stack

The Orb's third life is as a router. The same USB-C that carries the keyboard can present a USB Ethernet adapter and route the target's traffic through the Pi at 250 Mbit or more over USB 3.0, and the Pi can wrap that traffic in WireGuard or OpenVPN; in Tor, the onion-routing network, as a split or transparent proxy; in I2P, an anonymous overlay network; and in DNSCrypt v2 or DoH, two encrypted DNS transports, so no plaintext DNS leaves the box. At the v66 snapshot the catalogs held 226 resolvers and 187 anonymized relays, filterable by latency and by jurisdiction, with a No-Eyes filter that stays outside the 5, 9 and 14-Eyes alliances.

Three lessons from running it. Tor over a commercial VPN only bootstraps through an obfuscated transport such as AirVPN's SSL or SSH stealth modes; plain WireGuard exits get tar-pitted by Tor. Brave and Tor Browser bring their own Tor, so keep the Pi in split tunnel for them, and turn off the browser's Secure DNS or queries never reach the Pi. Paste VPN keys into the device's wizard, never into a chat with an agent.

Getting started

Flash image_vNN-aeon-magick.img.xz from the repository releases with Raspberry Pi Imager as a custom OS, without decompressing it first. Power the Pi on without Ethernet; after about 90 s offline it raises an aeon-setup hotspot whose captive portal lets you pick your WiFi. Then open https://aeon-magick.local/, accept the per-device self-signed certificate and set the admin password.

Wire the target's HDMI out to the capture device and its USB-A port to the Pi's USB-C, then take a snapshot. The 64 KB clipboard, which survives reboots, hands an agent a long credential without typing it character by character. The rest of what I publish is in the Aeon Forge catalog.

Sources

• Repository: AEON-7/aeon-magick-ai-computer-control, MIT license, with AGENTS.md, ARCHITECTURE.md, HANDOFF.md, BUILDING.md and the docs folder
• Headless Docker Orb: AEON-7/orb-server
• Benchmark pod: AEON-7/Aeon-Bench-Pod
• Pi 5 gadget regression: raspberrypi/linux issue 6289
• Lineage: PiKVM