Gemma 4 12B AEON Abliterated on a Mac: the MLX toolkit and server

Gemma 4 12B AEON Abliterated began on the NVIDIA side of my lab, where the BF16 weights and their FP8 and NVFP4 siblings run under vLLM on the Spark. This toolkit carries that same abliterated model to Apple Silicon, for Mac owners who want a local 12B that reads text, looks at images and listens to audio, with the refusal reflex removed, behind an OpenAI-style endpoint their existing tools already speak.

Gemma 4 12B is Google's gemma4_unified architecture, one set of weights for text, image and audio. Abliteration is a weight edit rather than a fine-tune: it finds the directions inside the model that encode refusal and projects them out, so the model keeps what it knows and loses the habit of saying no. Every number below came from one MacBook Pro with an M4 Pro and 48 GB of unified memory, the single pool that the CPU and GPU share.

What the toolkit rules out

The toolkit's operator guide, AGENTS.md, sets hard rules, and two of them decide the runtime. The first is no inference inside Docker on macOS. Containers are how the NVIDIA builds travel, but on a Mac, Docker runs Linux inside a virtual machine with no passthrough for Metal, Apple's GPU interface, so a containerized MLX server runs on the CPU alone. The toolkit's image on GHCR, GitHub's container registry, only delivers Metal speed when its scripts run on the Mac host.

The second is no vllm-metal for these models. As of June 2026 its documentation lists Gemma 4 as experimental with multimodal not supported, and its text-only path matches the model type gemma4 exactly, so it declines this gemma4_unified checkpoint. The text-only mlx_lm package cannot load gemma4_unified either and answers "Model type not supported".

What works is mlx-vlm, the vision and audio sibling of mlx_lm. Its mlx_vlm.server exposes /v1/chat/completions, the same route and request shape the OpenAI API uses, and accepts image_url and input_audio content with no extra flags. I built both quantizations with mlx 0.31.2 and mlx-vlm 0.6.1.

The one-paste quick start

Two builds ship. MLX-8bit, the near-lossless flagship, is 13.4 GB on disk and needs 24 GB of unified memory at minimum, 32 GB or more to be comfortable. MLXFP4, the compact build, is 9.3 GB and runs on a 16 GB Mac, happiest with 18 to 24 GB. Both run on any Apple Silicon Mac, M1 or newer. Pick one and paste the block: the first two lines install uv, Astral's Python package manager, and the third fetches Python 3.12 and mlx-vlm on first run, pulls the model from Hugging Face and starts the server on port 8080.

Code, shellcurl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv run --python 3.12 --with mlx-vlm -- python -m mlx_vlm.server --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLX-8bit --port 8080 --max-kv-size 32768

For the compact build, swap the model name and the context cap.

Code, shelluv run --python 3.12 --with mlx-vlm -- python -m mlx_vlm.server --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4 --port 8080 --max-kv-size 16384

The --max-kv-size caps are safe for each build's smallest Mac, and raising them is covered below; add --host 127.0.0.1 if the server should stay local.

Then call it like any OpenAI endpoint, with the model field set to the id the server printed at launch. One detail matters: the server defaults to greedy decoding, always the single most likely token, which can repeat or loop on long prompts, while both builds are tuned for temperature 1.0, top_p 0.95 and top_k 64, so set the temperature on every request.

Code, shellcurl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"<launched-id>","messages":[{"role":"user","content":"Explain mixed-precision quantization."}],"temperature":1.0}'

Images and speech through the same endpoint

Gemma 4 is encoder-free: there is no separate vision or audio tower, and image and audio fidelity live in two small projectors, embed_vision and embed_audio, which map pixels and waveforms into the space the text tokens occupy. Both builds keep them at bf16, the full 16-bit precision of the source, so the multimodal path works unchanged. An image goes in as an image_url entry, an http link or a data URI, and audio as input_audio carrying a base64 wav or mp3; the ellipsis below stands for your payload. Through the server, the model described my test image and transcribed speech correctly.

Code, shellcurl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"<launched-id>","messages":[{"role":"user","content":[{"type":"text","text":"Describe this image."},{"type":"image_url","image_url":{"url":"https://example.com/photo.jpg"}}]}],"temperature":1.0}'
curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"<launched-id>","messages":[{"role":"user","content":[{"type":"text","text":"Transcribe this."},{"type":"input_audio","input_audio":{"data":"data:audio/wav;base64,…","format":"wav"}}]}],"temperature":1.0}'

For transcription alone, the dedicated Qwen3 ASR server is the better tool; this model is for requests that mix a picture, a recording and a question.

What the M4 Pro measured

All figures are single stream and greedy, the median of five runs after warmup from the toolkit's benchmark.py, on macOS 26 with mlx-vlm 0.6.1. Text generation covers 256 tokens; the image column is image plus text over 140 tokens.

MLX single-stream generation is mostly bound by memory bandwidth, so the smaller build is faster, a base M4 or M3 lands lower, and an M4 Max or Ultra higher. An image costs almost nothing in speed and about half a gigabyte of RAM.

Context length and the optional drafter

The KV cache is the memory the model keeps for every token it has seen. On Gemma 4 it costs only about 74 KB per token, because only the 8 full-attention layers grow while the 40 sliding-window layers, which look back a fixed distance, stay capped at 1024 tokens, so even the full 256K context fits on a laptop. Peak RAM on the M4 Pro at 8K, 32K, 128K and 256K was about 12.3, 14, 21 and 31 GB for MLXFP4 and about 16.3, 18, 25 and 35 GB for MLX-8bit. My guidance by unified memory: 16 GB takes --max-kv-size 16384, 24 GB 32768, 32 GB 65536, 48 GB 131072, and 48 GB with long documents the full 262144. The cap rotates, evicting the oldest tokens, and --kv-bits 8 quantizes the cache to roughly double the context that fits in the same RAM.

Speculative decoding has a small draft model propose tokens that the large model verifies, so the output stays identical and only the speed changes. Google ships an MTP drafter for Gemma 4, google/gemma-4-12B-it-assistant, a 423M-parameter multi-token-prediction head. It is gated on Hugging Face, so run hf auth login once, then append three flags to either server command.

Code, shelluv run --python 3.12 --with mlx-vlm -- python -m mlx_vlm.server --model AEON-7/Gemma-4-12B-it-AEON-Abliterated-MLXFP4 --port 8080 --max-kv-size 16384 --draft-model google/gemma-4-12B-it-assistant --draft-kind mtp --draft-block-size 2

The gain is about 1.1 to 1.2x on these builds for about 0.9 GB more RAM, less than stock Gemma 4 sees because the drafter was tuned for stock weights. A block size of 2 is the benchmarked optimum; drafting deeper is slower.

Why there are two builds instead of one

Quantization stores weights in fewer bits so the model fits in less memory. MLX offers mxfp8 and mxfp4, microscaling formats in which blocks of 32 weights share one scale, and the toolkit's notes are blunt about the 4-bit one. NVFP4, NVIDIA's 4-bit format, is near-lossless; MLX mxfp4 is E2M1, two exponent bits and one mantissa bit per weight, and it is not. Only mxfp8 is, and MLX's own nvfp4 mode is not recommended on Metal. So there are two tiers.

The first FP4 attempt used mlx-vlm's stock mxfp4 predicate, and on paper it looked fine: a top-1 agreement of 0.908, the share of steps along the BF16 model's own greedy trajectory where the quantized model picks the same next token. Then the refusals came back, and long generations collapsed into repetition after about 200 to 300 tokens. The abliteration edit lives in o_proj and down_proj, the residual writers that add each layer's result back into the shared stream, on 24 of the 48 layers. The stock recipe put o_proj at 4 bits, along with the projectors and the tied output head, which shares its weights with the input embeddings: exactly where an abliterated multimodal model cannot afford to lose precision.

The recipe that survived

A tooling trap sat underneath: the stock --quant-predicate mixed_* recipes are affine-only and silently ignore --q-mode mxfp4, so a true mixed FP4 build needs a callable predicate that returns group size, bit width and mode per layer, which is recipe.py in the toolkit. MLXFP4 puts q_proj, k_proj, gate_proj and up_proj at mxfp4, holds o_proj, down_proj and v_proj at mxfp8, and keeps the projectors, the embeddings and the head at bf16. It lands at 0.885 top-1 with a median KL of 0.0042 nats, where KL divergence measures how far the quantized model's next-token probabilities drift from the original, and it averages 6.64 bits per quantizable weight.

MLX-8bit puts every linear layer of the decoder at mxfp8 and reaches 0.924 top-1 with a median KL of 0.0019 nats. The default full mxfp8 recipe scored 0.938 top-1 but quantizes the projectors, so I did not ship it. Both shipped builds passed the gates in validate.py: 0 of 8 harmful prompts refused, 0 of 5 benign prompts refused, and 3 of 3 clean generations of 512 tokens or more. The scripts preflight.py, convert_mixed_fp4.py and audit_quant.py reproduce the MLXFP4 build from the BF16 source; the 8-bit recipe is recipe_8bit.py.

Before you run it

These are uncensored models, and the cards say plainly that downstream safety and legal responsibility rest entirely with the operator. Each card carries an Arbitration Clause and recommends input and output filtering, logging and human review; the license is Google's Gemma license. The README still mentions hf auth login for private repos, but the three model repos are public and ungated; only the optional drafter needs the login. The other Apple Silicon story in this library is the Qwen 3.6 27B piece; the 31B Deckard and the 26B MoE cover the rest of the Gemma 4 family on the Spark, and the catalog has the whole shelf.

Sources

Repository and model cards:

• gemma4-aeon-abliterated-mlx-toolkit (GitHub)
• Container image (GHCR)
• Gemma-4-12B-it-AEON-Abliterated-MLX-8bit
• Gemma-4-12B-it-AEON-Abliterated-MLXFP4
• Gemma-4-12B-it-AEON-Abliterated-K4-BF16
• vllm-ultimate-dgx-spark (GitHub)

Upstream:

• google/gemma-4-12B-it
• google/gemma-4-12B-it-assistant (MTP drafter)
• mlx-vlm
• uv
• Gemma terms