Qwen 3.6 27B AEON Ultimate on Apple Silicon: native MLX quants

Most of my Qwen 3.6 27B work has lived on NVIDIA hardware under vLLM. A MacBook Pro is a different kind of machine: no CUDA, a great deal of memory in one shared pool, and that memory not especially fast. The question on a Mac is how few bytes per token the machine has to move while the model keeps its judgment.

My answer is two native MLX builds (MLX is Apple's array framework for the Metal GPU in its own chips): a fidelity build at 8 bits, 29.5 GB on disk, a compact build at 4 bits, 16 GB on disk, and a separate 821 MB drafter that gives a lossless speedup of up to 1.78x. The model underneath is a 27.4-billion-parameter hybrid decoder with a vision tower, a 256K context window and its refusal behavior removed. One honest note: the cards for this line now point to Qwen 3.8, which is built for vLLM on NVIDIA hardware, so on a Spark or an RTX card read the Qwen 3.8 article instead. On a Mac these builds are the ones I have, and they stay published for existing deployments and for the record.

Which Mac fits which build

Apple Silicon uses unified memory: the CPU and the GPU draw from one pool, so the weights, the working state and macOS itself must fit together. The FP4 build peaks at 17.07 GB while generating and the 8-bit build at 29.85 GB, and the rule of thumb in my runbook follows from those peaks: FP4 wants 24 GB or more, 8-bit wants 36 GB or more. A 32 GB Mac sits between the two, and the arithmetic says why the rule skips it: 29.85 GB of model leaves about 2 GB for everything else. If a build fails to load for lack of memory, the remedies are FP4 or a higher GPU wired limit through sysctl iogpu.wired_limit_mb, a system setting that is the Mac owner's call, not mine. Both builds need an M1 or newer; Intel Macs are not supported.

Every number here comes from one MacBook Pro with the Apple M4 Pro and 48 GB of unified memory (about 273 GB/s of bandwidth), running mlx-vlm from git main, greedy, after warmup, one request at a time. A base M4 or an M3 is somewhat slower and an M4 Max or Ultra notably faster; I have not measured them.

FP4 decodes 1.85x faster at 57 percent of the memory because single-stream decoding is bound by memory bandwidth: every token streams the whole model through the GPU once, so half the bytes per weight gives nearly double the speed. The price is precision, and the next two sections are about paying as little of it as possible.

Why the quantization is selective

Quantization stores weights in fewer bits than the bfloat16 they were trained in, and the usual way is uniform: pick a width and apply it to every large matrix. This model punishes that. Of its 64 layers, 48 are Gated DeltaNet layers, a Mamba-style state space model (SSM) that carries a compact recurrent state forward token by token instead of attending over the whole history, and only 16 are full attention layers. The tensors that govern the SSM dynamics (the short convolution, the A_log decay, the dt_bias, the decay gate and the beta projection) are tiny, high-leverage and numerically fragile. Quantize them like everything else and the recurrence is corrupted, which gives you not a slightly worse model but a different one.

So both builds keep a fixed list in BF16: linear_attn.conv1d, A_log, dt_bias, every norm, the in_proj_a and in_proj_b gates, the whole vision tower (333 tensors under model.visual) and the MTP head. An audit of the finished files found zero quantization scales on any of them.

Two recipes, one precision map

The 8-bit build applies affine 8-bit, integers with a scale and an offset per group of 64, to the MLP blocks, the three large linear_attn projections, the attention q, k, v and o projections, the embeddings and the output head: 402 quantized tensors at 8.634 bits per weight. The FP4 build uses mxfp4, a four-bit floating format with two exponent bits, one mantissa bit and a shared scale per group of 32, on the MLP, the linear attention projections and attention q and o, with 8-bit affine islands on attention k and v, embed_tokens and lm_head. The k and v islands exist because grouped-query attention leaves this model only 4 key-value heads for 24 query heads, so an error there carries about six times the leverage. If a KL check (how far the quantized model's output distribution drifts from the original) ever regresses, mlp.down_proj is the first tensor a -quality build would promote to 8-bit.

The abliteration, the removal of the model's refusal direction from its weights, lives in the two tensors that write into the residual stream, self_attn.o_proj and mlp.down_proj. Both builds keep enough range there for it to survive, and FP4 passed the gate I use for it: an in-character rogue-AI monologue with no refusal. Both recipes are round-to-nearest quant_predicate callables for mlx_vlm.convert, with no calibration data, and they live in the repository under scripts/.

A drafter made from the model's own head

Speculative decoding gets more than one token out of a large model per forward pass: a small drafter proposes a few tokens, the large model verifies the block in one pass, keeps the tokens it agrees with and substitutes its own token at the first disagreement. Done right, the output distribution is identical to the large model running alone, under greedy decoding and under sampling alike, so the speedup is lossless. This model carries its own multi-token prediction (MTP) head, a small extra layer that predicts further tokens from the hidden state, and I split that head out into a separate 821 MB drafter.

Block size 3 is the knee: block size 4 drafts more per round, but acceptance falls and net speed drops. With the drafter, FP4 runs about 3.2x faster than 8-bit alone, 26.5 against 8.2 tokens per second. Two caveats belong beside the headline. The 1.78x is the sweep figure, not a chat average: the speedup tracks how predictable the output is, from 1.73x on math with 90.1 percent acceptance down to 1.43x on chat with 67.1 percent. And the drafter costs about 1.5 to 1.6 GB of memory, so drop it only when memory is tight. The 8-bit build takes the same drafter, though I have no 8-bit MTP figure, and the drafter is never to be loaded alone: it is a head, not a chat model.

The quick start with uv

The stack is small: uv, the Python package manager from Astral, which also provisions Python 3.12 so you never touch the system Python, and mlx-vlm from its git main branch, because the PyPI release, 0.6.1, does not know the qwen3_5_vision tower. The --trust-remote-code flag is required because the model ships custom qwen3_5 code, and the first run downloads everything from Hugging Face.

Code, shellcurl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv run --python 3.12 --with "mlx-vlm @ git+https://github.com/Blaizzy/mlx-vlm" -- python -m mlx_vlm.server --model AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-MLX-FP4 --draft-model AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-MLX-MTP-Drafter --draft-kind mtp --draft-block-size 3 --port 8080 --trust-remote-code

Swap in AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-MLX-8bit for the fidelity build, and drop the three --draft-* flags to run without the drafter. The server is an OpenAI-style endpoint on port 8080; verify it with one request, and note the temperature, which is not optional.

Code, shellcurl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-MLX-FP4","messages":[{"role":"user","content":"In one sentence: what is gated DeltaNet?"}],"temperature":1.0,"max_tokens":64}'

The first request is slow while Metal compiles its kernels once, so send a throwaway one before you time anything. Vision works as on any OpenAI-style endpoint, with an image_url part in the message content; with 274 image tokens in the prompt, FP4 still decoded at 15.0 tokens per second. One more switch matters for agents. Out of the box the server re-reads the whole conversation on every request; set APC_ENABLED=1 in the environment before the same server command and it reuses the key-value state of the shared prefix. On FP4, over four turns sharing a prefix of about 1,560 tokens, time to first token fell from 14.3 seconds to 1.3, 1.2 and 1.2 seconds on turns two through four, roughly 11x. A proper --enable-prefix-caching flag is proposed in mlx-vlm pull request 1435.

What went wrong, and what to do about it

The runbook I wrote for a fresh Mac puts two gotchas at the top, because both look like a broken model and are not. Trap A is repetition collapse: the server defaults to greedy decoding, temperature 0, and on long prompts a greedy 27B model can fall into a loop. The fix is in the request, not the model. Send temperature 1.0 (typically with top_p 0.95 and top_k around 64) on every call, and make sure your client sends sampling parameters at all, because one that sends none falls back to greedy. Trap B is ValueError: Unsupported model type: qwen3_5_vision, which means you are on the PyPI mlx-vlm; install from git main, or as a last resort add qwen3_5_vision to the supported list in mlx_vlm/models/qwen3_vl/vision.py.

The validation is spot checks plus the structural audit, on the M4 Pro, and I want to be plain about that. Both builds read a test image of labeled shapes correctly and solved the classic five machines, five minutes, five widgets puzzle, the 8-bit build slightly more rigorously; FP4 wrote the rogue-AI monologue with no refusal; neither looped, and <think> mode works. There is no KL sweep or benchmark suite behind them; for that kind of evidence, AEON Bench is how I would gather it.

Finally, this is an uncensored model, and every card carries the same clause: "The absence of internal refusal behavior means the duty of care that would ordinarily rest partly with the model rests entirely with you." The license is inherited from Qwen. For a smaller Mac model with the same treatment, see Gemma 4 12B on a Mac; for everything else, the catalog.

Sources

• Repository: AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-MLX
• Model cards: MLX-8bit, MLX-FP4, MTP drafter, BF16 source
• Successor: Qwen3.8-27B-AEON-ULTIMATE-UNCENSORED-NVFP4-MIXED and its repository
• Siblings: NVFP4 + MTP for Blackwell and DFlash for A100 and H100
• mlx-vlm and pull request 1435
• uv documentation and Qwen on Hugging Face