Nemotron 3 Nano Omni AEON Ultimate, served uncensored on a DGX Spark

NVIDIA's Nemotron 3 Nano Omni is an unusual model to run at home. It is a mixture of experts (MoE; 30B-A3B: 30B total parameters, a 3B active slice per token), and 30 of its 52 layers are Mamba2 state-space layers, which carry context as a fixed-size state rather than a growing attention cache. A RADIO vision tower and a Parakeet audio encoder sit on top, so one model reads images, listens to speech, and reasons about both inside <think> tags before it answers. As shipped it refuses a great deal: in my hundred-prompt scan the base refused 53 of 100 prompts with thinking on and 99 of 100 with thinking off.

Nemotron-3-Nano-Omni-AEON-Ultimate-Uncensored is my abliterated version, its refusal behavior projected out of the weights, shipped as BF16 and NVFP4 (four-bit floating point) checkpoints with a source-built vLLM image and a deployment guide for the DGX Spark. The weights are alpha; the image and the recipe are stable. Here are both halves, including what did not work.

What abliteration is, and what it touched here

Abliteration removes a model's refusal behavior by finding the direction in its activations that separates refusal from an ordinary answer and projecting it out of the weights, with no finetune and no gradient step. The idea comes from Arditi and colleagues' 2024 finding that refusal is mediated by a single direction in the residual stream, the vector every layer reads from and adds to.

Here that meant modifying 2,998 tensors, every weight that writes into the residual stream: embedding rows, lm_head rows, attention and Mamba output projections, and the experts' shared and routed down projections. The 1,106 vision, audio, connector and projection tensors are byte-identical to NVIDIA's release. As far as I can tell it is the first public abliteration of the NemotronH architecture, and that hybrid is why it took eight tries.

Eight iterations to twelve dimensions

Version one projected out a single direction and left 60 percent of refusals in place. Version two left 97 percent in place because of where it measured: the last prompt position, position -1. Nemotron's chat template appends <think> and a newline to every prompt, so position -1 sits downstream of the point where the model has already decided to refuse. The signal peaks at position -4 and layer 24, where the separation between harmful and harmless activations reaches about 4.94.

Version three reported 100 percent compliance, inflated by validating on 80-token generations; the honest figure was 83 percent, which is why the final bench allows 1,200 new tokens.

From there the method became a subspace. I compute refusal directions as class-mean activation differences between harmful and harmless prompts at thirty (layer, position, thinking mode) triples: five layers, three positions, thinking on and off. I stack them, QR-decompose, and keep a column when its diagonal entry in R is at least half the first, which reduces thirty correlated directions to independent axes. Version seven kept 15 dimensions and broke multi-step arithmetic and analogy. Version eight kept 12, and that is the release: in BF16, on a 30-prompt subsample, roughly 0 percent refusals with thinking on and about 3 percent with thinking off, capabilities intact.

BF16 and NVFP4, two checkpoints and one trade

The BF16 checkpoint is 66.06 GB in 50 files; the NVFP4 checkpoint is 21.9 GB in 39 files, a 3.0x compression. NVFP4 is NVIDIA's four-bit floating-point format, in which each small group of weights shares a scale and Blackwell GPUs multiply the result natively; here the group size is 16, calibrated on 256 prompts with modelopt 0.43.0.

The recipe, MAMBA_MOE_NVFP4_AGGRESSIVE, puts the Mamba input and output projections of the 30 Mamba layers and the routed and shared expert up and down projections of 25 expert layers into NVFP4; attention, routers, the Mamba conv1d, lm_head, embeddings, norms and all multimodal towers stay BF16.

Quantization noise compounds the abliteration artifact: with thinking off, refusals rise from about 3 percent in BF16 to 16 of 100 in NVFP4 served through vLLM (seed 42, 1,200 new tokens). With thinking on, NVFP4 produced no real refusals in 100 prompts, but 10 never completed. BF16 is the fallback when memory allows, and the only choice on H100, H200 and A100, which have no native NVFP4.

The source-built image and its four patches

Serving this hybrid on the Spark needs a specific vLLM and a few repairs, so the image is built from source. It is vLLM v0.20.0 at commit 88d34c6409, which has the native multimodal class for this architecture, compiled for CUDA 13 with TORCH_CUDA_ARCH_LIST set to 12.0+PTX: sm_121a on the Spark's GB10, sm_120 on other Blackwell cards, where the Spark-specific patches are no-ops. Alongside sit FlashInfer v0.6.9 for the NVFP4 matrix-multiply (GEMM) kernel, my TurboQuant fork for KV-cache compression, transformers 5.7.0, librosa 0.11 and soundfile 0.13.

The four patches are small and idempotent:

1. patch_kv_cache_utils.py: hybrid cache groups can hand a block size of None to min() and crash; it filters None at four sites.
2. patch_cuda_optional_import.py: imports vllm._C_stable_libtorch with RTLD_LAZY, since sm_121a builds lack SM100-only symbols.
3. patch_cudagraph_align.py: aligns CUDA graph capture sizes for speculative decoding, where a small drafter proposes tokens that the main model verifies in one pass.
4. verify_nemotron_omni_registered.py: a build-time assertion that the architecture maps to the multimodal class and that RadioModel and ProjectedParakeet import; it halts the build if upstream regresses.

The image is about 9 GB compressed and 33 GB on disk; a rebuild takes 25 to 50 minutes on the Spark with a warm ccache, and the planned v9 weights will need only a checkpoint swap.

Deploying on the Spark

The tier-one target is a DGX Spark (GB10, sm_121a, 128 GB unified memory) with driver 580.x or newer, CUDA 13 on the host or the container's bundled 13.2, and Docker 25.x or newer with the nvidia-container-toolkit and compose. My documents disagree on free disk, 35 GB for NVFP4 and 90 GB for BF16 in the agent runbook against 60 GB and 100 GB in the setup guide; plan for the larger. A Hugging Face read token is optional and only avoids rate limits.

Code, shelldocker pull ghcr.io/aeon-7/vllm-nemotron-omni-aeon-ultimate:v1
export HF_TOKEN=your_read_token_here
mkdir -p ~/.cache/huggingface
curl -fsSL https://raw.githubusercontent.com/AEON-7/Nemotron-3-Nano-Omni-AEON-Ultimate-Uncensored/main/examples/docker-compose.yml -o docker-compose.yml
docker compose up -d
docker compose logs -f

Cold start takes 4 to 7 minutes for NVFP4 and 8 to 15 for BF16. Resident memory settles near 22 GiB for NVFP4 and 64 GiB for BF16, with a peak near 50 GiB during CUDA graph capture of NVFP4. The server then speaks the OpenAI-compatible API on port 8000.

Code, shellcurl -fsS http://localhost:8000/health && echo " ok"
curl -fsS http://localhost:8000/v1/models | python3 -m json.tool
curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "AEON-7/Nemotron-3-Nano-Omni-AEON-Ultimate-Uncensored-NVFP4", "messages": [{"role":"user","content":"What is 47 * 83? Show your reasoning."}], "max_tokens": 1200, "temperature": 0}'

Any other Blackwell card can skip the image; stock vLLM v0.20.0 or newer serves the NVFP4 checkpoint directly.

Code, shellvllm serve AEON-7/Nemotron-3-Nano-Omni-AEON-Ultimate-Uncensored-NVFP4 --trust-remote-code --max-model-len 200000 --gpu-memory-utilization 0.70

Flags that matter, and what the Spark measures

VLLM_TEST_FORCE_FP8_MARLIN picks the expert kernel: 0 selects the FlashInfer CUTLASS MoE path, which the README frames as the interactive choice on time to first token (TTFT); 1 selects MARLIN, with about 5 to 10 percent more aggregate throughput at eight concurrent requests. Set it explicitly; my sources disagree about the image's default.

The TTFT numbers are mixed: MARLIN's first token arrives sooner at a single stream and at the eight-request median, while at four concurrent requests with thinking on CUTLASS answers first at 90 ms against MARLIN's 124 ms. That last case is the one the README leans on, and it is why CUTLASS is my chat setting and MARLIN my batch setting.

The other flags: --max-model-len 200000 leaves KV-cache slack under the 256k upstream supports; drop it to 64000 for concurrency. --gpu-memory-utilization is 0.70 in the cards and 0.85 in the setup guide; the cards warn that above about 0.8 the shared pool page-thrashes, so I use 0.70. --max-num-seqs of 8 to 16 suits chat, 32 to 64 batch work at the cost of first-token latency. --enable-prefix-caching is off by default; enable it for retrieval and agent loops. On Blackwell cards with dedicated VRAM, the NVFP4 card suggests speculative decoding with {"method": "qwen3_5_mtp", "num_speculative_tokens": 3}.

What is still broken

The alpha label is not decoration. Roughly 10 of 100 thinking-on prompts repeat phrases inside <think> and never close the tag, and a larger token budget does not help. Multi-turn chat amplifies it, and sustained conversation shows brokenness my benches did not catch, because they measure refusal pass-rate, not chat quality. A v9 pass is planned.

Empty text after </think> means the budget ran out mid-thought; use max_tokens of 1,200 or more in reasoning mode, since about 18 percent of prompts overran at 600. Audio must be 16 kHz mono PCM WAV, and an image URL the container cannot reach should go in as a data URI.

The base model is under the NVIDIA Open Model Agreement, the scripts are MIT, and the cards put the safety layer on the operator who serves an uncensored model. The serving stack is in AEON vLLM Ultimate on the DGX Spark and AEON vLLM Ultimate on the RTX 5090; AEON Bench is my harness; the sibling releases are Qwen 3.8 27B, Ornith 1 35B, Gemma 4 26B A4B and Qwen 3.6 35B A3B, all listed in the catalog.

Sources

• Repository on GitHub and its DGX Spark setup guide
• Model cards: BF16 and NVFP4
• Container image: ghcr.io/aeon-7/vllm-nemotron-omni-aeon-ultimate, tag v1
• Base model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, under the NVIDIA Open Model Agreement
• Method: Arditi et al., Refusal in Language Models Is Mediated by a Single Direction
• Prior tooling: Maxime Labonne's abliteration write-up and Sumandora's remove-refusals-with-transformers