AEON Music Video: audio-reactive cuts with librosa and ffmpeg

The problem this tool solves is an old one. A finished song sits next to a folder of clips that have nothing to do with each other: a few renders from AEON Movie Maker, some stock footage, something shot on a phone. What is missing is the editor, the patient person who listens to the track over and over and cuts on the kick.

aeon-music-video is that editor: two MIT-licensed Python scripts that run on an ordinary CPU. reactive_compositor.py builds smooth, mood-driven edits with crossfades for videos of one to four minutes. dmt_flash_editor.py builds rapid hard-cut vertical edits of about 16 seconds for TikTok, Reels and Shorts, in the look the repo calls DMT or Tron: strobes, color fringing, glowing edges. Both listen to the song with librosa, a Python library for music analysis, and turn what they hear into instructions that ffmpeg, the command-line video tool, follows frame by frame. There is no model at runtime, no ComfyUI and no network call.

Why the edit step runs on a CPU

In the AEON Media Production family, AEON Music Maker makes the audio, AEON Movie Maker renders clips with the LTX 2.3 video model, and ComfyUI AEON Spark is the sibling repo for ComfyUI, the node-based interface for image and video models. Rendering clips with a video model is GPU work. Editing is different. Cutting clips to a beat is arithmetic on a waveform followed by one long ffmpeg command, and nothing in that needs a GPU. So this tool needs no ComfyUI at all, and the catalog lists the rest of the family.

Everything runs on the local machine; there is no remote mode because there is nothing to remote, so if the clips were rendered on a GPU box elsewhere, copy the audio and clips down first. It runs on Linux, macOS and Windows.

What librosa hears in the song

librosa loads the audio at its native sample rate, folded to mono, and computes four descriptions of it. The first is the beat grid from beat_track, which estimates the tempo and marks each beat. The second is onset strength, a curve of how sharply the energy rises at every instant: a kick drum is a spike, a sustained pad is nearly flat. The scripts normalise that curve to the range 0 to 1 and raise it to a gamma of 0.6 in the DMT editor and 0.7 in the smooth compositor; a gamma below 1 lifts the quieter hits so they still count as events. The third is RMS, root mean square energy, a running measure of loudness. The fourth is the spectral centroid, the center of mass of the spectrum, which tracks how bright the sound is.

The analysis hop is sized to the video frame rate, so each measurement lines up with one frame at the default 24 fps.

Turning the analysis into ffmpeg commands

ffmpeg has a filter called sendcmd that reads a text file of timed commands and changes the parameters of other filters while the video plays. The scripts write their analysis into such files, one row per event, each row four fields: TIME TARGET PARAM VALUE. Brightness in the eq filter pulses on every beat. Saturation follows RMS or onset strength. The hue filter drifts with the spectral centroid, up to 30 degrees in the smooth script and 60 degrees in the DMT editor. Zoom pulses on RMS peaks, with a default peak of 0.12 in the compositor. The flash is scaled for the mood: a brightness peak of 0.25 in the smooth video, and 0.7 in the DMT editor, where each flash decays over 60 ms.

sendcmd is strict. Write the file with brackets and a leading command keyword, as other ffmpeg command syntaxes do, and it answers with "Missing separator". The format is the bare four fields, and the rule in the repo's AGENTS.md is not to hand-edit the sendcmd files, because the scripts emit valid syntax.

The smooth compositor and its six moods

The compositor divides the song into segments, gives each one an intensity and a brightness value from the curves above, and chooses a clip for each segment from six mood buckets. You assign clips with --mood-clip calm:cosmic.mp4 and so on, or pass a flat list with --clip and skip the buckets.

Crossfades land on intensity drops. The default output is 832x480 at 24 fps, and horizontal clips suit 832x480 or 1920x1080. Resist the pile: three to six clips with distinct looks, matched to buckets, beat twelve that all look alike.

The DMT flash editor

The DMT editor crossfades nothing. It takes a schedule, a JSON list in which every entry names a clip, a trim start and a duration, and joins the cuts end to end. A clip longer than its slot is trimmed from trim_start; a shorter one is looped. The result is a vertical 720x1280 video of around 16 seconds built from 8 to 12 clips. A constant chromatic aberration, the color fringing of a cheap lens, comes from rgbashift with the red channel shifted 4 pixels one way and the blue 4 the other. A bloom, the soft glow around bright areas, comes from blurring a copy of the frame with gblur at sigma 8 and blending it back in screen mode.

The finished audio runs through loudnorm to -12 LUFS, the standard unit of integrated loudness, with a true peak ceiling of -0.5 and a loudness range of 5; the smooth compositor normalises to -14 LUFS with a true peak of -1.0 and a range of 7. Both scripts encode with libx264 at crf 18 and the medium preset, a quality setting close to visually lossless. On a modest CPU a 16 second DMT render takes around 30 to 60 seconds. Use vertical clips with a clear visual identity in their first two seconds.

Getting it running

You need Python 3.10 or newer with librosa 0.10, numpy 1.24, scipy 1.10 and soundfile 0.12 or later from requirements.txt, plus ffmpeg and ffprobe on PATH with the filters sendcmd, eq, hue, rgbashift, gblur, showcqt, avectorscope, loudnorm and acrossfade; setup.sh checks for sendcmd and rgbashift. Every variable in .env.example is optional, and a CLI flag always wins.

Code, shellgit clone https://github.com/AEON-7/aeon-music-video.git
cd aeon-music-video
cp .env.example .env
./setup.sh
python scripts/reactive_compositor.py --audio song.flac --mood-clip calm:cosmic.mp4 --mood-clip building:wireframe.mp4 --mood-clip explosive:fractal.mp4 -o music_video.mp4

The hard-cut mode takes a schedule instead.

Code, JSON[
  {"clip_path": "clip_01.mp4", "trim_start": 0.2, "duration": 2.0},
  {"clip_path": "clip_02.mp4", "trim_start": 0.5, "duration": 1.0}
]

Code, shellpython scripts/dmt_flash_editor.py --schedule schedule.json --audio song.flac -o dmt_flash.mp4

To update, run ./sync.sh; it shows a diff, stashes local edits around the pull, and takes --dry-run and --yes.

What went wrong, and which knob fixes it

The largest scar is the freeze. The obvious way to make a short clip fill a long slot is ffmpeg's loop filter, chained with trim and concat, and it stalls around 20 seconds into the render. The fix was to loop at the demuxer, the stage that reads the input file, with -stream_loop -1, and both scripts do that now.

Everything else is a knob.

• Stepped flashes: lower --brightness-peak.
• Hue drift too aggressive: lower --hue-deg or --hue-max-deg.
• Audio drifting against video: match --fps to the source clips (default 24).
• Too quiet: lower --lufs toward -9, at the cost of dynamic range.
• Dark or washed-out output: edit the eq baseline in the script; the defaults suit DMT and EDM material, not natural footage.
• Too much chromatic aberration or bloom: --no-chromatic and --no-bloom switch them off.

One honest note: the docs disagree with themselves in places. The README gives the loudness default as -12 LUFS while the smooth compositor defaults to -14, and the DMT edit is called 16 seconds, ten cuts and 8 to 12 clips in different files; the scripts are the truth.

That closes the pipeline: a song from AEON Music Maker, clips from AEON Movie Maker, this editor to cut them. AEON Radio Drama is another repo in the AEON family.

Sources

• Repository: AEON-7/aeon-music-video
• Maker account: AEON-7 on GitHub
• Companion repositories: aeon-music-maker, aeon-movie-maker, aeon-radio-drama, comfyui-aeon-spark
• Libraries: librosa, numpy, scipy, python-soundfile
• ffmpeg and its download page