[ingoau]

AutoMix analysis now finds where someone sings or raps at each end of a track, and the planner keeps one voice at a time in a blend.

Detector: musicnn (MTT_musicnn)

Candidate	Licence (code / weights)	Size	Result
musicnn MTT_musicnn (MTG-UPF)	ISC / ISC (the checkpoint ships in the ISC repo)	3.2 MB ONNX	Picked. Tells vocal from instrumental versions of the same songs on 3 s patches
YAMNet (Google)	Apache-2.0 / Apache-2.0 (Kaggle model page)	15 MB	Converted and ported; Singing/Rapping/Speech stay near 0 on dense rap/pop mixes, vocal or not
EfficientAT mn10_as	MIT / implied MIT	4.9 M params	Contrast only on 10 s windows, which costs ~12 s per track; short windows saturate
PANNs	MIT / CC-BY-4.0	21–327 MB	Not tried: CC-BY weights; Cnn14 frame-level too heavy
Essentia voice_instrumental / musicnn copies	AGPL / CC BY-NC-SA	–	Rejected: non-commercial
Schlüter ismir2015, kyungyunlee SVD	–	–	Rejected: no weights / no licence
Silero VAD, CREPE, RMVPE, basic-pitch	various	–	Rejected: speech or pitch, not vocals in a mix

Caveat, the same kind as Beat This!'s: MagnaTagATune audio is Magnatune's CC BY-NC-SA. The weights are ISC, and the data isn't redistributed. analysis/NOTICE has the provenance and hashes; scripts/musicnn-to-onnx.py reproduces the ONNX file. The Rust front end matches librosa/TF to 1e-3 (parity_with_reference, ignored, needs the script's output).

Cost: +1.3–1.5 s per track on one core, measured old against new analyser alternately on five tracks: +15–17 % on 3-minute tracks, +8 % on a 5-minute one. That's fixed per track: 24 non-overlapping 3 s patches over the first and last 35.9 s, which is the planner's longest overlap. The network is ~45 ms a patch (onnxruntime is no faster), so covering the full 60 s loudness minute would have cost ~2.3 s.

Record

• head_vocals / tail_vocals: a VocalCurve with a probability every 100 ms, held over each 3 s patch.
• ANALYSIS_VERSION 2, so the worker re-analyses everything.
• No migrations.

Planner

This applies only when both records have voice curves; otherwise behaviour is unchanged. "Apart" means A's last voice ends no more than 0.5 s after B's first.

Blends

• (a) The usual 8/16 bars when the voices are already apart. The handover moves to B's downbeat between the voices when the midpoint isn't between them.
• (b) Otherwise 4, then 2 bars, if that fits the instrumental gap.
• (c) Otherwise the shortest blend of at least 4 s, with the bass swap from B's entry and a 2 kHz −12 dB shelf on A up to the handover.

Beat-aligned fades drop bars (down to 4 s) to keep the voices apart.

Smart and rescue fades are not voice-aware. They are 3–12 s and have no beat grid.

Corpus (100 tracks, 500 random pairs)

• Beat-matched blends with overlapping voices: 31 of 40 before, 14 of 40 after. The 14 are voices all the way through.
• Median blend overlap: 19.2 s → 7.1 s.
• Plain fades: voices overlap in 199 of 460 either way (follow-up candidate).

Reference set: three vocal pairs added; six existing plans changed (all vocal pairs).

Before/after renders (hocket-mixlab --no-vocals pair … vs pair …), worth a listen:

• Little Light → Overdue: same 25.6 s TempoBlend, handover moved to where B's voice starts (a)
• Let's Ride → Lately: 21.4 s with the voices overlapping, now 10.7 s with them apart (b)
• Big → Black & White: 25.9 s, now 6.5 s shaped (c)

Not done

• No prod coordinator redeploy.
• Desktop and Android only read the plan, so they have no code changes.

🤖 Generated with Claude Code