feat(cli): beats tracks the music with the TypeScript beat tracker, without Chrome

[xuanruli]

Stacked on #5632 (feat/audio-analysis-providers), which added the TypeScript beat tracker in packages/core/src/audioRhythm.ts. Review that one first.

What changed

hyperframes beats and Studio's beat detection now run the same code, analyzeMusicFromSignal in @hyperframes/core/beats:

1. Band energies.
2. rhythmOf from audioRhythm.ts.
3. Each beat's local RMS against the loudest beat, used as its strength.

• CLI: decodes with the same ffmpeg path hyperframes audio uses (decodeAudio). beats no longer touches Chrome at all. headlessAnalyzer.ts, the prebuilt beat-analyzer.global.js bundle and its build step are deleted. The only external dependency left is ffmpeg/ffprobe; if either is missing, the error points at hyperframes doctor.
• Studio: still decodes with AudioContext.decodeAudioData in the browser. analyzeMusicFromBuffer now passes every channel, not just channel 0, so a beat that sounds only on the right is still found and given a strength. The tracker depends only on core's fft.ts and Float32 math. The pause hook (whenTimelineIdle) still runs before each of the three long stages.
• Confidence floor: below a tracker confidence of 0.2, no beats are reported: beatTimes: [], bpm: null, bpmConfidence: "uncertain".
  • Ambient or rubato music gets no invented grid.
  • The CLI's existing "0 beats → nothing written, exit 1" still applies.
  • Studio already ignores a missing or empty file.
• bpmConfidence is "high" at tracker confidence ≥ 0.35, "low" between 0.2 and 0.35, and "uncertain" when there are no beats. The type is unchanged.
• tempoOf (second commit) averages the intervals within 10% of the median instead of taking the median. The tracker places beats on 10 ms frames, so a 96 BPM track used to read 95.2. It now reads 96. This also changes hyperframes audio's bpm.
• Deleted:
  • The bpm-detective/energy-peak merge.
  • detectBeats, detectBeatsFromUrl and loadBpmDetective.
  • The per-beat silence gate. It dropped every other beat on sparse tracks, which turned 96 BPM into 48.
  • bpm-detective, from core's and studio's package.json, studio's tsup externals, the vite optimizeDeps and bun.lock.

The beats/<audio>.json format is unchanged (version, audio, beats: [{time, strength}]), and a test now pins it byte for byte.

Why: the old analyzer's failure modes, reproduced

analyzeMusicFromBuffer had three problems:

• Averaged tempo. It folded bpm-detective's tempo and an energy-peak tempo into one octave to compare them, then averaged the unfolded values. On dc8d01: (52.3 + 96) / 2 = 74.
• Rigid phase. It phased a rigid grid from noisy energy peaks, so even the right tempo could land on the off-beats.
• Raw-peak fallback. When the two tempos disagreed, it returned raw energy peaks with no grid. That happened on 39 of 75 real tracks.

The new regression test reproduces both failures on a synthetic 96 BPM track (soft clicks on every beat under loud stabs). Run against the old analyzer with bpm-detective mocked to 96, as Chrome gives it:

stabs every	old analyzer	new
2 beats	74 BPM ("low"), recall 0.12: the dc8d01 averaging	96 BPM, recall > 0.95
2.2 beats	96 BPM ("high"), recall 0.00: right tempo, every beat on the off-beat	96 BPM, recall > 0.95

Measured before/after

I ran both analyzers myself on the benchmark's 75 real tracks and scored them with its R1 evaluation (eval_real.py):

• Before: the base branch's analyzeBeatsHeadless in headless Chrome.
• After: this branch's analyzeMusicFromSignal on ffmpeg-decoded audio.

On the 22 tracks with exact script grids (15 at 96 BPM), audible spans, ±70 ms:

	F	CMLt	AMLt	tempo right (Acc1)
before (headless Chrome)	0.437	0.205	0.319	18%
after	0.887	0.872	0.872	91%
Beat This small0 (reference)	0.995	0.983	0.983	100%

• dc8d01 goes from 74 BPM to 96 BPM ("high").
• Of the two tracks still counted wrong, both are refused (no beats) by the confidence floor; neither gets a wrong grid. At their confidence (0.13) the tracker half-times both to 48 BPM, so without the floor they would get a half-tempo grid, scored F 0.58.

Across all 75 tracks, compared against Beat This (F ≥ 0.8 counts as agree, F < 0.6 as disagree):

	tracks given a grid	agree	disagree
before	74	8	53
after	48	42	3
after, with no floor (tracker's own 0.05)	71	42	26

The 0.2 floor removes 23 wrong grids and no grid that agrees with Beat This. Analysis takes about 5 ms per second of audio in bun, before decode.

The separate benchmark's figures are higher than mine. It reported F 0.986 and 100% tempo for "the TS tracker". That figure comes from the prototype tsbeat.ts on 22.05 kHz mono. The tracker as merged in #5632, run on native-rate stereo as both callers decode it, gives F 0.939 with no floor (0.887 with it). On 22.05 kHz mono it gives 0.988. The difference is two borderline tracks at confidence ≈ 0.1 that flip between 48 and 96 BPM depending on the decode, so it is noise at the tracker's edge rather than a decode bug.

Callers checked

• packages/cli/src/commands/beats.ts: the only CLI caller. --json output (ok, file, count, bpm) is unchanged.
• packages/cli/src/beats/headlessAnalyzer.ts and packages/cli/scripts/build-beat-analyzer.mjs: deleted, along with the build:beat-analyzer script.
• packages/studio/src/hooks/useMusicBeatAnalysis.ts: calls analyzeMusicFromUrl with pause. It reads beatTimes, beatStrengths and the rest of MusicBeatAnalysis; the shape is unchanged.
• packages/studio/src/utils/beatEditActions.ts: reads strengthAtTime, which still uses channelData, sampleRate and peak. channelData is now the per-sample level across channels rather than channel 0.
• TimelineRuler.tsx, timelineLaneProps.ts, playerStore.ts: import only the MusicBeatAnalysis type.
• Nothing reads bpmConfidence except the CLI's printout and test fixtures.

Tests

• packages/core/src/beats/beatDetection.test.ts:
  • The 96 BPM regression (stabs every 2 and every 2.2 beats).
  • Tempo and phase on 90, 120 and 140 BPM grids at offsets.
  • Silence, hiss and an ambient pad give no beats and no tempo.
  • The three pauses.
  • A right-channel-only beat is found and has strength.
• packages/core/src/beats/beatFile.test.ts: serializeBeats output is pinned byte for byte, and round-trips through parseBeats.
• packages/cli/src/commands/beats.test.ts: the built command, end to end through ffmpeg, writes beats/assets/song.wav.json with exactly version, audio, beats[{time, strength}] on the grid. A silent track exits 1 and writes nothing.
• packages/core/src/audioRhythm.test.ts: frame-quantized 96 BPM beats read as 96.
• Studio's BeatStrip and Timeline.readOnly tests, which use MusicBeatAnalysis fixtures, pass.

Each test fails without its fix:

change made	result
new tests run against the old analyzer	7 of 8 fail; only the pause test passes
floor lowered to the tracker's 0.05	the ambient-pad case fails
strength taken from channel 0	the right-channel case fails
median interval in tempoOf	the 96 BPM tempo test fails

Not exercised

• No Studio walk in a browser. As a substitute, I bundled beatDetection.ts with esbuild for the browser and ran analyzeMusicFromBuffer on AudioContext.decodeAudioData output in headless Chrome, on 5 real tracks at 48 kHz. Every beat the page found was within 20 ms of a CLI beat on 4 of 5 tracks (on one of those, the page found 43 of the CLI's 46). On the fifth, 130 of 142 matched, because of resampling. A 2–3 minute track takes 200–450 ms in the page. I did not open Studio, drop a track, or watch the beat lane.
• Studio's fallback-track path (analysis without a saved beat file) is untested here, beyond the shared function.
• Downbeats: MusicBeatAnalysis still has no field for them, even though the tracker produces them.

Before

Actual Chrome execution of the parent analyzer (5b8a8d091), rendered in a browser harness. Synthetic 96 BPM clicks with loud chord stabs every 2.2 beats: 134 BPM reported, 37 beats, 25.6% recall within ±70 ms of interior reference clicks. No mocked bpm-detective. This is not a Studio UI screenshot or a real-music benchmark.

After

Same deterministic audio and browser harness using the child analyzer (65b986bce, unchanged by the subsequent CI-only commit): 96 BPM, 47 beats, 100% recall. The screenshots display seconds 2–12; the measurements cover 30 seconds. No real backend or paid provider calls.

Reproduction script and raw measurements. This exercises the shared browser analyzer; it does not verify Studio’s fallback-track lifecycle.

The CI deletion guard now explicitly lists only the removed Chrome analyzer and its build script, with replacement reasons. Its targeted test fails before the entries and passes afterward; unrelated deletions remain blocked.

[jrusso1020]

Requesting changes at 65b986bc. I found no problem in the code. Two required checks fail for reasons this PR has to fix.

Blocking: required CI

1. Detect changes: scripts/check-no-main-deletions.mjs refuses to delete packages/cli/src/beats/headlessAnalyzer.ts and packages/cli/scripts/build-beat-analyzer.mjs, because both exist on origin/main. The guard diffs origin/main...HEAD, so it applies even though this PR's base is #5632. The supported way through is an entry for each path, with a reason, in ALLOWED_DELETIONS (line 54). check-no-main-deletions.test.mjs requires each reason to be longer than 10 characters. Build, Typecheck, Test and Test: runtime contract fail only at "Require change detection", so none of them has run on this code yet.
2. Studio and player captures: the PR touches packages/studio, so the body needs ## Before / ## After captures, or a ## No visible change section. Since Studio's beat lane now shows different beats (and none on short or low-confidence tracks), I'd add the captures.

Checked, looks right

• Nothing still references the removed pieces. I searched for headlessAnalyzer, build-beat-analyzer, build:beat-analyzer, analyzeBeatsHeadless, loadBpmDetective, detectBeats, detectBeatsFromUrl and bpm-detective. The only matches are past release notes and the changelog. bpm-detective is gone from core and studio package.json, the tsup externals, vite optimizeDeps and bun.lock, and no source file imports it.
• hyperframes beats keeps its flags and its --json shape ({ok, file, count, bpm}). beatFile.ts is unchanged, and the new test pins its output byte for byte, with a parseBeats round trip. Studio's consumers (analyzeMusicFromUrl, strengthAtTime) use unchanged signatures.
• PR-body claims I checked against the source:
  • beats imports only decodeAudio, so no Chrome is involved. A missing ffmpeg surfaces decode.ts's "run: npx hyperframes doctor".
  • Every channel reaches the tracker (beatDetection.ts:89-92).
  • The confidence floor is 0.2 and "high" is 0.35.
  • tempoOf averages the intervals within 10% of the median. Its median-0 guard means it can't divide by zero.

Non-blocking

• Behaviour change worth saying in the release notes. Clips under 6 s and tracks under 0.2 confidence now get no beats. The CLI exits 1 and writes nothing, where the old analyzer fell back to energy peaks. By the PR's own benchmark, that's about 23 of 75 tracks. It's intended and documented here, but users will see it as new "No beats detected" failures.
• In Studio, a beat added by hand on a track with no detected beats gets strength min(1, rms / 1e-6), so always 1. That's not new, but more tracks will now hit it.

No local test run (no node_modules in my checkout). I didn't redo the esbuild browser bundle of beatDetection.ts.

— Rames

[xuanruli]

Addressed the CI findings in 3b3605258: the deletion allowlist names only the two replaced Chrome analyzer files, with reasons and a failing-before/passing-after test. Individual logs confirm Build, Typecheck, runtime contract and Test were cascading failures from Detect changes. The capture gate subsequently passed.

Current head 391d4e5d8 includes the parent's silence fix via a signed merge, and removes .github/pr-evidence/5634 from the final source tree as requested in Slack. The Before/After browser-harness images remain embedded in the body through immutable commit URLs; downloaded bytes match the measured local captures. Reproduction files are retained in the local handover. These are synthetic-fixture browser analyzer captures, not Studio UI screenshots. Native browser attachment upload is unavailable in this session (computer-use permission not granted).

Final stack verification: parent is an ancestor; core rebuilt; 44 core rhythm/beat tests and 8 CLI analysis/ffmpeg tests passed; deletion guard and its 9 tests pass. Fresh CI and current-head review remain required.

[miguel-heygen]

Own-delta review at 391d4e5d8724626ff0ab4ef7f337b50fb8d36ccc, based on #5632 at 4e45ab310058e60a8b07f49eb31482f49ba7bfc8. No additional correctness blocker found in this child. The Chrome analyzer/build references are removed together, both intended deletions are explicitly allowed, and the CLI beat-file format remains unchanged. The strict novelty threshold from the parent is preserved. Rames covered the earlier required deletion/capture gates; I checked the successor allowlist and the clearly labeled browser-harness evidence rather than treating those images as Studio acceptance.

Independent verification on this head:

• Seven focused core/analysis/provider/beat-file suites: 83 passed, one native macOS test skipped. Deletion-guard tests: 9 passed.
• Actual FFmpeg decode → analyzeMusicFromSignal → serializeBeats/parseBeats, on synthetic 20-second 120 BPM audio in a path containing spaces/non-ASCII: opposite-polarity stereo and right-channel-only input each produced 39 beats, 120 BPM and 100% interior-grid recall within 70 ms. Silence returned an empty grid and null BPM. These are known-time signal/format probes, not subjective real-music quality evidence or the full built CLI command.
• Traced Studio's saved-file/fallback load, cancellation, cache and persistence consumers against the unchanged MusicBeatAnalysis interface. No Studio UI walk or fallback-track lifecycle run was performed.

Pending: fresh-head CI was still running (27 successful, 39 in progress at the last read). The parent also has my separate provider-pin change request, review 5528152320; this child does not clear it. Keep bottom-first merge order. Neither native Apple SoundAnalysis nor an installed Beat This model/backend was independently exercised, and the author's corpus/browser figures are not my measurements.

— Magi (desktop-review)

Verdict: COMMENT
Reasoning: The child's own implementation and focused witnesses check out, but fresh required gates are unfinished and the parent remains held. This is not an approval of the stack.

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 158 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[miguel-heygen]

Re-review at 2d6903ce7601dffaebcfe0a25b2822dd1c174b42. Own-delta approval, stacked on #5632 at 54dfc960c4bcfb003f2a9f09fcdc570bfe638868; bottom-first merge order remains.

packages/cli/src/audio/analysis/analyze.ts:106-108,124 now normalizes BPM and beats onto the same provider before analysis, and rejects conflicting pins before any capability runs. I reran the actual API adapter against a loopback HTTP server on this head: BPM-only and both beats/BPM orderings with bpm=signal made zero requests; both conflicting pin directions, with describe first, rejected with zero requests. Matching API pins made exactly one shared request, proving the transport witness is non-vacuous. My prior-head witness made an unwanted upload; the successor fixes it.

Independent focused tests: 88 passed, one native macOS test skipped. The complete successor diff is the shared-ladder pin correction plus five regressions; prior silence/beat-file fixes remain intact. No new own-code blocker found. All required checks now pass, including Windows render, build, typecheck and runtime contract. I read the completed aggregate accuracy log: 271/271 accurate, matching the measured base. One non-required regression shard remains running; no failure is reported, so this does not claim every check is finished.

Limits: synthetic loopback/provider tests, not a real backend, installed Beat This model, Apple native inference, subjective corpus evaluation, or live Studio/Desktop certification. The author’s benchmark and browser captures remain author evidence. No merge, queue, release or deployment performed.

— Magi (desktop-review)

Verdict: APPROVE
Reasoning: The ignored provider pin is fixed on this exact head with discriminating transport controls; current-head validation supports the code verdict.

[jrusso1020]

Approving at 2d6903ce. Both of my blockers from 65b986bc are fixed, and required CI on this head is green.

• Detect changes: ALLOWED_DELETIONS names exactly the two removed paths, each with a reason, and a test pins that beatDetection.ts is not allowlisted.
• Studio captures: the body has Before/After images, and .github/pr-evidence/5634/ is out of the tree.
• Parent fixes: 2d6903ce merges #5632's 54dfc960, which has the sections silence fix and the bpm= pin fix. git range-diff shows this PR's own five commits are identical to what I reviewed.

The non-blocking notes from my previous review still apply. The main one: short and low-confidence tracks now get no beats, and the CLI exits 1, so it's worth a line in the release notes.

Merge order: #5632 first, then this one.

— Rames

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
hyperframes	🟢 Ready	View Preview	Oct 15, 2026, 5:42 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[miguel-heygen]

Reaffirming approval at main-target head e0f89f6b3bd936ed02430c3826259874adb27e2d, based on 8e340d220ea40c994eb3b5cf69ae503eed267fe7 (the merged #5632).

Compared the complete child-v-main patch with the previously approved child-v-parent patch: all 20 non-manifest file patches are byte-identical. The sole differing patch is the generated hyperframes-cli skill hash; independently recomputed it from the exact head's 11 Git blobs and matched 8dea58f3309af1c5. The merge brings main's existing changes into the branch without an additional audio implementation change. No new wording in the child's own delta.

Fresh-head required checks all passed, including Windows rendering, Build, Typecheck, runtime contract and capture gate. Read the completed aggregate accuracy log: 271 accurate versus base 271, gate passes. Runtime job logs show core 4693 passed/1 skipped and CLI 4943 passed/38 skipped. At this review, 77 checks succeeded and 2 skipped; one non-required regression shard (3) remains running. This is a code approval; normal repository/queue gates still govern merge.

Prior independent audio probes remain applicable to the identical patch; no repeated local suite or native SoundAnalysis/BeatThis/real-music/Studio UI acceptance is claimed in this retarget pass. Read-only Git inspection; no task worktree or owned process created. No merge, publish or release action.

— Magi (desktop-review)

Verdict: APPROVE
Reasoning: Retarget preserves the approved own delta, generated manifest is consistent, and fresh required checks plus aggregate accuracy pass.

[miguel-heygen]

Reaffirming own-delta approval at 8c57a300fa6aed2da5a28508fb33996fee79164f, now targeting main at d4c8d76fff9a034227de3fff186aad9c5b1cd3c4.

All 20 non-manifest child file patches against current main are byte-identical to the approved e0f89f6b patch against its main base. The only differing patch is the generated skills manifest. Independently recomputed both updated hashes from exact-head Git blobs: hyperframes-cli f08c429fb0f8b82c (11 files), hyperframes-animation 1c91404cdf888540 (122 files). Both match; GitHub reports the merge commit signature valid. Imported shader changes are main's existing changes, not an additional child implementation delta.

Fresh-head required checks all passed, including Windows rendering, Build, Typecheck, runtime contract and captures. Studio and runtime test jobs passed. Read actual runtime logs: core 4693 passed/1 skipped, CLI 4943 passed/38 skipped. At review, 58 checks succeeded, 2 skipped and 18 additional checks were still running, including accuracy/regression work; I am not claiming that this head's aggregate accuracy gate has finished. Normal repository and merge-queue gates remain authoritative.

Prior independent audio probes remain applicable to the unchanged own patch. No new local test run, native-provider/BeatThis/real-music/Studio acceptance, paid call, merge or release action in this retarget pass. No worktree or owned process created.

— Magi (desktop-review)

Verdict: APPROVE
Reasoning: Main integration preserves the approved child patch; manifest consistency and fresh required gates are verified. Additional CI remains pending as disclosed above.

[jrusso1020]

Reaffirming my approval at 8c57a300, which targets main after the parent's squash merge (8e340d22) and a later sync with main (d4c8d76f).

I diffed this head against its merge base with main and compared that to my previously approved patch (54dfc960..2d6903ce). They are identical except for the hash lines in the regenerated skills manifest. There is no new code or skill wording, and the sync merge 8c57a300 is signed.

Required CI on this head is green, and there are no open code-scanning alerts.

— Rames