feat(cli): the api provider speaks the HeyGen audio analysis contract

[xuanruli]

What

The api rung of hyperframes audio (added in #5632) now speaks the contract of HeyGen's audio analysis route:

• Auth: the key goes in x-api-key instead of Authorization: Bearer.
• Idempotency: each request carries a new Idempotency-Key (a UUID). The CLI does not retry, so one request is one key.
• Upload: the audio is uploaded as 16 kHz mono 16-bit WAV. The channels are averaged, then resampled with a Hann-windowed sinc (16 zero crossings, cutoff at 0.9 of the lower Nyquist), so nothing above 8 kHz folds back in.
• Envelope: the answer is read as body.data ?? body, so { "data": … } and a bare answer both work.
• AudioApiResponse gains time_resolution_seconds. Nothing reads it.

Why

• HeyGen's API takes a HeyGen key only as x-api-key.
• The route requires Idempotency-Key, like /v3/audio/music and /v3/audio/transcriptions. A same-key retry replays the answer and is not charged again; the same key with a different body gets 409.
• The route answers in EF's standard data envelope.
• The model hears one channel at 16 kHz anyway. The route caps uploads at 32 MB and 600 s, so a 3-minute 44.1 kHz stereo song was at the limit and a 4-minute one was refused. At 16 kHz mono, ten minutes is 19.2 MB.

Related work

Refs #5632. The route is heygen-com/experiment-framework#57783. Desktop's gateway, which carries the call for a chat run, is heygen-com/hyperframes-internal#4049; it now takes the run token in x-api-key only.

How

• Every time is still in seconds from the start of the audio, so resampling does not change them.
• The snap step still measures beats on the signal as decoded, not on the upload.
• Resampling 600 s of 44.1 kHz audio takes about 1.8 s.

Test plan

• Unit tests updated: bunx vitest run src/audio in packages/cli passes 47/47.
  • The test server answers in the data envelope, plus one test with a bare answer.
  • Requests are checked for x-api-key with no Authorization, a UUID Idempotency-Key that differs per request, and a 16 kHz mono WAV.
  • A stereo 44.1 kHz test signal checks the upload. A 2 kHz tone in opposite phase between the channels must cancel. A 9 kHz tone must not alias to 7 kHz. A 440 Hz tone must survive at its level.
• Mutations, each reverted on its own, fail at least one test:
  • reading the raw body;
  • keeping only the first channel;
  • skipping the resample;
  • keeping the input rate;
  • a cutoff at the input Nyquist (aliasing);
  • no idempotency key;
  • a constant key;
  • Bearer auth;
  • snapping on the uploaded signal.
• Manual, end to end, all local:
  • Setup: this CLI → Desktop's real gateway (openHeygenGateway) → EF's real /v3/audio/analysis view in a local Flask app, with the model, credits and auth faked.
  • Run: hyperframes audio click120.wav --only beats,downbeats,bpm,sections,describe --quality best --json on a 12 s, 44.1 kHz stereo file.
  • All five capabilities came from api, with no warnings.
  • Three requests reached EF, each with the key in x-api-key, its own Idempotency-Key, and a 16 kHz mono WAV of 384,044 bytes. The old upload was 2,116,844 bytes.
  • Reverting each of the four changes broke the run:
    • without the envelope read, every api capability fell through;
    • without the resample, the route answered 400;
    • without the key, the route answered 400;
    • with Bearer, the gateway answered 401.
• Documentation updated: docs/packages/cli.mdx.
• Checks: oxlint, oxfmt, fallow audit --base origin/main (no issues), tsc for cli, core, studio and scripts, and commitlint.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
hyperframes	🟢 Ready	View Preview	Oct 15, 2026, 6:33 AM

💡 Tip: Enable Automations to automatically generate PRs for you.

[jrusso1020]

Approving at c6af0263. I found no blocker, and required CI on this head is green.

Resampling and aliasing. There is a real low-pass before decimation. resample builds a Hann-windowed sinc with 16 zero crossings and scales it by min(1, to/from) × 0.9, so for 44.1 kHz → 16 kHz the cutoff is 7.2 kHz.

• Weights are normalised to unity DC gain.
• The kernel is centred on i × step, so it is zero-phase and every time stays in seconds from the start, unshifted.
• Table indexing tops out at 16 × 512 + 1, inside the 16 × 512 + 2 table.
• The Hann transition is about 1.8 kHz wide around 7.2 kHz. Only a sliver between 8 and about 8.1 kHz can fold, and only into the top of the band. The 9 kHz → 7 kHz test covers the case that matters.
• Upsampling (a source under 16 kHz) gets a cutoff at 0.9 of its own Nyquist, which is fine.
• Snapping still measures on the decoded signal that analyze() receives; only ask() sees uploaded().

Auth. The key goes only in x-api-key, and the test checks that Authorization is absent and the key appears nowhere in the request JSON.

Idempotency and replay. randomUUID() runs once per ask() and ask() is never retried, so one request carries one key. perSignal memoises per signal, so asking beats and then downbeats reuses one request rather than sending a second key. The open edge is a client timeout (120 s default) after the server already did the work: a rerun is a new process with a new key, so it is charged again. That is inherent without a persisted key, and fine while the CLI does no retries. If a retry is ever added, it must reuse the key from the first attempt.

Envelope and compatibility. body?.data ?? body ?? {} reads both { data } and a bare answer. A null or empty answer becomes {}, which checked() refuses, so the ladder falls through rather than crashing. A service built to the #5632 contract with Bearer auth now gets no Authorization header and will refuse the call. That's acceptable because that contract shipped hours ago, and the request still describes its own audio (sampleRate: 16000, channels: 1).

Non-blocking:

1. Redirects now carry the key. fetch follows redirects by default and drops Authorization on a cross-origin hop, but it forwards custom headers like x-api-key. Moving off Bearer means a configured URL that redirects to another origin now receives the key. Adding redirect: "error" to the fetch call closes it, which is how the Desktop gateway's HeyGen calls are sent.
2. Averaging channels cancels content that is in opposite phase between them, as the 2 kHz test asserts. That's the right trade for a mono model, but music mixed with a wide out-of-phase element could lose it. Nothing to change; it's just worth knowing.

Not exercised: no call to the real service. The end-to-end run in the body used faked model, credits and auth, so it isn't live acceptance. Pricing is out of scope here; it lives on the server side.

— Rames

[xuanruli]

Addressed Rames' redirect finding in signed commit 0ac8b34: audio API requests now use redirect: "error", so a redirect cannot forward x-api-key to another origin. A two-server local regression test asserts that the destination receives no request. Removing the guard makes it fail with the forwarded test key.

Validation: 48/48 audio tests, CLI typecheck, oxlint/oxfmt, and normal commit hooks (including fallow and typechecks) passed. No live model/auth/ledger calls. This is a new head; prior-head approval does not satisfy the requested current-head Rames + Magi merge gate.

[miguel-heygen]

Reviewed 0ac8b347bd59f94599ce626e8f76aecd0b2d053e.

The low-pass resampling and request contract are well covered, and api.ts:161 now rejects redirects so the custom API-key header cannot follow them. Rames covered those areas; this adds a stronger witness for the channel-cancellation concern in his prior review.

Blocker — valid stereo becomes a silent model input (packages/cli/src/audio/analysis/providers/api.ts:85-90). uploaded() averages signed channels before resampling. For a one-second 440 Hz stereo signal with R = −L, the actual multipart upload contains entirely zero PCM samples. My loopback capture measured peaks [22937, 0] for the mono control and opposite-polarity stereo at this head. With the base provider restored, the same capture gives [22937, 22937]: the stereo content was retained. This is more than losing a wide instrument: an audible input can become complete silence for every API capability, causing missing or unrelated model answers. The authored stereo test has other in-phase content, so it does not catch this case.

Please preserve audible content when preparing mono input (including a phase-cancellation fallback), while retaining the 16 kHz and anti-aliasing contract, and add a pure opposite-polarity regression.

Evidence: the preceding focused suites passed 23 tests with one skip; the current-head provider suite and loopback witness were rerun after the redirect-only successor. The current-head provider suite passes 13/13, plus the separate upload witness (14 checks total). The prior head's hosted runtime log passed the provider tests; fresh successor CI is still pending at review time. No real gateway, Gemini, paid call or subjective audition was performed. EF deployment and pricing are separate unresolved gates.

— Magi (desktop-review)

Verdict: REQUEST CHANGES
Reasoning: The upload preprocessing newly erases a valid class of stereo inputs before the model sees them; green fixture tests on ordinary mixes do not establish preservation.

[xuanruli]

Addressed Magi review 5529003115 and Rames's follow-up in signed commit d98a0efd8.

The multipart-upload regressions first failed for L = −R, 162° phase separation, and opposite-phase channels with unequal gains. Mono preprocessing now compares average-signal energy with the loudest channel and uses that channel only when the average retains less than 10% of its energy. Ordinary averaging stays unchanged; 90° and in-phase controls pass, as does the existing mixed-tone cancellation/anti-aliasing test. The fallback still goes through the same 16 kHz resampler. The unequal-gain regression puts the louder channel second, so always selecting the first channel does not satisfy it.

53/53 audio tests pass; typecheck, oxlint/oxfmt and normal commit hooks pass. Fallow reports a nonblocking duplicated test-setup block. These are local loopback/fixture results, not subjective listening or live service acceptance. No paid call.

Please re-review the current head. Both current-head Rames and Magi approvals and fresh CI remain required; EF/gateway dependencies and pricing are still separate holds.

[miguel-heygen]

Re-reviewed d98a0efd8f76cf7f96845fed205d4b70fbd7715c. This fixes my upload-cancellation blocker and supersedes my earlier changes request.

packages/cli/src/audio/analysis/providers/api.ts:85-113 now falls back to the loudest channel only when the average retains less than 10% of its energy; the selected signal still passes through the existing anti-aliasing resampler. The unequal-gain test puts the louder channel second, and ordinary in-phase/90-degree averaging and the mixed-tone resampling controls remain intact. The redirect guard is unchanged.

I reran the actual multipart-upload witness: the mono control and L = −R stereo both retain PCM peak 22,937, instead of the stereo upload becoming zero. Four focused files pass 30 tests with one skip (18 authored API-provider cases plus the independent upload witness). Replacing only api.ts with the held head makes four checks fail: the three new destructive-phase cases and that independent witness. Source was restored and the disposable worktree retired cleanly. Required CI, including Windows rendering, now passes/skips.

No real gateway/Gemini/ledger, paid call or subjective listening acceptance was performed. EF #57783's durable billing, upload contract, pricing and gateway integration holds remain separate; this approval is limited to the CLI PR.

— Magi (desktop-review)

Verdict: APPROVE
Reasoning: The concrete silence regression is repaired with a bounded fallback and discriminating controls; current-head required CI is green.

[jrusso1020]

Re-approving at d98a0efd. Since c6af0263, which I approved, there are two commits: the redirect fix and the downmix fix.

Redirect (0ac8b347). ask() passes redirect: "error", so a 3xx from the configured endpoint fails the request instead of being followed with the x-api-key header. The new test uses two local servers, one returning 307 to the other, and asserts that the second never sees a request.

Downmix (d98a0efd). This fixes the cancellation Miguel flagged: averaging L and −R gave silence. monoInput still averages, but when the mix keeps under 10% of the loudest channel's energy it uploads that channel instead. I checked the five test cases by hand against the threshold:

• phase π: the mix is silent, so the loudest channel is sent (0.7).
• phase 0.9π: the mix amplitude is 0.7·cos(0.45π) ≈ 0.11, an energy ratio of about 2.4%, so the loudest channel is sent.
• phase π with one channel at 0.8: the mix amplitude is 0.07, a ratio of 1%, so the right channel is sent.
• phase π/2: the ratio is 50%, so the mix is kept at 0.7/√2.
• phase 0: the mix is kept at 0.7.

On the old uploaded(), the phase π case would upload silence, so the test pins the bug. With one channel, the mix equals that channel, so mono input is unchanged. The docs line describes the new rule.

CI is green at this head. I didn't run it locally.

— Rames