Skip to content

fix(producer): reject regression audio stream mismatches - #5228

Open
user-github-me wants to merge 7 commits into
heygen-com:mainfrom
user-github-me:fix/producer-audio-presence-regression
Open

user-github-me wants to merge 7 commits into
heygen-com:mainfrom
user-github-me:fix/producer-audio-presence-regression

Conversation

@user-github-me

@user-github-me user-github-me commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

A regression render can lose its audio track and still pass because the harness assigns correlation 1 when either side lacks audio. Require matching audio-stream presence before comparing envelopes. Both mismatch directions fail even with a zero correlation threshold; two silent renders still pass.

Preserve the reviewed silent pip-video-late-host baseline: 144 H.264 frames at 24 fps, six seconds, no audio stream. Its LFS content hash is unchanged. The current-main integration retains newer integration-test classifications and refreshes their reachability fingerprint.

Validation:

  • All 56 harness and audio tests pass with their declared Bun/Vitest runners, including four real render audio-presence cases.
  • The original pinned Chrome 148 Docker baseline validation passed compilation, all 100 visual checkpoints and audio comparison after regenerating the muted-video snapshot. This update preserves those exact MP4 bytes.
  • Full workspace build, producer typecheck, classification and reachability checks, lint, formatting, comment gates and normal commit hooks pass.

This PR shares the tested signed revision with #5297; the existing review history is preserved.

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change itself reads correctly, but the required regression check is red at this head, and the cause is this PR's new rule, not a flake. It failed the same way on a rerun.

What fails. Shard 2, pip-video-late-host:

✓ Visual quality: PASSED (0 failed frames, threshold: 5)
✗ Audio stream mismatch: rendered=missing, snapshot=present
✗ Audio quality: FAILED (correlation: -1.000, threshold: 0)
❌ Test FAILED: Pip video inside late-starting sub-composition host

Why. The fixture's only media element is a muted <video> (src/compositions/pip.html) with no data-has-audio. So a render with no audio stream is the correct output. The committed output/output.mp4, though, carries a real AAC track (volumedetect: mean -25.2 dB, max -4.5 dB). That snapshot was last regenerated in #3928. #4625 put this fixture into CI later and changed only the HTML, not the snapshot. Under the old rule a one-sided audio stream scored correlation 1, so the stale snapshot passed. Your stricter check now catches it, which is the behavior this PR is meant to add.

To land this, the PR needs to keep regression green:

  • Regenerate packages/producer/tests/pip-video-late-host/output/output.mp4 with the Docker baseline flow, so the snapshot matches the muted composition (no audio stream). Include it in this PR.
  • Then check whether any other fixture with a muted-only composition has an audio-bearing snapshot. Only shard 2 reached a verdict here. Shards 1, 3, 4, 7 and 9 were cancelled when shard 2 failed, so this run doesn't tell you whether they're clean.

I'll approve once regression is green at the new head.

— Rames

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants