feat(cli): judge text behind a transparent video by its decoded alpha in check

[WaterrrForever]

What

hyperframes check now judges text under a <video> by the frame the video actually shows at each sample. A transparent video, such as a remove-background cutout, covers text only where its decoded alpha is. An opaque video still covers its whole box, as it does today.

• A word partly behind the subject is an info text_occluded finding. Its coveredFraction and message give the most covered word's share, for example its most covered word "FASTER" is 60% hidden.
• A word that is 80% or more hidden, or two neighbouring words each at least half hidden, is an error. Neighbouring words can sit in one element or in sibling one-word elements, which is how captions are usually built.
• If a video's frame cannot be read (it is still seeking, has no decoded frame, is cross-origin, or reads back fully transparent), the audit treats the video as opaque, as it does today. It adds a new info finding, video_alpha_unread, that says why.
• data-layout-allow-occlusion on the text still silences the finding.

Why

To put text behind the speaker, an author stacks the footage <video>, then the captions, then a VP9-alpha cutout above them by z-index. The layout audit treated every <video> as opaque across its whole box but sampled an <img>'s real alpha (isOpaqueOccluder in layout-audit.browser.js). So every caption behind a WebM cutout failed text_occluded, including captions in the transparent part of the frame. Authors had to blanket-silence it with data-layout-allow-occlusion, and that also hid real problems, such as a word fully covered by the head.

On a sample project (1920x1080 footage, a synthetic head-and-shoulders VP9-alpha cutout, 7 caption elements, 9 samples):

	main	this PR
text_occluded	7 errors, all "hidden beneath an opaque element" (including NOW and with, which sit entirely in the transparent part)	2 errors ("today" is 100% behind…; "code" and "today" are 53% and 100%…), 3 infos (FASTER 60%, less 28%, BUILD 9%)

Related work

Refs #5651. That PR's skills/hyperframes/references/captions.md ("Text behind the speaker", line 44) currently says:

check treats a <video> as opaque across its whole box, so a WebM cutout fails every caption behind it with text_occluded: put data-layout-allow-occlusion on the cutout and judge from snapshot.

Once this lands, it should say:

check reads the cutout's alpha at each sample: a caption partly behind the person is reported as info with its most covered word's share, and a word 80% hidden or two neighbouring words each half hidden fail text_occluded. No data-layout-allow-occlusion is needed; keep it for text you mean to hide entirely, and put it on the text, not the cutout.

Line 45's "Measure the rendered word boxes against the matte…" can then point at check, which now does that measurement. Separately, the waiver is read from the text element itself, so putting it on the cutout never silenced anything.

How

All in packages/cli/src/commands/layout-audit.browser.js:

• Frame read. readVideoAlphaFrame draws the video's current frame into one shared scratch canvas (willReadFrequently), downscaled so its longer side is at most 480px, and keeps only the alpha channel. Results are cached per video for one auditLayout call, which is one sample time. A video is read only when a text probe's elementsFromPoint walk reaches it, which means it is painted above some text. Videos under no text are never decoded.
• Classification. If every pixel would occlude (alpha ≥ the existing 0.6 occluder threshold), the frame is opaque and nothing changes from today. If some pixels would not occlude, it is a matte, and isOpaqueOccluder judges each probe point by that pixel's alpha × the video's opacity chain. This is the same rule <img> already uses. The object-fit/object-position mapping was moved out of imageAlphaAt into paintedSourceMapping/sourcePixelAt, so images and videos share it. A frame that reads back fully transparent counts as unread, not as clear, so a failed draw cannot hide text.
• Coverage. When every grid hit on a text element comes from matte videos, the element takes the matte path. Each rendered word is a Range over a \S+ match, and its client rects are sampled at the matte's resolution (at most 48x48 per rect). Coverage is the area-weighted share under occluding alpha. These are the per-word, rendered-box measurements the retired embedded-captions check-occlusion.cjs made, and the thresholds (80% lost; two neighbours at 50%) are from its composition-craft.md. Mixed hits (an opaque box plus a matte) keep today's grid and atomic-label rules.
• Determinism. check seeks through the runtime, whose seek barrier (__hfWaitForSeekCompletion, fed by holdSeekBarrierUntilVideoLands in core's media.ts) holds until a seeking <video> fires seeked/loadeddata. The audit then refuses any frame that is still seeking or below HAVE_CURRENT_DATA, and reports it as video_alpha_unread rather than reading a stale frame. The Chromium test proves the read frame is the seeked one: a cutout whose subject switches sides at 0.5s gives opposite verdicts at 0.25s and 0.75s.
• video_alpha_unread is added to LayoutIssueCode and the parser's code list. The coveredFraction doc is updated. Docs that described the old behaviour are updated: docs/prompting/rules-and-anti-patterns.mdx said "a caption designed to sit behind a matted subject needs one" waiver. docs/guides/remove-background.mdx and docs/packages/cli.mdx gain a short paragraph, and so does skills/hyperframes-cli/references/lint-validate-inspect.md (with skills-manifest.json regenerated). No other skills are edited.

Performance. On the sample project, in chrome-headless-shell 152 on an M-series Mac, the in-page __hyperframesLayoutAudit call took a median of 2.3–2.5 ms per sample on main and 4.7–5.5 ms on this branch, over 3 runs of 9 samples each. That is about +3 ms per sample, or about +30 ms per check. Whole-check wall time was 11.6 s on main and 12.1–12.9 s on this branch, which is within run-to-run noise. Results were identical with --no-browser-gpu (SwiftShader) and with the default hardware mode.

Test plan

• Unit tests added: packages/cli/src/commands/layout-audit.video-alpha.browser.test.ts (happy-dom, mocked frame alpha). They cover:

  • an opaque frame occludes as before;
  • text under the transparent part is not flagged;
  • 30% cover → info coveredFraction: 0.3;
  • 85% → error;
  • one lost word fails an otherwise clear line;
  • neighbouring words at 63% each fail, within one element and across sibling elements;
  • a single half-covered word stays info;
  • the opt-out is respected;
  • one frame read per video per sample, and none for a video under no text;
  • four unread fallbacks (not decoded, still seeking, SecurityError, all-transparent) → opaque error plus video_alpha_unread.

• Browser test added: packages/cli/src/commands/layout-audit.video-alpha.chromium.test.ts. Its fixtures are VP9-alpha WebMs generated with ffmpeg in beforeAll (a few KB, nothing committed), served with byte ranges so Chromium can seek. It covers:

  • text fully behind an opaque video → error, coveredFraction: 1;
  • text behind the transparent part of a cutout → no finding;
  • a word half under the matte → info, coverage between 0.4 and 0.6;
  • a fully covered word → error;
  • two neighbouring half-covered words → error;
  • the seeked frame is the one read;
  • the opt-out is respected;
  • a preload="none" video → opaque error plus video_alpha_unread.

  It runs when PUPPETEER_EXECUTABLE_PATH is set and ffmpeg is on PATH, like layout-audit.chromium.test.ts, so CI skips it.

• Manual testing performed: check on the sample project above, before and after, in hardware and SwiftShader modes.

• Documentation updated

• Comments follow CONTRIBUTING.md "Comments"

Commands, run from the repo root unless noted:

PUPPETEER_EXECUTABLE_PATH=~/.cache/hyperframes/chrome/chrome-headless-shell/mac_arm-152.0.7977.30/chrome-headless-shell-mac-arm64/chrome-headless-shell \
  npx vitest run src/commands/layout-audit src/commands/contrast-occlusion.browser.test.ts src/utils/layoutAudit.test.ts src/utils/checkBrowser   # in packages/cli
  → 8 files, 339 tests passed
npx vitest run   # in packages/cli → 4941 passed, 10 failed in 5 files unrelated to this change (usage, fileWatcher.fs, whisper/manager.install, whisper/sherpa, renderCancellation.process; local-environment tests that do not import the layout audit)
npx tsc --noEmit -p packages/cli → clean
bun run lint → 0 warnings, 0 errors
npx oxfmt --check <changed files> → clean
npx -y fallow audit --base origin/main --threads 4 --fail-on-issues → No issues in 10 changed files
COMMENT_CHECK_BASE=origin/main node scripts/check-comment-citations.mjs → comments: OK
COMMENT_CHECK_BASE=origin/main node scripts/comment-ratchet.mjs → at or under base share
bun packages/cli/scripts/gen-skills-manifest.ts --check → in sync

No visible change

Nothing in packages/studio or packages/player changes, and no Studio or player UI changes. What changes is the text and JSON output of hyperframes check: text_occluded severity, coveredFraction and message for text under a transparent video, plus the new video_alpha_unread info finding. The finding counts above show the before and after.

🤖 Generated with Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
hyperframes	🟢 Ready	View Preview	Oct 15, 2026, 6:25 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

[github-actions]

Edit accuracy: accurate 271 (base branch 271), smooth 180 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

Unstable (2)

• textselect-none-px-r0-nested-z100: tracking 0, pressJump -, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0, pressJump -, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0, pressJump -, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true
• seqplainkeys-none-pct-r0-root-z100: tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo false, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true / tracking 0.06, pressJump 0, drop 0, reload 0, render 0.03, renderKey -, undo true, teleport true

[jrusso1020]

Reviewed at d62bc0cddb25bd7e00fb4a93987028c6d69f058e. Reading the matte instead of treating the whole video box as opaque is the right fix, and most of this checks out (details below). I'm requesting changes for one regression: a word hidden by a narrow part of the matte now gets no finding, where main reports an error.

Blocker: narrow matte coverage goes unreported

The per-word matte check (matteOcclusionIssue) runs only when occlusionCoverage returns an occluder (layout-audit.browser.js:1525, if (!occluder) return null;). With this PR, a probe point hits the video only where its alpha actually covers the text. The coarse grid is 9 columns × 3 rows per line, at 3/10/20/35/50/65/80/90/97% of the width, so neighbouring columns can be 15% of the line apart. A matte feature that fits between two columns gets no grid hit at all: an arm, a mic stand, a small subject, or a head under a long caption. In that case occluder stays null and the word-level check never runs.

Reproduced in chrome-headless-shell 152:

• Clip: VP9-alpha, opaque only at source x 76–104, shown at 2× (page x 152–208).
• Caption: ab cd ef gh ij … at 24px monospace, placed so the 20% and 35% probe columns land near 139px and 221px.
• main: text_occluded error, coveredFraction: 1.
• This PR: no finding, though gh is 100% hidden.

When the strip is moved onto a probe column, the PR correctly reports "gh" is 100% behind a transparent video's matte, so the word-level check itself works once it is reached.

Suggested fix: in occlusionCoverage, remember every matte-classified video the stack walk finds above the text, including at points where its alpha is clear. Then run matteOcclusionIssue whenever there are no opaque hits and at least one matte was seen, rather than only when occluder is set. A Chromium test with a strip between two probe columns would pin it.

Non-blocking

• A fully transparent frame counts as unread. That covers the cutout before the person walks in, or a fade. Each caption under it then becomes a hard text_occluded error, plus video_alpha_unread, which is the same false positive this PR removes elsewhere. The body says this is deliberate, so a failed draw can't hide text. It would help to say so in the docs, or to tell a real blank frame apart from a failed draw: readyState is already checked, so a frame that has been read and is all clear could be trusted.
• docs/packages/cli.mdx lists the unread causes as "not decoded yet, or cross-origin". The code also reports still seeking, fully transparent and no frame size.
• The Chromium suite is skipped in CI without PUPPETEER_EXECUTABLE_PATH. In the Test (runtime) log all 8 tests are skipped, so nothing in CI checks that VP9 alpha survives drawImage. The body says so, and it matches the existing layout-audit.chromium.test.ts.
• Mixed hits. Text under both an opaque box and a matte now counts only the matte pixels that actually cover it, not the whole video box. That's a small change from main, though a more accurate one.

Checked and fine

• Fallback. Every unread case falls back to opaque, which is the old behaviour: not decoded, seeking, SecurityError, draw error, no size, all clear. An opaque VP9 clip takes exactly the old path (Chromium test 1, coveredFraction: 1). H.264 reads as opaque. ProRes 4444 .mov never decodes in Chromium, so it also stays opaque.
• Shared canvas. A cross-origin read doesn't taint the shared scratch canvas for later same-origin reads: resizing it resets its state, and a same-origin cutout was still read after a cross-origin failure.
• Frame cache. The cache resets at the start of each auditLayout call, so frames don't carry over between samples. Frames are downscaled to at most 480px, and per-word sampling is capped at 48×48.
• Image path. The shared paintedSourceMapping/sourcePixelAt is byte-equivalent for <img>.
• Codes and waiver. The new code is in both LayoutIssueCode and the parser list, and the waiver is still checked first.
• Tests. I ran all 8 Chromium tests locally (chrome-headless-shell 152) and all 280 layout-audit tests, and they pass. With main's layout-audit.browser.js swapped in, 16 of the new tests fail.
• Manifest. gen-skills-manifest --check is in sync. CI is green; every check that ran passed.

— Rames

[miguel-heygen]

Reviewed exact head d62bc0cddb25bd7e00fb4a93987028c6d69f058e. Additive to Rames's narrow-matte finding: two other rendered-pixel regressions need fixes.

The per-audit frame cache, bounded canvas/sampling, new issue-code parser wiring and existing waiver path are consistent. Independently ran 209 layout tests, including all eight actual Chromium/FFmpeg tests; these pass. The actual hosted runtime log runs the 14 mocked-alpha cases but skips all eight Chromium cases.

Blocker — mirrored cutouts read alpha on the wrong side (packages/cli/src/commands/layout-audit.browser.js:1030, :1080, :1212). paintedSourceMapping uses only an axis-aligned bounding box, so transform:scaleX(-1) leaves source coordinates unmirrored. Using the PR's actual VP9 cutout fixture with that CSS transform, the screenshot shows the left SHIP fully visible and right SHIP fully hidden. The audit instead reports left SHIP 100% hidden and returns no finding for right SHIP. Base catches the hidden right word. Map viewport coordinates through the actual element transform, or conservatively retain opaque handling for unsupported transforms; add a real-browser mirrored fixture.

Blocker — a video's opaque CSS background disappears from occlusion (packages/cli/src/commands/layout-audit.browser.js:1235). Returning the matte result before considering the element's background treats clear media pixels as clear painted pixels. Add background:#123456 to the PR's cutout video and place SHIP behind its clear half: the actual Chromium screenshot contains no visible text, but this head returns no text_occluded finding. The identical base control returns the expected error. Preserve solid/background-image occlusion when resolving matte pixels; add a regression test.

Both witnesses used real decoded VP9 alpha in local Chromium, with actual screenshots, not mocked canvas output. No live Desktop/model/provider run. Existing CodeQL path alert 1172 reports its latest instance fixed; current CI has no failed/pending checks, which does not cover these two cases. Rames's separately demonstrated between-grid narrow-matte blocker also remains applicable.

— Magi (desktop-review)

Verdict: REQUEST CHANGES
Reasoning: The alpha exemption currently lets completely hidden words pass for mirrored cutouts and videos with opaque CSS backgrounds. Correct the painted-pixel mapping/composition before relaxing the gate.

[WaterrrForever]

Thanks both, fixed in 23c920f:

• Narrow matte (Rames): the probe walk now remembers every matte above the text, including at clear-alpha points, and runs the per-word check whenever there are no opaque hits and a matte was seen. Your strip-between-columns repro is now a Chromium test, plus a mocked twin so CI covers it.
• Mirrored cutout (Magi): viewport points map through flips from transform and scale on the video and its ancestors. A rotated, skewed or 3D-transformed video falls back to opaque with video_alpha_unread. Covered by Chromium scaleX(-1) and rotate(90deg) tests and mocked twins. This also fixes flipped <img> alpha.
• Opaque background (Magi): a video with an opaque CSS background keeps its whole box occluding, as on main.
• Docs: cli.mdx lists every unread cause and why a fully transparent frame counts as unread.

All 8 new tests fail on d62bc0c; 316 layout tests pass locally with the Chromium suite on. Required CI is green at 23c920f.

[jrusso1020]

Approved at 23c920f9b9. This resolves my request for changes at d62bc0cd.

The narrow-matte blocker is fixed. occluderAt now records every matte video it passes in the stack above the text, including at points where its alpha is clear. It only records hits above the text, because the walk still returns at the text element itself. When no probe hit something opaque and a matte was seen, occlusionCoverage returns that matte as the occluder and hands every seen matte to matteOcclusionIssue. The per-word check therefore runs even when no probe point lands on the matte's opaque pixels. If any probe does hit something opaque, the old whole-box path still applies.

The other two cases, also fixed:

• Mirrored cutouts. paintFlips walks the element and its ancestors, and flips the source lookup for a negative scaleX/scaleY from either transform or the scale property. A rotation, a skew or a 3D matrix anywhere up the chain returns null. The video is then treated as unread (opaque) with the reason given, rather than read misaligned.
• Opaque CSS background. A video with an opaque CSS background is no longer read as a matte, so its whole box occludes again.

Ran locally with Chromium 133 (PUPPETEER_EXECUTABLE_PATH set), all 30 of 30 layout-audit.video-alpha* tests pass at this head. With only layout-audit.browser.js reverted to d62bc0cd, all four new Chromium cases fail (narrow strip, mirrored, opaque background, rotated), so they test the fix.

Nits:

• The Chromium suite still skips in CI, because the CLI test job doesn't set PUPPETEER_EXECUTABLE_PATH. These four cases only run on a machine that sets it.
• The docs now list every unreadable cause, and explain that a fully transparent frame is treated as unread on purpose. That covers my earlier docs nit.

Required CI is green at this head. I didn't run a real composition through hyperframes check.

— Rames

[miguel-heygen]

Re-reviewed exact head 23c920f9b92be2c253cbe0d8599eeced043aa2e9. Both of my rendered-pixel blockers are fixed; Rames's narrow-matte repair is verified too.

• packages/cli/src/commands/layout-audit.browser.js:1057 tracks flips through the video and its ancestors; unsupported rotation/skew/3D falls back conservatively. My original real-Chromium mirrored-cutout witness now flags the hidden right word and leaves the visible left word clear. Additional ancestor transform:scaleX(-1) and standalone scale:-1 1 controls also pass.
• matteFrameFor preserves an opaque CSS background, so my original background witness again reports the fully hidden word.
• occlusionCoverage remembers mattes even at clear-alpha probe points, making the narrow-strip per-word check reachable.

Independent results: 249 layout tests pass with Chromium enabled (including 12 video-alpha Chromium cases and 32 other Chromium cases), plus three independent browser probes. Restoring only the previous d62bc0cd production audit makes exactly eight new regression tests fail (22 still pass). Manifest regeneration check passes. I read the complete four-file successor delta and actual hosted logs: the 18 mocked-alpha tests execute; the 12 Chromium cases still skip in CI, but ran locally here. All current required checks pass.

Limits: real browser/decoded synthetic VP9 fixtures, not live Desktop or a real project run through the complete CLI command. No model/provider calls, merge or release.

— Magi (desktop-review)

Verdict: APPROVE
Reasoning: The three demonstrated occlusion regressions are corrected and independently protected by passing browser witnesses and failing-old-code controls.