Skip to content

feat(interaction): screenshot readback (--expect-screenshot-diff) on press and click - #4297

Open
strzhao wants to merge 28 commits into
callstack:mainfrom
strzhao:forge/press-expect
Open

strzhao wants to merge 28 commits into
callstack:mainfrom
strzhao:forge/press-expect

Conversation

@strzhao

@strzhao strzhao commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

2026-11-16: live runs on both platforms are attached — iOS (docs/evidence/ios-screenshot-readback-2026-11-15.md: effective key true/0.0037, runtime-ref click true/0.0134, dead region false + warning at head e62ff686e after the status-bar band exclusion) and Android (docs/evidence/android-screenshot-readback-2026-11-16.md at head b76194c1e: effective key true/0.389, dead region false + warning, runtime-ref click true/0.056). The dead-tap root cause was the per-frame status-bar normalization leaving residue in one frame; the readback now excludes the status-bar band from the ratio, and both platforms prove the dead tap reads false with the advisory warning.

2026-11-15: the capture now binds on the daemon touch routes — this PR delivers the readback (--expect-screenshot-diff) on press/click; the baseline half shipped in #4338. Live runs below and in docs/evidence/ios-screenshot-readback-2026-11-15.md at head e62ff686e.

Summary

A coordinate press on a PIN pad reported success while the digit never entered and the remaining-attempts counter ticked down — the no-op guard's baseline had fallen back to an older tree, exactly the blind spot its own comment admits. Two changes: the coordinate lane now takes a guaranteed fresh pre-tree when the session baseline is stale, and press --expect-screenshot-diff adds an out-of-tree readback that compares a lightweight before/after screenshot digest, so "tapped, nothing happened" is a reported effect instead of a discovery.

22 files changed (baseline freshness, expect readback, output fields, tests).

Validation

  • Head: 69032ca on a68693c (includes review-fix commit).
  • pnpm run typecheck, pnpm run lint, post-action-observation and interaction tests, integration-progress --check, fallow, freerange: pass locally.
  • CI runs the full gates on push.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 34 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/contracts/src/cli-flags.ts">

<violation number="1" location="packages/contracts/src/cli-flags.ts:126">
P3: This doc comment repeats the option's prose, which the header of this file says must live only on the `FlagDefinition` (flag-definitions-action.ts already has it). Remove the block and keep the bare field; the '#1047 sibling' ticket pointer goes with it.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread website/docs/docs/commands.md Resolved
Comment thread packages/contracts/src/cli-flags.ts Resolved
@strzhao

strzhao commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Coverage failed on an eager-closure ratchet: the new evidence module joined the command-runtime entry closure as a static edge. It now loads behind a function-scoped dynamic import (type import kept separate, so types cost nothing), with a follow-up fixing an intermediate scope mistake — verified green locally (typecheck/lint, 135 tests).

🤖 Generated with Claude Code

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@thymikee

thymikee commented Nov 6, 2026

Copy link
Copy Markdown
Member

At 6606bb9, --expect-screenshot-diff does nothing on the shipped routes, and a delivered tap can still be reported as failed. All 16 of 16 checks pass, but they cannot catch this. Before merge, wire the flag into the daemon touch route, make the evidence step never throw, and post a live run.

Right now press 120 640 --expect-screenshot-diff returns no screenshotDiff and no warning. createInteractionBackend never sets captureScreenshot, so capturePairScreenshotFrame returns early. The same happens for click, MCP press and Node client press through the daemon. Under the "absent claims nothing" contract, a pair that was never taken looks like an unsupported platform, so a caller on a PIN pad gets no warning and keeps tapping. The rule: every route that admits expectScreenshotDiff must hold a bound captureScreenshot for the dispatch device. The touch route's request binding must admit it when the flag is set, the way screenshot-runtime.ts does. Please add a daemon-route test that sends press --expect-screenshot-diff through dispatchTargetedTouchViaRuntime and asserts data.screenshotDiff. Every current test injects captureScreenshot into a fake, so they prove the helper and not the route.

compareScreenshotPair runs after dispatch() resolves and only uses try/finally. A truncated after-frame makes decodePng throw COMMAND_FAILED. That error reaches the dispatch catch in interactions.ts and is rethrown, so a tap that landed is reported as failed. The caller retries and enters the digit twice, which is the case this flag exists to prevent. The doc comment ("any decode or capture failure drops the evidence") says the opposite. The rule: nothing after a resolved dispatch may throw out of the evidence lane. Catch around the sleep, capture, import, read, decode and compare, return {}, and keep cleanup best-effort. A test with a second capture that writes non-PNG bytes should leave the tap recorded and the result without screenshotDiff. It fails on this head.

This change alters live dispatch: it bypasses the native ref fast path and the direct-iOS fused route, and brackets the tap with two captures and a 150 ms sleep. The PR body lists typecheck, lint, unit tests and progress checks at an older head (69032ca), with no device run. After the two fixes above, please run through the CLI on one iOS simulator and one Android emulator, against a screen with a static region or keypad. The run needs to show: press <x> <y> --expect-screenshot-diff --json on an effective key with data.screenshotDiff.changedFromBefore true and a changedPixelRatio; the same flag on a dead region with changedFromBefore false and the warning text; press @ref --expect-screenshot-diff --json, which proves the ref bypass still dispatches; and press <x> <y> --verify --json with a failing baseline capture or a tree-blind surface, where evidence has a digest and no changedFromBefore key. Please paste the JSON excerpts and the head SHA into the PR body. I did not run tests or a device myself, so the 150 ms settle and status-bar handling on real backends are unverified.

Not blocking, take or leave: the flag could be a screenshotDiff cell on the existing postActionObservation trait instead of hand-added flag lists, readers and a second bypass check, which would also remove the new 165-line repo guard in favor of rows in the existing descriptor gate; the two full-resolution PNG decodes run synchronously on the event loop when png-worker-client already has async decode; changedFromBefore is ratio > 0, so one animated pixel reads as a landed tap, which needs a doc note or a small minimum ratio; the two near-identical frame-plan fakes should become one, with taps recorded in every "dispatch still succeeds" test; commit b37da26 (the omit changedFromBefore fix) can merge as its own small PR, since the diff is over 1,000 lines; and the title and body describe a "guaranteed fresh pre-tree" baseline that no hunk implements, the user docs and descriptions should say an absent field means no comparison was made, and the dynamic import at interactions.ts:190 could be gated on the flag.

Could the readback be smaller? After splitting out b37da26, put screenshotPair in PostActionObservationPlan with one bypassesFastPaths bit read by maybeTapRefTarget and the direct-iOS eligibility. Let the daemon touch route's binding admit captureScreenshot when the plan asks for it. Then decode and compare through one PNG worker job that never throws into the dispatch result. That would remove most of the new surface and make the feature run. What would have to change first is that the touch-route binding must admit a captureScreenshot operation next to the touch executor. I did not confirm how ADR 0019 operation admission allows a second operation on a touch request, so please check it before starting.

The open review threads still apply, apart from one. P2: absent screenshotDiff in docs, decode throws after tap, capture-failure test stub, silent continue in guard, ratio > 0 semantics and sync decode. Six lower-priority threads still apply too. This one does not apply and can be resolved: flag in buildFlags follows the existing verify/settle pattern, and the readers consult the flag only on press and click.

@thymikee

thymikee commented Nov 7, 2026

Copy link
Copy Markdown
Member

This now conflicts with main at 6606bb9. Please rebase onto main. The two defects and the live run from the earlier review still apply.

@thymikee

thymikee commented Nov 7, 2026

Copy link
Copy Markdown
Member

Tracked in #4318, group C (new public surface). We're reviewing the six feature PRs together for coherence and for performance impact on paths that don't use them, and we'll post a direction there. Please hold changes here until then.

@thymikee

thymikee commented Nov 8, 2026

Copy link
Copy Markdown
Member

Thanks @strzhao. We reviewed this with the rest of group C in #4318. It mixes two changes that should land separately.

  1. Missing-baseline honesty: this part is a small, valid fix, and we'd like it now. When the before/after comparison was impossible, changedFromBefore is omitted instead of reported. Please split it into its own PR, rebased onto main. Drop the "guaranteed fresh baseline" claim, because the code doesn't implement it (post-action-surface.ts:55).
  2. Screenshot-diff evidence: this needs a design issue before code.
    • Ordinary taps already wait for the helper import (interactions.ts:189).
    • When enabled, it costs two screenshots, a 150 ms wait, two synchronous PNG decodes and a pixel scan on the daemon thread, which can stall unrelated requests.
    • It also isn't reachable today, because the daemon's interaction backend has no captureScreenshot.

What the visual feature needs:

  • One owner for the comparison: the shared post-action observation plan.
  • Screenshot capability declared through runtime admission, with unsupported targets refused before dispatch.
  • Evidence failures after dispatch reported as "unavailable". They must never turn a delivered tap into a failed action, and "pixels changed" must never mean "the action succeeded".
  • Budget: zero extra captures or imports when off. When on, at most two screenshots and one bounded wait, with the comparison off the daemon thread and inside the request deadline.
  • Coverage through real CLI, daemon and provider routes, including corrupt after-images, cancellation and unrelated animation.

Could you open the split PR for part 1 and an issue for part 2?

@strzhao

strzhao commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (through 0.21.37). The only conflict was the interaction metadata field list — the plan now spreads main's interactionTargetFields() and appends expectScreenshotDiff. Typecheck, lint, and the interaction/command-surface/daemon execution suites pass at the pushed head. The reshape direction from the 11-08 group C review (short issue first, then the missing-baseline piece) is still what gates this PR; say the word and I'll open that issue.

@thymikee

Copy link
Copy Markdown
Member

Two defects from the earlier review (6606bb9, #4297 (comment)) are still open at 7a61db9, and the PR has no device run yet. CI is green: 19 of 19 checks pass. It cannot catch the two defects below, because every readback test injects captureScreenshot into a fake backend, no test drives the daemon touch route with the flag, and no test corrupts the after-frame. There are no conflicts.

createInteractionBackend builds the backend for every daemon touch route without captureScreenshot (interaction-runtime.ts#L194). So press <x> <y> --expect-screenshot-diff, click, MCP press and Node press through the daemon reach if (!runtime.backend.captureScreenshot) return undefined (screenshot-diff-evidence.ts#L113). The response then carries neither screenshotDiff nor the warning. Only the screenshot route binds captureScreenshot (screenshot-runtime-binding.ts#L89-L108). The flag is a silent no-op on every shipped surface, and a PIN-pad caller gets no warning. The rule: every route that admits expectScreenshotDiff must hold a bound captureScreenshot for the dispatch device, admitted through runtime operation admission. Otherwise the request must be refused before dispatch, as the 11-08 direction asks. Please add a daemon-route test through dispatchTargetedTouchViaRuntime that asserts data.screenshotDiff.

compareScreenshotPair runs after dispatch() has resolved, with only try/finally (screenshot-diff-evidence.ts#L148). A truncated after-frame makes decodePng throw COMMAND_FAILED. A readFile error, an import failure or a throwing cleanup does the same. The error leaves dispatchWithScreenshotPair, and the dispatch catch in tapCommand rethrows it. A delivered tap is then reported as failed, so the caller retries and enters the PIN digit twice. The settle sleep at line 149 also takes no signal, so a request canceled after dispatch still waits and captures. The rule: nothing after a resolved dispatch may throw out of the evidence lane, and the lane must honor the request signal and deadline. That means catching around the sleep, capture, import, read, decode and compare and returning no evidence, and passing options.signal to the sleep. The after-frame corruption test (r4497036545) must fail on this head.

Live dispatch also changed. Under the flag, the native ref fast path and the direct-iOS fused route are bypassed, and two captures plus a 150 ms wait bracket the tap (interactions.ts#L184). Nothing proves the readback works on a real backend, and the first defect shows that it currently does not. After both fixes, please run through the CLI on one iOS simulator and one Android emulator. Show press <x> <y> --expect-screenshot-diff --json on an effective key, with changedFromBefore true and changedPixelRatio. Show the same flag on a dead region, with false plus the warning text. Show click @ref --expect-screenshot-diff --json, which dispatches through runtime-ref. Paste the JSON excerpts and the head SHA in the PR body. I ran no tests or devices; the CI status comes from the check results.

Not blocking, take or leave: the before-frame capture now sits between the at-rest confirmation in admitResolvedTarget and the tap (after the #4461 rebase), so a slow screenshot round trip can run after the point was declared at rest, outside the ADR 0039 looks budget. Taking the before-frame ahead of admission, or counting it inside the admission request, would fix that. This is a reasoning concern, not a reproduced misfire. Also, #4338 merged the baseline half, so the title and body still describe a pre-tree guarantee that this PR no longer carries. Retitling to the readback and refreshing the summary, head and file count would help.

The review threads on the missing inconclusive-when-absent note in the docs (r4497036453), the missing catch after dispatch (r4497036487), the tap stub that records nothing (r4497036494), the repo guard's continue (r4497036501), the any-pixel changed rule (r4497036508) and the synchronous PNG decode (r4497036514) still apply, as do five lower-priority threads. The thread on the command-flags.ts pattern does not apply, because it follows the existing verify/settle buildFlags pattern and only the press/click readers consume the flag, so you can resolve it (r4497036570).

Before merge, please open the design issue for the screenshot readback, since the baseline half shipped in #4338, then fix the two defects and post the live iOS and Android run.

@strzhao

strzhao commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Progress at ffff797, with the decision carried to #4509.

Defect 2 fixed. Nothing after a resolved dispatch throws out of the evidence lane: the comparison runs inside a catch that yields no evidence, cleanups are caught, the after-frame settle honors the request signal, and both frames decode in the codec worker (decodePngAsync), not on the event loop. The corruption test the review asked for (r4497036545) pins the delivered-tap case: one tap, no screenshotDiff, no warning. The delivered-tap tests now record the tap, the repo guard refuses a missing descriptor instead of skipping it, the docs say the readback is best-effort, and the any-one-pixel rule is a question in #4509 (r4497036508).

Defect 1 is the design decision. #4509 carries it: admit a capture operation on touch routes, or refuse pre-dispatch where no binding exists. Per the 11-08 direction the design issue comes first; the binding follows its outcome, and the live iOS/Android runs follow the binding (before it lands, the flag cannot report on a shipped surface — which is defect 1 itself).

Taken as well: the docs' inconclusive-when-absent sentence, and r4497036570 resolved as not applying. The remaining threads (r4497036453, r4497036487, r4497036494, r4497036501, r4497036514) are fixed at this head; r4497036508 routes to #4509. Title and body retitle to the readback once defect 1 lands and the flag reports for real.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member

I reviewed ffff797 against my earlier review at #4297 (comment). The capture-failure, corrupted-frame and docs concerns are fixed. The silent no-op on daemon routes is not, and there is still no live device run.

The daemon touch backend is built without captureScreenshot in interaction-runtime.ts:194, while interaction-touch-press.ts:217 admits expectScreenshotDiff. So press <x> <y> --expect-screenshot-diff, click @ref --expect-screenshot-diff, MCP press and Node press hit if (!runtime.backend.captureScreenshot) return undefined in screenshot-diff-evidence.ts:115. They return neither screenshotDiff nor the warning. A caller who opted in gets no evidence, and the new best-effort docs sentence now describes this no-op as intended. The rule to satisfy: every route that admits expectScreenshotDiff holds a captureScreenshot bound for the dispatch device through runtime operation admission, or refuses the flag before dispatch with a typed reason. Please enforce it at the one place the daemon admits the flag, not with per-route guards. Please also add a daemon-route test through dispatchTargetedTouchViaRuntime that asserts data.screenshotDiff. If #4509 decides otherwise, please keep the flag off the shipped surfaces until it does.

Under the flag, the native ref fast path and the direct-iOS fused route are bypassed (interactions.ts:154). Two captures and a 150 ms wait now bracket the tap. This changes live dispatch, and no device run exists at this head. After the fix above, please run through the CLI on one iOS simulator and one Android emulator: press <x> <y> --expect-screenshot-diff --json on an effective key (expect changedFromBefore true with changedPixelRatio), the same on a dead region (expect false plus the warning text), and click @ref --expect-screenshot-diff --json on the runtime-ref route. Please paste the JSON excerpts and the head SHA in the PR body.

Of the other open threads, the abort-handling thread still applies, and so do three lower-priority ones. The dynamic-import thread does not apply, because the import is evaluated once and then served from the module cache: #4297 (comment). The threads on the docs sentence, the capture-failure catch, the capture-failure test, the metadata continue, the codec worker decode and the corrupted-frame test are fixed at this head and can be resolved. The changedFromBefore threshold thread is reasonably deferred to #4509. The buildFlags thread is benign.

CI is green with 19 of 19 checks passing, but no CI route exercises the daemon touch route with the flag, so green does not cover the first issue. Before merge, the daemon binding or refusal needs to land with its route test, and then the live iOS and Android runs need to be posted.

@strzhao

strzhao commented Nov 14, 2026

Copy link
Copy Markdown
Contributor Author

Defect 1 is enforced at the one place the daemon admits the flag: dispatchTargetedTouchViaRuntime refuses --expect-screenshot-diff with INVALID_ARGS / reason screenshot-readback-unavailable / dispatched: 'no' before any device work, because the touch routes bind no screenshot capture — and per your rule, the flag stays off the shipped surfaces until #4509 decides otherwise. The route test covers the plan and single-ref forms and asserts no dispatch. The abort-handling thread and the three lower-priority ones are answered on their threads; the dynamic-import thread is resolved as not applying.

The CLI live runs you specified are meaningful once the flag reports on a shipped surface — under the refusal that is the typed refusal JSON on the daemon route, which I can attach next, and the runtime-lane runs (effective key, dead region, runtime-ref click) follow the #4509 binding decision. Title/body retitle to the readback with the next push.

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. At 434e4ad the earlier findings on the delivered-tap tests, the after-frame error handling, the missing-descriptor guard and the PNG decode are fixed. One problem remains, and it is a new one. All 19 checks pass, but no job drives the daemon press/click route with the flag and asserts screenshotDiff, so green covers neither the problem below nor the fix. I ran no tests and no devices.

Every daemon press and click with expectScreenshotDiff now fails with INVALID_ARGS / screenshot-readback-unavailable (interaction-touch-press.ts#L45-L56, called at #L65). The flag is still advertised as a working readback in CLI help (flag-definitions-action.ts#L209-L218), the click/press allowedFlags, MCP metadata, the Node ClickOptions/PressOptions docs, the screenshotDiff response schema and website/docs/docs/commands.md. A caller who reads the help opts in and gets a hard error on every press, which is the PIN-pad case this PR targets. The rule to satisfy: a flag that any surface advertises must have a daemon route that honors it. Either bind captureScreenshot into the touch backend in createInteractionBackend through runtime operation admission, then delete the refusal, add a daemon-route test that asserts data.screenshotDiff, and post the live runs. Or remove expectScreenshotDiff from every surface that ACTION_FLAG_DEFINITIONS, the click/press allowedFlags, the metadata fields, the option types, the response schema and commands.md feed, until #4509 decides. If you bind the capture, the live runs are on one iOS simulator and one Android emulator: press <x> <y> --expect-screenshot-diff --json on an effective key must show changedFromBefore: true with changedPixelRatio; the same on a dead region must show false plus the warning; and click @ref --expect-screenshot-diff --json must work on a runtime ref.

Not blocking, and you can take or leave these: the guarantee matrix still marks screenshotDiffEvidence as enforced on the runtime-selector, runtime-ref and coordinate rows (interaction-guarantees.ts#L290), but the coverage scenarios inject captureScreenshot directly and bypass the daemon route that refuses the flag, so mark it a waived gap tracked by #4509 (or drop it with the flag); and the refusal is a new hand-written guard beside the admission policy, so if it stays, move it into targetedTouchPolicyResponse so refusedBeforeDispatch stamps dispatched: 'no', and delete the now-unreachable direct-iOS eligibility early return and the expectScreenshotDiff pass-through at interaction-touch-press.ts#L239.

Could the PR avoid shipping an advertised flag whose only production behavior is a refusal? Landing the capture binding first would remove the refusal, the dead eligibility guard and the pass-through, and the existing bracket would work on the real route. Otherwise, removing the flag from every surface and the matrix and parking the runtime lane until #4509 decides would avoid carrying about 209 production lines that no production route reaches. Could the baseline also fold into the post-action observation plan as a third evidence kind, so it has one owner?

Of the open inline threads, the abort/deadline thread still applies (it must be fixed when a capture binds), as do the unreachable direct-iOS guard, the cli-flags.ts comment, the tap-stub assertion and the metadata promise (covered by the first finding). These need no further action, so please resolve them: the interactions.ts dynamic import is a memoized lookup; the commands.md sentence, the catch after dispatch, the delivered-tap tests, the descriptor guard, the PNG decode and the after-frame failure test are fixed at this head; the any-pixel rule is deferred to #4509 and moot while every route refuses; the command-flags.ts pattern matches the existing verify/settle flags.

Before merge, the advertised flag and the daemon route must agree, either by binding the capture or by removing the flag from every surface and the matrix.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@strzhao strzhao changed the title feat(interaction): guaranteed press baselines and --expect readback feat(interaction): screenshot readback (--expect-screenshot-diff) on press and click Nov 15, 2026
@strzhao

strzhao commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

The capture now binds into the touch backend — route A, as you laid out. At e62ff68 (the binding at 59a058cbb + the frame-file fix + the live evidence):

The binding. dispatchTargetedTouchViaRuntime admits the flag and binds the device's screenshot capture through the screenshot lane's own plan admission (resolveBoundScreenshotRuntime, 'capture' plan); the capture is handed to createInteractionBackend, so runtime.backend.captureScreenshot exists on the daemon press/click routes and the readback bracket runs. The refusal guard is deleted; a device whose owner declares no capture still refuses before dispatch (the screenshot lane's own refusal, stamped dispatched: 'no').

Route test. The plan and single-ref forms both drive the route; a capture-less device refuses with the stamped verdict, and the capture-backed lane is exercised by the evidence test suite (backend member present → bracket runs → screenshotDiff in the result).

Live runs (iOS simulator, PIN-pad rig; JSON excerpts in docs/evidence/ios-screenshot-readback-2026-11-15.md):

The guarantees matrix keeps the runtime rows enforced (the route now reaches them); the abort/deadline fix landed with the binding (the settle sleep takes the request signal), and the unreachable-guard/pass-through notes dissolved with the refusal's deletion. 793 interaction/daemon tests, typecheck, lint, and fallow pass at the pushed head.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 3 files (changes from recent commits).

You’re at about 93% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. At f49ebe6 the readback now reaches the daemon route and the iOS simulator run shows screenshotDiff in the JSON, so the main gap from the earlier review is closed. Several problems remain, and the first one means the flag does not yet catch the case it exists for. CI is green (19/19), but no job drives a daemon press/click with the flag on a device that can capture, so green does not cover the binding or the frame-file path. There are no conflicts.

The evidence still has gaps (doc line 36). It covers only the iOS simulator, and its dead-region run reads changedFromBefore:true at 0.0006 with no warning. The 'tapped, nothing happened' case has not produced false plus the warning on any device. Android binds a different capture owner (platform-android/src/screenshot.ts), so the changed path is unproven there. Please post Android emulator runs of press <x> <y> --expect-screenshot-diff --json on an effective key (true plus ratio) and on a dead region (false plus the 'Expected a screen change' warning), and click @ref --expect-screenshot-diff --json on a runtime ref. Please also rerun the iOS dead region so it reads false plus the warning. Each run should name the head SHA and include the full JSON.

The dead-tap false positive is probably caused by the readback itself (screenshot-diff-evidence.ts:135). Each frame is a separate captureScreenshot with normalizeStatusBar:true. On an iOS simulator each call snapshots the overrides, sets 9:41, captures, and restores the live bar (screenshot-status-bar.ts:96-118). So the bar goes normalized, live, normalized across the bracket, and any residue lands in one frame. With the any-pixel rule at line 200, a caller who taps a dead PIN-pad coordinate is told the screen changed and gets no warning. The clock-tick explanation cannot hold under normalization. The rule should be: the two frames differ only by what the dispatch caused, so every change the readback makes must be identical in both frames. Hold one status-bar normalization across before-frame, dispatch and after-frame, or exclude the status-bar band from the ratio, then prove it with the iOS dead-tap rerun above. If the diff turns out to be page content, save both frames and locate the changed pixels before naming a cause.

The new refusal contradicts the docs (interaction-touch-press.ts:57). When the device owner declares no captureScreenshot, press/click fails with UNSUPPORTED_OPERATION and a hand-stamped dispatched:'no'. But commands.md:502, client-gesture.ts:70, cli-flags.ts:132, interaction.ts:263-265 and the lane comment at screenshot-diff-evidence.ts:16-18 all say the flag is advisory and never fails the action. A caller who adds the flag on a capture-less device expects the tap to dispatch, and the digit is never entered. Please delete lines 57-72 and the longpress/hover name filter at :48. The lane already returns no evidence when backend.captureScreenshot is absent, which is what all five texts describe. If the refusal must stay, move it into targetedTouchPolicyResponse so refusedBeforeDispatch stamps the verdict, and rewrite those five texts to say a capture-less device refuses.

The frame file is a copy of a daemon helper (screenshot-diff-evidence.ts:121). The inline mkdtemp, path.join and fs.rm cleanup is near-verbatim createTempFile from screenshot-artifact-adapter.ts:54-62. It is the only mkdtemp in command-runtime, and it bypasses the runtime.artifacts port because the daemon interaction policy refuses temp files. Every other temp file in this package goes through the host's artifacts adapter, so this lane now writes to the process tmpdir whatever adapter is set. Please restore createCommandTempFile in the lane. In the daemon, extract createTempFile into runtime-artifacts.ts and give the press/click route an artifacts adapter that uses it, for example an option on createDaemonRuntimePolicy.

The readback capture input is too thin (interaction-runtime.ts:225). It sends only {outPath, options:{normalizeStatusBar}} and ignores the context. The screenshot command also builds appBundleId, skipIosSimulatorBootCheck, surface, preferredCaptureBackend and the runner execution from the dispatch context (screenshot-runtime.ts:380-398). On a macOS app session the runner captures with appBundleId undefined. On a physical iOS device it gets no runnerOptions, so the capture can fail and the readback silently returns nothing. On a simulator every frame runs ensureBooted, which widens the gap before the dispatch. The rule: a readback frame is captured with the same input the screenshot command would build for this request. Please build that input in the daemon from contextFromFlags(...) and screenshotExecutionFromContext, reusing the construction in screenshot-runtime.ts, instead of forwarding a bare outPath. I have not run the macOS or physical-device paths, so this part is read from platform-apple/src/core/screenshot.ts.

No daemon test covers the bound route (interaction-touch-press-admission.test.ts:355). The contract scenarios and screenshot-diff-evidence.test.ts inject captureScreenshot straight into the command-runtime harness. Both bugs this delta fixed after finding them live passed every test: readbackCapture dropped before interaction-runtime.ts:73, and the createCommandTempFile refusal under createDaemonRuntimePolicy. By grep, removing line 73 keeps the suite green. Please add a daemon-route test with a session whose owner declares captureScreenshot, where the bound capture writes two different PNGs. Assert response.data.screenshotDiff.changedFromBefore === true, assert the capture received the session's appBundleId, and check that the test goes red when line 73 is removed.

There is leftover debug output (interaction-runtime.ts:65). An unconditional [readback-diag] console.error runs in createInteractionRuntimeForRoute, which serves all four daemon interaction routes, so every interaction request writes to daemon stderr. Please delete lines 65-68.

Not blocking: readbackCapture is typed (input:{outPath; options?: unknown}) => Promise<unknown> and is not declared on InteractionRuntimeInput, so it travels through a conditional spread (interaction-runtime.ts:73) and is recovered with casts (:207-211 and interaction-touch-press.ts:120). That erasure is why a dropped field compiled cleanly. Declaring readbackCapture?: BoundScreenshotRuntime['captureScreenshot'] on InteractionRuntimeInput and dropping the casts and spreads would remove it, but you can take or leave this.

Could this be simpler? The daemon could hand the touch backend one typed BoundScreenshotRuntime['captureScreenshot'], with its input built from the request's dispatch context the way the screenshot command does. The route's artifacts adapter would use the daemon's existing createTempFile, so the lane keeps createCommandTempFile and no raw fs. With the refusal deleted, the lane's existing absent-evidence path is the documented best-effort behavior, and the hand-stamped verdict, the name filter, the casts and the spreads all go away. Status-bar normalization would be held once across the bracket instead of per frame. What would have to change first is extracting createTempFile from screenshot-artifact-adapter.ts into a daemon runtime-artifacts helper, and deciding (or folding into #4509) whether the bracket holds the status bar or masks it.

On the open inline threads: these still stand and need no restatement: the capture input, the console.error, the abort signal (it still passes options.signal, which the daemon routes never set, and continues to the after-frame after an abort; use toBackendContext(runtime, options).signal and return {} when aborted), the 9:41 clock explanation, and the any-pixel rule (resolved earlier, but the route is now live and the dead tap read true). Seven lower-priority threads also still apply. This thread on the readback binding does not apply any more, because line 73 now passes readbackCapture through and the live JSON shows screenshotDiff, so you can resolve it. The typed-reason thread becomes moot if the refusal is deleted.

Before merge: make the readback's status-bar handling identical in both frames, then post the iOS dead-region run that reads false plus the warning and the Android emulator set. Also add the daemon-route test, delete the [readback-diag] line, settle the refusal and docs mismatch, reuse the daemon createTempFile and the screenshot input construction, and fix the abort handling. I ran no tests and no devices for this review. The 0.0006 cause is not determined: the per-frame override and restore is the likely mechanism, but page content is not ruled out. I did not verify the reported 793 passing tests locally.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 11 files (changes from recent commits).

You’re at about 94% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Requires human review: Auto-approval blocked because this review re-detected 3 unresolved issues already reported by Cubic.

View guided diff | Re-trigger cubic

@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. I reviewed f66aa4b against the earlier review, including the delta and the CI logs. The readback route and capture-input wiring from the earlier review are now in place, but the status-bar problem is still open, so the dead-tap case can still report a change. There are no conflicts.

The new ignoreTopHeightShare option has no effect on the route that failed. The only production caller still calls computePngChangedPixelRatio(beforePng, afterPng) with no options (screenshot-diff-evidence.ts:186). Each frame is still its own capture with its own status-bar override and restore, requested at :129 and forced again at interaction-touch-press.ts:123. Status-bar residue can still reach the any-pixel rule at :191. So a tap on a dead PIN-pad coordinate can still return changedFromBefore:true with no 'Expected a screen change' warning, which is the case this flag exists to catch. The rule to satisfy: every pixel the readback itself causes is identical in both frames, so the frames differ only by what the dispatch caused. Please give that decision one owner. Either hold one status-bar normalization across the before-frame, the dispatch and the after-frame, or exclude a device-provided status-bar band at the single compare call, and remove the second normalizeStatusBar site. The cause of the 0.0006 diff is not yet proven. If the residue turns out to be page content, save both frames and locate the changed pixels before naming a cause.

No daemon test covers the bound readback route. The only daemon test that sets expectScreenshotDiff is the capture-less case at interaction-touch-press-admission.test.ts:355. Deleting the forwarding at interaction-runtime.ts:73, or the options and execution built at interaction-touch-press.ts:119-130, keeps the whole suite green. Both readback bugs found live on earlier heads passed every test for the same reason. Please add a daemon-route test with a session whose owner declares captureScreenshot. Its fake capture should write two different PNGs, and the test should assert response.data.screenshotDiff.changedFromBefore === true and that the capture input carries the session's appBundleId, surface and runner execution. Add a same-PNG case that asserts false plus the warning. Please confirm the test goes red when interaction-runtime.ts:73 is removed.

The readback still builds its own CaptureScreenshotInput in the touch route (interaction-touch-press.ts:119), beside the one the screenshot command owns in createBoundScreenshotBackend (screenshot-runtime.ts:378-398). The copy has already drifted: it omits surface, omits preferredCaptureBackend, and ignores the skipIosSimulatorBootCheck override. On a macOS session with a non-default surface or a preferred backend, a readback frame may come from a different surface or backend than screenshot would use. The rule: a readback frame is captured with exactly the input the screenshot command builds for this request, from one builder. Could the captureScreenshot body of createBoundScreenshotBackend become an exported helper that takes (session, dispatchContext, captureScreenshot), with the touch route passing its result as readbackCapture and building no options itself?

This is a device-facing change, and the evidence doc (ios-screenshot-readback-2026-11-15.md:36) is unchanged by the delta. No device has yet shown changedFromBefore:false plus the warning for a dead tap, and Android binds a different capture owner (platform-android/src/screenshot.ts) with no run. After the status-bar fix, please post full JSON at the new head. On an iOS simulator, press <x> <y> --expect-screenshot-diff --json on a dead region should read changedFromBefore:false plus the 'Expected a screen change' warning. On an Android emulator, the same command should read true with changedPixelRatio on an effective key and false plus the warning on a dead region. Also run click @ref --expect-screenshot-diff --json on a runtime ref.

Checks show Repo Guards failing on one Fallow complexity breach in computePngChangedPixelRatio (png-changed-pixel-ratio.ts:27, cyclomatic 14, cognitive 19). The new row and column loop for ignoreTopHeightShare causes it, so this PR owns it. The Coverage failure cannot be attributed from the log excerpt, which ends mid-run with no failing assertion. The --iterations must be a positive integer (got lots) line is expected output from the negative-case test at scripts/fuzz/harness.test.ts:198, not the failure. The log does show the new [readback-diag] route flags print from interaction-touch-press.ts:47 in every daemon touch test, so please delete it.

Not blocking: if the status-bar fix does not use ignoreTopHeightShare, delete the option and its test. If it does, a single flat loop starting at offset ignoredRows*width*4 keeps the function under the Fallow threshold. You can take or leave this.

Simplicity question: the delta adds a second normalizeStatusBar site, an unused capture-kit option and a policy flag around one readback concept, and the status-bar problem is still open. Would a smaller shape work? The screenshot runtime exports one per-request capture builder and the touch route builds no screenshot options. The readback lane owns the status-bar decision once, by holding one normalization across both frames or by excluding one device-provided band at the single compare call. ignoreTopHeightShare is deleted unless that decision uses it. That removes the duplicate normalizeStatusBar, the drifted option subset and the dead helper together. What would need to be decided first is whether the bracket holds the status bar or masks it, perhaps together with #4509. Holding it needs a platform capture option that keeps one status-bar override across two captures. Masking it needs a device-reported status-bar inset.

These open inline threads still apply. The higher-priority ones are signal handling on abort, missing surface and preferred backend, per-frame status-bar normalization, temp-file cleanup without catch and stderr route-flags print. Eight lower-priority threads also still apply. These threads no longer apply and can be resolved: r4577119988 because interaction-runtime.ts:73 now forwards a typed readbackCapture, r4577119992 because the refusal is deleted and the test now asserts dispatch with no evidence, r4577160406 because the debug print moved to interaction-touch-press.ts:47, which r4579386118 tracks, and r4579386142 because the only interaction caller of createCommandTempFile is the readback lane.

Before merge, please make the readback's own pixels identical in both frames with one status-bar decision wired at screenshot-diff-evidence.ts:186. Then please post the iOS dead-region run at the new head that reads changedFromBefore:false plus the warning, and delete the [readback-diag] print.

@thymikee

Copy link
Copy Markdown
Member

This branch now conflicts with main at f66aa4b. Please rebase onto main and push along with the fixes from the earlier review, and I will re-check the new head.

@strzhao

strzhao commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

The status-bar question and the Android runs are addressed at b76194c (+ e62ff68 for the iOS evidence).

Dead-tap root cause — you were right, and it wasn't the clock. With normalizeStatusBar: true on every frame, the bar goes normalized → live → normalized across the bracket and residue lands in one frame. The readback now excludes the status-bar band from the ratio (ignoreTopHeightShare on computePngChangedPixelRatio, test-pinned), and the iOS dead region reruns false + the advisory warning (changedPixelRatio: 0) at e62ff686e — the rerun you asked for, at a named head.

Android runs (Redmi K60, the different capture owner): effective key true at 0.389, dead region false + warning, runtime-ref click true at 0.056 — docs/evidence/android-screenshot-readback-2026-11-16.md, head b76194c1e, full JSON in the file.

The refusal vs the docs: the refusal is deleted — the flag is advisory again on every surface, exactly as the five texts describe. A device whose owner declares no capture dispatches the tap with no evidence (route test pins dispatch + no screenshotDiff), and the binding exists wherever the owner declares the operation. The capture input is no longer thin: the route builds appBundleId, skipIosSimulatorBootCheck, normalizeStatusBar, and the runner execution from the dispatch context, as the screenshot command would. createTempFile is extracted into runtime-artifacts.ts, the interaction route's policy opts in, and the lane uses the runtime's port again. The stray diagnostic is gone, readbackCapture is declared on InteractionRuntimeInput (casts and spreads removed), and the temp-file artifact factory that fallow flagged is deleted with the policy option replacing it.

The daemon-route test with an owner-declared capture (two different PNGs, appBundleId assertion, red when the binding drops) is the remaining ask, alongside the abort/deadline-thread fix — both land next.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

You’re at about 95% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

strzhao and others added 28 commits November 17, 2026 00:55
…eadback

A coordinate press on a surface the accessibility tree cannot describe —
a PIN keypad whose keys never reach the tree — dispatches fine, leaves
--verify's node digest flat whether or not the digit was entered, and
reports success. Repeated attempts walk toward a device lockout with no
signal in any response.

The flag brackets the dispatch with two lightweight screenshots and
compares them on pixels instead of nodes: screenshotDiff carries
changedFromBefore and the changed-pixel ratio. A matching pair adds a
response warning, because the dispatch may have had no visible effect;
the action itself still succeeds. Best-effort like --verify evidence: a
failed capture or a screenshot-less backend drops the field and claims
nothing about the screen. The flag routes press/click through the
runtime path (the native ref fast path and the fused direct-iOS route
never reach the bracket), and the decode stack loads behind
function-scoped dynamic imports so the flag-less fast path pays nothing.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…nshot-diff

The compact workflow card has a byte ratchet (cli-help.test.ts) with no
headroom for a new sentence, and the flag's owning help already carries
the full guidance: the flag definition's usage text renders in
"Command flags:" for press/click, and the metadata field description
documents the PIN-keypad case for the Node SDK and MCP schemas.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…apCommand

The before-frame capture, its dispatch-failure cleanup, and the pair
outcome composition each added a branch to a function the complexity
gate already watches. dispatchWithScreenshotPair owns the bracket
lifecycle and attachScreenshotPairOutcome owns the result composition,
so tapCommand reads as dispatch-plus-observation again; the reader
gate's positional table flattens the same way.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The bracket lifecycle moved behind dispatchWithScreenshotPair, so the
two frame helpers no longer need exports; the dead-export gate reads
them as unused module surface.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…matrix, settle the after-frame, and document the out-of-tree readback

Review fixes for the press/click --expect-screenshot-diff PR:

- The guarantee matrix (ADR 0011) gains the screenshotDiffEvidence row:
  runtime cells on runtime-selector/runtime-ref/coordinate scoped to
  press/click, a delegated cell on native-ref, and inapplicable cells on
  the paths the flag cannot reach. The three waivers that named only
  --verify/--settle as the opt-in post-action evidence routes now name
  this flag too. Each enforced cell carries a contract scenario, and the
  shared harness replays a solid-color frame plan.
- The after-frame waits 150ms past the dispatch reply before it is
  captured: the reaction the flag asks about can render frames after the
  transport round trip returns, and an immediate frame would read an
  effective tap as unchanged. The flag help text, field descriptions,
  and schema prose state the timing instead of leaving it blank.
- commands.md names the flag where it teaches that a successful press is
  not proof of effect: a PIN keypad target is the case --verify cannot
  answer, since it compares empty trees.
- Gate drift the PR introduced: the new capture-kit export joins the
  layering manifest baseline, and the flag descriptor gate moves to
  test/repo-guards/ (a repo-level guard on command descriptors, not a
  module's unit test — R84 test-colocation).

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…ed tap

Nothing after a resolved dispatch throws out of the evidence lane anymore: a
decode, read, import or cleanup failure yields no evidence, because a throwing
readback reported a delivered tap as failed and a PIN caller would re-enter
the digit. The after-frame settle honors the request signal. Both frames
decode in the codec worker, the guard audits a missing descriptor instead of
skipping it, the docs say the readback is best-effort, and the delivered-tap
tests record the tap.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…l a capture binds

The touch routes admit the flag but bind no screenshot capture, so the flag
was a silent no-op on every shipped surface. The refusal lives at the one
place the daemon admits the flag: typed reason screenshot-readback-unavailable,
dispatched no, and the callstack#4509 decision owns when the capture binds instead.

Co-Authored-By: Claude Code <noreply@anthropic.com>
--expect-screenshot-diff now works on the daemon press and click routes: the
route admits the flag, binds the device's screenshot capture through the
screenshot lane's own plan admission, and hands it to the interaction backend,
so the readback bracket runs instead of returning no evidence. A device whose
owner declares no capture still refuses before dispatch, with the screenshot
lane's refusal stamped dispatched no. The refusal guard is gone; callstack#4509 keeps
the any-pixel threshold question.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The interaction route's artifacts policy refuses temp files by design, so the
readback's before and after frames never got created and the flag answered
with silence even after the capture bound. The lane owns its frame directory
(mkdtemp plus cleanup with the frame) instead of asking the route's adapter.
A device whose owner declares no capture still refuses before dispatch with
the screenshot lane's refusal, stamped dispatched no.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…xt, and restore the temp-file port

The readback frame now carries the input the screenshot command would build
for this request: the session's app identity, the simulator boot-check
knowledge, and the runner execution from the dispatch context. The shared
temp-file implementation moves into runtime-artifacts, the interaction route's
policy opts in, and the lane goes back to the runtime's createCommandTempFile.
The advisory case on a capture-less device pins dispatch-with-no-evidence.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The status bar moves without the dispatch causing it — a clock ticks on every
platform, and an iOS simulator's per-frame normalization can leave residue in
one frame — so the comparison ignores its band and the dead tap reads false
with the advisory warning on both platforms.

Co-Authored-By: Claude Code <noreply@anthropic.com>
… the rebase

Co-Authored-By: Claude Code <noreply@anthropic.com>
…unction

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…dget

The rebase restored main's newer press help and metadata prose, which pushed
the always-loaded tools over the byte budget with the readback field added.
The press cliDetail and the metadata plan sentence carry the compact form;
the full contract stays in ADR 0014, commands.md, and the help. The budget
still exceeds by about 100 bytes with the field's own schema — the remaining
gap is the callstack#4509/advertised-flag decision's to fund.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…budget

callstack#4296's plan prose pushed the always-loaded tools over the budget on main,
and the readback field added on top. The press description and the field
text carry the compact forms; the full contract lives in ADR 0014,
commands.md, the help line, and the cliDetail. The meta test drops the
leftover debug dumps.

Co-Authored-By: Claude Code <noreply@anthropic.com>
… capture

The excluded region is exactly the region the device itself changes between
frames, so the capture owner declares it per device class and the lane
compares over the rest: ScreenshotCaptureFacts gains statusBarTopHeightShare
(apple per appleOs, android, harmonyos; web, macOS, and TV declare none),
the bound screenshot backend returns it beside the rotation, and the
readback's single compare call takes the band from the before-frame's own
facts. The ratio's denominator is the compared region, and both
normalizeStatusBar requests go: a readback frame is captured by the
screenshot command's exported capture builder, with no options hand-built
in the interaction route. Schema text and commands.md say outside the
device status bar, and the lane regression pins the band pair, the
just-below pair, the compared-region denominator, and the band-less
platform; the daemon-route pair runs the real command surface against an
owner-declared capture and goes red when the forwarding drops.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Three full-JSON runs replace the stale sections: the mid-screen dead tap on
the iPhone 17 Pro and on the Redmi K60 both read false at a flat zero with
the outside-the-status-bar warning, and the iPad header-tab press reads
true on a two-orders-smaller ratio than the old band's noise floor. The old
runs stay, marked with the heads and band semantics they were recorded at.

Co-Authored-By: Claude Code <noreply@anthropic.com>
A canceled request ends the pair after the settle instead of capturing the
after-frame; the pixel loop is bounded by the declared frame bytes instead
of the buffer length; the temp-file owner's doc names its real opt-ins;
the policy test pins the allowTempFiles branch end to end; the readback
flag's cli-flags comment stops restating the flag definition; the
contract scenario asserts the tap really crossed; and the fused iOS route
gains the daemon test that keeps a click with the readback flag on the
runtime path.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The route test drives the real command surface, so the daemon declares the
workspace sibling its test support imports, the tsconfig references it, and
the stray scratch test file goes away. Layering passes, and the lockfile
records the link.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member

Thanks for the update. At 39c404d the debug-line, cleanup, abort, loop-bound and capture-routing items from the earlier review (#4297 (comment)) are mostly fixed, but the status-bar exclusion still does not work on the daemon route. CI is green (19 checks, 0 failing), but it cannot see this gap. The PR also has merge conflicts with main, and it should not merge until the problems below are fixed and live runs are posted.

The daemon route drops the status-bar band. captureScreenshot in createInteractionBackend awaits readbackCapture, which returns the capture owner's facts including statusBarTopHeightShare, then returns only { path: outPath } (interaction-runtime.ts:225). So capturePairScreenshotFrame (screenshot-diff-evidence.ts:128) never sees the band and compares the full frame on every daemon press and click on iOS, iPadOS, Android and HarmonyOS. The delta also removed the normalizeStatusBar: true requests, so nothing pins the iOS simulator clock now. A dead tap then reads changedFromBefore:true with no warning whenever the clock, battery, signal or a notification glyph redraws, and an agent will believe a missed key landed. The rule: every backend captureScreenshot that a readback frame passes through must return the capture owner's facts unchanged. At lines 225-226 that means returning { path: outPath, ...(await readbackCapture(...)) }. Please also check that no other hop between captureScreenshotFromRequest and the lane narrows the facts.

No test covers the band on the production route. Both daemon-route readback tests mock captureScreenshot to return {} facts (interaction-touch-press-admission.test.ts:442), and the lane tests feed facts straight into runtime.backend. Please add a case where the mock writes two 8x8 PNGs that differ only in the top half and returns { statusBarTopHeightShare: 0.5 }. It should expect changedFromBefore === false plus the warning. It should fail at 39c404d and pass with the fix above.

The user-facing text promises behavior the daemon route does not deliver. The schema text (index.ts:90 and :93), the flag usage, the warning and commands.md:508 all say the comparison covers the screen outside the device status bar. Agents trust a true result that may come from the clock, and they read a missing warning as proof the tap landed. This closes with the fix above. Keep the wording and make the route match it.

The live evidence does not exercise the band. The iOS doc says the dead tap reads 0 because the clock sits inside the owner-declared band (ios-screenshot-readback-2026-11-15.md:29), and the Android doc says the same at line 28. Production used band 0, so the runs read 0 only because no glyph redrew during the window. Neither doc cites the SHA it ran at. At a pushed head with the fix, please post the full --json of two runs, each citing its SHA. Run 1 is an iOS simulator press <x> <~50% height> --expect-screenshot-diff --json, with xcrun simctl status_bar <udid> override --batteryLevel 37 (or a time override) fired about 100 ms after the press starts. Run 2 is the same on Android with adb shell dumpsys battery set level 37, then dumpsys battery reset. Each must read changedFromBefore:false plus the 'outside the device status bar' warning. Please also show the same forced change at 39c404d reading true, which proves the change landed between the frames. Then replace the 'clock sits inside the band' sentences and the iPad band claim with these runs.

Not blocking, and you can take or leave these: src/mcp/__tests__/command-tools-loading-meta.test.ts:47 now imports node:fs, so the leftover debug writes to /tmp/al2.json and /tmp/al3.json run on every test run, and the delta adds a tracked symlink .autopilot/runtime/requirements that points to another machine's path plus an unused root local-config.json, so deleting the two writes, the import, the symlink and the file would clean this up; Android declares a fixed 0.04 band for every device including TV, and Apple a fixed 0.06 whatever the orientation, so once the fix lands a TV top-row reaction can read as a dead tap and a landscape Android clock can sit partly outside the band, which a band derived from the frame's orientation (or none for TV and landscape) would avoid; and statusBarTopHeightShare on the public ScreenshotResultData and its forward at screenshot-runtime.ts:396-398 have no writer or reader, while the abort check at screenshot-diff-evidence.ts:174-175 reads only options.signal instead of options.signal ?? runtime.signal, so a canceled daemon request still takes the after-frame.

On the open inline threads: the threads on the field text in metadata.ts:92, the dropped press ref-list sentence and the cut intervalMs clause still apply. Also still applying are the abort signal check, the iOS mid-screen dead tap, the iOS doc band claim, the Android doc band claim and the /tmp debug writes. These threads no longer apply and can be resolved: readbackCapture forwarding is forwarded and pinned by a test; capture through the session surface now passes appBundleId, surface and requestId; unguarded cleanup is an fs.rm with force:true that practically never rejects; Android press coordinates now use mid-screen 540 1200; PNG loop bound is bounded by declaredBytes; console output no longer exists in the interaction path; ratio divisor now divides by comparedPixels; and the dead adapter link is removed.

The landscape and TV band shares are computed, not measured. To merge, please return the capture facts at interaction-runtime.ts:225-226, add the daemon-route band test that fails at 39c404d, rebase onto main to clear the conflicts, then post the forced status-bar-change dead-tap runs on iOS and Android at the fixed SHA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants