Repository navigation
feat(interaction): screenshot readback (--expect-screenshot-diff) on press and click #4297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
strzhao
wants to merge
28
commits into
callstack:main
Choose a base branch
from
strzhao:forge/press-expect
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,833
−106
Open
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
ce4b936
feat(interaction): press/click --expect-screenshot-diff out-of-tree r…
strzhao 06c5f93
docs(interaction): drop the workflow-card sentence for --expect-scree…
strzhao 8705e33
refactor(interaction): move the screenshot bracket lifecycle out of t…
strzhao cabe726
refactor(interaction): keep the screenshot frame helpers module-private
strzhao b070fb7
fix(interaction): register --expect-screenshot-diff in the guarantee …
strzhao 6ad2a6c
refactor(interaction): load the evidence lane behind a dynamic import
strzhao 1d733a5
fix(interaction): hoist the evidence import above its outer use
strzhao 183e9a2
style(interaction): format after the import hoist
strzhao 6d0a8f6
fix(interaction): keep the screenshot readback from failing a deliver…
strzhao eb07e21
fix(daemon): refuse --expect-screenshot-diff on the touch routes unti…
strzhao 6ac03f2
fix(daemon): bind the admitted screenshot capture into the touch backend
strzhao eae227c
fix(daemon): give the screenshot readback its own frame files
strzhao d21e630
style(interaction): drop the unused catch bindings
strzhao b3b133f
docs(evidence): run the screenshot readback live on the daemon routes
strzhao 7f00ff9
refactor(daemon): pass the readback capture without a second branch
strzhao 4edfbe2
feat(daemon): bind the readback capture input from the dispatch conte…
strzhao 15d2860
docs(evidence): run the screenshot readback live on Android
strzhao d71396c
fix(interaction): exclude the status-bar band from the readback ratio
strzhao 6c95bcd
test(daemon): restore the multi-ref block and the advisory test after…
strzhao fe413d0
refactor(capture-kit): split the top-band counting out of the ratio f…
strzhao 335aafc
chore(daemon): remove the leftover route-flags diagnostic
strzhao d4c1f09
fix(interaction): keep the plan text inside the always-loaded tool bu…
strzhao 4593a28
fix(interaction): bring the always-loaded tools back inside the byte …
strzhao 92dd7dd
fix(interaction): make the status-bar band a device fact owned by the…
strzhao 12e7513
docs(evidence): re-run the readback live at the device-fact band head
strzhao d62cbcf
fix(interaction): close the remaining review threads on the readback
strzhao a6776c3
fix(platform-apple): import the capture facts type the band helper names
strzhao 39c404d
fix(daemon): declare the command surface the readback route test wires
strzhao File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| /Users/stringzhao/workspace/agent-device/.autopilot/runtime/requirements |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # Screenshot readback live on Android, 2026-11-16 | ||
|
|
||
| Evidence for #4297's Android side, re-run at the head that makes the | ||
| status-bar band a device fact (`platform-android` declares | ||
| `statusBarTopHeightShare: 0.04`; the lane compares over the rest and divides | ||
| by the compared region). Redmi K60 (adb serial `5p4hqglvpzxshiuw`), the | ||
| rig page (a static body with a TAB A header tab) served over `adb reverse` | ||
| loopback HTTP through the device's own capture owner. | ||
|
|
||
| ## Mid-screen dead region: `press 540 1200 --expect-screenshot-diff --json` | ||
|
|
||
| The tap lands mid-screen (~50% of the frame height) on the page's blank | ||
| body. Full JSON: | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "data": { | ||
| "x": 540, "y": 1200, | ||
| "message": "Tapped (540, 1200)", | ||
| "targetKind": "point", | ||
| "screenshotDiff": { "changedFromBefore": false, "changedPixelRatio": 0 }, | ||
| "warning": "Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot." | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The Android status-bar clock sits inside the owner-declared band now, and | ||
| the dead tap reads a flat zero — the same answer iOS gives, from each | ||
| platform's own declared band instead of a shared constant. | ||
|
|
||
| ## Earlier runs, recorded at head f66aa4bad (+ the fixed 6% band) | ||
|
|
||
| The runs below predate the device-fact band; they are kept for the | ||
| dispatch-route coverage they show, not for their band semantics. | ||
|
|
||
| - Effective key `press 540 500` (Settings root, opened a sub-page): | ||
| `changedFromBefore: true`, ratio 0.389. | ||
| - Runtime-ref `click @e8` (USB row): `changedFromBefore: true`, ratio 0.056. | ||
| - The old dead-region run pressed (540, 60), which lands ON the status bar — | ||
| inside any band, so it could not show a content dead tap; the mid-screen | ||
| run above replaces it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| # Screenshot readback live on the daemon press/click routes, 2026-11-15 | ||
|
|
||
| Evidence for #4297, re-run at the head that makes the status-bar band a | ||
| device fact (the capture owner declares `statusBarTopHeightShare`; the lane | ||
| compares over the rest and divides by the compared region). The rig is a | ||
| local page with a static body and a header-tab button whose label toggles | ||
| TAB A ↔ TAB B, served to Safari over loopback HTTP. | ||
|
|
||
| ## Mid-screen dead region: `press 201 500 --expect-screenshot-diff --json` | ||
|
|
||
| iPhone 17 Pro simulator (`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, iOS 26.5). | ||
| The tap lands mid-screen (~57% of the frame height) on the page's static | ||
| body. Full JSON: | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "data": { | ||
| "x": 201, "y": 500, | ||
| "message": "Tapped (201, 500)", | ||
| "targetKind": "point", | ||
| "screenshotDiff": { "changedFromBefore": false, "changedPixelRatio": 0 }, | ||
| "warning": "Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot." | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The ratio is flat zero: the clock that used to read as a 0.0006 through the | ||
| old blanket band sits inside the owner-declared band now, and the warning | ||
| says what the comparison actually covered. | ||
|
|
||
| ## Top-of-screen reaction (web header tab): `press 410 118 --expect-screenshot-diff --json` | ||
|
|
||
| iPad (A16) simulator, same rig. The press hits the TAB A button's center | ||
| from `get attrs @e39` (rect y 93–143); the only pixels that change are the | ||
| button's label. Full JSON: | ||
|
|
||
| ```json | ||
| { | ||
| "success": true, | ||
| "data": { | ||
| "x": 410, "y": 118, | ||
| "message": "Tapped (410, 118)", | ||
| "targetKind": "point", | ||
| "screenshotDiff": { "changedFromBefore": true, "changedPixelRatio": 0.00034415047540305914 } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| A near-top reaction reads `true` at a ratio two orders below the old | ||
| blanket band's noise floor: the band is the device's own, not a fixed 6% | ||
| guess. The visual after-state names TAB B. | ||
|
|
||
| ## Earlier runs, recorded at head 9aee85d6a | ||
|
|
||
| The effective-key and runtime-ref runs below predate the band change; they | ||
| are kept for the dispatch-route coverage they show, not for their ratios. | ||
|
|
||
| - Effective key `press 111 300`: `changedFromBefore: true`, ratio 0.0037. | ||
| - Runtime-ref `click @e16`: `changedFromBefore: true`, ratio 0.0134. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| { "server": { "devPort": 4392, "hostname": "localhost", "enableHttps": false } } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.