Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
ce4b936
feat(interaction): press/click --expect-screenshot-diff out-of-tree r…
strzhao Nov 6, 2026
06c5f93
docs(interaction): drop the workflow-card sentence for --expect-scree…
strzhao Nov 6, 2026
8705e33
refactor(interaction): move the screenshot bracket lifecycle out of t…
strzhao Nov 6, 2026
cabe726
refactor(interaction): keep the screenshot frame helpers module-private
strzhao Nov 6, 2026
b070fb7
fix(interaction): register --expect-screenshot-diff in the guarantee …
strzhao Nov 6, 2026
6ad2a6c
refactor(interaction): load the evidence lane behind a dynamic import
strzhao Nov 6, 2026
1d733a5
fix(interaction): hoist the evidence import above its outer use
strzhao Nov 6, 2026
183e9a2
style(interaction): format after the import hoist
strzhao Nov 6, 2026
6d0a8f6
fix(interaction): keep the screenshot readback from failing a deliver…
strzhao Nov 13, 2026
eb07e21
fix(daemon): refuse --expect-screenshot-diff on the touch routes unti…
strzhao Nov 14, 2026
6ac03f2
fix(daemon): bind the admitted screenshot capture into the touch backend
strzhao Nov 15, 2026
eae227c
fix(daemon): give the screenshot readback its own frame files
strzhao Nov 15, 2026
d21e630
style(interaction): drop the unused catch bindings
strzhao Nov 15, 2026
b3b133f
docs(evidence): run the screenshot readback live on the daemon routes
strzhao Nov 15, 2026
7f00ff9
refactor(daemon): pass the readback capture without a second branch
strzhao Nov 15, 2026
4edfbe2
feat(daemon): bind the readback capture input from the dispatch conte…
strzhao Nov 15, 2026
15d2860
docs(evidence): run the screenshot readback live on Android
strzhao Nov 16, 2026
d71396c
fix(interaction): exclude the status-bar band from the readback ratio
strzhao Nov 16, 2026
6c95bcd
test(daemon): restore the multi-ref block and the advisory test after…
strzhao Nov 16, 2026
fe413d0
refactor(capture-kit): split the top-band counting out of the ratio f…
strzhao Nov 16, 2026
335aafc
chore(daemon): remove the leftover route-flags diagnostic
strzhao Nov 16, 2026
d4c1f09
fix(interaction): keep the plan text inside the always-loaded tool bu…
strzhao Nov 16, 2026
4593a28
fix(interaction): bring the always-loaded tools back inside the byte …
strzhao Nov 16, 2026
92dd7dd
fix(interaction): make the status-bar band a device fact owned by the…
strzhao Nov 16, 2026
12e7513
docs(evidence): re-run the readback live at the device-fact band head
strzhao Nov 16, 2026
d62cbcf
fix(interaction): close the remaining review threads on the readback
strzhao Nov 16, 2026
a6776c3
fix(platform-apple): import the capture facts type the band helper names
strzhao Nov 16, 2026
39c404d
fix(daemon): declare the command surface the readback route test wires
strzhao Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .autopilot/runtime/requirements
42 changes: 42 additions & 0 deletions docs/evidence/android-screenshot-readback-2026-11-16.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Screenshot readback live on Android, 2026-11-16

Evidence for #4297's Android side, re-run at the head that makes the
status-bar band a device fact (`platform-android` declares
`statusBarTopHeightShare: 0.04`; the lane compares over the rest and divides
by the compared region). Redmi K60 (adb serial `5p4hqglvpzxshiuw`), the
rig page (a static body with a TAB A header tab) served over `adb reverse`
loopback HTTP through the device's own capture owner.

## Mid-screen dead region: `press 540 1200 --expect-screenshot-diff --json`

The tap lands mid-screen (~50% of the frame height) on the page's blank
body. Full JSON:

```json
{
"success": true,
"data": {
"x": 540, "y": 1200,
"message": "Tapped (540, 1200)",
"targetKind": "point",
"screenshotDiff": { "changedFromBefore": false, "changedPixelRatio": 0 },
"warning": "Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot."
}
}
```

The Android status-bar clock sits inside the owner-declared band now, and
the dead tap reads a flat zero — the same answer iOS gives, from each
platform's own declared band instead of a shared constant.

## Earlier runs, recorded at head f66aa4bad (+ the fixed 6% band)

The runs below predate the device-fact band; they are kept for the
dispatch-route coverage they show, not for their band semantics.

- Effective key `press 540 500` (Settings root, opened a sub-page):
`changedFromBefore: true`, ratio 0.389.
- Runtime-ref `click @e8` (USB row): `changedFromBefore: true`, ratio 0.056.
- The old dead-region run pressed (540, 60), which lands ON the status bar —
inside any band, so it could not show a content dead tap; the mid-screen
run above replaces it.
60 changes: 60 additions & 0 deletions docs/evidence/ios-screenshot-readback-2026-11-15.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Screenshot readback live on the daemon press/click routes, 2026-11-15

Evidence for #4297, re-run at the head that makes the status-bar band a
device fact (the capture owner declares `statusBarTopHeightShare`; the lane
compares over the rest and divides by the compared region). The rig is a
local page with a static body and a header-tab button whose label toggles
TAB A ↔ TAB B, served to Safari over loopback HTTP.

## Mid-screen dead region: `press 201 500 --expect-screenshot-diff --json`

iPhone 17 Pro simulator (`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, iOS 26.5).
The tap lands mid-screen (~57% of the frame height) on the page's static
body. Full JSON:

```json
{
"success": true,
"data": {
"x": 201, "y": 500,
"message": "Tapped (201, 500)",
"targetKind": "point",
"screenshotDiff": { "changedFromBefore": false, "changedPixelRatio": 0 },
"warning": "Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot."
}
}
```

The ratio is flat zero: the clock that used to read as a 0.0006 through the
old blanket band sits inside the owner-declared band now, and the warning
says what the comparison actually covered.

## Top-of-screen reaction (web header tab): `press 410 118 --expect-screenshot-diff --json`

iPad (A16) simulator, same rig. The press hits the TAB A button's center
from `get attrs @e39` (rect y 93–143); the only pixels that change are the
button's label. Full JSON:

```json
{
"success": true,
"data": {
"x": 410, "y": 118,
"message": "Tapped (410, 118)",
"targetKind": "point",
"screenshotDiff": { "changedFromBefore": true, "changedPixelRatio": 0.00034415047540305914 }
}
}
```

A near-top reaction reads `true` at a ratio two orders below the old
blanket band's noise floor: the band is the device's own, not a fixed 6%
guess. The visual after-state names TAB B.

## Earlier runs, recorded at head 9aee85d6a

The effective-key and runtime-ref runs below predate the band change; they
are kept for the dispatch-route coverage they show, not for their ratios.

- Effective key `press 111 300`: `changedFromBefore: true`, ratio 0.0037.
- Runtime-ref `click @e16`: `changedFromBefore: true`, ratio 0.0134.
1 change: 1 addition & 0 deletions local-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{ "server": { "devPort": 4392, "hostname": "localhost", "enableHttps": false } }
4 changes: 4 additions & 0 deletions packages/capture-kit/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,10 @@
"types": "./src/png-rgb-difference.ts",
"default": "./src/png-rgb-difference.ts"
},
"./png-changed-pixel-ratio": {
"types": "./src/png-changed-pixel-ratio.ts",
"default": "./src/png-changed-pixel-ratio.ts"
},
"./png-size": {
"types": "./src/png-size.ts",
"default": "./src/png-size.ts"
Expand Down
21 changes: 21 additions & 0 deletions packages/capture-kit/src/png-changed-pixel-ratio.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,24 @@ describe('computePngChangedPixelRatio', () => {
).toEqual({ status: 'dimension_mismatch' });
});
});

test('computePngChangedPixelRatio ignores the top band share when asked', () => {
const first = { width: 4, height: 4, data: new Uint8Array(4 * 4 * 4).fill(10) };
const second = { width: 4, height: 4, data: new Uint8Array(4 * 4 * 4).fill(10) };
// Second frame: top half (rows 0-1) jumps to 200, bottom half (rows 2-3) to 30.
for (let y = 0; y < 2; y += 1) {
for (let x = 0; x < 4; x += 1) second.data[(y * 4 + x) * 4] = 200;
}
for (let y = 2; y < 4; y += 1) {
for (let x = 0; x < 4; x += 1) second.data[(y * 4 + x) * 4] = 30;
}

const full = computePngChangedPixelRatio(first, second);
expect(full).toEqual({ status: 'compared', changedPixelRatio: 1 });

// With the top half excluded, the ignored rows drop out of the comparison:
// the ratio counts the bottom half's changed pixels over that compared
// region, so every pixel the pair was compared over changed.
const half = computePngChangedPixelRatio(first, second, { ignoreTopHeightShare: 0.5 });
expect(half).toEqual({ status: 'compared', changedPixelRatio: 1 });
});
34 changes: 31 additions & 3 deletions packages/capture-kit/src/png-changed-pixel-ratio.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,18 @@ export type PngChangedPixelRatioResult =
*
* Dimensions must match. Callers comparing frames of different sizes are asking a different
* question, and returning a made-up ratio would hide it.
*
* `ignoreTopHeightShare` cuts the top share of the frame out of the comparison entirely, for
* bands whose pixels move without the dispatch causing it — the device's own status bar ticks
* between the bracket's frames, and the readback must answer for what the dispatch changed, not
* for the wall clock. The ratio runs over the compared region: both the changed count and the
* denominator count the rows below the band, so the share is the share of the region the pair
* was actually compared over.
*/
export function computePngChangedPixelRatio(
first: PngRgbImage,
second: PngRgbImage,
options: Readonly<{ ignoreTopHeightShare?: number }> = {},
): PngChangedPixelRatioResult {
if (first.width !== second.width || first.height !== second.height) {
return { status: 'dimension_mismatch' };
Expand All @@ -32,8 +40,29 @@ export function computePngChangedPixelRatio(
return { status: 'dimension_mismatch' };
}

// Rows at or above the cut are outside the compared region on both sides.
const ignoredRows =
options.ignoreTopHeightShare && options.ignoreTopHeightShare > 0
? Math.min(first.height, Math.floor(first.height * options.ignoreTopHeightShare))
: 0;
const comparedPixels = totalPixels - ignoredRows * first.width;
if (comparedPixels <= 0) return { status: 'compared', changedPixelRatio: 0 };

return {
status: 'compared',
changedPixelRatio: countChangedPixels(first, second, ignoredRows) / comparedPixels,
};
}

/** Counts pixels whose RGB differs outside the ignored top rows. */
function countChangedPixels(first: PngRgbImage, second: PngRgbImage, ignoredRows: number): number {
const declaredBytes = first.width * first.height * 4;
let changedPixels = 0;
for (let offset = 0; offset + 3 < first.data.length; offset += 4) {
for (
let offset = ignoredRows * first.width * 4;
offset + 3 < declaredBytes && offset + 3 < first.data.length;
offset += 4
) {
if (
first.data[offset] !== second.data[offset] ||
first.data[offset + 1] !== second.data[offset + 1] ||
Expand All @@ -42,6 +71,5 @@ export function computePngChangedPixelRatio(
changedPixels += 1;
}
}

return { status: 'compared', changedPixelRatio: changedPixels / totalPixels };
return changedPixels;
}
10 changes: 10 additions & 0 deletions packages/command-registry/src/flag-definitions-action.ts
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,16 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [
projectConfig: true,
recorded: false,
},
{
key: 'expectScreenshotDiff',
names: ['--expect-screenshot-diff'],
type: 'boolean',
usageLabel: '--expect-screenshot-diff',
usageDescription:
'Press/click: compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report screenshotDiff in the response; use it when the target lives outside the accessibility tree (a PIN keypad) and --verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect outside the device status bar',
projectConfig: true,
recorded: false,
},
{
key: 'settle',
names: ['--settle'],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,7 @@ export function createInteractionDevice(
Pick<
AgentDeviceBackend,
| 'captureSnapshot'
| 'captureScreenshot'
| 'resolveGestureViewport'
| 'tap'
| 'tapTarget'
Expand All @@ -366,6 +367,9 @@ export function createInteractionDevice(
platform: overrides.platform ?? 'ios',
captureSnapshot: async (...args) =>
overrides.captureSnapshot ? await overrides.captureSnapshot(...args) : { snapshot },
captureScreenshot: overrides.captureScreenshot
? async (...args) => (await overrides.captureScreenshot?.(...args)) ?? {}
: undefined,
resolveGestureViewport: overrides.resolveGestureViewport
? async (...args) => await overrides.resolveGestureViewport?.(...args)
: undefined,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import { isFillableType } from '@agent-device/contracts/snapshot';
import { attachResolvedInteractionTarget } from '@agent-device/host-kit/interaction-outcome';
import { sleep, toBackendContext } from '../../runtime-common.ts';
import { toBackendResult, type RuntimeCommand } from '../../runtime-types.ts';
import type { ScreenshotPairOutcome } from './screenshot-diff-evidence.ts';
import {
applyPostActionObservation,
planPostActionObservation,
Expand Down Expand Up @@ -50,6 +51,14 @@ export type PressCommandOptions = CommandContext &
* before the first press; see tapRefPlanCommand.
*/
refs?: readonly string[];
/**
* Opt-in: bracket the dispatch with two lightweight screenshots and attach
* `screenshotDiff` to the result, so a tap on a surface the accessibility
* tree cannot describe (a PIN keypad) still answers "did the screen react
* at all". Best-effort — a capture failure drops the evidence; a matching
* pair adds a warning instead of failing the action.
*/
expectScreenshotDiff?: boolean;
/** ADR 0012 step 4: replay-only post-resolution guard; see resolution.ts. */
expectedResolvedTarget?: ExpectedResolvedTarget;
/** #1654: a mutating `find`'s already-resolved node; see resolution.ts. */
Expand Down Expand Up @@ -152,9 +161,12 @@ async function tapCommand(
return await tapRefPlanCommand(runtime, options, action);
}
const observation = planPostActionObservation(options);
const nativeRefTap = observation.needsPreActionBaseline
? null
: await maybeTapRefTarget(runtime, options, action);
// --expect-screenshot-diff brackets the dispatch below, which the native ref
// fast path never reaches — route through the runtime path like verify/settle.
const nativeRefTap =
observation.needsPreActionBaseline || options.expectScreenshotDiff === true
? null
: await maybeTapRefTarget(runtime, options, action);
if (nativeRefTap) return nativeRefTap;

// One request object for the resolution and for the re-resolutions #3471's gate may spend, so the
Expand All @@ -174,17 +186,28 @@ async function tapCommand(
const atRest = await admitResolvedTarget(runtime, options, resolutionRequest, resolved);
const point = requireResolvedPoint(atRest);
const pointObservation = pointObservationOf(atRest);
const dispatchTap = runtime.backend.tap;
// Dynamic import: the readback evidence lane stays off the command-runtime entry closure.
const { attachScreenshotPairOutcome, dispatchWithScreenshotPair } =
await import('./screenshot-diff-evidence.ts');
let backendResult;
let screenshotOutcome: ScreenshotPairOutcome | undefined;
try {
backendResult = await runtime.backend.tap(toBackendContext(runtime, options), point, {
button: options.button,
count: options.count,
intervalMs: options.intervalMs,
holdMs: options.holdMs,
jitterPx: options.jitterPx,
doubleTap: options.doubleTap,
...(pointObservation ? { pointObservation } : {}),
});
({ result: backendResult, outcome: screenshotOutcome } = await dispatchWithScreenshotPair(
runtime,
options,
options.expectScreenshotDiff === true,
Comment thread
strzhao marked this conversation as resolved.
Resolved
async () =>
await dispatchTap(toBackendContext(runtime, options), point, {
button: options.button,
count: options.count,
intervalMs: options.intervalMs,
holdMs: options.holdMs,
jitterPx: options.jitterPx,
doubleTap: options.doubleTap,
...(pointObservation ? { pointObservation } : {}),
}),
));
} catch (error) {
// Resolution is complete before the backend call. Preserve it out of
// band so a daemon-level failure corroboration can still record the same
Expand All @@ -197,10 +220,10 @@ async function tapCommand(
runtime,
options,
atRest,
{
...atRest,
...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}),
},
attachScreenshotPairOutcome(
{ ...atRest, ...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}) },
screenshotOutcome,
),
observation,
);
}
Expand Down
Loading
Loading