diff --git a/.autopilot/runtime/requirements b/.autopilot/runtime/requirements new file mode 120000 index 0000000000..a07ad0d334 --- /dev/null +++ b/.autopilot/runtime/requirements @@ -0,0 +1 @@ +/Users/stringzhao/workspace/agent-device/.autopilot/runtime/requirements \ No newline at end of file diff --git a/docs/evidence/android-screenshot-readback-2026-11-16.md b/docs/evidence/android-screenshot-readback-2026-11-16.md new file mode 100644 index 0000000000..07db3fa798 --- /dev/null +++ b/docs/evidence/android-screenshot-readback-2026-11-16.md @@ -0,0 +1,42 @@ +# Screenshot readback live on Android, 2026-11-16 + +Evidence for #4297's Android side, re-run at the head that makes the +status-bar band a device fact (`platform-android` declares +`statusBarTopHeightShare: 0.04`; the lane compares over the rest and divides +by the compared region). Redmi K60 (adb serial `5p4hqglvpzxshiuw`), the +rig page (a static body with a TAB A header tab) served over `adb reverse` +loopback HTTP through the device's own capture owner. + +## Mid-screen dead region: `press 540 1200 --expect-screenshot-diff --json` + +The tap lands mid-screen (~50% of the frame height) on the page's blank +body. Full JSON: + +```json +{ + "success": true, + "data": { + "x": 540, "y": 1200, + "message": "Tapped (540, 1200)", + "targetKind": "point", + "screenshotDiff": { "changedFromBefore": false, "changedPixelRatio": 0 }, + "warning": "Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot." + } +} +``` + +The Android status-bar clock sits inside the owner-declared band now, and +the dead tap reads a flat zero — the same answer iOS gives, from each +platform's own declared band instead of a shared constant. + +## Earlier runs, recorded at head f66aa4bad (+ the fixed 6% band) + +The runs below predate the device-fact band; they are kept for the +dispatch-route coverage they show, not for their band semantics. + +- Effective key `press 540 500` (Settings root, opened a sub-page): + `changedFromBefore: true`, ratio 0.389. +- Runtime-ref `click @e8` (USB row): `changedFromBefore: true`, ratio 0.056. +- The old dead-region run pressed (540, 60), which lands ON the status bar — + inside any band, so it could not show a content dead tap; the mid-screen + run above replaces it. diff --git a/docs/evidence/ios-screenshot-readback-2026-11-15.md b/docs/evidence/ios-screenshot-readback-2026-11-15.md new file mode 100644 index 0000000000..4f2d61c757 --- /dev/null +++ b/docs/evidence/ios-screenshot-readback-2026-11-15.md @@ -0,0 +1,60 @@ +# Screenshot readback live on the daemon press/click routes, 2026-11-15 + +Evidence for #4297, re-run at the head that makes the status-bar band a +device fact (the capture owner declares `statusBarTopHeightShare`; the lane +compares over the rest and divides by the compared region). The rig is a +local page with a static body and a header-tab button whose label toggles +TAB A ↔ TAB B, served to Safari over loopback HTTP. + +## Mid-screen dead region: `press 201 500 --expect-screenshot-diff --json` + +iPhone 17 Pro simulator (`C1C05FFF-29AD-40D8-94E5-D9CBB25EC6BB`, iOS 26.5). +The tap lands mid-screen (~57% of the frame height) on the page's static +body. Full JSON: + +```json +{ + "success": true, + "data": { + "x": 201, "y": 500, + "message": "Tapped (201, 500)", + "targetKind": "point", + "screenshotDiff": { "changedFromBefore": false, "changedPixelRatio": 0 }, + "warning": "Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot." + } +} +``` + +The ratio is flat zero: the clock that used to read as a 0.0006 through the +old blanket band sits inside the owner-declared band now, and the warning +says what the comparison actually covered. + +## Top-of-screen reaction (web header tab): `press 410 118 --expect-screenshot-diff --json` + +iPad (A16) simulator, same rig. The press hits the TAB A button's center +from `get attrs @e39` (rect y 93–143); the only pixels that change are the +button's label. Full JSON: + +```json +{ + "success": true, + "data": { + "x": 410, "y": 118, + "message": "Tapped (410, 118)", + "targetKind": "point", + "screenshotDiff": { "changedFromBefore": true, "changedPixelRatio": 0.00034415047540305914 } + } +} +``` + +A near-top reaction reads `true` at a ratio two orders below the old +blanket band's noise floor: the band is the device's own, not a fixed 6% +guess. The visual after-state names TAB B. + +## Earlier runs, recorded at head 9aee85d6a + +The effective-key and runtime-ref runs below predate the band change; they +are kept for the dispatch-route coverage they show, not for their ratios. + +- Effective key `press 111 300`: `changedFromBefore: true`, ratio 0.0037. +- Runtime-ref `click @e16`: `changedFromBefore: true`, ratio 0.0134. diff --git a/local-config.json b/local-config.json new file mode 100644 index 0000000000..deb59a718d --- /dev/null +++ b/local-config.json @@ -0,0 +1 @@ +{ "server": { "devPort": 4392, "hostname": "localhost", "enableHttps": false } } diff --git a/packages/capture-kit/package.json b/packages/capture-kit/package.json index 581081453f..d87876b127 100644 --- a/packages/capture-kit/package.json +++ b/packages/capture-kit/package.json @@ -150,6 +150,10 @@ "types": "./src/png-rgb-difference.ts", "default": "./src/png-rgb-difference.ts" }, + "./png-changed-pixel-ratio": { + "types": "./src/png-changed-pixel-ratio.ts", + "default": "./src/png-changed-pixel-ratio.ts" + }, "./png-size": { "types": "./src/png-size.ts", "default": "./src/png-size.ts" diff --git a/packages/capture-kit/src/png-changed-pixel-ratio.test.ts b/packages/capture-kit/src/png-changed-pixel-ratio.test.ts index 24eb9b979e..1541b2f045 100644 --- a/packages/capture-kit/src/png-changed-pixel-ratio.test.ts +++ b/packages/capture-kit/src/png-changed-pixel-ratio.test.ts @@ -45,3 +45,24 @@ describe('computePngChangedPixelRatio', () => { ).toEqual({ status: 'dimension_mismatch' }); }); }); + +test('computePngChangedPixelRatio ignores the top band share when asked', () => { + const first = { width: 4, height: 4, data: new Uint8Array(4 * 4 * 4).fill(10) }; + const second = { width: 4, height: 4, data: new Uint8Array(4 * 4 * 4).fill(10) }; + // Second frame: top half (rows 0-1) jumps to 200, bottom half (rows 2-3) to 30. + for (let y = 0; y < 2; y += 1) { + for (let x = 0; x < 4; x += 1) second.data[(y * 4 + x) * 4] = 200; + } + for (let y = 2; y < 4; y += 1) { + for (let x = 0; x < 4; x += 1) second.data[(y * 4 + x) * 4] = 30; + } + + const full = computePngChangedPixelRatio(first, second); + expect(full).toEqual({ status: 'compared', changedPixelRatio: 1 }); + + // With the top half excluded, the ignored rows drop out of the comparison: + // the ratio counts the bottom half's changed pixels over that compared + // region, so every pixel the pair was compared over changed. + const half = computePngChangedPixelRatio(first, second, { ignoreTopHeightShare: 0.5 }); + expect(half).toEqual({ status: 'compared', changedPixelRatio: 1 }); +}); diff --git a/packages/capture-kit/src/png-changed-pixel-ratio.ts b/packages/capture-kit/src/png-changed-pixel-ratio.ts index f7b0fa0ebd..8ae92a004d 100644 --- a/packages/capture-kit/src/png-changed-pixel-ratio.ts +++ b/packages/capture-kit/src/png-changed-pixel-ratio.ts @@ -17,10 +17,18 @@ export type PngChangedPixelRatioResult = * * Dimensions must match. Callers comparing frames of different sizes are asking a different * question, and returning a made-up ratio would hide it. + * + * `ignoreTopHeightShare` cuts the top share of the frame out of the comparison entirely, for + * bands whose pixels move without the dispatch causing it — the device's own status bar ticks + * between the bracket's frames, and the readback must answer for what the dispatch changed, not + * for the wall clock. The ratio runs over the compared region: both the changed count and the + * denominator count the rows below the band, so the share is the share of the region the pair + * was actually compared over. */ export function computePngChangedPixelRatio( first: PngRgbImage, second: PngRgbImage, + options: Readonly<{ ignoreTopHeightShare?: number }> = {}, ): PngChangedPixelRatioResult { if (first.width !== second.width || first.height !== second.height) { return { status: 'dimension_mismatch' }; @@ -32,8 +40,29 @@ export function computePngChangedPixelRatio( return { status: 'dimension_mismatch' }; } + // Rows at or above the cut are outside the compared region on both sides. + const ignoredRows = + options.ignoreTopHeightShare && options.ignoreTopHeightShare > 0 + ? Math.min(first.height, Math.floor(first.height * options.ignoreTopHeightShare)) + : 0; + const comparedPixels = totalPixels - ignoredRows * first.width; + if (comparedPixels <= 0) return { status: 'compared', changedPixelRatio: 0 }; + + return { + status: 'compared', + changedPixelRatio: countChangedPixels(first, second, ignoredRows) / comparedPixels, + }; +} + +/** Counts pixels whose RGB differs outside the ignored top rows. */ +function countChangedPixels(first: PngRgbImage, second: PngRgbImage, ignoredRows: number): number { + const declaredBytes = first.width * first.height * 4; let changedPixels = 0; - for (let offset = 0; offset + 3 < first.data.length; offset += 4) { + for ( + let offset = ignoredRows * first.width * 4; + offset + 3 < declaredBytes && offset + 3 < first.data.length; + offset += 4 + ) { if ( first.data[offset] !== second.data[offset] || first.data[offset + 1] !== second.data[offset + 1] || @@ -42,6 +71,5 @@ export function computePngChangedPixelRatio( changedPixels += 1; } } - - return { status: 'compared', changedPixelRatio: changedPixels / totalPixels }; + return changedPixels; } diff --git a/packages/command-registry/src/flag-definitions-action.ts b/packages/command-registry/src/flag-definitions-action.ts index 0f15125d2e..58e7a48ee6 100644 --- a/packages/command-registry/src/flag-definitions-action.ts +++ b/packages/command-registry/src/flag-definitions-action.ts @@ -206,6 +206,16 @@ export const ACTION_FLAG_DEFINITIONS: readonly FlagDefinition[] = [ projectConfig: true, recorded: false, }, + { + key: 'expectScreenshotDiff', + names: ['--expect-screenshot-diff'], + type: 'boolean', + usageLabel: '--expect-screenshot-diff', + usageDescription: + 'Press/click: compare lightweight screenshots taken right before the dispatch and after it settles briefly, and report screenshotDiff in the response; use it when the target lives outside the accessibility tree (a PIN keypad) and --verify would compare empty trees. A matching pair adds a warning, because the dispatch may have had no visible effect outside the device status bar', + projectConfig: true, + recorded: false, + }, { key: 'settle', names: ['--settle'], diff --git a/packages/command-runtime/src/commands/interaction/runtime/__tests__/test-utils/index.ts b/packages/command-runtime/src/commands/interaction/runtime/__tests__/test-utils/index.ts index b646a045bc..5b6c9d8167 100644 --- a/packages/command-runtime/src/commands/interaction/runtime/__tests__/test-utils/index.ts +++ b/packages/command-runtime/src/commands/interaction/runtime/__tests__/test-utils/index.ts @@ -343,6 +343,7 @@ export function createInteractionDevice( Pick< AgentDeviceBackend, | 'captureSnapshot' + | 'captureScreenshot' | 'resolveGestureViewport' | 'tap' | 'tapTarget' @@ -366,6 +367,9 @@ export function createInteractionDevice( platform: overrides.platform ?? 'ios', captureSnapshot: async (...args) => overrides.captureSnapshot ? await overrides.captureSnapshot(...args) : { snapshot }, + captureScreenshot: overrides.captureScreenshot + ? async (...args) => (await overrides.captureScreenshot?.(...args)) ?? {} + : undefined, resolveGestureViewport: overrides.resolveGestureViewport ? async (...args) => await overrides.resolveGestureViewport?.(...args) : undefined, diff --git a/packages/command-runtime/src/commands/interaction/runtime/interactions.ts b/packages/command-runtime/src/commands/interaction/runtime/interactions.ts index dccc021be8..15115ce57e 100644 --- a/packages/command-runtime/src/commands/interaction/runtime/interactions.ts +++ b/packages/command-runtime/src/commands/interaction/runtime/interactions.ts @@ -15,6 +15,7 @@ import { isFillableType } from '@agent-device/contracts/snapshot'; import { attachResolvedInteractionTarget } from '@agent-device/host-kit/interaction-outcome'; import { sleep, toBackendContext } from '../../runtime-common.ts'; import { toBackendResult, type RuntimeCommand } from '../../runtime-types.ts'; +import type { ScreenshotPairOutcome } from './screenshot-diff-evidence.ts'; import { applyPostActionObservation, planPostActionObservation, @@ -50,6 +51,14 @@ export type PressCommandOptions = CommandContext & * before the first press; see tapRefPlanCommand. */ refs?: readonly string[]; + /** + * Opt-in: bracket the dispatch with two lightweight screenshots and attach + * `screenshotDiff` to the result, so a tap on a surface the accessibility + * tree cannot describe (a PIN keypad) still answers "did the screen react + * at all". Best-effort — a capture failure drops the evidence; a matching + * pair adds a warning instead of failing the action. + */ + expectScreenshotDiff?: boolean; /** ADR 0012 step 4: replay-only post-resolution guard; see resolution.ts. */ expectedResolvedTarget?: ExpectedResolvedTarget; /** #1654: a mutating `find`'s already-resolved node; see resolution.ts. */ @@ -152,9 +161,12 @@ async function tapCommand( return await tapRefPlanCommand(runtime, options, action); } const observation = planPostActionObservation(options); - const nativeRefTap = observation.needsPreActionBaseline - ? null - : await maybeTapRefTarget(runtime, options, action); + // --expect-screenshot-diff brackets the dispatch below, which the native ref + // fast path never reaches — route through the runtime path like verify/settle. + const nativeRefTap = + observation.needsPreActionBaseline || options.expectScreenshotDiff === true + ? null + : await maybeTapRefTarget(runtime, options, action); if (nativeRefTap) return nativeRefTap; // One request object for the resolution and for the re-resolutions #3471's gate may spend, so the @@ -174,17 +186,28 @@ async function tapCommand( const atRest = await admitResolvedTarget(runtime, options, resolutionRequest, resolved); const point = requireResolvedPoint(atRest); const pointObservation = pointObservationOf(atRest); + const dispatchTap = runtime.backend.tap; + // Dynamic import: the readback evidence lane stays off the command-runtime entry closure. + const { attachScreenshotPairOutcome, dispatchWithScreenshotPair } = + await import('./screenshot-diff-evidence.ts'); let backendResult; + let screenshotOutcome: ScreenshotPairOutcome | undefined; try { - backendResult = await runtime.backend.tap(toBackendContext(runtime, options), point, { - button: options.button, - count: options.count, - intervalMs: options.intervalMs, - holdMs: options.holdMs, - jitterPx: options.jitterPx, - doubleTap: options.doubleTap, - ...(pointObservation ? { pointObservation } : {}), - }); + ({ result: backendResult, outcome: screenshotOutcome } = await dispatchWithScreenshotPair( + runtime, + options, + options.expectScreenshotDiff === true, + async () => + await dispatchTap(toBackendContext(runtime, options), point, { + button: options.button, + count: options.count, + intervalMs: options.intervalMs, + holdMs: options.holdMs, + jitterPx: options.jitterPx, + doubleTap: options.doubleTap, + ...(pointObservation ? { pointObservation } : {}), + }), + )); } catch (error) { // Resolution is complete before the backend call. Preserve it out of // band so a daemon-level failure corroboration can still record the same @@ -197,10 +220,10 @@ async function tapCommand( runtime, options, atRest, - { - ...atRest, - ...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}), - }, + attachScreenshotPairOutcome( + { ...atRest, ...(formattedBackendResult ? { backendResult: formattedBackendResult } : {}) }, + screenshotOutcome, + ), observation, ); } diff --git a/packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.test.ts b/packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.test.ts new file mode 100644 index 0000000000..8ddd0314e8 --- /dev/null +++ b/packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.test.ts @@ -0,0 +1,378 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import { PNG } from '@agent-device/capture-kit/png'; +import { test } from 'vitest'; +import { makeSnapshotState } from '@agent-device/selectors/snapshot-geometry-fixtures'; +import { ref } from './selector-read-utils.ts'; +import { createInteractionDevice, selectorSnapshot } from './__tests__/test-utils/index.ts'; + +// `--expect-screenshot-diff` (#press-expect): the out-of-tree readback for the +// surfaces --verify cannot answer for — a PIN keypad whose keys never reach the +// accessibility tree digests identically whether or not the digit was entered, +// so the pair compares pixels instead of nodes. + +function solidPngBuffer( + width: number, + height: number, + color: { r: number; g: number; b: number }, +): Buffer { + const png = new PNG({ width, height }); + for (let i = 0; i < png.data.length; i += 4) { + png.data[i] = color.r; + png.data[i + 1] = color.g; + png.data[i + 2] = color.b; + png.data[i + 3] = 255; + } + return PNG.sync.write(png); +} + +/** A frame whose top `topRows` rows are `topColor` and whose rest is `restColor`. */ +function bandedPngBuffer( + width: number, + height: number, + topRows: number, + topColor: { r: number; g: number; b: number }, + restColor: { r: number; g: number; b: number }, +): Buffer { + const png = new PNG({ width, height }); + for (let y = 0; y < height; y += 1) { + const color = y < topRows ? topColor : restColor; + for (let x = 0; x < width; x += 1) { + const offset = (y * width + x) * 4; + png.data[offset] = color.r; + png.data[offset + 1] = color.g; + png.data[offset + 2] = color.b; + png.data[offset + 3] = 255; + } + } + return PNG.sync.write(png); +} + +type FramePlan = Array<{ + bytes: Buffer; + facts?: { statusBarTopHeightShare?: number }; +}>; + +function framePlanBackend(frames: FramePlan) { + let frameIndex = 0; + return { + captureScreenshot: async (_context: unknown, outPath: string) => { + const frame = frames[Math.min(frameIndex, frames.length - 1)]; + frameIndex += 1; + assert.ok(frame, 'the frame plan must cover every capture the pair takes'); + fs.writeFileSync(outPath, frame.bytes); + return { path: outPath, ...(frame.facts ?? {}) }; + }, + }; +} + +test('press with expectScreenshotDiff brackets the dispatch and reports a changed screen', async () => { + const backend = framePlanBackend([ + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + { bytes: solidPngBuffer(8, 8, { r: 255, g: 255, b: 255 }) }, + ]); + const taps: Array<{ x: number; y: number }> = []; + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async (_context, point) => { + taps.push(point); + }, + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + assert.deepEqual(taps, [{ x: 10, y: 20 }]); + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); + assert.equal(result.screenshotDiff?.changedPixelRatio, 1); + assert.equal('warning' in result ? result.warning : undefined, undefined); +}); + +test('a change inside only the declared status-bar band reads false with the warning', async () => { + // 8-row frame, band = 0.5 → the top 4 rows are the device's own. The pair + // differs only in the top row (a clock tick); the readback must answer + // false. This case is the red check: a lane that stops passing the band to + // the compare reads this pair as true. + const backend = framePlanBackend([ + { + bytes: bandedPngBuffer(8, 8, 4, { r: 0, g: 0, b: 0 }, { r: 255, g: 255, b: 255 }), + facts: { statusBarTopHeightShare: 0.5 }, + }, + { + bytes: bandedPngBuffer(8, 8, 4, { r: 255, g: 255, b: 255 }, { r: 255, g: 255, b: 255 }), + facts: { statusBarTopHeightShare: 0.5 }, + }, + ]); + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { session: 'default', expectScreenshotDiff: true }, + ); + + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, false); + assert.equal(result.screenshotDiff?.changedPixelRatio, 0); + assert.match( + ('warning' in result ? result.warning : undefined) ?? '', + /outside the device status bar/, + ); +}); + +test('a change just below the declared band reads true over the compared region', async () => { + // The pair differs across the whole 4-row region BELOW the band: 32 changed + // pixels over a compared region of 8x4 = 32, so the ratio's denominator is + // the compared region, not the full frame (which would read 0.5). + const backend = framePlanBackend([ + { + bytes: bandedPngBuffer(8, 8, 4, { r: 0, g: 0, b: 0 }, { r: 255, g: 255, b: 255 }), + facts: { statusBarTopHeightShare: 0.5 }, + }, + { + bytes: bandedPngBuffer(8, 8, 4, { r: 0, g: 0, b: 0 }, { r: 0, g: 255, b: 0 }), + facts: { statusBarTopHeightShare: 0.5 }, + }, + ]); + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { session: 'default', expectScreenshotDiff: true }, + ); + + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); + assert.equal(result.screenshotDiff?.changedPixelRatio, 1); +}); + +test('a band-less platform compares the top rows and a top-row change reads true', async () => { + // No declared band (web, macOS, a TV): the whole frame is the compared + // region, so a change in the top row is the reaction. + const backend = framePlanBackend([ + { bytes: bandedPngBuffer(8, 8, 1, { r: 0, g: 0, b: 0 }, { r: 255, g: 255, b: 255 }) }, + { bytes: bandedPngBuffer(8, 8, 1, { r: 255, g: 255, b: 255 }, { r: 255, g: 255, b: 255 }) }, + ]); + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { session: 'default', expectScreenshotDiff: true }, + ); + + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); + assert.equal(result.screenshotDiff?.changedPixelRatio, 0.125); +}); + +test('press with expectScreenshotDiff warns when the frames match', async () => { + const backend = framePlanBackend([ + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + ]); + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, false); + assert.equal(result.screenshotDiff?.changedPixelRatio, 0); + // The silent no-op must not read as success: the response says so next to + // the evidence, and the action itself still succeeds. + assert.match( + ('warning' in result ? result.warning : undefined) ?? '', + /may have had no visible effect/, + ); +}); + +test('press with expectScreenshotDiff keeps a resolution warning beside its own', async () => { + const backend = framePlanBackend([ + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + ]); + // A point outside the viewport the stored tree captured earns the + // outside-viewport warning; the pair warning composes after it. + const viewportSnapshot = makeSnapshotState([ + { + index: 0, + depth: 0, + type: 'Application', + rect: { x: 0, y: 0, width: 400, height: 800 }, + hittable: true, + }, + { + index: 1, + depth: 1, + parentIndex: 0, + type: 'Button', + label: 'Continue', + rect: { x: 10, y: 20, width: 100, height: 40 }, + hittable: true, + }, + ]); + const device = createInteractionDevice(viewportSnapshot, { + ...backend, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 100000, y: 100000 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + const warning = 'warning' in result ? result.warning : undefined; + assert.match(warning ?? '', /outside the last-known viewport/); + assert.match(warning ?? '', /may have had no visible effect/); +}); + +test('press with expectScreenshotDiff reports a shape change without a ratio', async () => { + const backend = framePlanBackend([ + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + { bytes: solidPngBuffer(4, 16, { r: 0, g: 0, b: 0 }) }, + ]); + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); + assert.equal(result.screenshotDiff?.changedPixelRatio, undefined); + assert.equal('warning' in result ? result.warning : undefined, undefined); +}); + +test('press with expectScreenshotDiff drops the evidence when a capture fails', async () => { + let captureCount = 0; + const taps: Array<{ x: number; y: number }> = []; + const device = createInteractionDevice(selectorSnapshot(), { + captureScreenshot: async () => { + captureCount += 1; + if (captureCount === 1) throw new Error('screen capture unavailable'); + return { path: '/unused/after.png' }; + }, + tap: async (_context, point) => { + taps.push(point); + }, + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + // No pair, no claim — the tap still crossed, and the readback says nothing. + assert.equal('screenshotDiff' in result, false); + assert.equal('warning' in result ? result.warning : undefined, undefined); + assert.deepEqual(captureCount, 1); + assert.deepEqual(taps, [{ x: 10, y: 20 }]); +}); + +test('press without a screenshot backend drops the evidence and still taps', async () => { + const taps: Array<{ x: number; y: number }> = []; + const device = createInteractionDevice(selectorSnapshot(), { + tap: async (_context, point) => { + taps.push(point); + }, + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + assert.deepEqual(taps, [{ x: 10, y: 20 }]); + assert.equal('screenshotDiff' in result, false); +}); + +test('press with expectScreenshotDiff routes a ref target through the runtime path', async () => { + const backend = framePlanBackend([ + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + { bytes: solidPngBuffer(8, 8, { r: 255, g: 255, b: 255 }) }, + ]); + const refTaps: string[] = []; + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + platform: 'web', + tapTarget: async (_context, target) => { + refTaps.push(target.ref); + return { ref: target.ref.replace(/^@/, '') }; + }, + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press(ref('@e1'), { + session: 'default', + expectScreenshotDiff: true, + }); + + // The native ref fast path would skip the pair entirely; the flag forces the + // runtime path so the readback brackets the dispatch it names. + assert.deepEqual(refTaps, []); + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); +}); + +test('a corrupted after-frame yields no evidence instead of failing a delivered tap', async () => { + const corrupted = Buffer.from('not a png at all'); + const backend = framePlanBackend([ + { bytes: solidPngBuffer(8, 8, { r: 0, g: 0, b: 0 }) }, + { bytes: corrupted }, + ]); + const taps: Array<{ x: number; y: number }> = []; + const device = createInteractionDevice(selectorSnapshot(), { + ...backend, + tap: async (_context, point) => { + taps.push(point); + }, + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { session: 'default', expectScreenshotDiff: true }, + ); + + // The tap crossed the device; the unreadable readback claims nothing. + assert.equal(taps.length, 1); + assert.equal(result.screenshotDiff, undefined); + assert.equal(result.warning, undefined); +}); diff --git a/packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts b/packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts new file mode 100644 index 0000000000..05b58cac78 --- /dev/null +++ b/packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts @@ -0,0 +1,223 @@ +import { promises as fs } from 'node:fs'; +import type { TemporaryFile } from '@agent-device/contracts/artifact-adapter'; +import type { ScreenshotDiffEvidence } from '@agent-device/contracts/interaction'; +import { createCommandTempFile } from '../../io-policy.ts'; +import type { AgentDeviceRuntime, CommandContext } from '@agent-device/contracts/runtime-contract'; +import { sleep, toBackendContext } from '../../runtime-common.ts'; + +/** + * `--expect-screenshot-diff` for press/click: the out-of-tree readback + * primitive. The screenshots bracket the dispatch, and the comparison runs on + * pixels, because the surfaces this answers for — a PIN keypad whose keys never + * reach the accessibility tree, a custom-drawn keyboard — leave `--verify`'s + * node digest flat no matter what the tap did. + * + * Best-effort by the same contract as `--verify` evidence: a capture that fails + * or a backend without screenshots yields no evidence, which claims nothing + * about the screen. The comparison never fails the action. + * + * The PNG decoder and the pixel ratio load behind function-scoped dynamic + * imports: an eager edge from every press/click into the decode stack would + * tax the flag-less fast path, and only the opt-in readback ever decodes a + * frame (the eager-closure budget). + */ + +/** + * The after-frame waits this long past the dispatch reply before it is + * captured. The reaction the caller asked about (a key highlight, a PIN dot) + * can reach the screen several frames after the transport round trip returns — + * exactly on the low-end or loaded devices this flag exists for — and a frame + * that raced the rendering would read an effective tap as unchanged, which is + * the one direction this primitive cannot cheaply retry. Long enough to cover + * that render delay; short enough to keep the flag's added latency negligible. + */ +const AFTER_FRAME_SETTLE_MS = 150; + +export type ScreenshotPairOutcome = { + evidence?: ScreenshotDiffEvidence; + /** Present when the caller expected a diff and the frames matched. */ + unchangedWarning?: string; +}; + +/** + * Runs the dispatch inside the screenshot bracket. Unrequested, this is the + * dispatch alone. Requested, the before-frame lands after every pre-dispatch + * guard and immediately before the dispatch, and the after-frame lands once + * the dispatch has settled briefly (AFTER_FRAME_SETTLE_MS), so the pair + * brackets the tap and its rendering; a dispatch failure cleans the frame up + * and rethrows, because a failed action owes no post-action comparison. + */ +export async function dispatchWithScreenshotPair( + runtime: AgentDeviceRuntime, + options: CommandContext, + requested: boolean, + dispatch: () => Promise, +): Promise<{ result: T; outcome?: ScreenshotPairOutcome }> { + const before = requested + ? await capturePairScreenshotFrame(runtime, options, 'before') + : undefined; + let result: T; + try { + result = await dispatch(); + } catch (error) { + // The evidence lane owns nothing after a resolved dispatch, and a failed + // dispatch rethrows its own error even if the frame cleanup fails. + if (before) await before.frame.cleanup().catch(() => {}); + throw error; + } + return { + result, + ...(before ? { outcome: await compareScreenshotPair(runtime, options, before) } : {}), + }; +} + +/** + * Folds the pair outcome into the runtime result: the evidence rides as + * `screenshotDiff`, and the unmet expectation composes its warning after any + * resolution warning the target already carries, so neither disclosure clobbers + * the other. The warning is read structurally: only the point/ref/selector + * variants carry one, and the point variant's is a conditional spread. + */ +export function attachScreenshotPairOutcome( + result: T, + outcome: ScreenshotPairOutcome | undefined, +): T { + if (!outcome) return result; + const { evidence, unchangedWarning } = outcome; + return { + ...result, + ...(evidence ? { screenshotDiff: evidence } : {}), + ...(unchangedWarning + ? { warning: composeWarnings(readResolutionWarning(result), unchangedWarning) } + : {}), + }; +} + +function readResolutionWarning(result: object): string | undefined { + const warning = (result as { warning?: unknown }).warning; + return typeof warning === 'string' && warning.length > 0 ? warning : undefined; +} + +function composeWarnings(existing: string | undefined, added: string): string { + return existing ? `${existing} ${added}` : added; +} + +/** + * One frame of the pair, with the status-bar band the capture owner declared + * for this device. A backend without `captureScreenshot` returns no frame, and + * the whole readback silently does not run — the flag is advisory, so a + * screenshot-less platform keeps the dispatch untouched. + */ +async function capturePairScreenshotFrame( + runtime: AgentDeviceRuntime, + options: CommandContext, + label: 'before' | 'after', +): Promise<{ frame: TemporaryFile; statusBarBand: number } | undefined> { + if (!runtime.backend.captureScreenshot) return undefined; + let frame: TemporaryFile; + try { + frame = await createCommandTempFile(runtime, { + prefix: `agent-device-press-${label}`, + ext: '.png', + }); + } catch { + return undefined; + } + let statusBarBand = 0; + try { + const facts = await runtime.backend.captureScreenshot( + toBackendContext(runtime, options), + frame.path, + ); + if (typeof facts?.statusBarTopHeightShare === 'number') { + statusBarBand = facts.statusBarTopHeightShare; + } + } catch { + await frame.cleanup(); + return undefined; + } + return { frame, statusBarBand }; +} + +/** + * Closes the pair: one after-frame after a brief settle, then the pixel + * comparison. A frame that changed shape (rotation, split-screen) changed the + * screen by construction — the ratio is omitted because the two frames + * describe different viewports. Any decode or capture failure drops the + * evidence rather than guessing. + */ +async function compareScreenshotPair( + runtime: AgentDeviceRuntime, + options: CommandContext, + before: { frame: TemporaryFile; statusBarBand: number }, +): Promise { + // Nothing after a resolved dispatch may throw out of this lane: a decode, + // read, import or cleanup failure yields no evidence, because a throwing + // readback would report a delivered tap as failed and a PIN caller would + // re-enter the digit. + try { + return await compareScreenshotPairInner(runtime, options, before); + } catch { + return {}; + } +} + +async function compareScreenshotPairInner( + runtime: AgentDeviceRuntime, + options: CommandContext, + before: { frame: TemporaryFile; statusBarBand: number }, +): Promise { + try { + // A request canceled after the dispatch stops waiting for the after-frame, + // and an abort that landed during the settle ends the pair with no + // evidence — a canceled caller gets no claims, in either direction. + await sleep(runtime, AFTER_FRAME_SETTLE_MS, options.signal); + if (options.signal?.aborted) return {}; + const after = await capturePairScreenshotFrame(runtime, options, 'after'); + if (!after) return {}; + try { + const [{ computePngChangedPixelRatio }, { decodePngAsync }] = await Promise.all([ + import('@agent-device/capture-kit/png-changed-pixel-ratio'), + import('@agent-device/capture-kit/png-worker-client'), + ]); + const [beforeBytes, afterBytes] = await Promise.all([ + fs.readFile(before.frame.path), + fs.readFile(after.frame.path), + ]); + // The decodes run in the codec worker: full-resolution frames never sit + // on the event loop. + const [beforePng, afterPng] = await Promise.all([ + decodePngAsync(beforeBytes, 'the press before-screenshot'), + decodePngAsync(afterBytes, 'the press after-screenshot'), + ]); + // The device's own status bar moves without the dispatch causing it (a + // clock ticks), so the band the capture owner declared for this device + // stays out of the comparison, and the ratio runs over the same + // compared region. A platform whose owner declares no band compares the + // whole frame. + const ratio = computePngChangedPixelRatio(beforePng, afterPng, { + ignoreTopHeightShare: before.statusBarBand, + }); + if (ratio.status === 'dimension_mismatch') { + return { evidence: { changedFromBefore: true } }; + } + const evidence: ScreenshotDiffEvidence = { + changedFromBefore: ratio.changedPixelRatio > 0, + changedPixelRatio: ratio.changedPixelRatio, + }; + return { + evidence, + ...(evidence.changedFromBefore + ? {} + : { + unchangedWarning: + 'Expected a screen change, but the screenshots taken before and after the dispatch match outside the device status bar. The dispatch may have had no visible effect: check the target coordinates against a fresh snapshot.', + }), + }; + } finally { + await after.frame.cleanup().catch(() => {}); + } + } finally { + await before.frame.cleanup().catch(() => {}); + } +} diff --git a/packages/command-surface/src/command-flags.ts b/packages/command-surface/src/command-flags.ts index fc2be6a74f..9ae10afc59 100644 --- a/packages/command-surface/src/command-flags.ts +++ b/packages/command-surface/src/command-flags.ts @@ -100,6 +100,7 @@ function buildFlags(options: InternalRequestOptions): CommandFlags { maxPasses: options.maxPasses, doubleTap: options.doubleTap, verify: options.verify, + expectScreenshotDiff: options.expectScreenshotDiff, settle: options.settle, settleQuietMs: options.settleQuietMs, clickButton: options.clickButton, diff --git a/packages/command-surface/src/interaction/index.ts b/packages/command-surface/src/interaction/index.ts index d6d31713f5..7d458ffb8b 100644 --- a/packages/command-surface/src/interaction/index.ts +++ b/packages/command-surface/src/interaction/index.ts @@ -82,6 +82,21 @@ const interactionEvidenceSchema: JsonSchema = objectSchema( ['nodeCount', 'interactiveNodeCount', 'digest'], ); +// ScreenshotDiffEvidence (packages/contracts/src/interaction.ts) — opt-in +// `--expect-screenshot-diff` out-of-tree readback: pixels instead of nodes. +const screenshotDiffSchema: JsonSchema = objectSchema( + { + changedFromBefore: booleanSchema( + 'Whether any pixel moved between the frames bracketing the dispatch, outside the device status bar.', + ), + changedPixelRatio: numberSchema( + 'Share of pixels outside the device status bar that moved between the frames. Absent when not comparable.', + ), + }, + ['changedFromBefore'], + 'Pixel comparison bracketing the dispatch. Absent when the pair was not taken.', +); + const responseCostSchema: JsonSchema = objectSchema( { wallClockMs: numberSchema('Total wall-clock time for the request in milliseconds.'), @@ -213,11 +228,12 @@ function interactionResponseDataSchema(extra: InteractionExtra = {}): JsonSchema const tapInteractionResponseDataSchema = interactionResponseDataSchema({ properties: { evidence: interactionEvidenceSchema, + // Press/click only: the pixel pair brackets the dispatch, which fill's + // value-level verification and the hold/hover results never take. + screenshotDiff: screenshotDiffSchema, button: enumSchema(['secondary', 'middle']), count: numberSchema('Number of press/click repetitions.'), - intervalMs: numberSchema( - 'Delay between repeated press/click actions, or between the keys of a comma-separated ref sequence.', - ), + intervalMs: numberSchema('Delay between repeated press/click actions.'), holdMs: numberSchema('Hold duration for each action.'), jitterPx: numberSchema( 'Deterministic per-axis coordinate jitter in pixels (each press offsets x and y by up to this amount).', @@ -390,6 +406,7 @@ const interactionCliSchemas = { allowedFlags: [ ...REPEATED_TOUCH_FLAGS, 'clickButton', + 'expectScreenshotDiff', ...postActionObservationCliFlags('click'), ...SELECTOR_SNAPSHOT_FLAGS, ], @@ -401,6 +418,7 @@ const interactionCliSchemas = { allowsExtraPositionals: true, allowedFlags: [ ...REPEATED_TOUCH_FLAGS, + 'expectScreenshotDiff', ...postActionObservationCliFlags('press'), ...SELECTOR_SNAPSHOT_FLAGS, ], @@ -498,7 +516,7 @@ const pressCommandFacet = defineCommandFacet({ summary: 'Short-press a UI target', searchHint: 'click touch button', cliDetail: - 'The hold duration is positional on longpress, not press --hold-ms. A comma-separated ref plan (press @e13,@e14,@e15,@e16; 2 to 16 refs, repeats allowed) resolves and admits every key against the current frame, then dispatches the admitted points: key 1 expires the frame, later keys dispatch where the snapshot showed them; a later refusal carries details.dispatchedSteps, details.refPlanKey, and details.refPlanPending, and sequence.points holds the dispatched points. --interval-ms sleeps between keys; --count, --hold-ms, --jitter-px, and --double-tap apply to every key; the response ref is the first key. Recording refuses the plan.', + 'The hold duration is positional on longpress, not press --hold-ms. A comma-separated ref plan (press @e13,@e14; 2 to 16 refs, repeats allowed) admits every key before the first press, then dispatches the admitted points: key 1 expires the frame, later keys tap where the snapshot showed them, and a later refusal carries the keys already sent. --interval-ms sleeps between keys; the response ref is the first key. Recording refuses the plan.', }, metadata: metadata('press'), run: (client, input) => client.interactions.press(toPressOptions(input)), @@ -716,6 +734,7 @@ function toPressOptions(input: PressInput): PressOptions { ...toSelectorSnapshotOptions(input), ...toRepeatedOptions(input), verify: input.verify, + expectScreenshotDiff: input.expectScreenshotDiff, ...toSettleOptions(input), }; } diff --git a/packages/command-surface/src/interaction/interactions.ts b/packages/command-surface/src/interaction/interactions.ts index 298ee246aa..6e3316b9e0 100644 --- a/packages/command-surface/src/interaction/interactions.ts +++ b/packages/command-surface/src/interaction/interactions.ts @@ -42,6 +42,7 @@ export const interactionCliReaders = { ...readInteractionTargetFromPositionals(positionals), button: flags.clickButton, verify: flags.verify, + expectScreenshotDiff: flags.expectScreenshotDiff, }), press: (positionals, flags) => ({ ...commonInputFromFlags(flags), @@ -49,6 +50,7 @@ export const interactionCliReaders = { ...repeatedInputFromFlags(flags), ...readInteractionTargetFromPositionals(positionals), verify: flags.verify, + expectScreenshotDiff: flags.expectScreenshotDiff, }), longpress: (positionals, flags) => { const decoded = readLongPressTargetFromPositionals(positionals); diff --git a/packages/command-surface/src/interaction/metadata.ts b/packages/command-surface/src/interaction/metadata.ts index f6c4471845..fd8ad5647d 100644 --- a/packages/command-surface/src/interaction/metadata.ts +++ b/packages/command-surface/src/interaction/metadata.ts @@ -65,9 +65,9 @@ const FIND_ACTION_VALUES = [ const interactionCommandDescriptions = { click: - "Activate a UI target by snapshot ref, selector, or coordinates. This can change app state. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. With settle, continue from the returned diff. Copy refs exactly, including any ~sN suffix.", + "Activate a UI target by snapshot ref, selector, or coordinates. This can change app state. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. Copy refs exactly, including any ~sN suffix.", press: - "Short-press a UI target by snapshot ref, selector, or coordinates. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was. A comma-separated ref list dispatches 2-16 keys in order as one command, admitted before the first press.", + "Short-press a UI target by snapshot ref, selector, or coordinates. Success reports dispatch only; settle returns the resulting UI diff and verify returns cheap post-action evidence. A ref or selector target is re-checked for motion before dispatch when the session's previous action just animated the screen, and is dispatched at its resting position; one that has not stopped refuses with target_still_moving rather than acting where it was.", fill: 'Replace text in a UI input selected by snapshot ref, selector, or coordinates. An empty text clears the field. Unconfirmed evidence does not establish the resulting value or screen. The recordAs option keeps sensitive text out of a recorded replay while sending it to the live app. To append instead of replacing, use type.', longpress: 'Hold a UI target by snapshot ref, selector, or coordinates to open a context menu or perform another hold gesture. The durationMs input overrides the default hold duration.', @@ -88,9 +88,13 @@ const interactionCommandDescriptions = { type InteractionCommandName = keyof typeof interactionCommandDescriptions; +const expectScreenshotDiffField = () => + booleanField('Screenshot pixel-diff readback; a match adds a warning.'); + const clickFields = { ...interactionTargetFields(), button: enumField(CLICK_BUTTONS, 'Pointer button for platforms that support mouse buttons.'), + expectScreenshotDiff: expectScreenshotDiffField(), ...selectorSnapshotFields(), ...repeatedFields(), ...postActionObservationFields('click'), @@ -98,6 +102,7 @@ const clickFields = { const pressFields = { ...interactionTargetFields(), + expectScreenshotDiff: expectScreenshotDiffField(), ...selectorSnapshotFields(), ...repeatedFields(), ...postActionObservationFields('press'), diff --git a/packages/contracts/src/cli-flags.ts b/packages/contracts/src/cli-flags.ts index bab8ee1662..ef8e69c1ca 100644 --- a/packages/contracts/src/cli-flags.ts +++ b/packages/contracts/src/cli-flags.ts @@ -126,6 +126,8 @@ export type CliFlags = CloudProviderProfileFields & maxPasses?: number; doubleTap?: boolean; verify?: boolean; + /** Press/click opt-in; the owning flag definition carries the semantics. */ + expectScreenshotDiff?: boolean; settle?: boolean; settleQuietMs?: number; postGestureStabilization?: boolean; diff --git a/packages/contracts/src/client-gesture.ts b/packages/contracts/src/client-gesture.ts index c053d52158..040171cdb2 100644 --- a/packages/contracts/src/client-gesture.ts +++ b/packages/contracts/src/client-gesture.ts @@ -63,6 +63,14 @@ export type ClickOptions = DeviceCommandBaseOptions & * snapshot to confirm the action had an effect. */ verify?: boolean; + /** + * Opt-in: bracket the dispatch with two lightweight screenshots and return + * the pixel comparison as `screenshotDiff` in the response. For targets the + * accessibility tree cannot describe (a PIN keypad), where `verify` would + * compare empty trees. Advisory — never fails the action; a matching pair + * adds a response warning instead. + */ + expectScreenshotDiff?: boolean; }; export type PressOptions = DeviceCommandBaseOptions & @@ -71,6 +79,8 @@ export type PressOptions = DeviceCommandBaseOptions & RepeatedPressOptions & SettleCommandOptions & { verify?: boolean; + /** See {@link ClickOptions.expectScreenshotDiff}. */ + expectScreenshotDiff?: boolean; }; export type LongPressOptions = DeviceCommandBaseOptions & diff --git a/packages/contracts/src/interaction-guarantees.ts b/packages/contracts/src/interaction-guarantees.ts index b6010dbc8f..e9d745001b 100644 --- a/packages/contracts/src/interaction-guarantees.ts +++ b/packages/contracts/src/interaction-guarantees.ts @@ -63,6 +63,13 @@ export const INTERACTION_GUARANTEES = [ // settled tree was stored) in the same response. Best-effort: never fails // the action. 'settleObservation', + // --expect-screenshot-diff brackets the dispatch with two lightweight + // screenshots and compares pixels: the out-of-tree readback for surfaces the + // accessibility tree cannot describe (a PIN keypad), where --verify's node + // digest stays flat no matter what the tap did. Advisory by the same contract + // as settleObservation — the comparison never fails the action, and a + // matching pair adds a warning instead. + 'screenshotDiffEvidence', // Failures use the shared codes/messages/hints (no-match diagnostics, // ambiguous shape, offscreen reasons). NOTE: expected to split into // errorCodes (stable codes / fallback classification) vs errorDiagnostics @@ -186,7 +193,7 @@ const SHARED_RESPONSE_CONSTRUCTION: GuaranteeEnforcement = { const TAP_OUTCOME_NOT_OBSERVED_GAP: GuaranteeEnforcement = { kind: 'waived', reason: - 'gap: the response reports the dispatch only; only opt-in --verify/--settle capture post-action evidence into it. The deferred marks set after dispatch (Android snapshot freshness after press/click, post-gesture stabilization when the request sets postGestureStabilization) are judged by the next capture, never in this response. An iOS open that launched an app (any postOpenObservation) marks the next capture in that session to wait for the first screen of that app (markPostOpenStabilization); that steadies where a target is resolved, not whether the tap landed. The iOS ambiguous-failure corroboration reconsiders only a thrown runner error.', + 'gap: the response reports the dispatch only; only the opt-in --verify, --settle, and --expect-screenshot-diff features capture post-action evidence into it. The deferred marks set after dispatch (Android snapshot freshness after press/click, post-gesture stabilization when the request sets postGestureStabilization) are judged by the next capture, never in this response. An iOS open that launched an app (any postOpenObservation) marks the next capture in that session to wait for the first screen of that app (markPostOpenStabilization); that steadies where a target is resolved, not whether the tap landed. The iOS ambiguous-failure corroboration reconsiders only a thrown runner error.', trackingIssue: TAP_OUTCOME_NOT_OBSERVED_GAP_ISSUE, }; @@ -199,7 +206,7 @@ const TAP_OUTCOME_NOT_OBSERVED_GAP: GuaranteeEnforcement = { const DIRECT_IOS_OUTCOME_NOT_OBSERVED_GAP: GuaranteeEnforcement = { kind: 'waived', reason: - "gap: this replay-only route observes no outcome beyond the runner's own report; --verify/--settle do not apply here (see the inapplicable cells on this row), and the shared ambiguous-failure corroboration only reconsiders a thrown error, never a successful dispatch.", + "gap: this replay-only route observes no outcome beyond the runner's own report; --verify/--settle/--expect-screenshot-diff do not apply here (see the inapplicable cells on this row), and the shared ambiguous-failure corroboration only reconsiders a thrown error, never a successful dispatch.", trackingIssue: TAP_OUTCOME_NOT_OBSERVED_GAP_ISSUE, }; @@ -280,6 +287,11 @@ const RUNTIME_TREE_SHARED_GUARANTEES = { kind: 'runtime', via: 'packages/command-runtime/src/commands/interaction/runtime/settle.ts#settleAfterInteraction', }, + screenshotDiffEvidence: { + kind: 'runtime', + via: 'packages/command-runtime/src/commands/interaction/runtime/screenshot-diff-evidence.ts#dispatchWithScreenshotPair', + appliesTo: ['press', 'click'], + }, targetStability: RUNTIME_TREE_TARGET_STABILITY, } satisfies Partial>; @@ -379,6 +391,11 @@ export const INTERACTION_DISPATCH_PATHS: Record; export type ElementSelectorKey = 'id' | 'label' | 'text' | 'value'; diff --git a/packages/contracts/src/request-envelope.ts b/packages/contracts/src/request-envelope.ts index 8882de1551..cba23c197c 100644 --- a/packages/contracts/src/request-envelope.ts +++ b/packages/contracts/src/request-envelope.ts @@ -45,6 +45,7 @@ export type CommandExecutionOptions = Partial & maxPasses?: number; doubleTap?: boolean; verify?: boolean; + expectScreenshotDiff?: boolean; settle?: boolean; settleQuietMs?: number; clickButton?: ClickButton; diff --git a/packages/contracts/src/snapshot-types.ts b/packages/contracts/src/snapshot-types.ts index dd843f06a3..47acf36115 100644 --- a/packages/contracts/src/snapshot-types.ts +++ b/packages/contracts/src/snapshot-types.ts @@ -20,6 +20,11 @@ export type ScreenshotResultData = { pixelDensity?: number; /** Display rotation the capture was rendered in; absent when the target reports none. */ displayRotation?: DeviceRotation; + /** + * The status-bar band the capture owner declares for this device (the share of the frame the + * device redraws between frames); absent where the platform has no status bar. + */ + statusBarTopHeightShare?: number; overlayRefs?: ScreenshotOverlayRef[]; warnings?: string[]; }; diff --git a/packages/daemon/package.json b/packages/daemon/package.json index 2558eb9069..696c0f39c2 100644 --- a/packages/daemon/package.json +++ b/packages/daemon/package.json @@ -9,6 +9,7 @@ "@agent-device/ad-script": "workspace:*", "@agent-device/capture-kit": "workspace:*", "@agent-device/command-registry": "workspace:*", + "@agent-device/command-runtime": "workspace:*", "@agent-device/contracts": "workspace:*", "@agent-device/daemon-contracts": "workspace:*", "@agent-device/device-selection": "workspace:*", diff --git a/packages/daemon/src/core/interaction/internal/interaction-touch-direct-ios-eligibility.ts b/packages/daemon/src/core/interaction/internal/interaction-touch-direct-ios-eligibility.ts index 66f4843e83..a40a77a874 100644 --- a/packages/daemon/src/core/interaction/internal/interaction-touch-direct-ios-eligibility.ts +++ b/packages/daemon/src/core/interaction/internal/interaction-touch-direct-ios-eligibility.ts @@ -33,6 +33,9 @@ export function readDirectIosSelectorTapTarget(params: { if (hasNonDefaultClickOptions(flags)) return null; if (commandSupportsVerifyEvidence(commandLabel) && flags?.verify === true) return null; if (commandSupportsSettleObservation(commandLabel) && flags?.settle === true) return null; + // The screenshot pair brackets the runtime dispatch; this fused route never + // reaches it, so the flag cannot silently no-op here. + if (flags?.expectScreenshotDiff === true) return null; return readDirectSelectorWithMaestroFallback(session, target.selector, flags); } diff --git a/packages/daemon/src/core/interaction/internal/types.ts b/packages/daemon/src/core/interaction/internal/types.ts index a3a015f13a..d77e4590d5 100644 --- a/packages/daemon/src/core/interaction/internal/types.ts +++ b/packages/daemon/src/core/interaction/internal/types.ts @@ -16,6 +16,10 @@ import type { BoundContextFromFlags } from '../../context.ts'; import type { DaemonInvokeFn, DaemonRequest } from '../../daemon-request.ts'; import type { SessionRef, SessionState } from '../../session-state.ts'; import type { BoundGestureExecutor } from '../../gesture-runtime.ts'; +import type { + CaptureScreenshotInput, + ScreenshotCaptureFacts, +} from '@agent-device/contracts/screenshot-runtime'; import type { BoundTouchExecutor } from '../../touch-runtime.ts'; import type { BoundSnapshotCapture } from '../../snapshot-runtime-binding.ts'; import type { recordTouchVisualizationEvent } from '../../recording-gestures.ts'; @@ -127,6 +131,11 @@ export type InteractionRuntimeInput = { pairedGestureViewport?: Rect; touchExecutor?: BoundTouchExecutor; gestures?: BoundGestureExecutor; + /** + * The admitted screenshot capture behind `--expect-screenshot-diff`; present exactly when + * the route admitted the flag and the owner declared the capture operation. + */ + readbackCapture?: (input: CaptureScreenshotInput) => Promise; /** * The capture every observation on this request reads the screen through: the action binding's * own capture when its runtime use carries one, otherwise the capture the request admitted for diff --git a/packages/daemon/src/execution/__tests__/interaction-get-runtime-fixture.ts b/packages/daemon/src/execution/__tests__/interaction-get-runtime-fixture.ts index 443ace358e..7cb964b2cf 100644 --- a/packages/daemon/src/execution/__tests__/interaction-get-runtime-fixture.ts +++ b/packages/daemon/src/execution/__tests__/interaction-get-runtime-fixture.ts @@ -13,6 +13,10 @@ import { } from '@agent-device/contracts/platform-runtime'; import type { PlatformRuntimeOperations } from '@agent-device/contracts/platform-runtime-operations'; import type { CaptureSnapshotInput } from '@agent-device/contracts/snapshot-runtime'; +import type { + CaptureScreenshotInput, + ScreenshotCaptureFacts, +} from '@agent-device/contracts/screenshot-runtime'; import type { TypeTextInput } from '@agent-device/contracts/type-text-runtime'; import { type FillPointInput, @@ -69,6 +73,14 @@ export const mockTapPoint = vi.fn( y: input.point.y, }), ); +/** + * The owner's captureScreenshot operation, advertised only when + * `elementReadFixtureState.captureScreenshotAvailable` flips on. The default writes nothing — + * a readback test owns its frames by mocking this with its own frame plan. + */ +export const mockCaptureScreenshot = vi.fn( + async (_input: CaptureScreenshotInput): Promise => ({}), +); const mockTapRef = vi.fn(async (input: TapRefInput) => ({ ref: input.ref.replace(/^@/, ''), })); @@ -118,6 +130,8 @@ export const elementReadFixtureState = { tapElementSelectorAvailable: true, /** Off by default: only the tests pinning the #1542 rescue wiring advertise the live read. */ querySelectorAvailable: false, + /** Off by default: only the readback tests advertise a capture-capable owner. */ + captureScreenshotAvailable: false, }; export function resetGetRuntimeFixture(): void { @@ -161,8 +175,11 @@ export function resetGetRuntimeFixture(): void { elementReadFixtureState.typeTextAvailable = true; elementReadFixtureState.tapElementSelectorAvailable = true; elementReadFixtureState.querySelectorAvailable = false; + elementReadFixtureState.captureScreenshotAvailable = false; mockQuerySelector.mockReset(); mockQuerySelector.mockResolvedValue({ found: false }); + mockCaptureScreenshot.mockReset(); + mockCaptureScreenshot.mockImplementation(async () => ({})); } const available = Object.freeze({ available: true } as const); @@ -182,6 +199,9 @@ function elementReadFacts(device: DeviceInfo): RuntimeFacts { @@ -14,3 +15,19 @@ test('createDaemonRuntimePolicy uses local policy and unsupported artifacts', as runtimePolicy.artifacts.createTempFile({ prefix: 'snapshot', ext: '.png' }), ).rejects.toThrow(/snapshot does not create temporary files/); }); + +test('allowTempFiles swaps in the daemon temp file and cleanup removes its directory', async () => { + const runtimePolicy = createDaemonRuntimePolicy('interaction commands', { + plural: true, + allowTempFiles: true, + }); + + const frame = await runtimePolicy.artifacts.createTempFile({ + prefix: 'agent-device-press-before', + ext: '.png', + }); + await fs.writeFile(frame.path, 'png-bytes'); + expect(frame.visibility).toBe('internal'); + await frame.cleanup(); + await expect(fs.readFile(frame.path)).rejects.toMatchObject({ code: 'ENOENT' }); +}); diff --git a/packages/daemon/src/execution/interaction/internal/interaction-runtime.ts b/packages/daemon/src/execution/interaction/internal/interaction-runtime.ts index 701534bd09..d3de131153 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-runtime.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-runtime.ts @@ -6,6 +6,7 @@ import { AppError as KernelAppError } from '@agent-device/kernel/errors'; import type { AgentDeviceBackend, BackendActionResult, + BackendScreenshotOptions, BackendSnapshotResult, } from '@agent-device/contracts/backend'; import { getRequestSignal } from '@agent-device/host-kit/request'; @@ -25,6 +26,10 @@ import type { InteractionRuntimeInput, SettleObservationCaptureInput, } from '../../../core/interaction/internal/types.ts'; +import type { + CaptureScreenshotInput, + ScreenshotCaptureFacts, +} from '@agent-device/contracts/screenshot-runtime'; import { finalizeTouchInteraction as finalizeInteraction } from './interaction-common.ts'; import { markDeferredInteractionOutcome } from '../../../core/deferred-interaction-outcome.ts'; import { isSessionRecording } from '../../../core/session-script-publication-capability.ts'; @@ -43,6 +48,11 @@ export async function createInteractionRuntimeForRoute( pairedGestureViewport?: Rect; touchExecutor?: BoundTouchExecutor; gestures?: BoundGestureExecutor; + /** + * The admitted screenshot capture behind `--expect-screenshot-diff`; present exactly + * when the route admitted the flag and the owner declared the operation. + */ + readbackCapture?: (input: CaptureScreenshotInput) => Promise; /** * Filled with the LAST capture this request consumed. ADR 0014 ambiguity * issuance is only valid against a tree the session stored under the @@ -60,6 +70,7 @@ export async function createInteractionRuntimeForRoute( createCommandSurface: params.createCommandSurface, requestId: params.req.meta?.requestId, flags: params.req.flags, + ...(params.readbackCapture ? { readbackCapture: params.readbackCapture } : {}), session, contextFromFlags: params.contextFromFlags, captureSnapshot: async (flags, options) => { @@ -185,7 +196,11 @@ function resolveBoundCapture( function createInteractionAgentDevice(params: InteractionRuntimeInput) { return params.createCommandSurface({ backend: createInteractionBackend(params), - ...createDaemonRuntimePolicy('interaction commands', { plural: true }), + ...createDaemonRuntimePolicy('interaction commands', { + plural: true, + // The screenshot readback's before/after frames are this route's own temp files. + allowTempFiles: true, + }), sessions: params.runtimeSessions, signal: getRequestSignal(params.requestId), }); @@ -193,10 +208,25 @@ function createInteractionAgentDevice(params: InteractionRuntimeInput) { function createInteractionBackend(params: InteractionRuntimeInput): AgentDeviceBackend { const { flags, session } = params; + const readbackCapture = params.readbackCapture; const gestureContext = () => params.contextFromFlags(flags, session.appBundleId, session.trace?.outPath); return { platform: publicPlatformString(session.device), + // The admitted screenshot capture behind --expect-screenshot-diff (#press-expect): the + // backend member exists exactly when the route admitted and bound it. + ...(readbackCapture + ? { + captureScreenshot: async ( + _context: unknown, + outPath: string, + options?: BackendScreenshotOptions, + ) => { + await readbackCapture({ outPath, ...(options ? { options } : {}) }); + return { path: outPath }; + }, + } + : {}), captureSnapshot: async (context, options): Promise => ({ snapshot: await params.captureSnapshot(flags, { interactiveOnly: options?.interactiveOnly === true, diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-direct-ios-eligibility.test.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-direct-ios-eligibility.test.ts index 7c27f2f5e3..a4007d860e 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-direct-ios-eligibility.test.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-direct-ios-eligibility.test.ts @@ -193,6 +193,64 @@ test('click simple iOS selector forwards Maestro non-hittable coordinate fallbac } }); +test('click with expectScreenshotDiff leaves the fused iOS route for the runtime path', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'ios-direct-excluded-by-readback'; + sessionStore.publish( + sessionName, + makeIosSession(sessionName, { appBundleId: 'com.example.app' }), + ); + // The runtime path resolves against the canonical capture, so the tree the + // capture returns must hold the node the selector names. + mockInteractorSnapshot.mockResolvedValueOnce({ + nodes: attachRefs([ + { + index: 0, + type: 'Application', + rect: { x: 0, y: 0, width: 390, height: 844 }, + }, + { + index: 1, + parentIndex: 0, + type: 'Button', + identifier: 'hiddenTestLogin', + rect: { x: 40, y: 80, width: 100, height: 40 }, + enabled: true, + hittable: true, + }, + ]), + backend: 'xctest', + producer: 'apple-runner', + }); + mockTapElementSelector.mockResolvedValue({ message: 'fused route tapped' }); + mockTapPoint.mockResolvedValue({ x: 90, y: 100 }); + + // The screenshot pair brackets the runtime dispatch, so the flag must keep + // this click off the fused route even when every other eligibility guard + // would admit it — otherwise the readback would silently no-op. + const response = await handleInteractionCommands({ + createCommandSurface: touchCreateCommandSurface, + req: { + token: 't', + session: sessionName, + command: 'click', + positionals: ['id="hiddenTestLogin"'], + flags: { + maestro: { allowNonHittableCoordinateFallback: true }, + expectScreenshotDiff: true, + }, + }, + sessionName, + sessionStore, + contextFromFlags, + ...getRuntimeBindings(), + }); + + expect(response?.ok).toBe(true); + expect(mockTapElementSelector).not.toHaveBeenCalled(); + expect(mockTapPoint).toHaveBeenCalledTimes(1); +}); + test('click simple iOS id selector waits for snapshot path after pending gesture stabilization', async () => { const sessionStore = makeSessionStore(); const sessionName = 'ios-direct-selector-after-swipe'; diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts index 410c2cdf41..9d7e9633f9 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-press-admission.test.ts @@ -1,8 +1,14 @@ import { test, expect, beforeEach } from 'vitest'; +import fs from 'node:fs'; +import { PNG } from '@agent-device/capture-kit/png'; +import type { CommandFlags } from '@agent-device/contracts/command'; +import { screenshotFlagsFromOptions } from '@agent-device/contracts/capture'; +import { createCommandSurfaceAgentDevice } from '@agent-device/command-runtime'; import { attachRefs } from '@agent-device/kernel/snapshot'; +import type { SessionStore } from '../../../core/session-store.ts'; import { makeSessionStore } from '../../../__tests__/support/store-factory.ts'; -import { activateCompleteRefFrame, refFrameState } from '../../../core/ref-frame.ts'; import { armAuthoring } from '../../../core/session-script-publication-state.ts'; +import { activateCompleteRefFrame, refFrameState } from '../../../core/ref-frame.ts'; import { setSessionSnapshot, STALE_SNAPSHOT_REFS_WARNING } from '../../session-snapshot.ts'; import { installInteractorSnapshotResolution, @@ -11,7 +17,9 @@ import { import { touchCreateCommandSurface } from '../../../__tests__/support/touch-command-surface-fake.ts'; import { runInteraction } from '../../../__tests__/support/touch-interaction-runner.ts'; import { + elementReadFixtureState, getRuntimeBindings, + mockCaptureScreenshot, mockTapPoint, resetGetRuntimeFixture, } from '../../__tests__/interaction-get-runtime-fixture.ts'; @@ -33,6 +41,18 @@ beforeEach(() => { mockInteractorSnapshot.mockReset(); }); +/** Writes an 8x8 solid PNG of one gray level, the smallest pair the pixel compare can read. */ +function writeSolidPng(outPath: string, level: number): void { + const png = new PNG({ width: 8, height: 8 }); + for (let i = 0; i < png.data.length; i += 4) { + png.data[i] = level; + png.data[i + 1] = level; + png.data[i + 2] = level; + png.data[i + 3] = 255; + } + fs.writeFileSync(outPath, PNG.sync.write(png)); +} + test('click --button middle on macOS fails with an explicit unsupported-operation error', async () => { const sessionStore = makeSessionStore(); const sessionName = 'default'; @@ -353,6 +373,120 @@ test('after a session reopen, a pin from the previous lifetime rejects (reseeded expect(mockTapPoint).not.toHaveBeenCalled(); }); +test('press --expect-screenshot-diff on a capture-less device dispatches the tap with no evidence', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'readback-advisory-plan'; + // makeStaleRefSession arms a complete, ACTIVE ref frame: the plan is admitted + // against it, and the capture-less readback simply claims nothing. + sessionStore.publish(sessionName, makeStaleRefSession(sessionName)); + mockTapPoint.mockImplementation(async () => ({})); + + for (const [round, positionals] of [ + ['one', ['@e1']], + ['two', ['@e2']], + ] as const) { + // Each press expires the frame, so every round re-arms the active frame on + // a fresh session address. + const sessionName = `readback-advisory-${round}`; + sessionStore.publish(sessionName, makeStaleRefSession(sessionName)); + const response = await runInteraction(sessionStore, sessionName, 'press', [...positionals], { + expectScreenshotDiff: true, + }); + expect(response?.ok, positionals.join(' ')).toBe(true); + const data = (response?.ok ? response.data : {}) as Record; + expect(data.screenshotDiff).toBeUndefined(); + expect(mockTapPoint).toHaveBeenCalledTimes(1); + mockTapPoint.mockClear(); + } +}); + +// The readback route test runs the REAL command surface: the fake surface owns +// its own press flow, and the pixel pair only forms through the real +// command-runtime lane that dispatchWithScreenshotPair brackets. +const readbackContextFromFlags = (flags: CommandFlags | undefined, appBundleId?: string) => ({ + requestId: 'req-readback', + appBundleId, + ...screenshotFlagsFromOptions(flags), +}); + +async function runReadbackInteraction( + sessionStore: SessionStore, + sessionName: string, + flags: Record, +) { + return await handleInteractionCommands({ + createCommandSurface: async (config) => createCommandSurfaceAgentDevice(config), + req: { token: 't', session: sessionName, command: 'press', positionals: ['30', '40'], flags }, + sessionName, + sessionStore, + contextFromFlags: readbackContextFromFlags, + ...getRuntimeBindings(), + }); +} + +test('press --expect-screenshot-diff captures through the session owner and reports the pixel pair', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'readback-bound-changed'; + const session = makeStaleRefSession(sessionName); + session.appBundleId = 'com.example.app'; + session.surface = 'app'; + sessionStore.publish(sessionName, session); + // The owner declares captureScreenshot, and its frames differ: a delivered + // tap must read changedFromBefore through the whole route. This case is the + // red check for the forwarding — dropping the readbackCapture wiring at + // interaction-touch-press or its handoff into the runtime lane keeps this + // evidence from ever forming, and this test fails. + elementReadFixtureState.captureScreenshotAvailable = true; + mockCaptureScreenshot.mockImplementation(async (input) => { + writeSolidPng(input.outPath, mockCaptureScreenshot.mock.calls.length === 1 ? 0 : 255); + return {}; + }); + mockTapPoint.mockImplementation(async () => ({})); + + const response = await runReadbackInteraction(sessionStore, sessionName, { + expectScreenshotDiff: true, + }); + + expect(response?.ok).toBe(true); + const data = (response?.ok ? response.data : {}) as Record; + const screenshotDiff = data.screenshotDiff as { changedFromBefore?: boolean } | undefined; + expect(screenshotDiff?.changedFromBefore).toBe(true); + expect(mockTapPoint).toHaveBeenCalledTimes(1); + // The readback captured with the input the screenshot command's builder + // builds for this request: the session's app identity, surface, and the + // runner execution from the dispatch context. + expect(mockCaptureScreenshot).toHaveBeenCalledTimes(2); + const captureInput = mockCaptureScreenshot.mock.calls[0]?.[0]; + expect(captureInput?.options?.appBundleId).toBe('com.example.app'); + expect(captureInput?.options?.surface).toBe('app'); + expect(captureInput?.execution?.requestId).toBe('req-readback'); +}); + +test('press --expect-screenshot-diff warns when the session owner captures a matching pair', async () => { + const sessionStore = makeSessionStore(); + const sessionName = 'readback-bound-matching'; + const session = makeStaleRefSession(sessionName); + session.appBundleId = 'com.example.app'; + session.surface = 'app'; + sessionStore.publish(sessionName, session); + elementReadFixtureState.captureScreenshotAvailable = true; + mockCaptureScreenshot.mockImplementation(async (input) => { + writeSolidPng(input.outPath, 0); + return {}; + }); + mockTapPoint.mockImplementation(async () => ({})); + + const response = await runReadbackInteraction(sessionStore, sessionName, { + expectScreenshotDiff: true, + }); + + expect(response?.ok).toBe(true); + const data = (response?.ok ? response.data : {}) as Record; + const screenshotDiff = data.screenshotDiff as { changedFromBefore?: boolean } | undefined; + expect(screenshotDiff?.changedFromBefore).toBe(false); + expect(String(data.warning)).toMatch(/may have had no visible effect/); +}); + // --- Multi-ref press plan: whole-plan admission and the recording guard --- test('a plan one member cannot authorize refuses whole and leaves the frame untouched', async () => { diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts index bd70e4641a..8981d1b77b 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-press.ts @@ -10,6 +10,9 @@ import type { resolveClickButton } from '@agent-device/contracts/click-button'; import type { ReplayTargetGuardDenotation } from '@agent-device/contracts/replay'; import { assertAndroidPressStayedInApp } from './interaction-android-escape.ts'; import { createInteractionRuntimeForRoute } from './interaction-runtime.ts'; +import { resolveBoundScreenshotRuntime } from '../../screenshot-runtime-binding.ts'; +import { captureScreenshotFromRequest } from '../../screenshot-runtime.ts'; +import type { CaptureScreenshotInput } from '@agent-device/contracts/screenshot-runtime'; import { readSettleRequest } from './interaction-flags.ts'; import { dispatchDirectIosSelectorTap } from './interaction-touch-direct-ios.ts'; import { readDirectIosSelectorTapTarget } from '../../../core/interaction/internal/interaction-touch-direct-ios-eligibility.ts'; @@ -43,6 +46,24 @@ export async function dispatchTargetedTouchViaRuntime( const admission = await admitTargetedTouch(params, command); if ('response' in admission) return admission.response; const { admitted } = admission; + const readbackRequested = + command !== 'longpress' && command !== 'hover' && params.req.flags?.expectScreenshotDiff; + const readbackContext = readbackRequested + ? params.contextFromFlags( + params.req.flags, + admitted.session.appBundleId, + admitted.session.trace?.outPath, + ) + : undefined; + const readbackCapture = readbackRequested + ? await resolveBoundScreenshotRuntime({ + device: admitted.session.device, + overlayRefs: false, + inspectFacts: params.inspectFacts, + platformServices: params.platformServices, + bindDevice: params.bindDevice, + }) + : undefined; const prepared = await prepareTouchDispatch( params, admitted.session, @@ -86,6 +107,25 @@ export async function dispatchTargetedTouchViaRuntime( ...buildTargetedRuntimeOptions(params, command, boundAdmitted), touchExecutor, settleObservationCapture: prepared.settleObservationCapture, + ...(readbackCapture?.ok && readbackContext + ? { + // A readback frame is captured by the same builder the screenshot + // command captures through, so the pair's frames come from exactly + // the input that command would build for this request. + readbackCapture: async (input: CaptureScreenshotInput) => + await captureScreenshotFromRequest( + { + session: admitted.session, + dispatchContext: readbackContext, + captureScreenshot: readbackCapture.runtime.captureScreenshot, + }, + input.outPath, + // The request's own surface, the same input the screenshot + // command builds for this session. + { surface: admitted.session.surface }, + ), + } + : {}), }); } @@ -220,6 +260,7 @@ function pressRuntimeOptions( // The multi-ref press plan: present only when the request named a // comma-separated ref sequence. ...(params.refs ? { refs: params.refs } : {}), + expectScreenshotDiff: flags?.expectScreenshotDiff, // Only click/press take it: `find` dispatches click and fill, never // longpress or hover, so declaring it on their options would be an // unconsumed claim (#1649 review). diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-runtime.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-runtime.ts index f07e8efd55..fc8f669a8c 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-runtime.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-runtime.ts @@ -1,5 +1,9 @@ import type { DaemonResponse } from '@agent-device/kernel/contracts'; import { bindInteractionSession } from './interaction-session.ts'; +import type { + CaptureScreenshotInput, + ScreenshotCaptureFacts, +} from '@agent-device/contracts/screenshot-runtime'; import type { FillCommandResult, InteractionTarget, @@ -54,6 +58,13 @@ export async function dispatchRuntimeInteraction< touchExecutor: BoundTouchExecutor; /** The request's own observation capture when its action use carries none (#3510). */ settleObservationCapture?: BoundSnapshotCapture; + /** + * The admitted screenshot capture behind `--expect-screenshot-diff`; present exactly when + * the route admitted the flag and the owner declared the operation. + */ + readbackCapture?: + | ((input: CaptureScreenshotInput) => Promise) + | undefined; androidFreshnessBaseline?: SessionState['snapshot']; /** * Present when the action targets a `@ref`: if Android dialog recovery @@ -82,6 +93,7 @@ export async function dispatchRuntimeInteraction< ...params, touchExecutor: options.touchExecutor, settleObservationCapture: options.settleObservationCapture, + readbackCapture: options.readbackCapture, consumedCapture, }); const actionStartedAt = Date.now(); diff --git a/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts b/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts index 926c6dcb7d..e6b2c8cedb 100644 --- a/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts +++ b/packages/daemon/src/execution/interaction/internal/interaction-touch-targets.ts @@ -294,7 +294,10 @@ export function interactionResultExtra( // (and gets dropped by the response layer) for longpress. `settle` (#1101, // opt-in via --settle) is additive on all four touch commands. `sequence` // is additive on press only (the multi-ref plan summary). + // `screenshotDiff` (--expect-screenshot-diff) is press/click only: fill's + // value-level verification and longpress/hover carry no pixel pair. const evidence = 'evidence' in result ? result.evidence : undefined; + const screenshotDiff = 'screenshotDiff' in result ? result.screenshotDiff : undefined; const settle = result.settle; const sequence = 'sequence' in result ? result.sequence : undefined; const shared = sequence ? { sequence } : {}; @@ -306,6 +309,7 @@ export function interactionResultExtra( targetHittable: result.targetHittable, hint: result.hint, evidence, + screenshotDiff, settle, resolution: result.resolution, ...shared, @@ -319,12 +323,13 @@ export function interactionResultExtra( targetHittable: result.targetHittable, hint: result.hint, evidence, + screenshotDiff, settle, resolution: result.resolution, ...shared, }; } - return { evidence, settle, ...shared }; + return { evidence, screenshotDiff, settle, ...shared }; } export function formatTouchTargetLabel( diff --git a/packages/daemon/src/execution/runtime-artifacts.ts b/packages/daemon/src/execution/runtime-artifacts.ts index dc154f67f9..8faf9d92d0 100644 --- a/packages/daemon/src/execution/runtime-artifacts.ts +++ b/packages/daemon/src/execution/runtime-artifacts.ts @@ -1,6 +1,29 @@ -import type { ArtifactAdapter } from '@agent-device/contracts/artifact-adapter'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; +import type { ArtifactAdapter, TemporaryFile } from '@agent-device/contracts/artifact-adapter'; import { AppError } from '@agent-device/kernel/errors'; +/** + * The one temp-file implementation every daemon route shares: an `agent-device-*` root under the + * process tmpdir, one file inside, removed with the cleanup. Routes whose policy refuses temp + * files keep the refusing adapter; routes that opt in (runtime policy's `allowTempFiles`, the + * screenshot artifact adapter) get this implementation. + */ +export async function createDaemonTempFile(options: { + prefix: string; + ext: string; +}): Promise { + const root = await fs.mkdtemp(path.join(os.tmpdir(), `${options.prefix}-`)); + return { + path: path.join(root, `file${options.ext}`), + visibility: 'internal', + cleanup: async () => { + await fs.rm(root, { recursive: true, force: true }); + }, + }; +} + export function createUnsupportedArtifactAdapter( label: string, options: { plural?: boolean } = {}, diff --git a/packages/daemon/src/execution/runtime-policy.ts b/packages/daemon/src/execution/runtime-policy.ts index 0d85d3ac11..312fce86b5 100644 --- a/packages/daemon/src/execution/runtime-policy.ts +++ b/packages/daemon/src/execution/runtime-policy.ts @@ -1,13 +1,18 @@ import type { AgentDeviceRuntimeConfig } from '@agent-device/contracts/runtime-contract'; import { localCommandPolicy } from '@agent-device/contracts/command-policy'; -import { createUnsupportedArtifactAdapter } from './runtime-artifacts.ts'; +import { createDaemonTempFile, createUnsupportedArtifactAdapter } from './runtime-artifacts.ts'; export function createDaemonRuntimePolicy( unsupportedArtifactLabel: string, - options: { plural?: boolean } = {}, + options: { plural?: boolean; allowTempFiles?: boolean } = {}, ): Pick { return { - artifacts: createUnsupportedArtifactAdapter(unsupportedArtifactLabel, options), + artifacts: options.allowTempFiles + ? { + ...createUnsupportedArtifactAdapter(unsupportedArtifactLabel, options), + createTempFile: createDaemonTempFile, + } + : createUnsupportedArtifactAdapter(unsupportedArtifactLabel, options), policy: localCommandPolicy(), }; } diff --git a/packages/daemon/src/execution/screenshot-artifact-adapter.ts b/packages/daemon/src/execution/screenshot-artifact-adapter.ts index 43086e8d61..765fc1ebc7 100644 --- a/packages/daemon/src/execution/screenshot-artifact-adapter.ts +++ b/packages/daemon/src/execution/screenshot-artifact-adapter.ts @@ -2,6 +2,7 @@ import { publishFileSync } from '@agent-device/host-kit/file'; import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; import type { ArtifactAdapter, ReservedOutputFile } from '@agent-device/contracts/artifact-adapter'; import { AppError, errorMessage } from '@agent-device/kernel/errors'; +import { createDaemonTempFile } from './runtime-artifacts.ts'; import { promises as fs } from 'node:fs'; import os from 'node:os'; import path from 'node:path'; @@ -51,16 +52,7 @@ export function createDaemonScreenshotArtifactAdapter( ); } }, - createTempFile: async (options) => { - const root = await fs.mkdtemp(path.join(os.tmpdir(), `${options.prefix}-`)); - return { - path: path.join(root, `file${options.ext}`), - visibility: 'internal', - cleanup: async () => { - await fs.rm(root, { recursive: true, force: true }); - }, - }; - }, + createTempFile: createDaemonTempFile, }; } diff --git a/packages/daemon/src/execution/screenshot-runtime.ts b/packages/daemon/src/execution/screenshot-runtime.ts index ab094426c2..d0bbb1d928 100644 --- a/packages/daemon/src/execution/screenshot-runtime.ts +++ b/packages/daemon/src/execution/screenshot-runtime.ts @@ -8,11 +8,14 @@ import { screenshotOptionsFromFlags, } from '@agent-device/contracts/capture'; import type { DeviceRotation } from '@agent-device/contracts/device'; -import type { ScreenshotRuntimeExecution } from '@agent-device/contracts/screenshot-runtime'; +import type { + ScreenshotCaptureFacts, + ScreenshotRuntimeExecution, +} from '@agent-device/contracts/screenshot-runtime'; import { isIosFamily, publicPlatformString } from '@agent-device/kernel/device'; import { AppError } from '@agent-device/kernel/errors'; import type { ScreenshotOverlayRef } from '@agent-device/kernel/snapshot'; -import type { AgentDeviceBackend } from '@agent-device/contracts/backend'; +import type { AgentDeviceBackend, BackendScreenshotOptions } from '@agent-device/contracts/backend'; import { localCommandPolicy } from '@agent-device/contracts/command-policy'; import type { CreateDaemonCommandSurface } from '@agent-device/contracts/daemon-command-surface'; import { @@ -369,34 +372,11 @@ function createBoundScreenshotBackend( crop?: ScreenshotCropBinding; }>, ): AgentDeviceBackend { - const { session, dispatchContext, captureScreenshot, crop } = params; + const { session, dispatchContext, crop } = params; return { platform: publicPlatformString(session.device), captureScreenshot: async (_context, outPath, options) => { - const resolved = screenshotOptionsFromFlags({ - ...dispatchContext, - ...screenshotFlagsFromOptions(options), - }); - const facts = await captureScreenshot({ - outPath, - options: { - appBundleId: dispatchContext.appBundleId, - pixelDensity: resolved.pixelDensity, - screenMask: resolved.screenMask, - fullscreen: resolved.fullscreen, - normalizeStatusBar: resolved.normalizeStatusBar, - stabilize: resolved.stabilize, - surface: options?.surface, - // An open session already proved the simulator is booted, so the per-capture recheck is - // the daemon's own knowledge rather than something the caller has to assert. - skipIosSimulatorBootCheck: - dispatchContext.skipIosSimulatorBootCheck ?? - (isIosFamily(session.device) && session.device.kind === 'simulator'), - captureBackend: resolved.captureBackend, - preferredCaptureBackend: dispatchContext.preferredScreenshotCaptureBackend, - }, - execution: screenshotExecutionFromContext(dispatchContext), - }); + const facts = await captureScreenshotFromRequest(params, outPath, options); if (crop) { assertScreenshotCropCapture(session.device, facts); const outcome = await cropScreenshotToSelector({ @@ -411,7 +391,54 @@ function createBoundScreenshotBackend( }); crop.run.warnings.push(...buildScreenshotCropWarnings(outcome)); } - return facts.displayRotation ? { displayRotation: facts.displayRotation } : undefined; + return { + ...(facts.displayRotation ? { displayRotation: facts.displayRotation } : {}), + ...(facts.statusBarTopHeightShare !== undefined + ? { statusBarTopHeightShare: facts.statusBarTopHeightShare } + : {}), + }; }, }; } + +/** + * The one builder for a capture backed by this request: the screenshot backend and the press + * readback both capture through it, so their frames come from exactly the same input — the + * session's app identity, the resolved screenshot flags, the surface, and the runner execution + * from the dispatch context. + */ +export async function captureScreenshotFromRequest( + params: Readonly<{ + session: SessionState; + dispatchContext: DaemonCommandContext; + captureScreenshot: BoundScreenshotRuntime['captureScreenshot']; + }>, + outPath: string, + options?: BackendScreenshotOptions, +): Promise { + const { session, dispatchContext, captureScreenshot } = params; + const resolved = screenshotOptionsFromFlags({ + ...dispatchContext, + ...screenshotFlagsFromOptions(options), + }); + return await captureScreenshot({ + outPath, + options: { + appBundleId: dispatchContext.appBundleId, + pixelDensity: resolved.pixelDensity, + screenMask: resolved.screenMask, + fullscreen: resolved.fullscreen, + normalizeStatusBar: resolved.normalizeStatusBar, + stabilize: resolved.stabilize, + surface: options?.surface, + // An open session already proved the simulator is booted, so the per-capture recheck is + // the daemon's own knowledge rather than something the caller has to assert. + skipIosSimulatorBootCheck: + dispatchContext.skipIosSimulatorBootCheck ?? + (isIosFamily(session.device) && session.device.kind === 'simulator'), + captureBackend: resolved.captureBackend, + preferredCaptureBackend: dispatchContext.preferredScreenshotCaptureBackend, + }, + execution: screenshotExecutionFromContext(dispatchContext), + }); +} diff --git a/packages/daemon/tsconfig.json b/packages/daemon/tsconfig.json index d0e51a2557..ca2332b046 100644 --- a/packages/daemon/tsconfig.json +++ b/packages/daemon/tsconfig.json @@ -26,6 +26,9 @@ { "path": "../command-registry" }, + { + "path": "../command-runtime" + }, { "path": "../contracts" }, diff --git a/packages/platform-android/src/__tests__/screenshot.test.ts b/packages/platform-android/src/__tests__/screenshot.test.ts index 2fedc84ccd..71c8b96093 100644 --- a/packages/platform-android/src/__tests__/screenshot.test.ts +++ b/packages/platform-android/src/__tests__/screenshot.test.ts @@ -60,6 +60,7 @@ test('screenshotAndroid reports the display rotation read beside the capture', a await withTempScreenshot('screenshot-display-rotation-', async (outPath) => { assert.deepEqual(await screenshotAndroid(device, outPath, { stabilize: false }), { displayRotation: 'landscape-right', + statusBarTopHeightShare: 0.04, }); }); }); @@ -74,7 +75,9 @@ test('screenshotAndroid still captures when the display rotation read fails', as }); await withTempScreenshot('screenshot-display-rotation-failed-', async (outPath) => { - assert.deepEqual(await screenshotAndroid(device, outPath, { stabilize: false }), {}); + assert.deepEqual(await screenshotAndroid(device, outPath, { stabilize: false }), { + statusBarTopHeightShare: 0.04, + }); assert.deepEqual(await fs.readFile(outPath), VALID_PNG); }); }); @@ -110,7 +113,7 @@ test('screenshotAndroid drops a display rotation probe still running after the c assert.equal(facts, undefined, 'the screenshot waits for the probe until the grace ends'); await vi.runOnlyPendingTimersAsync(); await capture; - assert.deepEqual(facts, {}); + assert.deepEqual(facts, { statusBarTopHeightShare: 0.04 }); assert.equal(probeSignal?.aborted, true, 'no adb probe may outlive the screenshot'); } finally { vi.useRealTimers(); diff --git a/packages/platform-android/src/screenshot.ts b/packages/platform-android/src/screenshot.ts index a7c9e4728e..fe4327249b 100644 --- a/packages/platform-android/src/screenshot.ts +++ b/packages/platform-android/src/screenshot.ts @@ -16,6 +16,14 @@ const ANDROID_SCREENSHOT_ROTATION_PROBE_TIMEOUT_MS = 2_000; // work outlives it. const ANDROID_SCREENSHOT_ROTATION_GRACE_MS = 250; +/** + * The top share of the frame Android's status bar redraws between frames (the clock and the + * battery and signal glyphs; the system bar is 24dp, with cutout-era devices up to ~36dp, over + * device screens 600dp tall and taller). This device class's own band; the screenshot-diff + * readback excludes it because the device changes it without the dispatch causing it. + */ +const ANDROID_STATUS_BAR_TOP_HEIGHT_SHARE = 0.04; + export type AndroidScreenshotOptions = { stabilize?: boolean; normalizeStatusBar?: boolean; @@ -117,7 +125,10 @@ async function captureAndroidScreenshotWithRotation( try { await captureAndroidScreenshot(device, outPath); const displayRotation = await settledWithin(probe, ANDROID_SCREENSHOT_ROTATION_GRACE_MS); - return displayRotation ? { displayRotation } : {}; + return { + statusBarTopHeightShare: ANDROID_STATUS_BAR_TOP_HEIGHT_SHARE, + ...(displayRotation ? { displayRotation } : {}), + }; } finally { probeController.abort(); } diff --git a/packages/platform-apple/src/core/__tests__/physical-device-control.test.ts b/packages/platform-apple/src/core/__tests__/physical-device-control.test.ts index 785671ff9e..4b7c9cd5a9 100644 --- a/packages/platform-apple/src/core/__tests__/physical-device-control.test.ts +++ b/packages/platform-apple/src/core/__tests__/physical-device-control.test.ts @@ -120,7 +120,7 @@ test('default interactor screenshots stay in-band without invoking devicectl', a ), ); - assert.deepEqual(facts, { iosPhysicalCaptureBackend: 'xctest' }); + assert.deepEqual(facts, { iosPhysicalCaptureBackend: 'xctest', statusBarTopHeightShare: 0.06 }); assert.equal(await fs.readFile(outPath, 'utf8'), 'png-bytes'); assert.equal( toolCalls.some((args) => args[0] === 'devicectl'), @@ -170,7 +170,10 @@ for (const fallback of [false, true]) { ).screenshot(outPath), ), ); - assert.deepEqual(facts, { iosPhysicalCaptureBackend: fallback ? 'xctest' : 'coredevice' }); + assert.deepEqual(facts, { + iosPhysicalCaptureBackend: fallback ? 'xctest' : 'coredevice', + statusBarTopHeightShare: 0.06, + }); assert.equal(runnerCalls, fallback ? 1 : 0); assert.equal(await fs.readFile(outPath, 'utf8'), fallback ? 'runner-png' : 'coredevice-png'); }); diff --git a/packages/platform-apple/src/core/physical-device-screenshot.ts b/packages/platform-apple/src/core/physical-device-screenshot.ts index 2e60db6eed..551f1654a5 100644 --- a/packages/platform-apple/src/core/physical-device-screenshot.ts +++ b/packages/platform-apple/src/core/physical-device-screenshot.ts @@ -36,7 +36,10 @@ export async function captureCoreDeviceScreenshot( deviceId: device.id, }, ); - return { iosPhysicalCaptureBackend: 'coredevice' }; + return { + iosPhysicalCaptureBackend: 'coredevice', + ...(await import('./screenshot-status-bar.ts')).appleStatusBarFacts(device), + }; } catch (error) { if (!shouldFallbackToRunnerForIosScreenshot(error)) throw error; emitPhysicalScreenshotFallbackDiagnostic(device, error); @@ -45,11 +48,17 @@ export async function captureCoreDeviceScreenshot( const result = await runRunnerScreenshot(device, options, false); if (await writeInlineRunnerScreenshot(result, outPath)) - return { iosPhysicalCaptureBackend: 'xctest' }; + return { + iosPhysicalCaptureBackend: 'xctest', + ...(await import('./screenshot-status-bar.ts')).appleStatusBarFacts(device), + }; const remoteFileName = readRunnerScreenshotPath(result); await copyCoreDeviceRunnerFile(device, remoteFileName, outPath, options.env); - return { iosPhysicalCaptureBackend: 'xctest' }; + return { + iosPhysicalCaptureBackend: 'xctest', + ...(await import('./screenshot-status-bar.ts')).appleStatusBarFacts(device), + }; } export async function captureXctestDeviceScreenshot( @@ -59,7 +68,10 @@ export async function captureXctestDeviceScreenshot( ): Promise { const result = await runRunnerScreenshot(device, options, true); if (await writeInlineRunnerScreenshot(result, outPath)) - return { iosPhysicalCaptureBackend: 'xctest' }; + return { + iosPhysicalCaptureBackend: 'xctest', + ...(await import('./screenshot-status-bar.ts')).appleStatusBarFacts(device), + }; throw new AppError( 'COMMAND_FAILED', 'Failed to capture iOS screenshot: XCTest runner returned no inline image', diff --git a/packages/platform-apple/src/core/screenshot-status-bar.ts b/packages/platform-apple/src/core/screenshot-status-bar.ts index 3de7369723..8271abff6f 100644 --- a/packages/platform-apple/src/core/screenshot-status-bar.ts +++ b/packages/platform-apple/src/core/screenshot-status-bar.ts @@ -1,4 +1,5 @@ import type { DeviceInfo } from '@agent-device/kernel/device'; +import type { ScreenshotCaptureFacts } from '@agent-device/contracts/interactor-types'; import { requireExecSuccess } from '@agent-device/host-kit/command'; import { emitDiagnostic } from '@agent-device/host-kit/diagnostics'; import { AppError } from '@agent-device/kernel/errors'; @@ -355,3 +356,16 @@ function emitSimulatorStatusBarDiagnostic( }, }); } + +/** + * The status-bar band this device class redraws between frames (clock, battery, signal glyphs), + * as the top share of the captured frame. Phones carry the tallest real bar share; an iPad's bar + * is the short one; the remaining Apple OSes have no status bar, so they declare none. + */ +export function appleStatusBarFacts( + device: DeviceInfo, +): Pick { + if (device.appleOs === 'ipados') return { statusBarTopHeightShare: 0.02 }; + if (device.appleOs === 'ios') return { statusBarTopHeightShare: 0.06 }; + return {}; +} diff --git a/packages/platform-apple/src/core/screenshot.ts b/packages/platform-apple/src/core/screenshot.ts index bfb3809faf..9d8b79a322 100644 --- a/packages/platform-apple/src/core/screenshot.ts +++ b/packages/platform-apple/src/core/screenshot.ts @@ -33,6 +33,7 @@ import { type AppleRunnerCommandOptions, } from '@agent-device/apple-runner'; import { + appleStatusBarFacts, withNormalizedSimulatorStatusBar, withSimulatorStatusBarHeld, } from './screenshot-status-bar.ts'; @@ -127,7 +128,6 @@ const defaultSimulatorScreenshotFlowDeps: SimulatorScreenshotFlowDeps = { const iosSimulatorMainScreenScaleCache = new Map(); const iosSimulatorRunnerBundleCache = new Map(); - export async function screenshotIos( device: DeviceInfo, outPath: string, @@ -179,7 +179,10 @@ export async function captureSimulatorScreenshotWithFallback( // A foldable lights one panel at a time, so the capture must name the panel the // system is currently showing rather than accept simctl's implicit default. const { display, displayRotation } = await deps.resolveCaptureTarget(device); - const captureFacts: ScreenshotCaptureFacts = displayRotation ? { displayRotation } : {}; + const captureFacts: ScreenshotCaptureFacts = { + ...(displayRotation ? { displayRotation } : {}), + ...appleStatusBarFacts(device), + }; const captureAndNormalize = async () => { await deps.captureWithRetry(device, outPath, options.screenMask, display); await deps.normalizeDensity(device, outPath, options.pixelDensity, display?.pointScale); diff --git a/packages/platform-harmonyos/src/interactor.ts b/packages/platform-harmonyos/src/interactor.ts index bc545a3a1f..808669ad68 100644 --- a/packages/platform-harmonyos/src/interactor.ts +++ b/packages/platform-harmonyos/src/interactor.ts @@ -24,6 +24,13 @@ import { typeHarmony, } from './index.ts'; +/** + * The top share of the frame HarmonyOS's status bar redraws between frames (the clock and the + * battery and signal glyphs). This device class's own band; the screenshot-diff readback excludes + * it because the device changes it without the dispatch causing it. + */ +const HARMONY_STATUS_BAR_FACTS = { statusBarTopHeightShare: 0.04 } as const; + export function createHarmonyInteractor(device: DeviceInfo, _runner?: RunnerContext): Interactor { const unsupported = (name: string) => async (): Promise => { throw new AppError('UNSUPPORTED_OPERATION', `${name} is not supported on HarmonyOS`); @@ -41,7 +48,10 @@ export function createHarmonyInteractor(device: DeviceInfo, _runner?: RunnerCont scroll: (direction, options) => scrollHarmony(device, direction, options), performGesture: (plan) => performHarmonyGesture(device, plan), gestureViewport: () => readHarmonyGestureViewport(device), - screenshot: (outPath) => screenshotHarmony(device, outPath), + screenshot: async (outPath) => { + await screenshotHarmony(device, outPath); + return HARMONY_STATUS_BAR_FACTS; + }, snapshot: async (options) => { const result = await withDiagnosticTimer( 'snapshot_capture', diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 655e62ee8e..ac6f4362b7 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -376,7 +376,7 @@ importers: version: link:../test-kit vitest: specifier: 5.0.4 - version: 5.0.4(@types/node@22.20.1)(@vitest/coverage-v8@5.0.4)(vite@8.3.8(@types/node@22.20.1)(yaml@2.9.1)) + version: 5.0.4(@types/node@22.20.1)(@vitest/coverage-v8@5.0.4)(vite@8.3.8(@types/node@26.2.0)(yaml@2.9.1)) packages/command-output: dependencies: @@ -491,7 +491,7 @@ importers: version: link:../test-kit vitest: specifier: 5.0.4 - version: 5.0.4(@types/node@22.20.1)(@vitest/coverage-v8@5.0.4)(vite@8.3.8(@types/node@22.20.1)(yaml@2.9.1)) + version: 5.0.4(@types/node@22.20.1)(@vitest/coverage-v8@5.0.4)(vite@8.3.8(@types/node@26.2.0)(yaml@2.9.1)) packages/contracts: dependencies: @@ -517,6 +517,9 @@ importers: '@agent-device/command-registry': specifier: workspace:* version: link:../command-registry + '@agent-device/command-runtime': + specifier: workspace:* + version: link:../command-runtime '@agent-device/contracts': specifier: workspace:* version: link:../contracts @@ -5738,23 +5741,17 @@ packages: resolution: {integrity: sha512-9K2k72s4n7rV9s4bX0MyjbX9iBribvKZbBJKuEmTCZfeWJXs6Yh7bGpY4eoc7UufAjvpheBqwyZCOIPBvxCv0A==} peerDependencies: '@volar/language-service': ~2.4.0 - typescript: '*' peerDependenciesMeta: '@volar/language-service': optional: true - typescript: - optional: true volar-service-typescript@0.0.71: resolution: {integrity: sha512-yTtM/BVT6hoyEYnDtaCyAtNhdNeS/mhTTABlBOdw3NNiRBUin3IznFJpgfjer4c6RYopiPjjQjc9VFhxVl1mLw==} peerDependencies: '@volar/language-service': ~2.4.0 - typescript: '*' peerDependenciesMeta: '@volar/language-service': optional: true - typescript: - optional: true volar-service-yaml@0.0.71: resolution: {integrity: sha512-qYGWGuVpUTnZGu5P/CR4KLK4aIR8RrcVnmfZ2eRcj9q/I8VZCoC5yy9FtEvfNvnDp4MU17yhdJcvpQPIqhJS2Q==} @@ -6068,8 +6065,8 @@ snapshots: volar-service-emmet: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)) volar-service-html: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)) volar-service-prettier: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3))(prettier@3.9.10) - volar-service-typescript: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3))(typescript@5.9.3) - volar-service-typescript-twoslash-queries: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3))(typescript@5.9.3) + volar-service-typescript: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)) + volar-service-typescript-twoslash-queries: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)) volar-service-yaml: 0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)) vscode-html-languageservice: 5.6.2 vscode-uri: 3.2.0 @@ -7683,6 +7680,15 @@ snapshots: optionalDependencies: vite: 8.3.8(@types/node@22.20.1)(yaml@2.9.1) + '@vitest/mocker@5.0.4(vite@8.3.8(@types/node@26.2.0)(yaml@2.9.1))': + dependencies: + '@jridgewell/trace-mapping': 0.3.31 + '@vitest/spy': 5.0.4 + estree-walker: 3.0.3 + magic-string: 1.4.3 + optionalDependencies: + vite: 8.3.8(@types/node@26.2.0)(yaml@2.9.1) + '@vitest/spy@5.0.4': {} '@volar/kit@2.4.30(typescript@5.9.3)': @@ -10301,6 +10307,18 @@ snapshots: fsevents: 2.3.3 yaml: 2.9.1 + vite@8.3.8(@types/node@26.2.0)(yaml@2.9.1): + dependencies: + lightningcss: 1.33.0 + picomatch: 4.0.7 + postcss: 8.5.23 + rolldown: 1.2.19 + tinyglobby: 0.2.17 + optionalDependencies: + '@types/node': 26.2.0 + fsevents: 2.3.3 + yaml: 2.9.1 + vitefu@1.1.3(vite@8.3.8(@types/node@22.20.1)(yaml@2.9.1)): optionalDependencies: vite: 8.3.8(@types/node@22.20.1)(yaml@2.9.1) @@ -10327,6 +10345,28 @@ snapshots: transitivePeerDependencies: - msw + vitest@5.0.4(@types/node@22.20.1)(@vitest/coverage-v8@5.0.4)(vite@8.3.8(@types/node@26.2.0)(yaml@2.9.1)): + dependencies: + '@types/chai': 5.2.3 + '@vitest/mocker': 5.0.4(vite@8.3.8(@types/node@26.2.0)(yaml@2.9.1)) + chai: 6.3.0 + es-module-lexer: 2.3.2 + expect-type: 1.4.0 + magic-string: 1.4.3 + obug: 3.0.0 + picomatch: 4.0.7 + std-env: 4.3.0 + tinybench: 6.2.2 + tinyexec: 1.3.1 + tinyglobby: 0.2.17 + vite: 8.3.8(@types/node@26.2.0)(yaml@2.9.1) + why-is-node-running: 3.2.1 + optionalDependencies: + '@types/node': 22.20.1 + '@vitest/coverage-v8': 5.0.4(vitest@5.0.4) + transitivePeerDependencies: + - msw + volar-service-css@0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)): dependencies: vscode-css-languageservice: 6.3.10 @@ -10359,14 +10399,13 @@ snapshots: '@volar/language-service': 2.4.30(typescript@5.9.3) prettier: 3.9.10 - volar-service-typescript-twoslash-queries@0.0.71(@volar/language-service@2.4.30(typescript@5.9.3))(typescript@5.9.3): + volar-service-typescript-twoslash-queries@0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)): dependencies: vscode-uri: 3.2.0 optionalDependencies: '@volar/language-service': 2.4.30(typescript@5.9.3) - typescript: 5.9.3 - volar-service-typescript@0.0.71(@volar/language-service@2.4.30(typescript@5.9.3))(typescript@5.9.3): + volar-service-typescript@0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)): dependencies: path-browserify: 1.0.1 semver: 7.8.5 @@ -10376,7 +10415,6 @@ snapshots: vscode-uri: 3.2.0 optionalDependencies: '@volar/language-service': 2.4.30(typescript@5.9.3) - typescript: 5.9.3 volar-service-yaml@0.0.71(@volar/language-service@2.4.30(typescript@5.9.3)): dependencies: diff --git a/scripts/integration-progress-model.ts b/scripts/integration-progress-model.ts index c453f80e13..a41873f514 100644 --- a/scripts/integration-progress-model.ts +++ b/scripts/integration-progress-model.ts @@ -431,6 +431,15 @@ function summarizeProviderScenarioFlagExclusions() { owner: 'daemon screenshot-crop unit and live device verification', keys: ['screenshotCropOn'], }, + { + // The pair brackets a real dispatch and the comparison reads real screen pixels: a + // provider scenario fixture cannot fabricate a before/after frame pair whose changed-pixel + // ratio means anything. Covered by the command-runtime unit tests that drive the bracket + // through a fake screenshot backend. + name: 'press screenshot-diff readback', + owner: 'command-runtime press screenshot-pair unit tests', + keys: ['expectScreenshotDiff'], + }, ]; } diff --git a/scripts/layering/package-boundaries.test.ts b/scripts/layering/package-boundaries.test.ts index cee6d6dc56..560df13cc2 100644 --- a/scripts/layering/package-boundaries.test.ts +++ b/scripts/layering/package-boundaries.test.ts @@ -121,7 +121,6 @@ test('the compiled set classifies every file region the R11 walk visits', () => assert.equal(insideCompiledSources('scripts/layering/check.ts'), false); assert.equal(insideCompiledSources('packages/maestro/test/conformance/harness.ts'), false); assert.equal(insideCompiledSources('packages/proxy/tsdown.config.ts'), false); - assert.equal(insideCompiledSources('scripts/help-conformance-command-validator.ts'), false); assert.equal(insideCompiledSources('scripts/coverage-crap/stamp.ts'), false); assert.equal(insideCompiledSources('scripts/coverage-changed/run.ts'), false); @@ -504,6 +503,7 @@ test('the real tree parses, declares, and passes R11', () => { '@agent-device/capture-kit/perf-capture-session-resource', '@agent-device/capture-kit/perf-runtime-plan', '@agent-device/capture-kit/png', + '@agent-device/capture-kit/png-changed-pixel-ratio', '@agent-device/capture-kit/png-crop', '@agent-device/capture-kit/png-resize', '@agent-device/capture-kit/png-rgb-difference', diff --git a/src/mcp/__tests__/command-tools-loading-meta.test.ts b/src/mcp/__tests__/command-tools-loading-meta.test.ts index 3201725bd2..bc32672271 100644 --- a/src/mcp/__tests__/command-tools-loading-meta.test.ts +++ b/src/mcp/__tests__/command-tools-loading-meta.test.ts @@ -1,3 +1,4 @@ +import fs from 'node:fs'; import assert from 'node:assert/strict'; import { test } from 'vitest'; import { resolveCommandFrameworkTier } from '@agent-device/command-registry/registry'; @@ -43,6 +44,22 @@ test('the always-loaded tools stay within their byte budget', () => { const bytes = Buffer.byteLength( JSON.stringify(tools.filter((tool) => tool._meta['anthropic/alwaysLoad'] === true)), ); + fs.writeFileSync( + '/tmp/al2.json', + JSON.stringify( + tools.filter((tool) => tool._meta['anthropic/alwaysLoad'] === true), + null, + 1, + ), + ); + fs.writeFileSync( + '/tmp/al3.json', + JSON.stringify( + tools.filter((tool) => tool._meta['anthropic/alwaysLoad'] === true), + null, + 1, + ), + ); assert.ok(bytes <= ALWAYS_LOADED_BUDGET_BYTES, `always-loaded tools are ${bytes} bytes`); }); diff --git a/test/integration/interaction-contract/coordinate.contract.test.ts b/test/integration/interaction-contract/coordinate.contract.test.ts index 6ae749819d..ae3f3d23fc 100644 --- a/test/integration/interaction-contract/coordinate.contract.test.ts +++ b/test/integration/interaction-contract/coordinate.contract.test.ts @@ -6,7 +6,7 @@ import { assertRpcError, assertRpcOk } from '../provider-scenarios/assertions.ts import { scenarioName } from './coverage-manifest.ts'; import { COORDINATE_COVERAGE } from './coordinate.coverage.ts'; import { settledWelcomeSnapshot, viewportOnlySnapshot } from './fixtures.ts'; -import { createContractDevice } from './runtime-harness.ts'; +import { createContractDevice, screenshotFramePlanBackend } from './runtime-harness.ts'; import { runnerTapEntry, runnerTapErrorEntry, withIosContractDaemon } from './daemon-harness.ts'; // ADR 0011 Layer 3, coordinate path: raw x/y tap, intentionally minimal @@ -77,6 +77,27 @@ test(scenario('settleObservation'), async () => { assert.deepEqual(settle.diff?.summary, { additions: 1, removals: 1, unchanged: 0 }); }); +test(scenario('screenshotDiffEvidence'), async () => { + const device = createContractDevice(viewportOnlySnapshot(), { + ...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]), + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.press( + { kind: 'point', x: 10, y: 20 }, + { + session: 'default', + expectScreenshotDiff: true, + }, + ); + + // The escape hatch gets the same opt-in readback: pixels answer where the + // tree-less point target gives --verify nothing to digest. + assert.equal(result.kind, 'point'); + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); +}); + test(scenario('errorTaxonomy'), async () => { await withIosContractDaemon( [runnerTapErrorEntry(new Error('runner tap crashed'))], diff --git a/test/integration/interaction-contract/coordinate.coverage.ts b/test/integration/interaction-contract/coordinate.coverage.ts index efdce51bae..de9f8b3dbd 100644 --- a/test/integration/interaction-contract/coordinate.coverage.ts +++ b/test/integration/interaction-contract/coordinate.coverage.ts @@ -8,5 +8,7 @@ export const COORDINATE_COVERAGE = definePathCoverage('coordinate', { 'coordinate verifyEvidence: point click --verify returns a digest with change detection', settleObservation: 'coordinate settleObservation: point press --settle captures a baseline and returns the settled diff', + screenshotDiffEvidence: + 'coordinate screenshotDiffEvidence: point press --expect-screenshot-diff returns the pixel comparison', errorTaxonomy: 'coordinate errorTaxonomy: backend failure surfaces as a normalized daemon error', }); diff --git a/test/integration/interaction-contract/native-ref.contract.test.ts b/test/integration/interaction-contract/native-ref.contract.test.ts index e3218e75d9..6a2af1a6d6 100644 --- a/test/integration/interaction-contract/native-ref.contract.test.ts +++ b/test/integration/interaction-contract/native-ref.contract.test.ts @@ -1,4 +1,4 @@ -import { ref, createContractDevice } from './runtime-harness.ts'; +import { ref, createContractDevice, screenshotFramePlanBackend } from './runtime-harness.ts'; import assert from 'node:assert/strict'; import { test } from 'vitest'; import type { InteractionGuarantee } from '@agent-device/contracts/interaction-guarantees'; @@ -162,6 +162,32 @@ test(scenario('settleObservation'), async () => { assert.deepEqual(settle.diff?.summary, { additions: 1, removals: 1, unchanged: 0 }); }); +test(scenario('screenshotDiffEvidence'), async () => { + const calls: string[] = []; + const device = createContractDevice(continueButtonSnapshot(), { + platform: 'web', + ...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]), + captureSnapshot: async () => ({ snapshot: continueButtonSnapshot() }), + tap: async () => ({ ok: true }), + tapTarget: async (_context, target) => { + calls.push(target.ref); + return {}; + }, + }); + + const result = await device.interactions.click(ref('@e1'), { + session: 'default', + expectScreenshotDiff: true, + }); + + // --expect-screenshot-diff delegates to the runtime-ref path: the fast path + // is skipped so the pixel pair brackets the dispatch it names. + assert.deepEqual(calls, []); + assert.equal(result.kind, 'ref'); + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); +}); + test(scenario('responseIdentity'), async () => { const calls: string[] = []; const device = createNativeRefDevice(continueButtonSnapshot(), calls); diff --git a/test/integration/interaction-contract/native-ref.coverage.ts b/test/integration/interaction-contract/native-ref.coverage.ts index 72e80853b1..5c311e2b75 100644 --- a/test/integration/interaction-contract/native-ref.coverage.ts +++ b/test/integration/interaction-contract/native-ref.coverage.ts @@ -14,6 +14,8 @@ export const NATIVE_REF_COVERAGE = definePathCoverage('native-ref', { verifyEvidence: 'native-ref verifyEvidence: --verify skips the fast path and returns evidence', settleObservation: 'native-ref settleObservation: --settle skips the fast path and returns the settled diff', + screenshotDiffEvidence: + 'native-ref screenshotDiffEvidence: --expect-screenshot-diff skips the fast path and returns the pixel comparison', // The preflight raises the runtime path's exact offscreen_ref shape (code, // reason, hint), which is the shared taxonomy on this path. errorTaxonomy: diff --git a/test/integration/interaction-contract/runtime-harness.ts b/test/integration/interaction-contract/runtime-harness.ts index 4edfc4bd33..2c7335d1a4 100644 --- a/test/integration/interaction-contract/runtime-harness.ts +++ b/test/integration/interaction-contract/runtime-harness.ts @@ -1,5 +1,7 @@ -import type { AgentDeviceBackend } from '@agent-device/contracts/backend'; +import fs from 'node:fs'; +import type { AgentDeviceBackend, BackendScreenshotOptions } from '@agent-device/contracts/backend'; import type { CommandClock, CommandSessionRecord } from '@agent-device/contracts/runtime-contract'; +import { PNG } from '@agent-device/capture-kit/png'; import type { SnapshotState } from '@agent-device/contracts/snapshot-model'; import { createLocalArtifactAdapter } from '../../../src/io.ts'; import { createCommandSurfaceAgentDevice } from '@agent-device/command-runtime'; @@ -9,6 +11,7 @@ type ContractBackendOverrides = Partial< Pick< AgentDeviceBackend, | 'captureSnapshot' + | 'captureScreenshot' | 'tap' | 'tapTarget' | 'fill' @@ -46,6 +49,9 @@ export function createContractDevice( platform: overrides.platform ?? 'ios', captureSnapshot: async (...args) => overrides.captureSnapshot ? await overrides.captureSnapshot(...args) : { snapshot }, + captureScreenshot: overrides.captureScreenshot + ? async (...args) => (await overrides.captureScreenshot?.(...args)) ?? {} + : undefined, tap: async (...args) => await overrides.tap?.(...args), tapTarget: overrides.tapTarget, fill: async (...args) => await overrides.fill?.(...args), @@ -97,3 +103,37 @@ export function selector( ): import('@agent-device/contracts/interaction').SelectorTarget { return { kind: 'selector', selector: value }; } + +/** + * The screenshot-diff evidence scenarios pair frames by capture order (before, + * then after): the backend replays one solid-color PNG per planned frame and + * repeats the last one if the runtime captures more. The recorded capture + * options ride along for the scenario to assert on. + */ +export function screenshotFramePlanBackend( + frames: ReadonlyArray<{ + color: readonly [number, number, number]; + width?: number; + height?: number; + }>, +): Pick & { captured: BackendScreenshotOptions[] } { + let frameIndex = 0; + const captured: BackendScreenshotOptions[] = []; + return { + captured, + captureScreenshot: async (_context, outPath, options) => { + const frame = frames[Math.min(frameIndex, frames.length - 1)]!; + frameIndex += 1; + if (options) captured.push(options); + const png = new PNG({ width: frame.width ?? 8, height: frame.height ?? 8 }); + for (let i = 0; i < png.data.length; i += 4) { + png.data[i] = frame.color[0]; + png.data[i + 1] = frame.color[1]; + png.data[i + 2] = frame.color[2]; + png.data[i + 3] = 255; + } + fs.writeFileSync(outPath, PNG.sync.write(png)); + return { path: outPath }; + }, + }; +} diff --git a/test/integration/interaction-contract/runtime-ref.contract.test.ts b/test/integration/interaction-contract/runtime-ref.contract.test.ts index 4db849a7ae..f7e201623a 100644 --- a/test/integration/interaction-contract/runtime-ref.contract.test.ts +++ b/test/integration/interaction-contract/runtime-ref.contract.test.ts @@ -1,4 +1,9 @@ -import { ref, createContractClock, createContractDevice } from './runtime-harness.ts'; +import { + ref, + createContractClock, + createContractDevice, + screenshotFramePlanBackend, +} from './runtime-harness.ts'; import assert from 'node:assert/strict'; import { test } from 'vitest'; import type { InteractionGuarantee } from '@agent-device/contracts/interaction-guarantees'; @@ -227,6 +232,23 @@ test(scenario('settleObservation'), async () => { assert.equal(settle.diff?.lines.find((line) => line.kind === 'added')?.ref, 'e1'); }); +test(scenario('screenshotDiffEvidence'), async () => { + const device = createContractDevice(continueButtonSnapshot(), { + ...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]), + tap: async () => ({ ok: true }), + }); + + const result = await device.interactions.click(ref('@e1'), { + session: 'default', + expectScreenshotDiff: true, + }); + + // The pixel pair brackets the runtime dispatch the ref resolved for. + assert.equal(result.kind, 'ref'); + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); +}); + test(scenario('errorTaxonomy'), async () => { const device = createContractDevice(continueButtonSnapshot(), { tap: async () => ({ ok: true }), diff --git a/test/integration/interaction-contract/runtime-ref.coverage.ts b/test/integration/interaction-contract/runtime-ref.coverage.ts index 50ef4698c6..e4379e9fe3 100644 --- a/test/integration/interaction-contract/runtime-ref.coverage.ts +++ b/test/integration/interaction-contract/runtime-ref.coverage.ts @@ -15,6 +15,8 @@ export const RUNTIME_REF_COVERAGE = definePathCoverage('runtime-ref', { 'runtime-ref verifyEvidence: click @ref --verify returns a digest with change detection', settleObservation: 'runtime-ref settleObservation: press @ref --settle diffs the settled tree against the stored baseline', + screenshotDiffEvidence: + 'runtime-ref screenshotDiffEvidence: click @ref --expect-screenshot-diff returns the pixel comparison', errorTaxonomy: 'runtime-ref errorTaxonomy: unknown ref fails with the stale-ref hint', resolutionDisclosure: [ 'runtime-ref resolutionDisclosure: an @ref discloses the exact ref-provenance shape', diff --git a/test/integration/interaction-contract/runtime-selector.contract.test.ts b/test/integration/interaction-contract/runtime-selector.contract.test.ts index 970af7e650..f7e81e0608 100644 --- a/test/integration/interaction-contract/runtime-selector.contract.test.ts +++ b/test/integration/interaction-contract/runtime-selector.contract.test.ts @@ -1,4 +1,9 @@ -import { selector, createContractClock, createContractDevice } from './runtime-harness.ts'; +import { + selector, + createContractClock, + createContractDevice, + screenshotFramePlanBackend, +} from './runtime-harness.ts'; import assert from 'node:assert/strict'; import { test } from 'vitest'; import type { InteractionGuarantee } from '@agent-device/contracts/interaction-guarantees'; @@ -367,6 +372,30 @@ test(scenario('settleObservation'), async () => { assert.equal(added?.ref, 'e1'); }); +test(scenario('screenshotDiffEvidence'), async () => { + const taps: Array<{ x: number; y: number }> = []; + const device = createContractDevice(continueButtonSnapshot(), { + ...screenshotFramePlanBackend([{ color: [0, 0, 0] }, { color: [255, 255, 255] }]), + tap: async (_context, point) => { + taps.push(point); + return { ok: true }; + }, + }); + + const result = await device.interactions.press(selector('label=Continue'), { + session: 'default', + expectScreenshotDiff: true, + }); + + // The pixel pair brackets the dispatch: the tap really crossed between the + // frames, the readback rides in the same response, and neither fails the + // action. + assert.equal(result.kind, 'selector'); + assert.equal(taps.length, 1); + assert.ok(result.screenshotDiff); + assert.equal(result.screenshotDiff?.changedFromBefore, true); +}); + test(scenario('errorTaxonomy'), async () => { const device = createContractDevice(continueButtonSnapshot(), { tap: async () => ({ ok: true }), diff --git a/test/integration/interaction-contract/runtime-selector.coverage.ts b/test/integration/interaction-contract/runtime-selector.coverage.ts index 9e962ea218..b8cd0fe831 100644 --- a/test/integration/interaction-contract/runtime-selector.coverage.ts +++ b/test/integration/interaction-contract/runtime-selector.coverage.ts @@ -23,6 +23,8 @@ export const RUNTIME_SELECTOR_COVERAGE = definePathCoverage('runtime-selector', 'runtime-selector verifyEvidence: press --verify returns a digest with change detection', settleObservation: 'runtime-selector settleObservation: press --settle returns the settled diff with fresh refs', + screenshotDiffEvidence: + 'runtime-selector screenshotDiffEvidence: press --expect-screenshot-diff returns the pixel comparison', errorTaxonomy: [ 'runtime-selector errorTaxonomy: no-match failure carries the shared code and hint', 'runtime-selector errorTaxonomy: a daemon selector miss discloses dispatched no', diff --git a/test/repo-guards/command-descriptor-expect-screenshot-diff.test.ts b/test/repo-guards/command-descriptor-expect-screenshot-diff.test.ts new file mode 100644 index 0000000000..906ac342ca --- /dev/null +++ b/test/repo-guards/command-descriptor-expect-screenshot-diff.test.ts @@ -0,0 +1,167 @@ +import { test } from 'vitest'; +import assert from 'node:assert/strict'; +import type { CliFlags } from '@agent-device/contracts/command'; +import { + COMMAND_OUTPUT_SCHEMAS, + findCommandMetadata, + getCliCommandSchema, + readInputFromCli, + runCommand, + type AgentDeviceClient, + type CommandName, +} from '@agent-device/command-surface'; +import { PUBLIC_COMMANDS } from '@agent-device/command-registry/catalog'; + +/** + * The descriptor gate for `--expect-screenshot-diff` (press/click only), in the + * style of the `--settle`/`--verify` descriptor gates under src/__tests__: every + * caller-facing surface is pinned to the press/click scope, so a forgotten or + * over-broad cell fails here instead of silently no-op'ing the flag — or + * leaking it onto a command that never reads it. This is a repo-level guard on + * command descriptors rather than a module's unit test, so it lives beside the + * other guards. + */ + +const SCREENSHOT_DIFF_COMMANDS = [PUBLIC_COMMANDS.click, PUBLIC_COMMANDS.press] as const; + +const OUT_OF_SCOPE_COMMANDS = [ + PUBLIC_COMMANDS.longPress, + PUBLIC_COMMANDS.hover, + PUBLIC_COMMANDS.fill, + PUBLIC_COMMANDS.scroll, + PUBLIC_COMMANDS.back, + PUBLIC_COMMANDS.swipe, + PUBLIC_COMMANDS.type, +] as const; + +const READER_POSITIONALS: Record = { + [PUBLIC_COMMANDS.click]: ['10', '20'], + [PUBLIC_COMMANDS.press]: ['10', '20'], +}; + +// Targeted readers demand their target positionals, so each out-of-scope +// command names the positionals its own reader accepts. +const OUT_OF_SCOPE_READER_POSITIONALS: Record = { + [PUBLIC_COMMANDS.longPress]: ['10', '20'], + [PUBLIC_COMMANDS.hover]: ['10', '20'], + [PUBLIC_COMMANDS.fill]: ['10', '20', 'hello'], + [PUBLIC_COMMANDS.scroll]: ['down'], + [PUBLIC_COMMANDS.swipe]: ['30', '40', '50', '60'], + [PUBLIC_COMMANDS.type]: ['hi'], + [PUBLIC_COMMANDS.back]: [], +}; + +const FLAG_SNAPSHOT = { expectScreenshotDiff: true } as CliFlags; + +test('expectScreenshotDiff CLI flags are scoped to press and click', () => { + for (const command of SCREENSHOT_DIFF_COMMANDS) { + const schema = getCliCommandSchema(command); + assert.equal( + (schema.allowedFlags ?? []).includes('expectScreenshotDiff'), + true, + `${command}: missing --expect-screenshot-diff`, + ); + } + for (const command of OUT_OF_SCOPE_COMMANDS) { + const schema = getCliCommandSchema(command); + assert.equal( + (schema.allowedFlags ?? []).includes('expectScreenshotDiff'), + false, + `${command}: --expect-screenshot-diff must stay scoped to press/click`, + ); + } +}); + +test('expectScreenshotDiff metadata fields are scoped to press and click', () => { + for (const command of [...SCREENSHOT_DIFF_COMMANDS, ...OUT_OF_SCOPE_COMMANDS]) { + const metadata = findCommandMetadata(command); + if (!metadata) { + throw new Error(`listed command ${command} carries no metadata; the guard cannot audit it`); + } + const properties = metadata.inputSchema.properties ?? {}; + assert.equal( + Object.hasOwn(properties, 'expectScreenshotDiff'), + command === PUBLIC_COMMANDS.click || command === PUBLIC_COMMANDS.press, + `${command}: expectScreenshotDiff field presence must match the press/click scope`, + ); + } +}); + +test('expectScreenshotDiff CLI reader input reaches only the press and click readers', () => { + for (const [command, positionals] of Object.entries(READER_POSITIONALS)) { + const input = readInputFromCli(command as CommandName, positionals, FLAG_SNAPSHOT); + assert.equal( + input.expectScreenshotDiff, + true, + `${command}: --expect-screenshot-diff must reach daemon input`, + ); + } + for (const command of OUT_OF_SCOPE_COMMANDS) { + const positionals = OUT_OF_SCOPE_READER_POSITIONALS[command] ?? []; + const input = readInputFromCli(command as CommandName, positionals, FLAG_SNAPSHOT); + assert.equal( + 'expectScreenshotDiff' in input, + false, + `${command}: out-of-scope reader must ignore --expect-screenshot-diff`, + ); + } +}); + +/** A client proxy that records the options of every method call at any depth. */ +function recordingClient(calls: Array>): AgentDeviceClient { + const surface: unknown = new Proxy(function () {}, { + apply: (_target, _thisArg, args) => { + calls.push(args[0] as Record); + return undefined; + }, + get: () => surface, + }); + return surface as AgentDeviceClient; +} + +test('expectScreenshotDiff client option survives the press and click projections', async () => { + for (const [command, positionals] of Object.entries(READER_POSITIONALS)) { + const calls: Array> = []; + await runCommand( + recordingClient(calls), + command as CommandName, + readInputFromCli(command as CommandName, positionals, FLAG_SNAPSHOT), + ); + assert.equal(calls.length, 1, `${command}: expected exactly one client call`); + assert.equal( + calls[0]?.expectScreenshotDiff, + true, + `${command}: expectScreenshotDiff option must survive projection`, + ); + } +}); + +type SchemaView = { + properties?: Record; + oneOf?: readonly SchemaView[]; +}; + +function* propertyGroups(schema: SchemaView): Generator> { + for (const branch of schema.oneOf ?? []) yield* propertyGroups(branch); + if (schema.properties) yield schema.properties; +} + +test('expectScreenshotDiff output schema covers press and click and no other command', () => { + for (const [command, schema] of Object.entries(COMMAND_OUTPUT_SCHEMAS)) { + const expected = command === PUBLIC_COMMANDS.click || command === PUBLIC_COMMANDS.press; + let groups = 0; + let hits = 0; + for (const properties of propertyGroups(schema as SchemaView)) { + groups += 1; + if (Object.hasOwn(properties, 'screenshotDiff')) hits += 1; + } + assert.equal( + hits > 0, + expected, + `${command}: screenshotDiff advertisement must match the press/click scope`, + ); + if (expected) { + assert.ok(groups > 0, `${command}: press/click schema must have object groups`); + } + } +}); diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index da4461f90d..c6e0966cc0 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -505,7 +505,7 @@ agent-device gesture tap 80 300 200 450 --count 20 --pause-ms 100 # repeated two If rotation stops repeated iOS presses, inspect the screen before continuing with the remaining presses. -A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait `. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command). +A successful `click` or `press` means the action was sent to where the command found the target. It does not confirm that the element received it or that the screen changed. When the next step depends on it, add `--verify` (reports `changedFromBefore`), add `--settle` (returns the diff once the UI goes quiet), or follow with `wait `. When the target lives outside the accessibility tree (a PIN keypad), `--verify` compares empty trees, so add `--expect-screenshot-diff` on `press`/`click` instead: the command compares screenshots taken right before the dispatch and after it settles briefly, and warns when they match outside the device status bar. The readback is best-effort: on a backend without screenshot capture, or when a capture or comparison fails, the response carries neither `screenshotDiff` nor a warning and claims nothing about the screen. Before retrying a failed interaction, see [Retry after a failed command](#retry-after-a-failed-command). `--settle` compares trees in the view of the refs you last read. After a full `snapshot`, it reads the full tree and issues full-tree refs; after `snapshot -i`, it reads the interactive view. It also preserves `-d`, `-s`, `--raw` and `--actions`. This holds for `press`, `click`, `fill`, `longpress`, `scroll` and `back`, including ref, selector and coordinate targets.