Skip to content
Open
22 changes: 19 additions & 3 deletions docs/adr/0025-foldable-apple-panels.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Only inference answers the second question.
| More than one integrated panel | Pass `--display=<panel name>` on every `simctl io screenshot`. Never use the implicit default. |
| Zero or multiple lit panels with a `primary` panel | Name the `primary` panel. Emit `apple_display_capture_ambiguous`. Report pose as `unknown`. |
| No sole lit panel and no `primary` panel | Refuse with `COMMAND_FAILED` and `details.reason: capture-panel-unidentified`. Never choose a panel by array order. |
| One integrated panel | Preserve pre-panel behavior exactly: no display flag, no pose, unchanged scale probe. |
| One integrated panel | Preserve pre-panel behavior exactly: no display flag, no pose, unchanged scale probe. Once a resolved table shows one panel, read the table beside the capture for `displayRotation` only. |
| Density normalization | Use the captured panel's `pointScale`. For runner fallback, use that capture's reported pixels-per-point. |
| Pose report | From panel power alone, report only `closed`, `fully-open`, or `unknown`, never a narrower pose. `fold` reports an exact pose only after hinge readings verify it. |
| Pose request | `agent-device fold <closed\|half-open\|open>` sends private HID inside the simulator. Then it reads the hinge angle back inside the simulator until the angle agrees. For `half-open`, two consecutive readings must agree. If the readings never agree, refuse the pose. |
Expand Down Expand Up @@ -100,8 +100,8 @@ It is the only reader of panel tables, and it selects the source.
The inventory reports measured `primary`, panel power, geometry, and orientation.
It leaves exact pose to the app under test through `UIHinge.status`.
If a caller needs pose, it asks the operator and takes another snapshot.
- **No cache.**
The probe executes per capture.
- **No cached lit-panel inventory.**
The probe executes per capture. The daemon caches only each device's multi-panel verdict, a hardware fact (see the 2026-11-16 amendment).
An operator changes the lit panel through a fold or open action.
Cached inventory could resume capture of the dark panel and recreate the failure.
`IOS_APPLE_DISPLAY_PROBE_TIMEOUT_MS` limits the probe to 5s, below the subsequent 20s screenshot deadline.
Expand Down Expand Up @@ -755,3 +755,19 @@ Consequences:
- `runner.log` prints `AGENT_DEVICE_RUNNER_LIT_PANEL_WINDOW lit=<id>(<w>x<h>) windows=<n> outcome=<reason> frame=(…)`. `SYNTHESIZED_DISPATCH reference=` must equal that frame. Snapshot points are translated by that frame's origin, as for a placed iPad window, and a synthesized turn is about the lit panel's screen in the app's interface orientation, not the window's shape: an app beside the inner panel's 80 pt status strip still maps into the whole screen. `SYNTHESIZED_RECORD displayID=` must equal `lit=`.
- A recording keeps the panel that its `recordStart` named. The runner cannot read panel power, so it does not notice a fold during the recording: SpringBoard keeps windows on the panel that went dark, and later frames show that dark panel. Start a new recording after a fold.
- Open gap: on the folded home screen, XCTest reports the page icons with the inner panel's layout in both panels' windows. The Dock and the page indicator report the outer panel correctly.

## Amendment: a single-panel table read runs beside the capture (2026-11-16)

This amendment narrows the capture rule "first resolve the panel table" to devices known to be single-panel.
The display inventory caches each device's multi-panel verdict from its latest resolved table for the daemon's lifetime; an unresolved read does not change it.
When the verdict is single-panel, the capture names no display, so the table read only supplies `displayRotation`.
That read then starts beside `simctl io screenshot` with a 2s budget, in both simulator sets.
A read still running 250ms after the image is written and the status bar is released is aborted, and the capture reports no rotation, as for an unresolved table.
A multi-panel device or a device with no resolved verdict keeps the read before the capture, so a fold still names the lit panel.
If a read beside a capture resolves to a multi-panel table, including one that refuses with `capture-panel-unidentified`, that capture has already gone out without `--display`. It takes simctl's implicit default and can be the dark-panel black image.
The capture then emits `apple_display_single_panel_count_stale` with a retake hint, and the resolved read replaces the verdict, so the next capture reads the table before it captures and names the lit panel.

Rejected alternatives:

- **A cached panel table: rejected.** The lit panel changes on a fold, and rotation changes without an agent-device command.
- **Waiting for the read without a bound: rejected.** Rotation is optional metadata and must not delay the image.
4 changes: 4 additions & 0 deletions packages/capture-kit/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@
"types": "./src/capture-admission/audio-probe-session-resource.ts",
"default": "./src/capture-admission/audio-probe-session-resource.ts"
},
"./capture-fact-grace": {
"types": "./src/capture-fact-grace.ts",
"default": "./src/capture-fact-grace.ts"
},
"./durable-capture": {
"types": "./src/durable-capture/index.ts",
"default": "./src/durable-capture/index.ts"
Expand Down
21 changes: 21 additions & 0 deletions packages/capture-kit/src/capture-fact-grace.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
/**
* Waits at most `graceMs` for optional capture metadata that is read beside the image, such as
* the display rotation. A read still running at the deadline yields `undefined`; the caller then
* aborts it and reports the fact as absent.
*/
export async function resolvedWithinGrace<T>(
pending: Promise<T>,
graceMs: number,
): Promise<T | undefined> {
let timer: NodeJS.Timeout | undefined;
try {
return await Promise.race([
pending,
new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), graceMs);
}),
]);
} finally {
clearTimeout(timer);
}
}
17 changes: 2 additions & 15 deletions packages/platform-android/src/screenshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { emitAndroidAdbDiagnostic, requireAndroidAdbHost } from './adb-host.ts';
import { androidAdbResultError } from './adb-executor.ts';
import { probeAndroidDisplayRotation } from './input-actions.ts';
import type { AndroidHelperScreenshot } from './automation-helper.ts';
import { resolvedWithinGrace } from '@agent-device/capture-kit/capture-fact-grace';

// PNG file signature: 0x89 P N G \r \n 0x1A \n
const PNG_SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
Expand Down Expand Up @@ -164,26 +165,12 @@ async function captureAndroidScreencapWithRotation(
throw new AppError('COMMAND_FAILED', 'Failed to capture screenshot');
}
await writeAndroidPng(bytes, outPath);
return rotationFacts(await settledWithin(probe, ANDROID_SCREENSHOT_ROTATION_GRACE_MS));
return rotationFacts(await resolvedWithinGrace(probe, ANDROID_SCREENSHOT_ROTATION_GRACE_MS));
} finally {
probeController.abort();
}
}

async function settledWithin<T>(pending: Promise<T>, graceMs: number): Promise<T | undefined> {
let timer: NodeJS.Timeout | undefined;
try {
return await Promise.race([
pending,
new Promise<undefined>((resolve) => {
timer = setTimeout(() => resolve(undefined), graceMs);
}),
]);
} finally {
clearTimeout(timer);
}
}

async function writeAndroidPng(bytes: Buffer, outPath: string): Promise<void> {
// On multi-display devices (e.g. Galaxy Z Fold), adb screencap may write a
// warning to stdout before the PNG data. Strip any leading garbage by
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import {
establishedLitPanel,
buildInventory,
parseCoreDeviceDisplays,
knownAppleMultiScreen,
readPanelPower,
resolveAppleCaptureDisplay,
queryAppleDisplayInventory,
Expand Down Expand Up @@ -567,6 +568,84 @@ describe('resolveAppleCaptureTarget', () => {
});
});

describe('knownAppleMultiScreen', () => {
beforeEach(() => {
mockRunXcrun.mockClear();
mockEmitDiagnostic.mockClear();
mockReadHostTextFile.mockReset();
mockReadHostTextFile.mockResolvedValue('');
});

test('keeps the panel count of the latest resolved table, not of an unresolved one', async () => {
const device = { ...IOS_TEST_SIMULATOR, id: `${IOS_TEST_SIMULATOR.id}-panel-count` };
assert.equal(knownAppleMultiScreen(device), undefined);

mockReadHostTextFile.mockResolvedValueOnce(JSON.stringify(IPHONE_17_SINGLE_PANEL));
await queryAppleDisplayInventory(device, {});
assert.equal(knownAppleMultiScreen(device), false);

mockRunXcrun.mockResolvedValueOnce({
exitCode: 1,
stdout: '',
stderr: 'The specified device was not found',
} as Awaited<ReturnType<typeof runXcrun>>);
assert.equal((await queryAppleDisplayInventory(device, {})).unresolved, true);
assert.equal(knownAppleMultiScreen(device), false);

mockReadHostTextFile.mockResolvedValueOnce(JSON.stringify(IPHONE_DUO_CLOSED));
await queryAppleDisplayInventory(device, {});
assert.equal(knownAppleMultiScreen(device), true);
});

test('a refused multi-panel table still records the multi-panel verdict', async () => {
// A single-panel capture's read beside it relies on this to send the next capture through the
// table read, even when the table names no panel to capture.
const device = { ...IOS_TEST_SIMULATOR, id: `${IOS_TEST_SIMULATOR.id}-unidentified` };
mockReadHostTextFile.mockResolvedValueOnce(JSON.stringify(IPHONE_17_SINGLE_PANEL));
await queryAppleDisplayInventory(device, {});
assert.equal(knownAppleMultiScreen(device), false);

mockReadHostTextFile.mockResolvedValueOnce(
JSON.stringify({
result: {
displays: IPHONE_DUO_CLOSED.result.displays.map((entry) => ({
...entry,
primary: false,
backlightState: 'activeOn',
})),
},
}),
);
await assert.rejects(
resolveAppleCaptureTarget(device),
(error: unknown) =>
error instanceof AppError && error.details?.reason === 'capture-panel-unidentified',
);
assert.equal(knownAppleMultiScreen(device), true);
});

test('a read its caller aborted reports the cancellation, not a toolchain failure', async () => {
// The fold hint points at this diagnostic, so a CoreDevice read its caller aborted must still
// fire it and say why. A guest read rethrows the cancellation instead and reaches no hint.
const device = { ...IOS_TEST_SIMULATOR, id: `${IOS_TEST_SIMULATOR.id}-aborted-read` };
const controller = new AbortController();
mockRunXcrun.mockImplementationOnce(async () => {
controller.abort();
throw new Error('aborted');
});
const inventory = await queryAppleDisplayInventory(device, { signal: controller.signal });
assert.equal(inventory.unresolved, true);
const unresolved = mockEmitDiagnostic.mock.calls
.map(([event]) => event)
.filter((event) => event.phase === 'apple_display_inventory_unresolved');
assert.equal(unresolved.length, 1);
assert.equal(unresolved[0]?.data?.reason, 'probe-canceled');
assert.equal(unresolved[0]?.level, 'info', 'a canceled read is expected, not a warning');
assert.equal(unresolved[0]?.data?.hint, undefined);
assert.equal(knownAppleMultiScreen(device), undefined);
});
});

describe('appleSimulatorDisplayArgvFragment', () => {
test('names the panel for capture commands that share the same implicit default', () => {
assert.deepEqual(appleSimulatorDisplayArgvFragment(outerPanel), ['--display=LCD']);
Expand Down Expand Up @@ -621,6 +700,7 @@ describe('captureSimulatorScreenshotWithFallback runner fallback', () => {
pixelDensity: 2,
deps: {
ensureBooted: async () => {},
isKnownSinglePanel: () => false,
resolveCaptureTarget: async () => ({ display: litPanel }),
captureWithRetry: async () => {
throw new Error('simctl screenshot failed');
Expand Down Expand Up @@ -653,6 +733,7 @@ describe('captureSimulatorScreenshotWithFallback runner fallback', () => {
pixelDensity: 2,
deps: {
ensureBooted: async () => {},
isKnownSinglePanel: () => false,
resolveCaptureTarget: async () => ({ display: outerPanel }),
captureWithRetry: async () => {
throw new Error('simctl screenshot failed');
Expand Down Expand Up @@ -680,6 +761,7 @@ describe('captureSimulatorScreenshotWithFallback runner fallback', () => {
pixelDensity: 2,
deps: {
ensureBooted: async () => {},
isKnownSinglePanel: () => false,
resolveCaptureTarget: async () => ({ display: outerPanel }),
captureWithRetry: async () => {},
normalizeDensity: async (_device, _path, _density, sourcePixelDensity) => {
Expand Down
Loading
Loading