Skip to content

feat(ios): name the dismiss controls when back changes nothing - #4628

Open
thymikee wants to merge 4 commits into
mainfrom
t3/ios-back-dismiss-controls
Open

thymikee wants to merge 4 commits into
mainfrom
t3/ios-back-dismiss-controls

Conversation

@thymikee

@thymikee thymikee commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Summary

iOS back still only pops the navigation stack (BackButton, else the verified edge swipe) and never presses Close or Cancel. On back_swipe_no_change (e.g. a full-screen modal) the refusal adds details.dismissControls: [{ selector, label, identifier? }]: hittable top-band buttons named back, close, cancel or xmark*, never Done/Save. The hint names a lone one. Help (shortened to fit the always-loaded MCP budget), commands.md and RUNNER_PROTOCOL.md say iOS back does not close modals or sheets, unlike Android's system back. 9 files.

  • Selector, not ref: back expires the ref frame before it swipes (ADR 0014), so each entry carries role=button label="…" or id="…". A control whose label or identifier another button shares is omitted.
  • Cost: one bounded tree read, only on this refusal.

Validation

iPhone 17 Pro simulator, iOS 27.0, at 8a3d7f0. The rebase onto fce2fa6 changed no back code.

  • Wallet Top up: base 26a8a1b refused with no list. Now --in-app and --system list role=button label="Close", and press with it lands on Home.
  • transactions/tx-0011: pops to Home via the verified swipe (its header "Back" is a JS button, not BackButton); path unchanged.
  • Home root: back_swipe_no_change, dismissControls absent.
  • 01-settings replay passes; its back pressed BackButton.
  • 7 focused iOS XCUITests passed at 8a3d7f0. At 2df99ee the macOS models tests, MCP budget and doc guards, and apple-runner vitest pass. CI owns the full gates.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.35 MB 6.36 MB +5.6 kB
Package (unpacked) 6.35 MB 6.36 MB +5.6 kB
Package (download) 1.93 MB 1.93 MB +1.5 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 21.7 ms 20.2 ms -1.5 ms
CLI --help 44.2 ms 44.1 ms -0.1 ms

thymikee and others added 3 commits November 17, 2026 12:20
iOS back stays the navigation pop: the BackButton press, else the
verified edge swipe. When the swipe changes nothing, such as on a
full-screen modal, the refusal now lists the hittable Back, Close,
Cancel or xmark buttons in the window's top band as
details.dismissControls, each with a selector the caller can press
without a new snapshot. The hint names a lone control. Back never
presses them: Close or Cancel can discard a draft or end a flow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the dismiss controls with an exact predicate and their snapshot
frames, ask hittability last, and have the runner name the attribute
each control is pressed by. A control whose label or identifier another
button shares is left out, since its selector would match both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… back help

The every-field ErrorPayload fixture sets dismissControls so the encoder
contract covers it. The back description is rewritten shorter than
before, so the always-loaded MCP tools stay within their byte budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 2 still open · 0 no longer raised

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift:748 Dismiss controls are read from every window but judged in one window's frame space
packages/apple-runner/src/runner-contract.ts:592 Ambiguity is checked among buttons only, but the emitted id= selector matches any role

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (fce2fa6247c5).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of 8a3d7f07 · 17.11.2026 11:22 UTC

Apex Sentinel · ✅ Approve

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift:711 Ambiguity is checked among buttons only, but the emitted selector matches any element type
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift:749 dismissControls is read from every window, so a sheet's button can be pressed for the wrong window

Reviewed with this repository's own Sentinel configuration (fce2fa6247c5).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 8a3d7f0. The verdict and the full list are in the summary comment.

@thymikee
thymikee force-pushed the t3/ios-back-dismiss-controls branch from 8a3d7f0 to 2df99ee Compare November 17, 2026 11:22

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 findings moved to a new line on 2df99ee. The verdict and the full list are in the summary comment.

Comment on lines +748 to +761
let deadlineMs = currentUptimeMs() + Self.navigationBarBackReadBudgetMs
// Snapshot frames are in the window's own space, which a placed iPad window starts off the screen origin.
let window = CGRect(origin: .zero, size: context.geometry.windowOnScreen.rect.size)
let read = heldNavigationBarRead(.dismissControlLookup) { () -> [DismissControlPayload]? in
guard let roots = try? elementSearchRoots(root: app, litDisplayID: requestLitPanel?.displayID) else { return nil }
let elements = roots.flatMap {
$0.descendants(matching: .button).matching(Self.dismissControlPredicate).allElementsBoundByIndex
}
var buttons: [TopBandButton] = []
for element in elements {
guard currentUptimeMs() < deadlineMs, let snapshot = try? element.snapshot() else { return nil }
buttons.append(TopBandButton(label: snapshot.label, identifier: snapshot.identifier, frame: snapshot.frame))
}
return Self.dismissControls(buttons, window: window) { elements[$0].isHittable }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor · Dismiss controls are read from every window but judged in one window's frame space

On a multi-window iPad, topBandDismissControls collects buttons from every elementSearchRoots window into one flat list but tests each against a rect built from context.geometry.windowOnScreen.rect.size, so a button in another window gets compared in the wrong coordinate space—a control below the band in its own window is listed while one inside it is dropped, and a listed selector presses the button in the non-focused window. The flat list also makes the shared-attribute check cross windows, so a Close in the focused sheet is silently omitted because an unrelated background window also has a Close.

Suggested change

Pair each window's roots with its own rect and run both the band test and the shared-attribute test within one window, listing only controls from the window actually swiped on.

Details

Also raised here

Also raised at this location by persona swift-ios: Dismiss controls are read from every window but filtered against one synthesized window rect — On iPad, where the app has several windows on the lit display, a control that back cannot act on is listed as the way out. elementSearchRoots returns every lit window and the button query runs over all of them, while the band test uses a single rect CGRect(origin: .zero, size: context.geometry.windowOnScreen.rect.size). Snapshot frames are in each window's own space (as the comment at line 749 itself says, and as navigationBarBackState(_:holdsButton:geometry:) compensates for at line 453), so a button in a secondary window near that window's own origin has a midY inside the synthesized band and passes, even though that window sits elsewhere on screen.

persona: javascript, swift-ios, senior-developer

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c13ce51. topBandDismissControls lists candidates only from the window back swiped on (context.resolvedWindow, or every search root when none resolved), so each frame is judged in its own window's space. Uniqueness is still checked against the buttons of every search root, because press resolves against the whole tree: a same-named Close in a second window makes the entry ambiguous and drops it. testDismissControlsComeFromTheBackWindowAndAreUniqueAcrossEveryWindow covers the root choice and the cross-window collision.

Comment on lines +592 to +597
const selector =
pressBy === 'identifier' && id !== undefined
? `id=${JSON.stringify(id)}`
: pressBy === 'label' && label.length > 0
? `role=button label=${JSON.stringify(label)}`
: undefined;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor · Ambiguity is checked among buttons only, but the emitted id= selector matches any role

A non-button element (image or group) carrying the same accessibility identifier as a pressable button causes the press to be refused as ambiguous. The Swift uniqueness check at line 731 compares only buttons, but the host turns pressBy: identifier into a bare id="…" selector with no role constraint.

Suggested change
const selector =
pressBy === 'identifier' && id !== undefined
? `id=${JSON.stringify(id)}`
: pressBy === 'label' && label.length > 0
? `role=button label=${JSON.stringify(label)}`
: undefined;
const selector =
pressBy === 'identifier' && id !== undefined
? `role=button id=${JSON.stringify(id)}`
: pressBy === 'label' && label.length > 0
? `role=button label=${JSON.stringify(label)}`
: undefined;

Suggested change

Constrain the id= selector to the button role so it matches what the runner checked for ambiguity, or check identifier uniqueness across every element type, not only buttons.

Details

persona: senior-developer, swift-ios

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if it is wrong — I will re-check the cited code.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c13ce51. Both selector forms now carry role=button: role=button label="…" and role=button id="…" (the bare id="…" form is gone). A two-term selector does not take the direct iOS runner lookup, so press resolves it in the snapshot tree, where role=button admits only XCUIElementTypeButton nodes. That is the set the runner checks for uniqueness. packages/selectors/src/internal/match.test.ts pins this: with a StaticText and an Image that share the label and identifier, both forms resolve uniquely to the button, and a bare label="Close" is ambiguous. The runner also compares text the way the selector engine does (trimmed, lowercased, whitespace collapsed) and searches with CONTAINS[c], so a button that reads the same to press always counts as a collision.

… selectors

Both dismiss control selectors now carry role=button, so press resolves
them against buttons only, the set the runner checks for uniqueness;
a selectors test pins that scope for the label and id forms. The runner
lists controls only from the window back swiped on and checks each
label or identifier against the buttons of every window, compared the
way the selector engine normalizes text. The test file is renamed for
the module it covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thymikee

Copy link
Copy Markdown
Member Author

The change at c13ce51 reads well, but the live device run is stale. It covers 8a3d7f0, and c13ce51 changed the runner read afterwards: listing roots now come from context.resolvedWindow, and uniqueness is split across two queries. A modal whose Close button is not under the resolved window element would silently lose its list, and nothing validated at this head would show it. Please rerun at c13ce51 on an iOS simulator. Back on Wallet Top up should return reason back_swipe_no_change with details.dismissControls equal to [{selector:'role=button label="Close"',label:'Close'}] and the single-control hint. Then press 'role=button label="Close"' should land on Home. Back at the Home root should have no dismissControls. testInAppBackNeverPressesABarButtonOnlyTitledBack and the two new Navigation unit tests should pass at c13ce51. The changed read is here: https://github.com/callstack/agent-device/blob/c13ce51/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Navigation.swift#L783

Of the open inline threads, #4628 (comment) (frame space) is outdated and fixed at c13ce51, because listing now uses dismissControlRoots(resolvedWindow:) (Navigation.swift:720-722, :783). #4628 (comment) (selector role) is also fixed, because both selector forms carry role=button (runner-contract.ts:595-598) and iOS role=button maps only to XCUIElementTypeButton. Please resolve both.

Not blocking, and you can take or leave these. A listed selector should name exactly one node in the tree press resolves against, but uniqueness at Navigation.swift:748 compares the raw XCTest label while press matches the snapshot's aggregated label (RunnerTests+SnapshotAcquisition.swift:189, 286-297), so a button with an empty label and a child text 'Close' would be missed; compare on the snapshot label or drop pressBy=label in that case. selectorText (Navigation.swift:676) copies normalizeText from packages/selectors, so one shared fixture case such as ' Cancel\t' vs 'cancel' asserted on both sides would tie them together. The new selectors test at match.test.ts:282 pins existing behavior and is fine as a contract pin. When resolvedWindow is nil, matching() runs the same button query twice (Navigation.swift:775), so reusing the first pass's snapshots would avoid that.

The lint failure is oxlint max-lines on scripts/layering/package-boundaries.test.ts (1001 lines). This PR does not touch scripts/, and the file is 1001 lines on main at fce2fa6 too, so it is not caused by this change. There are no conflicts. The live claims in the PR body are from 8a3d7f0. Before merge, we need the live run at c13ce51 described above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant