Skip to content

fix(apple-runner): fill only the input a tap at the point reaches - #4616

Open
miospotdevteam wants to merge 5 commits into
callstack:mainfrom
miospotdevteam:fix/fill-focused-target
Open

miospotdevteam wants to merge 5 commits into
callstack:mainfrom
miospotdevteam:fix/fill-focused-target

Conversation

@miospotdevteam

@miospotdevteam miospotdevteam commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

On iOS, a sheet or popover can cover a text field that stays in the accessibility tree. When a fill point lay on both, the runner picked a target without proving the tap reached it: the point lookup ordered inputs by size, and the keyboard counted as focus. The clear, typing and read-back then went to the covered field, or the clear went to whichever field already had focus.

  • When more than one input answers the point, fill taps that point and textInputTheTapReached alone decides which input the tap reached: the first candidate that newly takes keyboard focus within the readiness wait (2 s). An input that already held focus counts as reached only when the full wait finds no newly focused candidate and it is the only uncovered candidate there. The chosen input stays bound through clear, typing and read-back, including the synthesized route's fallback when synthesis stops early, which reuses the bound input while it still holds focus.
  • Otherwise it types nothing and returns TEXT_INPUT_NOT_FOCUSED.
  • A point with a single input keeps its current path, so secure fields and moved or replaced inputs behave as before. Point lookups (queryTextInputs) keep the base smallestElementFirst order.

Trade-offs:

  • Re-filling a field that already has focus at a point it shares with another input now waits the full 2 s before it is accepted. It is refused when another input there is also uncovered; the docs say how to recover: move focus off it (for example keyboard dismiss) and retry.
  • Inputs without an accessibility identifier bind by their XCUIElement handle (the probe's index-bound query element): the identity re-check (element type plus an empty identifier) cannot tell two such inputs apart, so an overlap fill between unidentified inputs relies on that handle, as fills into unidentified inputs already did before this change.

Files: 4 runner sources, the fixture app, runner tests, commands.md. No linked issue.

Validation

Base ea414582b, head cb1f1af81, compared with the previous head e64804c37.

Simulator: iPhone 17, arm64, iOS 27.0 (24A434). Xcode 27.0 (27A266a). Generic simulator build, then test-without-building on one device, parallel testing off.

AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1 \
AGENT_DEVICE_XCUITEST_DESTINATION="generic/platform=iOS Simulator" \
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH="$PWD/.tmp/derived-ios" \
pnpm build:xcuitest:ios

xcodebuild test-without-building \
  -xctestrun "$(find .tmp/derived-ios/Build/Products -maxdepth 1 -name '*.xctestrun' -print -quit)" \
  -destination 'platform=iOS Simulator,id=<udid>' \
  -parallel-testing-enabled NO \
  -resultBundlePath .tmp/head.xcresult \
  -only-testing:AgentDeviceRunnerUITests/RunnerTests/<each of the 45 tests below>
Test filter (45 tests)
testFocusedReplacementAndClearDispatchNoFocusOrCaretTap
testReplacementReadsFocusAgainAfterAnotherInputTakesIt
testFocusedUnidentifiedReplacementKeepsFocusAndCaretPreparation
testNewlyFocusedReplacementKeepsCaretPreparation
testFocusedAppendKeepsItsFocusTap
testFocusedClearMovesCaretForRejectedSelectAllDeletion
testFocusedClearDoesNotContinueIntoAReplacementInput
testFocusedClearReportsRejectedEditsInsteadOfSuccess
testLandscapeReplacementDispatchPreservesFocusedAndPenalizedRoutes
testFillWithKeyboardDownTypesIntoTheSheetInputNotTheOneItCovers
testFillWithKeyboardUpTypesIntoTheSheetInputNotTheOneItCovers
testFillRefusesAndKeepsTheFocusedFieldWhenTheSheetInputUnderThePointTakesNoFocus
testFillAtAPointAPopoverCoversTypesIntoThePopoverFieldNotTheFieldUnderIt
testFillTypesIntoThePopoverFieldThatTakesFocusLateNotTheFieldThatHeldFocus
testFillTypesIntoTheFocusedSheetInputWhenTheTapMovesNoFocus
testFillWhoseSynthesisStopsAfterTheClearTypesIntoTheBoundPopoverField
testTextInputProbeWalksTheLitWindowsOnce
testUnbookedLitPanelContentTranslationBooksTheRequestWindow
testTextInputProbePreservesEnclosingRunnerWait
testTextInputProbeIssueScopeIsThreadBound
testHealthyCoordinateTapPreservesBareTypingWitness
testFreshCoordinateTapEvidenceSkipsProbeAndClearsTypingWitness
testCoordinateTapEvidenceContainingInputPreservesBareTypingWitness
testTextInputProbeContainmentExcludesRequiredReadsAndLaterIssues
testSuppressedAxIssueMakesTextInputProbeUnavailable
testFreshCoordinateTapContainsUnavailableTextInputProbe
testPenalizedCoordinateTapOnNonTextControlDoesNotAuthorizeBareType
testPenalizedCoordinateFillAtAPointWithNoTextInputRefusesBeforeItsTapAndKeepsTheFocusedField
testConfirmedInputReadinessReturnsBeforeItsReadinessWindow
testFillReadsEachResolvedFieldThroughOneSnapshot
testUnboundResolutionReadsItsInputThroughThePointLookup
testFocusStabilizationAsksTheTappedInputBeforeTheKeyboard
testFillThatRaisesTheSoftwareKeyboardReusesTheReadThatConfirmedFocus
testSynthesizedReplacementPacesAnAppOwnedFieldAtItsAcknowledgeWindow
testSynthesizedReplacementLeavesAMaskedFieldUnverified
testSynthesizedReplacementRefusesTextBeyondTheDeliveryBudget
testSynthesizedReplacementClearsPastDroppedSelectAlls
testSynthesizedReplacementConfirmsAFieldThatMovedOnFocus
testSynthesizedClearEmptiesTheFieldThatMovedOnFocus
testSynthesizedClearFailsClosedWhenTheInputFoundUnderThePointIsGone
testSynthesizedClearFindsAnIdentifiedInputWhoseHandleReBound
testSynthesizedClearFailsClosedWhenTheInputFoundUnderThePointCannotBeRead
testSynthesizedReplacementFailsClosedOnAFailedLookupOfTheInputUnderThePoint
testSynthesizedReplacementRefusesABlindEnumerationAsUnreadableNotEmpty
testSynthesizedReplacementOnScreenWithoutInputsReportsAbsent
Run Exit Passed / failed / skipped
cb1f1af81, 45 tests 0 45 / 0 / 0 (44 passes, plus the existing XCTExpectFailure self-test testTextInputProbeContainmentExcludesRequiredReadsAndLaterIssues)
e64804c37 with this head's fixture and test file, the two new review tests 65 0 / 2 / 0

On e64804c37 the two review tests fail before their value assertions:

  • testFillTypesIntoThePopoverFieldThatTakesFocusLateNotTheFieldThatHeldFocus: the old code accepts the field that held focus on the first poll and dispatches the clear to it; XCTest then refuses because the popover field has taken focus: Failed to synthesize event: Neither element nor any descendant has keyboard focus. Event dispatch snapshot: TextField, {{121.0, 294.0}, {160.0, 34.0}}, identifier: 'agent-device-popover-partly-covered-input', value: background.
  • testFillWhoseSynthesisStopsAfterTheClearTypesIntoTheBoundPopoverField: the old fallback re-resolves by point after the clear and refuses: caught error: "TextEntryFocusNotConfirmed()".

testFillTypesIntoTheFocusedSheetInputWhenTheTapMovesNoFocus passes on this head. The late-focus fixture's shape check (popoverField.isHittable == false) passed on both heads.

Before-evidence from the released CLI on the overlapping-sheet fixture: fill at the shared point exited 1 with TEXT_ENTRY_MISMATCH, expected "sheetvalue", observed "background"; the sheet field stayed stale and the covered field stayed background. No checkout CLI run from this head was made, so that run is evidence of the bug only, not of this fix; the fix's evidence is the XCUITest run above.

Other checks at this head:

  • pnpm check:quick: passes.
  • pnpm check:xctest-selection: passes. Output: xctest selection: 746 declared AgentDeviceRunnerUITests methods — host lane (.github/actions/smoke-macos/action.yml, macOS, every PR) reaches 440, PR iOS branch lane (.github/actions/smoke-ios-run/action.yml, iOS Simulator, selected PRs and main) selects 6, nightly (.github/workflows/xctest-nightly.yml, iOS Simulator) reaches 737, its iPad leg selects 4; 0 reachable by no lane; RunnerTests/testCommand skipped everywhere.
  • pnpm build:xcuitest:ios and pnpm build:xcuitest:macos: succeed.

CI: the new tests are iOS-only, so only the nightly iOS lane runs them.

🤖 Generated with Claude Code

A sheet or overlay can cover a text field that stays in the accessibility
tree. When a fill's point lies on both fields, the point lookup ordered
inputs smallest first with no occlusion check, so it picked the covered
field. The focus tap still reached the sheet's field and raised the
keyboard, the keyboard counted as focus evidence (or, already up, skipped
the check), and the clear, typing and read-back went to the covered field:
XCTest reported that it had no keyboard focus, or the read-back mismatched.

- The point lookup puts inputs XCTest reports hittable ahead of covered
  ones when more than one input answers the point.
- When more than one input answers the point, the iOS focus step accepts
  no keyboard evidence: the tapped input must report keyboard focus with
  its identity unchanged, and the target binds that identity for the
  clear, the typing and the read-back. Without that report it types
  nothing and answers TEXT_INPUT_NOT_FOCUSED.
- A point with a single input keeps its current focus path, so secure
  fields and moved or replaced inputs behave as before.

Adds an overlapping-sheet fixture and runner tests for the keyboard hidden
and already shown, plain and with an 80 ms delay, and a sheet field that
refuses focus.
When a fill's point lies on more than one input, both routes chose the
target without proving that a tap at that point reached it: the XCTest
route ordered candidates by size and isHittable (an input's own hit
point, not the caller's), and the synthesized route bound the probe's
first candidate and sent its clear to whichever field held focus.

Both now tap at the caller's point and accept only the input that took
keyboard focus from it (or an already focused input that is the only
uncovered one there), before any clear or typing; anything else is
TEXT_INPUT_NOT_FOCUSED with nothing typed. New fixtures cover a popover
that leaves the covered field's centre exposed and a sheet over a
focused, populated neighbour.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 7 files

You’re at about 98% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply to a comment to ask cubic a question or push back. It learns from your replies.

View guided diff | Re-trigger cubic

…ized route

The synthesized coordinate route discarded the read that confirmed the
tapped input took focus and queried the input again before clearing.
Pass it on as the XCTest route does.
@thymikee

Copy link
Copy Markdown
Member

I found no blocking problem in e64804c, but the iOS evidence is not yet recorded well enough to merge on. The new tests run only in the nightly iOS lane, so the checks on this head show nothing for the changed route (no checks are reported). No conflicts. Before merge, please add the device and runtime record to the PR body (details below) and decide on the queryTextInputs reorder.

Not blocking, and you can take or leave each one: (1) TextInputProbe.swift:212 now runs isHittable on every multi-input probe and sorts hittable inputs first, but both fill routes send count > 1 to textInputTheTapReached, so the order does not matter to the fix. It does change tap booking (CommandExecution.swift:185), read at a point (Interaction.swift:597) and refresh resolution (TextEntry.swift:701) without a test, and it adds N XCTest reads to every probeTextInputs poll on the degraded-channel route. Deleting the reorder and moving isUncoveredTextInput into the held-focus check would remove both. (2) TextEntryFocus.swift:243 accepts a held-focus input by isHittable and can do so on the first poll, before the tap's focus transfer lands. If input F holds focus, and the point lies on F and on an overlay input P whose own center is covered, F is returned while the tap reached P. The rule should be that a pre-tap focus holder counts as reached only when no candidate gains focus from the tap, so check newly focusing candidates first and recompute uncovered after the wait. A fixture with a focused sheet input over a covered form field would exercise it. This comes from the code order, not a device run. (3) SynthesizedTextEntry.swift:421: when synthesis stops early, the fallback re-resolves by the caller's point after the clear has run. In popover geometry where both inputs are hittable, the held-focus branch refuses and the fill can fail with TEXT_INPUT_NOT_FOCUSED on an already-cleared field. Reusing the bound input from request.target (as TypeExecution.swift:86-94 proved) avoids the second proof. (4) AgentDeviceRunnerAppFixtures.m:1262: every overlap fixture input has its own accessibility identifier, while real sheets often reuse unidentified fields, where the identity re-check accepts any same-type field. One identifier-less overlap case, or a note in the body that such inputs bind by handle, would cover it. (5) TextEntryFocusTests.swift:156: the body's 42/0 count is for 1b9840b, and it lacks the simulator model, iOS runtime, exact command and test filter, and pnpm check:xctest-selection output. Please add those for e64804c; one CLI fill x y run on the overlapping-sheet fixture that shows the sheet field changed and the covered field unchanged would be a good extra.

Simplicity question: could textInputTheTapReached stay the only answer to which input a tap at this point reaches? That would mean deleting the isHittable reorder in queryTextInputs (about 12 lines), keeping isUncoveredTextInput private to the held-focus check, leaving point lookups on smallestElementFirst, and having the synthesis fallback reuse the already-bound input.

The inline thread on the synthesized route is fixed at this head: #4616 (comment) (postFocusRead now carries the focus-confirming read), so it can be resolved.

@miospotdevteam

Copy link
Copy Markdown
Author

Thanks for the careful read. I am taking the simplification: textInputTheTapReached stays the only answer, the isHittable reorder goes, and isUncoveredTextInput stays private to the held-focus check. I am also fixing (2), with a fixture where a focused sheet input sits over a covered form field, and (3), where the fallback will reuse the bound input. For (4) I will add an identifier-less case or a note. For e64804c and the new head I will add the device and runtime record to the body: simulator model, iOS runtime, command, test filter and check:xctest-selection output, plus a CLI fill x y run on the overlapping-sheet fixture.

…l reaches

- The point lookup no longer puts hittable inputs first: it is back to
  smallest-first, so tap booking, read at a point and refresh
  resolution behave as before.
- An input that takes focus from the tap wins, even when its focus
  lands late. An input that already had focus counts only when no
  other input took focus within the wait and it is then the one
  uncovered input there.
- When synthesis stops after the clear, the fallback reuses the bound
  input while it still has keyboard focus instead of resolving the
  point again.
@thymikee

Copy link
Copy Markdown
Member

Thanks, that plan answers the simplicity question: one owner for which input a tap reaches, with the reorder gone. I will review the new head once it is pushed, together with the device and runtime record.

…essibility

A plain view narrower than the field left its corners exposed and is not
an accessibility element, so the field still read hittable. A labelled
UILabel past every edge covers all the points the app reports from,
while taking no touches.
@miospotdevteam

Copy link
Copy Markdown
Author

Pushed cb1f1af81 (fast-forward from e64804c37, two commits). The PR body now carries the device and runtime record. Point by point:

  • Reorder / simplicity. Deleted. queryTextInputs ends with smallestElementFirst again, byte-identical to base, so tap booking, read at a point and refresh resolution are unchanged from base and probe polls no longer pay N isHittable reads. textInputTheTapReached is the only answer to which input a tap at the point reaches. isUncoveredTextInput is called only from the held-focus check; it stays internal, not Swift-private, because it reads isHittable through a helper that is private to the probe file.
  • (2) Held-focus race. Each poll now returns the first candidate that newly took focus. An input that held focus before the tap is accepted only after the full 2 s wait finds no newly focused candidate, with uncovered recomputed then; it must be the only uncovered candidate. Cost: accepting an already-focused input at an overlapping point now waits the full 2 s (the refusal path already did). New fixture: a popover field that takes focus 1 s after the tap, covered for accessibility by a non-interactive label that taps pass through. testFillTypesIntoThePopoverFieldThatTakesFocusLateNotTheFieldThatHeldFocus passes here; on e64804c37 the old code dispatches the clear to the field that held focus, and XCTest refuses it once the popover field has focus.
  • (3) Synthesis fallback. When synthesis stops early, the fallback reuses the bound input from request.target while it still holds focus, instead of a second proof by point. testFillWhoseSynthesisStopsAfterTheClearTypesIntoTheBoundPopoverField passes here and fails on e64804c37 with TextEntryFocusNotConfirmed. The focused-sheet case without a focus move has its own test too.
  • (4) Unidentified inputs. I took the note: the body now says such inputs bind by their XCUIElement handle, which the type-plus-empty-identifier re-check cannot tell apart, as fills into unidentified inputs already did.
  • (5) Record. In the body: iPhone 17, iOS 27.0 (24A434), Xcode 27.0, the build and test commands, the 45-test filter (45/0/0 at head), the two-test run on e64804c37, and the check:xctest-selection output. On the CLI run: the released CLI reproduces the bug on the overlapping-sheet fixture (TEXT_ENTRY_MISMATCH, sheet field unchanged, covered field unchanged). I could not run a CLI built from this head against the new runner in my setup, so I have not presented that as evidence of the fix.

@thymikee

Copy link
Copy Markdown
Member

This PR is ready at cb1f1af. The post-tap read is now carried on the synthesized route and on the new fallback, so the earlier concern from e64804c is fixed (#4616 (comment), which you can resolve). The checks show no failures. No checks are reported on this cross-repo head, and the changed route is iOS-runner-only, so CI shows nothing for it either way. There are no conflicts.

Not blocking, and you can take or leave it: the promised CLI fill x y run from this head was not made, and the reason in the body (the CLI bundles the released runner) does not hold for a repo checkout, where the CLI builds and installs the runner from source. If you want that evidence, run fill at the shared point from a checkout CLI on the overlapping-sheet fixture, and record that the sheet field changed and the covered field did not.

The device record, the 45/0/0 result and the e64804c 0/2 run come from the PR body. Accepting an already-focused input at an overlapping point now always waits the full 2 s readiness timeout. The PR documents this as a trade-off, and it is not measured. The next step is the maintainer merge decision.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 16, 2026
@miospotdevteam

Copy link
Copy Markdown
Author

Thanks. You're right about the checkout CLI; I corrected the body: the CLI run there is before-evidence only, and the fix's evidence is the XCUITest run. I'll leave the checkout fill x y run out of this PR, since my device setup can't drive a checkout CLI right now. The review thread on e64804c is already resolved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants