Skip to content

feat(keyboard): report the return key in keyboard status - #4602

Open
thymikee wants to merge 1 commit into
mainfrom
feat/keyboard-status-return-key
Open

thymikee wants to merge 1 commit into
mainfrom
feat/keyboard-status-return-key

Conversation

@thymikee

@thymikee thymikee commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Summary

keyboard status now answers what the visible keyboard's return key offers, on both platforms.

$ agent-device keyboard status          # iOS, before: UNSUPPORTED_OPERATION
Keyboard visible: yes                   # after
Return key: done
Return key label: done

Android parses the focused editor's imeOptions out of the dumpsys input_method output the visibility read already fetched, so no extra adb call; IME_FLAG_NO_ENTER_ACTION outranks the action bits, with the app's actionLabel beside it. iOS gains a read-only keyboardStatus runner command that reads the keyboard's own key elements, so a localized return key keeps its wording; keyboard enter's probe order is unchanged. A label is reported only for a key the read identified as the return key: the one whose wording names an action, or the one beside the space bar — so a number or phone pad, which has none, answers unknown with no label. Both project into KEYBOARD_RETURN_KEYS; only Android establishes none. Limrun iOS serves status too. 41 files touched.

Validation

At 4f1bcd0bc9, rebased on current main. pnpm typecheck, format, lint, build, generate:capabilities green; 616 files / 7020 tests. Swift runner unit tests, AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1: 18 cases, 0 failures. No local check:affected receipt for this head; gates are CI's.

Device evidence (iOS Simulator 26.2, this head's runner; Android emulator 17):

  • Contacts → new contact → Phone field (phone pad, Delete bottom-trailing): Return key: unknown, no label line.
  • Settings search, Polish keyboard: unknown / szukaj; English: search / search.
  • Android: Settings search → search; tester email → done.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.33 MB 6.34 MB +10.1 kB
Package (unpacked) 6.33 MB 6.34 MB +10.1 kB
Package (download) 1.92 MB 1.92 MB +3.1 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 53.1 ms 63.4 ms +10.2 ms
CLI --help 115.6 ms 134.6 ms +19.0 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 40 files

You’re at about 96% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread website/docs/docs/commands.md Resolved
Comment thread packages/contracts/src/keyboard.ts Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (a30accf38081).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of fd62e5fd · 16.11.2026 18:32 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 1 no longer raised (1 marked fixed)

No open findings.

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (ea414582b7fa).

Review of dbe52010 · 16.11.2026 17:31 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 1 still open · 1 no longer raised

1 finding(s): 1 minor.

🟡 Minor (1)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift:207 iOS return-key label can only ever be one of nine English probe labels

2 findings below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (cb335952017e).

Review of 3a4d56c3 · 16.11.2026 16:56 UTC

Apex Sentinel · ✅ Approve

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+Keyboard.swift:207 iOS return-key read can only name the nine probe labels
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/UnitTests/RunnerTests+ModelsTests.swift:272 Merge-base exemption now skips witness assertion for volumeUp/volumeDown

Reviewed with this repository's own Sentinel configuration (b3c1a3c8abd9).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 3a4d56c. The verdict and the full list are in the summary comment.

@thymikee

Copy link
Copy Markdown
Member Author

The keyboard help text describes unknown wrongly, so this needs one fix before merge (reviewed at 3a4d56c).

The help string at https://github.com/callstack/agent-device/blob/3a4d56c/packages/command-surface/src/system/index.ts#L322 says unknown means "a key is there that this read cannot name". On iOS, a visible keyboard with no return control (number pad, phone pad) or with a key outside the probe labels also answers unknown. Agents that read this help will assume a submit key exists and try keyboard enter on a number pad. The same claim is open in the commands.md thread, so fixing only the website leaves the help agents read. Please state one rule on every surface that describes unknown (the help string, commands.md, the capabilities note and the contracts doc in keyboard.ts): unknown means the read could not name the return key, including when it found none, and only Android can establish none.

Not blocking, take or leave: the iOS status read classifies the element's localized label instead of the matched probe label or identifier, so a Polish "szukaj" key answers unknown (RunnerTests+Keyboard.swift:213); route and continue in KEYBOARD_RETURN_KEYS have no producer, so the vocabulary could match what producers emit (packages/contracts/src/keyboard.ts:19); and the daemon defaults a visible keyboard's missing return key to unknown, which could move into the WebDriver provider with the field required on visible results (keyboard-runtime.ts:156).

The open inline threads still apply. The review threads on visionOS eligibility (#4602 (comment)), comma truncation of the action label (#4602 (comment)) and the ModelsTests early return (#4602 (comment)) hold, as do the threads on the none type (#4602 (comment)) and the help wording (#4602 (comment)). Four lower-priority threads also still apply.

The live device runs are reported in the PR body. The checks show 16 of 17 passing, and the one pending is a review bot that does not exercise this code. There are no conflicts. Before merge, please fix the unknown wording on every surface and resolve the holding threads.

@thymikee
thymikee added this pull request to stack #4606 November 16, 2026 17:21
Base automatically changed from refactor/apple-runner-return-key-lookup to main November 16, 2026 17:22
@thymikee
thymikee force-pushed the feat/keyboard-status-return-key branch 6 times, most recently from 5981d97 to fd62e5f Compare November 16, 2026 18:23
@thymikee

Copy link
Copy Markdown
Member Author

Review round addressed at fd62e5fd79 — 10 threads, each replied to and resolved.

Contracts and platform mapping

  • The Apple keyboard status variant excludes returnKey: 'none', which only Android's IME_ACTION_NONE reading establishes; the Apple mapper accepts the shared vocabulary minus that word and degrades anything else to unknown.
  • keyboard status is gated on the handheld Apple leaf. visionOS previously would have answered "keyboard hidden" for a keyboard that may be up, because the runner's visibility read is #if os(iOS). dismiss and enter keep their existing eligibility.
  • The owner matrix now names each owner's legs from its own declarations: Android and the Apple runner three (the Apple runner on iOS/iPadOS), HarmonyOS dismiss/enter, WebDriver all three on Android and dismiss only on iOS.

iOS status read no longer needs English
It reads the keyboard's own key elements: a control whose wording kind(of:) maps is the return key in any language; otherwise the bottom-trailing measurable key is, the slot Apple's keyboards give it. Its trimmed wording is always reported, so the Polish keyboard answers unknown with returnKeyLabel: "szukaj" on-device, and the unknown docs now say what each platform's read actually proved. findKeyboardReturnControl/probeLabels are unchanged press-only code.

Android
A custom actionLabel keeps its commas and stops at the next key=value field, }/], or end of line, so Save, now is no longer truncated to Save.

Help, docs and tests
Help text and the output schema each have their own assertion over the twelve returnKey words; returnKeyLabel is described as trimmed everywhere users read it. The merge-base witness exemption is now per assertion and covers only keyboardStatus, so volumeUp/volumeDown are back in the witness column (they were never in the replay column).

Also: the Swift trailing-comma guard CI flagged is fixed — that syntax needs Swift 6.1+ — verified across apple/runner.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the update. I reviewed fd62e5f against the earlier findings (#4602 (comment)). Those are fixed, but the new iOS return-key fallback adds one problem. CI is green: 22 checks, 0 not passing at fd62e5f. There are no conflicts.

When no observed key's wording maps to a known return kind, observed(visible:keyboardKeys:) takes the worded key with the largest (maxY, maxX) as the return key (RunnerKeyboardStatus.swift#L75). A keyboard with no return key at all (number pad, decimal pad, phone pad: OTP and phone fields) has Delete in that bottom-trailing slot. Status would then answer unknown with returnKeyLabel set to "Delete". The help and commands.md say the label is the wording the return key itself shows, so an agent would be told a return key exists on an OTP field. That is the false positive the earlier wording fix removed. The fixtures only cover QWERTY layouts, where Delete sits above the last row (RunnerTests+KeyboardReturnKeyTests.swift#L128-L154). The rule the code must satisfy: the label comes only from a control the read identifies as the return key, and a keyboard with no identifiable return control answers unknown with no label. Could you limit the geometric fallback to the key right of the space bar in the space bar's row (no space bar, no candidate)? Please add a number-pad fixture (digits, 0, Delete bottom-right, no space bar) that expects unknown and a nil label. For live proof, run keyboard status on a numberPad or phonePad field on the iOS Simulator. It should print Return key: unknown and no Return key label line. This comes from reading the code, so a number-pad run will confirm it. Could the dictation or globe buttons on Face ID iPhones, if exposed under app.keyboards.buttons, also win that slot on a localized keyboard? Your Polish simulator run reported "szukaj".

The review threads on the keyboard variant, visionOS refusal, unknown wording, the merge-base witness, the dumpsys label parse, the help word list, the runtime plan doc and the whitespace doc are all fixed at this head; please resolve them. The volumeUp/volumeDown witness thread is also fixed (thread). Its other thread was marked as no longer raised, and the cited probe code is replaced by observedKeyboardKeys (thread). Please resolve both too.

Not blocking, take or leave: the head commit message still says the iOS read "reads the same return control keyboard enter presses" and that unknown "says a key is there this read cannot name". Both are false now, so please reword it, or the squash body, to match the PR body.

Before merge, the number-pad case needs the fix, a fixture and a live run.

`keyboard status` now answers what the visible keyboard's return key offers, on both
platforms. Android parses the focused editor's `imeOptions` action out of the `dumpsys
input_method` output the visibility read already fetched, so no extra adb call is made;
`IME_FLAG_NO_ENTER_ACTION` outranks the action bits, and an app's `actionLabel` is reported
beside its action kind. iOS gains a read-only runner status command that reads the
keyboard's own key elements, so a localized return key keeps its wording instead of
needing English to be named; `keyboard enter`'s probe list and order are unchanged.

Both readings project into one vocabulary (`KEYBOARD_RETURN_KEYS` in the contracts package):
`none` is only Android's to establish, `unknown` means this read could not name the key
rather than that the keyboard has one, and a hidden keyboard reports neither field. A label
is reported only for a control the read identified as the return key, so a number, decimal
or phone pad — whose bottom-trailing key is Delete and which has no return key at all —
answers `unknown` with no label. Snapshot output and its device calls are unchanged.

- Apple: `keyboardStatus` runner command (read-only, app-windows surface), `RunnerKeyboardStatus`
- Android: `parseAndroidReturnKey` in `device-input-state.ts`
- Daemon: projects `returnKey`/`returnKeyLabel`; a visibility-only owner publishes `unknown`
- Output: `Return key:` and `Return key label:` lines, plus the JSON fields
@thymikee
thymikee force-pushed the feat/keyboard-status-return-key branch from fd62e5f to 4f1bcd0 Compare November 16, 2026 21:33
@thymikee

Copy link
Copy Markdown
Member Author

Fixed at 4f1bcd0bc9 (rebased on current main; CI was green at fd62e5f).

The rule. returnKeyLabel now comes only from a control the read identifies as the return key: one whose wording kind(of:) names, or the one in the return key's slot. The geometric fallback is anchored on the space bar rather than on the keyboard's bottom-right corner — the candidate must carry wording, share the space bar's row, and start at its right edge (spaceBar.maxX - 2, nearest wins). No space bar, no candidate, so a pad answers unknown with no label. (RunnerKeyboardStatus.returnKeySlot)

The anchor is a control whose wording is whitespace and nothing else, which is how the tree carries the space bar (' ', measured on English and Polish layouts); it is picked as the widest such control so a zero-frame padding element cannot impersonate it. That meant observedKeyboardKeys had to stop dropping whitespace-only wording, which is why the space bar is now in the observed set and never a candidate itself — no return key's wording is whitespace.

Dictation and globe. Neither can win the slot on the layout I measured (iOS 26.2, iPhone 17, Polish QWERTY + emoji). Dictation is exposed as a Dictate button at y=811, above the keyboard's top edge at y=952, so it is the app's control and not under app.keyboards at all. The keys beside the space bar were cyfry at x=5 and Emoji at x=54, against the space bar's 103→301, so both are left of it and the space-bar row rules already exclude them; no separate globe control was exposed inside the keyboard window. The rule is positional, so a layout that put one on the space bar's right would still be excluded only by nonReturnKeyLabels — that list is the backstop, and its wording is English, which is why position is the guard. That is why the Polish run still reported szukaj.

Fixtures (RunnerTests+KeyboardReturnKeyTests.swift): the phone pad I captured (digits, 0, Shift, Delete bottom-trailing, no space bar) → unknown + nil label; a keyboard with Globe/Dictation beside the space bar, and one with a globe alone in the slot → never reported as the return key; wording outranks a populated slot; and the no-candidate set now covers a pad with no space bar and a space bar with nothing beside it.

Live, this head's runner, iOS Simulator 26.2. Contacts → Add → Insert add phone (phone pad focused):

$ agent-device keyboard status
Keyboard visible: yes
Return key: unknown
$ agent-device keyboard status --json
{ "platform": "ios", "action": "status", "visible": true, "returnKey": "unknown" }

No Return key label line, and no returnKeyLabel field. Re-run of the localization case: Settings search with the Polish keyboard → unknown / Return key label: szukaj; with the English keyboard → search / search.

Also done: the commit message no longer says the read "reads the same return control keyboard enter presses" or that unknown "says a key is there this read cannot name"; the help text, commands.md, KEYBOARD_RETURN_KEYS docs and the keyboard variant doc now state that a label appears only for an identified return key. All ten threads are resolved, including the two keyboard enter ones. Validation: 18 Swift unit cases 0 failures, 616 files / 7020 tests, typecheck, format, lint, build and the Swift trailing-comma guard clean.

@thymikee

Copy link
Copy Markdown
Member Author

The earlier findings from fd62e5f are fixed at 4f1bcd0, and one new problem remains in the iOS slot fallback.

In RunnerKeyboardStatus.swift:117, the slot rule takes the nearest worded control right of the space bar. On the iPad full keyboard, the return key ends the third row. The bottom row right of the space bar holds the numbers key and then the hide-keyboard key. If no key maps through kind(of:), for example on a localized keyboard or with an unmapped return type, the rule picks the numbers key. Status then reports returnKey unknown with returnKeyLabel set to that key's wording. This contradicts the new text in commands.md that a label appears only for a key the platform identified as the return key. The rule holds on iPhone only. The slot candidate should be the space bar's immediate right neighbour and also the trailing worded control of its row, and otherwise there should be no candidate. Please add an iPad bottom-row fixture (numbers key and hide key right of the space bar, localized return key in row 3) that expects unknown and a nil label. Please also add one live keyboard status run on an iPad simulator with a non-English keyboard on a search field. It should show no "Return key label" line, or the real return key's wording.

Could one positional rule replace the nearest-right rule and the English nonReturnKeyLabels denylist at RunnerKeyboardStatus.swift:121 and :135-138? The candidate would be the space bar's right neighbour that is also its row's trailing key. That excludes pads, the iPad numbers and hide keys, and a globe or dictation key on the right, so the denylist and its fixture could be deleted.

Of the earlier inline threads, all ten are fixed at this head and can be resolved: #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment), #4602 (comment).

The phone-pad live run covers the number-pad case. The iPad layout claim comes from how iPadOS lays out its keyboard, not a device run, so the fixture and live run will confirm or refute it. There are no conflicts. CI is green, with 22 checks and none failing at 4f1bcd0. Before merge, the iOS slot fallback must stop an iPad layout from supplying returnKeyLabel, with the iPad fixture and the live iPad run above.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant