Repository navigation
feat(keyboard): report the return key in keyboard status #4602
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
+1,195
−121
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
143 changes: 143 additions & 0 deletions
143
apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerKeyboardStatus.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| import Foundation | ||
| import CoreGraphics | ||
|
|
||
| /// One `keyboard status` answer: whether the software keyboard is on screen and, when it is, what | ||
| /// its return key offers. | ||
| /// | ||
| /// `returnKey` is a `RunnerKeyboardReturnKey` raw value, or `nil` when the keyboard is hidden — a | ||
| /// hidden keyboard has no return key to report, which is a different fact from one this read cannot | ||
| /// name. `returnKeyLabel` is the wording the return key itself carries, with surrounding whitespace | ||
| /// trimmed, so a localized or an app's custom key stays readable even where the kind is `unknown`. | ||
| /// It is present only for a control this read identified as the return key: a keyboard whose return | ||
| /// key cannot be identified — a number or phone pad has none — answers `unknown` with no label, so a | ||
| /// caller is never told a key exists on a layout that has no key to press. | ||
| struct RunnerKeyboardStatus: Equatable { | ||
| let visible: Bool | ||
| let returnKey: String? | ||
| let returnKeyLabel: String? | ||
| } | ||
|
|
||
| extension RunnerKeyboardStatus { | ||
| /// One control the status read observed on the visible keyboard: the wording it carries in the | ||
| /// accessibility tree, and the frame it occupies. `frame` is what locates the return key by slot — | ||
| /// the control immediately right of the space bar in the space bar's own row. | ||
| struct ObservedKeyboardKey { | ||
| let text: String? | ||
| let frame: CGRect | ||
| } | ||
|
|
||
| /// What identifies Apple's space bar in the accessibility tree: wording that is whitespace and | ||
| /// nothing else. This read measured a single space on an English and a Polish iOS layout, so the | ||
| /// anchor survives the localization the positional rule exists for. A pad's keys all carry digit or | ||
| /// `Delete` wording, so a pad has no anchor, and therefore no candidate. | ||
| private static func isSpaceBar(_ text: String?) -> Bool { | ||
| guard let text, !text.isEmpty else { return false } | ||
| return text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty | ||
| } | ||
|
|
||
| /// Builds the answer from what the runner observed. The keyboard decides whether a return key is | ||
| /// reported at all: a hidden one has none, which is a different fact from one this read cannot | ||
| /// name. Only Android's `IME_ACTION_NONE` can establish that a field offers no action key, so | ||
| /// `none` is never this runner's answer. | ||
| /// | ||
| /// The key's own wording decides the kind, through the one rule `keyboard enter` probes with | ||
| /// (`RunnerKeyboardReturnKey.kind(of:)`), so the two commands can never disagree about the same | ||
| /// key. The label itself is kept as the accessibility tree carries it apart from its surrounding | ||
| /// whitespace, so a localized or an app's custom key stays readable even where the kind is | ||
| /// `unknown`. | ||
| static func observed( | ||
| visible: Bool, | ||
| returnControlText: String? | ||
| ) -> RunnerKeyboardStatus { | ||
| guard visible else { | ||
| return RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil) | ||
| } | ||
| guard let returnControlText else { | ||
| return RunnerKeyboardStatus( | ||
| visible: true, | ||
| returnKey: RunnerKeyboardReturnKey.unknown.rawValue, | ||
| returnKeyLabel: nil | ||
| ) | ||
| } | ||
| let label = returnControlText.trimmingCharacters(in: .whitespacesAndNewlines) | ||
| return RunnerKeyboardStatus( | ||
| visible: true, | ||
| returnKey: RunnerKeyboardReturnKey.kind(of: label).rawValue, | ||
| returnKeyLabel: label.isEmpty ? nil : label | ||
| ) | ||
| } | ||
|
|
||
| /// Builds the answer from every control the read saw on the visible keyboard, so it does not need to | ||
| /// read English to find the return key. A control whose wording `kind(of:)` names a kind for is the | ||
| /// return key on any keyboard in any language, and its wording is the label. When none maps — a fully | ||
| /// localized keyboard — the return key is the control in the slot Apple's keyboards give it: the | ||
| /// keyboard's `returnKeySlot`, immediately right of the space bar in the space bar's own row. | ||
| /// | ||
| /// The slot is anchored on the space bar rather than on the keyboard's bottom-right corner because | ||
| /// the corner is not the return key on every layout: a number, decimal or phone pad has no space bar | ||
| /// and puts Delete in that corner, so reporting the corner would tell a caller an OTP field has a | ||
| /// return key. No space bar, no candidate. A candidate is also never a dictation or globe control: | ||
| /// those sit left of the space bar, and the slot is right of it. | ||
| /// | ||
| /// `returnKeyLabel` therefore comes only from a control this read identified as the return key. A | ||
| /// visible keyboard with no identifiable return control answers `unknown` with no label, which is the | ||
| /// honest reading of a pad that has no return key at all. | ||
| static func observed( | ||
| visible: Bool, | ||
| keyboardKeys: [ObservedKeyboardKey] | ||
| ) -> RunnerKeyboardStatus { | ||
| guard visible else { | ||
| return RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil) | ||
| } | ||
| let worded = keyboardKeys.filter { key in | ||
| !(key.text ?? "").trimmingCharacters(in: .whitespacesAndNewlines).isEmpty | ||
| } | ||
| if let named = worded.first(where: { RunnerKeyboardReturnKey.kind(of: $0.text ?? "") != .unknown }) { | ||
| return observed(visible: true, returnControlText: named.text) | ||
| } | ||
| return observed(visible: true, returnControlText: returnKeySlot(in: keyboardKeys)?.text) | ||
| } | ||
|
|
||
| /// The worded control in the return key's slot, or `nil` when the keyboard exposes no space bar to | ||
| /// anchor the slot on. See `observed(visible:keyboardKeys:)` for why the slot is space-bar-relative. | ||
| /// A candidate carries wording, shares the space bar's row, starts at its right edge, and is the | ||
| /// nearest such control. The space bar itself can never be a candidate: its wording is whitespace, | ||
| /// which is what identifies it and what no return key's wording ever is. | ||
| private static func returnKeySlot(in keys: [ObservedKeyboardKey]) -> ObservedKeyboardKey? { | ||
| // The space bar is the widest key on Apple's keyboards by several times any other, so taking the | ||
| // widest whitespace-worded control cannot mistake a padding or divider element for it. | ||
| guard let spaceBar = keys | ||
| .filter({ isSpaceBar($0.text) && !$0.frame.isEmpty }) | ||
| .max(by: { $0.frame.width < $1.frame.width }) | ||
| else { | ||
| return nil | ||
| } | ||
| let spaceEdge = spaceBar.frame.maxX | ||
| let spaceMidY = spaceBar.frame.midY | ||
| return keys | ||
| .filter { candidate in | ||
| let wording = (candidate.text ?? "").trimmingCharacters(in: .whitespacesAndNewlines) | ||
| return !wording.isEmpty | ||
| && !nonReturnKeyLabels.contains(wording.lowercased()) | ||
| && !candidate.frame.isEmpty | ||
| && candidate.frame.minY <= spaceMidY | ||
| && candidate.frame.maxY >= spaceMidY | ||
| && candidate.frame.minX >= spaceEdge - returnKeySlotTolerance | ||
| } | ||
| .min { lhs, rhs in (lhs.frame.minX, lhs.frame.maxX) < (rhs.frame.minX, rhs.frame.maxX) } | ||
| } | ||
|
|
||
| /// Controls Apple's keyboards expose that are never the return key, by their accessibility wording: | ||
| /// the dictation key, the input-language (globe) key, and the emoji-switch key. On every layout this | ||
| /// read measured, all three sit left of the space bar, so the positional rule already excludes them; | ||
| /// this covers a layout that puts one on the right. A localized keyboard spells them differently, | ||
| /// which is why position is the guard and this list only backs it up. | ||
| private static let nonReturnKeyLabels: Set<String> = [ | ||
| "dictation", "start dictation", "stop dictation", "globe", "next keyboard", | ||
| "select next keyboard", "emoji", | ||
| ] | ||
|
|
||
| /// Keys sit edge to edge, so the slot's left edge is the space bar's right edge; this absorbs the | ||
| /// fractions of a point a keyboard's layout may round away. | ||
| private static let returnKeySlotTolerance: CGFloat = 2 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.