Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,11 @@ import Foundation
/// `Emergency Call` places a call, which no caller of this should ever report as a submit. Every
/// other Apple type has a case above.
///
/// `probeLabels` is the order `findKeyboardReturnControl(app:)` presses in. It is the runner's own
/// vocabulary, so it is not derived from the cases: it carries the two spellings of the plain
/// return key, in the order that found a hittable control before this rule existed.
/// `probeLabels` is the order `findKeyboardReturnControl(app:)` presses in. It is the
/// runner's own vocabulary, so it is not derived from the cases: it carries the two spellings of the
/// plain return key, in the order that found a hittable control before this rule existed. The
/// `keyboard status` read does not use it: it reads the keyboard's own key elements, so it needs no
/// English to find the return key.
enum RunnerKeyboardReturnKey: String {
case go
case search
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
import Foundation
import CoreGraphics

/// One `keyboard status` answer: whether the software keyboard is on screen and, when it is, what
/// its return key offers.
///
/// `returnKey` is a `RunnerKeyboardReturnKey` raw value, or `nil` when the keyboard is hidden — a
/// hidden keyboard has no return key to report, which is a different fact from one this read cannot
/// name. `returnKeyLabel` is the wording the return key itself carries, with surrounding whitespace
/// trimmed, so a localized or an app's custom key stays readable even where the kind is `unknown`.
/// It is present only for a control this read identified as the return key: a keyboard whose return
/// key cannot be identified — a number or phone pad has none — answers `unknown` with no label, so a
/// caller is never told a key exists on a layout that has no key to press.
struct RunnerKeyboardStatus: Equatable {
let visible: Bool
let returnKey: String?
let returnKeyLabel: String?
}

extension RunnerKeyboardStatus {
/// One control the status read observed on the visible keyboard: the wording it carries in the
/// accessibility tree, and the frame it occupies. `frame` is what locates the return key by slot —
/// the control immediately right of the space bar in the space bar's own row.
struct ObservedKeyboardKey {
let text: String?
let frame: CGRect
}

/// What identifies Apple's space bar in the accessibility tree: wording that is whitespace and
/// nothing else. This read measured a single space on an English and a Polish iOS layout, so the
/// anchor survives the localization the positional rule exists for. A pad's keys all carry digit or
/// `Delete` wording, so a pad has no anchor, and therefore no candidate.
private static func isSpaceBar(_ text: String?) -> Bool {
guard let text, !text.isEmpty else { return false }
return text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
}

/// Builds the answer from what the runner observed. The keyboard decides whether a return key is
/// reported at all: a hidden one has none, which is a different fact from one this read cannot
/// name. Only Android's `IME_ACTION_NONE` can establish that a field offers no action key, so
/// `none` is never this runner's answer.
///
/// The key's own wording decides the kind, through the one rule `keyboard enter` probes with
/// (`RunnerKeyboardReturnKey.kind(of:)`), so the two commands can never disagree about the same
/// key. The label itself is kept as the accessibility tree carries it apart from its surrounding
/// whitespace, so a localized or an app's custom key stays readable even where the kind is
/// `unknown`.
static func observed(
visible: Bool,
returnControlText: String?
) -> RunnerKeyboardStatus {
guard visible else {
return RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil)
}
guard let returnControlText else {
return RunnerKeyboardStatus(
visible: true,
returnKey: RunnerKeyboardReturnKey.unknown.rawValue,
returnKeyLabel: nil
)
}
let label = returnControlText.trimmingCharacters(in: .whitespacesAndNewlines)
return RunnerKeyboardStatus(
visible: true,
returnKey: RunnerKeyboardReturnKey.kind(of: label).rawValue,
returnKeyLabel: label.isEmpty ? nil : label
)
}

/// Builds the answer from every control the read saw on the visible keyboard, so it does not need to
/// read English to find the return key. A control whose wording `kind(of:)` names a kind for is the
/// return key on any keyboard in any language, and its wording is the label. When none maps — a fully
/// localized keyboard — the return key is the control in the slot Apple's keyboards give it: the
/// keyboard's `returnKeySlot`, immediately right of the space bar in the space bar's own row.
///
/// The slot is anchored on the space bar rather than on the keyboard's bottom-right corner because
/// the corner is not the return key on every layout: a number, decimal or phone pad has no space bar
/// and puts Delete in that corner, so reporting the corner would tell a caller an OTP field has a
/// return key. No space bar, no candidate. A candidate is also never a dictation or globe control:
/// those sit left of the space bar, and the slot is right of it.
///
/// `returnKeyLabel` therefore comes only from a control this read identified as the return key. A
/// visible keyboard with no identifiable return control answers `unknown` with no label, which is the
/// honest reading of a pad that has no return key at all.
static func observed(
visible: Bool,
keyboardKeys: [ObservedKeyboardKey]
) -> RunnerKeyboardStatus {
guard visible else {
return RunnerKeyboardStatus(visible: false, returnKey: nil, returnKeyLabel: nil)
}
let worded = keyboardKeys.filter { key in
!(key.text ?? "").trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
}
if let named = worded.first(where: { RunnerKeyboardReturnKey.kind(of: $0.text ?? "") != .unknown }) {
return observed(visible: true, returnControlText: named.text)
}
return observed(visible: true, returnControlText: returnKeySlot(in: keyboardKeys)?.text)
}

/// The worded control in the return key's slot, or `nil` when the keyboard exposes no space bar to
/// anchor the slot on. See `observed(visible:keyboardKeys:)` for why the slot is space-bar-relative.
/// A candidate carries wording, shares the space bar's row, starts at its right edge, and is the
/// nearest such control. The space bar itself can never be a candidate: its wording is whitespace,
/// which is what identifies it and what no return key's wording ever is.
private static func returnKeySlot(in keys: [ObservedKeyboardKey]) -> ObservedKeyboardKey? {
// The space bar is the widest key on Apple's keyboards by several times any other, so taking the
// widest whitespace-worded control cannot mistake a padding or divider element for it.
guard let spaceBar = keys
.filter({ isSpaceBar($0.text) && !$0.frame.isEmpty })
.max(by: { $0.frame.width < $1.frame.width })
else {
return nil
}
let spaceEdge = spaceBar.frame.maxX
let spaceMidY = spaceBar.frame.midY
return keys
.filter { candidate in
let wording = (candidate.text ?? "").trimmingCharacters(in: .whitespacesAndNewlines)
return !wording.isEmpty
&& !nonReturnKeyLabels.contains(wording.lowercased())
&& !candidate.frame.isEmpty
&& candidate.frame.minY <= spaceMidY
&& candidate.frame.maxY >= spaceMidY
&& candidate.frame.minX >= spaceEdge - returnKeySlotTolerance
}
.min { lhs, rhs in (lhs.frame.minX, lhs.frame.maxX) < (rhs.frame.minX, rhs.frame.maxX) }
}

/// Controls Apple's keyboards expose that are never the return key, by their accessibility wording:
/// the dictation key, the input-language (globe) key, and the emoji-switch key. On every layout this
/// read measured, all three sit left of the space bar, so the positional rule already excludes them;
/// this covers a layout that puts one on the right. A localized keyboard spells them differently,
/// which is why position is the guard and this list only backs it up.
private static let nonReturnKeyLabels: Set<String> = [
"dictation", "start dictation", "stop dictation", "globe", "next keyboard",
"select next keyboard", "emoji",
]

/// Keys sit edge to edge, so the slot's left edge is the space bar's right edge; this absorbs the
/// fractions of a point a keyboard's layout may round away.
private static let returnKeySlotTolerance: CGFloat = 2
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ extension CommandType {
return .none
case .tap, .longPress, .drag, .type, .swipe, .scroll, .findText, .querySelector, .readText,
.snapshot, .backInApp, .gesture, .gestureViewport, .appSwitcher,
.keyboardDismiss, .sequence:
.keyboardDismiss, .keyboardStatus, .sequence:
return .appWindows
case .screenshot, .recordStart:
return .observedScreen
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,23 @@ extension RunnerTests {
wasVisible: result.wasVisible
)
)
case .keyboardStatus:
var status: RunnerKeyboardStatus?
withBoundedInteractionIdleTimeoutIfSupported(activeApp, waits: .bothSkipped) {
status = keyboardStatus(app: activeApp)
}
guard let result = status else {
return Response(ok: false, error: ErrorPayload(message: "keyboardStatus produced no result"))
}
return Response(
ok: true,
data: DataPayload(
message: "keyboardStatus",
visible: result.visible,
keyboardReturnKey: result.returnKey,
keyboardReturnKeyLabel: result.returnKeyLabel
)
)
case .alert:
let action = (command.action ?? "get").lowercased()
let deadline = alertDeadline ?? Date().addingTimeInterval(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,45 @@ extension RunnerTests {
return visibleKeyboardFrame(app: app) != nil
}

/// What `keyboard status` answers with: the keyboard's visibility and, when it is up, the return
/// key the focused field offers. It reads the keyboard's own key elements rather than looking a
/// control up by English name, so a localized keyboard (a return key reading `szukaj`) is reported
/// with its own wording instead of a label-less `unknown`. `RunnerKeyboardStatus` decides which
/// observed control is the return key, through the same `kind(of:)` rule `keyboard enter` probes
/// with, so the two commands can never disagree about the wording they can name.
func keyboardStatus(app: XCUIApplication) -> RunnerKeyboardStatus {
guard isKeyboardVisible(app: app) else {
return RunnerKeyboardStatus.observed(visible: false, returnControlText: nil)
}
return RunnerKeyboardStatus.observed(visible: true, keyboardKeys: observedKeyboardKeys(app: app))
}

/// Every control the visible keyboard exposes with its wording and frame. The keyboard is another
/// process, so every element read is a round trip: one pass over its keys and one over its buttons
/// is what a read can afford, and it is the same two queries a snapshot's keyboard band makes. An
/// element that answers no wording at all is dropped; wording that is only whitespace is kept,
/// because that is how the accessibility tree carries the space bar and the positional rule anchors
/// the return key's slot on it. Off iOS nothing is queried and nothing is reported, so a caller must
/// not read the empty answer as a keyboard with no return key; `keyboard status` is not advertised
/// there.
func observedKeyboardKeys(app: XCUIApplication) -> [RunnerKeyboardStatus.ObservedKeyboardKey] {
#if os(iOS)
let keyboard = app.keyboards.firstMatch
let elements: [XCUIElement] =
safely("KEYBOARD_STATUS_KEYS", [XCUIElement]()) { keyboard.keys.allElementsBoundByIndex }
+ safely("KEYBOARD_STATUS_BUTTONS", [XCUIElement]()) { keyboard.buttons.allElementsBoundByIndex }
return elements.compactMap { element in
guard let label = safely("KEYBOARD_STATUS_KEY_LABEL", { element.label }), !label.isEmpty else {
return nil
}
let frame = safely("KEYBOARD_STATUS_KEY_FRAME", CGRect.zero) { element.frame }
return RunnerKeyboardStatus.ObservedKeyboardKey(text: label, frame: frame)
}
#else
return []
#endif
}
Comment thread
thymikee marked this conversation as resolved.
Resolved

func dismissKeyboard(app: XCUIApplication) -> RunnerKeyboardDismissOutcome {
let keyboard = app.keyboards.firstMatch
let wasVisible = isKeyboardVisible(app: app)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ enum CommandType: String, Codable, CaseIterable {
case volumeDown
case keyboardDismiss
case keyboardReturn
case keyboardStatus
case alert
case sequence
case gesture
Expand Down Expand Up @@ -300,7 +301,7 @@ extension Command {
.keyboardDismiss, .keyboardReturn, .sequence, .gesture:
return .interaction

case .findText, .readText, .gestureViewport:
case .findText, .readText, .gestureViewport, .keyboardStatus:
return .appRead(retainsJournalResponseJson: true)

case .snapshot:
Expand Down Expand Up @@ -536,6 +537,12 @@ struct DataPayload: Codable {
var wasVisible: Bool?
var dismissed: Bool?
var keyboardDismissMechanism: String?
/// The kind of return key the visible keyboard offers, on a `keyboard status` answer. Absent when
/// the keyboard is hidden, which is not the same claim as a key this read could not name.
var keyboardReturnKey: String?
/// The wording that return key carries, verbatim, so a localized or an app's custom key stays
/// readable. Absent when the key carries none, or there is no key to read.
var keyboardReturnKeyLabel: String?
var orientation: String?
var gestureFallback: String?
var gestureFallbackMessage: String?
Expand Down
Loading
Loading