Add native simulator automation CLI

Author: DjDeveloperr

AGENTS.md

@@ -22,9 +22,11 @@ The native side should own anything that depends on macOS frameworks, `xcrun sim
 - `server/src/simulators/registry.rs`
   Tracks Rust-side simulator session state and lazy native attachment by UDID.
 - `cli/XCWSimctl.*`
-  Wraps `xcrun simctl` for discovery, lifecycle management, app launching, URL opening, and screenshot capture.
+  Wraps `xcrun simctl` for discovery, lifecycle management, app launching, URL opening, appearance toggles, and simulator log capture.
 - `cli/DFPrivateSimulatorDisplayBridge.*`
   Owns headless private display frames plus HID-based touch and keyboard injection.
+- `cli/XCWAccessibilityBridge.*`
+  Owns private CoreSimulator accessibility snapshots through `AccessibilityPlatformTranslation`.
 - `cli/XCWPrivateSimulatorSession.*`
   Owns one private display bridge per booted simulator plus selectable HEVC/H.264 encode.
 - `cli/native/XCWNativeBridge.*`
@@ -33,8 +35,6 @@ The native side should own anything that depends on macOS frameworks, `xcrun sim
   Wraps one Objective-C private simulator session handle for the Rust registry.
 - `cli/XCWPrivateSimulatorBooter.*`
   Uses private `CoreSimulator` APIs for direct simulator boot when available, with `simctl` as the fallback path.
-- `cli/XCWPrivateSimulatorChromeBridge.*`
-  Experimental private `SimulatorKit` chrome bridge for simulator chrome exploration.
 - `cli/XCWChromeRenderer.*`
   Renders Apple’s CoreSimulator device-type PDF chrome assets into PNGs for the browser.
 - `client/src/app/App.tsx`
@@ -60,10 +60,10 @@ The native side should own anything that depends on macOS frameworks, `xcrun sim
 Private simulator behavior is implemented locally in:
 
 - Boot path: `cli/XCWPrivateSimulatorBooter.*`
-- Chrome asset bridge: `cli/XCWPrivateSimulatorChromeBridge.*`
 - Full live display bridge: `cli/DFPrivateSimulatorDisplayBridge.*`
+- Accessibility bridge: `cli/XCWAccessibilityBridge.*`
 
-The current repo uses the private boot path and private display bridge directly. The browser streams frames from that bridge and injects touch and keyboard events through the same native session layer.
+The current repo uses the private boot path, private display bridge, and private accessibility translation bridge directly. The browser streams frames from that bridge, injects touch and keyboard events through the same native session layer, inspects accessibility through `AccessibilityPlatformTranslation`, and renders device chrome from `cli/XCWChromeRenderer.*`.
 
 ## Build and Run
 
@@ -105,8 +105,26 @@ Useful direct commands:
 ./build/xcode-canvas-web list
 ./build/xcode-canvas-web boot <udid>
 ./build/xcode-canvas-web shutdown <udid>
+./build/xcode-canvas-web erase <udid>
+./build/xcode-canvas-web install <udid> /path/to/App.app
+./build/xcode-canvas-web uninstall <udid> com.example.App
 ./build/xcode-canvas-web open-url <udid> https://example.com
 ./build/xcode-canvas-web launch <udid> com.apple.Preferences
+./build/xcode-canvas-web pasteboard set <udid> "hello"
+./build/xcode-canvas-web pasteboard get <udid>
+./build/xcode-canvas-web screenshot <udid> --output screen.png
+./build/xcode-canvas-web describe-ui <udid>
+./build/xcode-canvas-web tap <udid> 120 240
+./build/xcode-canvas-web tap <udid> --label "Continue" --wait-timeout-ms 5000
+./build/xcode-canvas-web swipe <udid> 200 700 200 200
+./build/xcode-canvas-web gesture <udid> scroll-down
+./build/xcode-canvas-web pinch <udid> --start-distance 160 --end-distance 80
+./build/xcode-canvas-web rotate-gesture <udid> --radius 100 --degrees 90
+./build/xcode-canvas-web key-sequence <udid> --keycodes h,e,l,l,o
+./build/xcode-canvas-web key-combo <udid> --modifiers cmd --key a
+./build/xcode-canvas-web type <udid> "hello"
+./build/xcode-canvas-web button <udid> lock --duration-ms 1000
+./build/xcode-canvas-web home <udid>
 ```
 
 ## Expectations For Future Changes
@@ -120,5 +138,5 @@ Useful direct commands:
 ## Near-Term Roadmap
 
 - Compose the private frame stream and CoreSimulator chrome into a single server-side render path.
-- Add richer input surfaces such as gesture synthesis, text entry helpers, and chrome-button actions on top of the HID bridge.
-- Add simulator creation, erase, pasteboard, install, and log streaming commands.
+- Keep private Indigo multi-touch packet assumptions documented when Xcode runtimes change.
+- Add simulator creation and log streaming commands.

README.md

@@ -22,12 +22,22 @@
 
 ## Install
 
+Requirements:
+
+- macOS
+- Xcode or Command Line Tools
+- Rust toolchain (`cargo`)
+- Node.js 18+
+
 Install the published CLI globally:
 
 ```sh
 npm install -g xcode-canvas-web
 ```
 
+The npm package builds the native Rust/Objective-C CLI during `postinstall`; it
+is not a prebuilt cross-platform binary.
+
 Install the current local checkout globally from source:
 
 ```sh
@@ -43,6 +53,8 @@ xcode-canvas-web serve --port 4310
 ```
 
 Then open [http://127.0.0.1:4310](http://127.0.0.1:4310).
+To focus a specific simulator, open
+[http://127.0.0.1:4310?device=UDID](http://127.0.0.1:4310?device=UDID).
 
 The Rust server exposes HTTP on the requested port and WebTransport on `port + 1`.
 The browser bootstrap comes from `GET /api/health`, which returns the WebTransport URL template,
@@ -56,22 +68,80 @@ Enable the per-user background service with `launchd`:
 xcode-canvas-web service on --port 4310
 ```
 
+Restart it:
+
+```sh
+xcode-canvas-web service restart
+```
+
 Disable it:
 
 ```sh
 xcode-canvas-web service off
 ```
 
+Restart the CoreSimulator service layer when `simctl` reports a stale service
+version or the live display gets stuck before the first frame:
+
+```sh
+xcode-canvas-web core-simulator restart
+```
+
+You can also start or stop the CoreSimulator service layer explicitly:
+
+```sh
+xcode-canvas-web core-simulator start
+xcode-canvas-web core-simulator shutdown
+```
+
 ## CLI
 
 ```sh
 xcode-canvas-web list
 xcode-canvas-web boot <udid>
 xcode-canvas-web shutdown <udid>
+xcode-canvas-web erase <udid>
+xcode-canvas-web install <udid> /path/to/App.app
+xcode-canvas-web uninstall <udid> com.example.App
 xcode-canvas-web open-url <udid> https://example.com
 xcode-canvas-web launch <udid> com.apple.Preferences
+xcode-canvas-web toggle-appearance <udid>
+xcode-canvas-web pasteboard set <udid> "hello"
+xcode-canvas-web pasteboard get <udid>
+xcode-canvas-web screenshot <udid> --output screen.png
+xcode-canvas-web describe-ui <udid>
+xcode-canvas-web describe-ui <udid> --point 120,240
+xcode-canvas-web tap <udid> 120 240
+xcode-canvas-web tap <udid> --label "Continue" --wait-timeout-ms 5000
+xcode-canvas-web swipe <udid> 200 700 200 200
+xcode-canvas-web gesture <udid> scroll-down
+xcode-canvas-web pinch <udid> --start-distance 160 --end-distance 80
+xcode-canvas-web rotate-gesture <udid> --radius 100 --degrees 90
+xcode-canvas-web touch <udid> 0.5 0.5 --phase began --normalized
+xcode-canvas-web touch <udid> 120 240 --down --up --delay-ms 800
+xcode-canvas-web key <udid> enter
+xcode-canvas-web key-sequence <udid> --keycodes h,e,l,l,o
+xcode-canvas-web key-combo <udid> --modifiers cmd --key a
+xcode-canvas-web type <udid> "hello"
+xcode-canvas-web type <udid> --file message.txt
+xcode-canvas-web button <udid> lock --duration-ms 1000
+xcode-canvas-web batch <udid> --step "tap --label Continue" --step "type 'hello'"
+xcode-canvas-web dismiss-keyboard <udid>
+xcode-canvas-web home <udid>
+xcode-canvas-web app-switcher <udid>
+xcode-canvas-web rotate-left <udid>
+xcode-canvas-web rotate-right <udid>
+xcode-canvas-web chrome-profile <udid>
+xcode-canvas-web logs <udid> --seconds 30 --limit 200
 ```
 
+`describe-ui` uses the built-in private CoreSimulator accessibility bridge and
+does not shell out to AXe. Coordinate commands accept screen coordinates from
+the accessibility tree by default; pass `--normalized` to send `0.0..1.0`
+coordinates directly. The CLI intentionally does not implement screenshot-based
+video streaming, MJPEG output, or screen recording; the live visual path remains
+the web UI's WebTransport stream.
+
 ## NativeScript Inspector
 
 NativeScript apps can connect directly to the running server from JS and expose
@@ -88,8 +158,8 @@ if (__DEV__) {
 
 The runtime connects to `GET /api/inspector/connect` as a WebSocket. The Rust
 server prefers connected NativeScript inspectors for hierarchy requests and
-falls back to the Swift TCP inspector or AXe when no matching app inspector is
-available.
+falls back to the Swift TCP inspector or the built-in native accessibility
+bridge when no matching app inspector is available.
 
 ## VS Code
 

cli/DFPrivateSimulatorDisplayBridge.h

@@ -4,7 +4,6 @@
 NS_ASSUME_NONNULL_BEGIN
 
 @class DFPrivateSimulatorDisplayBridge;
-@class DFPrivateSimulatorChromeButton;
 
 typedef NS_ENUM(NSInteger, DFPrivateSimulatorTouchPhase) {
     DFPrivateSimulatorTouchPhaseBegan = 0,
@@ -30,33 +29,40 @@ NS_SWIFT_NAME(PrivateSimulatorDisplayBridge)
 
 - (instancetype)init NS_UNAVAILABLE;
 - (instancetype)new NS_UNAVAILABLE;
-- (nullable instancetype)initWithUDID:(NSString *)udid error:(NSError * _Nullable * _Nullable)error NS_DESIGNATED_INITIALIZER NS_SWIFT_NAME(init(udid:));
+- (nullable instancetype)initWithUDID:(NSString *)udid error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(init(udid:));
+- (nullable instancetype)initWithUDID:(NSString *)udid
+                         attachDisplay:(BOOL)attachDisplay
+                                 error:(NSError * _Nullable * _Nullable)error NS_DESIGNATED_INITIALIZER;
 
 @property (nonatomic, weak, nullable) id<DFPrivateSimulatorDisplayBridgeDelegate> delegate;
-@property (nonatomic, readonly) NSView *displayView;
 @property (nonatomic, readonly, getter=isDisplayReady) BOOL displayReady;
 @property (nonatomic, readonly) NSString *displayStatus;
 
-- (void)activateDisplayIfNeeded;
 - (nullable CVPixelBufferRef)copyPixelBuffer CF_RETURNS_RETAINED;
 
 - (BOOL)sendTouchAtNormalizedX:(double)normalizedX
                    normalizedY:(double)normalizedY
                          phase:(DFPrivateSimulatorTouchPhase)phase
                          error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(sendTouch(normalizedX:normalizedY:phase:));
 
+- (BOOL)sendMultiTouchAtNormalizedX1:(double)normalizedX1
+                          normalizedY1:(double)normalizedY1
+                          normalizedX2:(double)normalizedX2
+                          normalizedY2:(double)normalizedY2
+                                phase:(DFPrivateSimulatorTouchPhase)phase
+                                error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(sendMultiTouch(normalizedX1:normalizedY1:normalizedX2:normalizedY2:phase:));
+
 - (BOOL)sendKeyCode:(uint16_t)keyCode
           modifiers:(NSUInteger)modifiers
               error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(sendKey(keyCode:modifiers:));
-
-- (BOOL)sendKeyEvent:(NSEvent *)event
-               error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(sendKey(event:));
-
-- (NSArray<DFPrivateSimulatorChromeButton *> *)availableChromeButtons NS_SWIFT_NAME(availableChromeButtons());
-- (BOOL)pressChromeButtonWithIdentifier:(NSString *)identifier
-                                  error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(pressChromeButton(identifier:));
+- (BOOL)sendKeyCode:(uint16_t)keyCode
+                down:(BOOL)down
+               error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(sendKey(keyCode:down:));
 
 - (BOOL)pressHomeButton:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(pressHomeButton());
+- (BOOL)pressHardwareButtonNamed:(NSString *)buttonName
+                       durationMs:(NSUInteger)durationMs
+                            error:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(pressHardwareButton(named:durationMs:));
 
 - (BOOL)rotateRight:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(rotateRight());
 - (BOOL)rotateLeft:(NSError * _Nullable * _Nullable)error NS_SWIFT_NAME(rotateLeft());
@@ -65,18 +71,4 @@ NS_SWIFT_NAME(PrivateSimulatorDisplayBridge)
 
 @end
 
-NS_SWIFT_NAME(PrivateSimulatorChromeButton)
-@interface DFPrivateSimulatorChromeButton : NSObject
-
-- (instancetype)init NS_UNAVAILABLE;
-- (instancetype)new NS_UNAVAILABLE;
-
-@property (nonatomic, copy, readonly) NSString *identifier;
-@property (nonatomic, copy, readonly) NSString *title;
-@property (nonatomic, copy, readonly) NSString *toolTip;
-@property (nonatomic, copy, readonly) NSString *accessibilityLabel;
-@property (nonatomic, copy, readonly) NSString *summary;
-
-@end
-
 NS_ASSUME_NONNULL_END