Skip to content

feat(iosdevice): stream and control an iPhone's display with scrios (hmi: option) - #1359

Draft
kirkbrauer wants to merge 4 commits into
feat/hmi-adbfrom
feat/hmi-iosdevice
Draft

kirkbrauer wants to merge 4 commits into
feat/hmi-adbfrom
feat/hmi-iosdevice

Conversation

@kirkbrauer

Copy link
Copy Markdown
Member

Draft until scrios 0.1.0 is released (testing on Linux, macOS and Windows in progress).

Stacked on the Android backend PR, which is stacked on the HMI interface PR; the base of this PR is that branch.

  • pin SCRIOS_VERSION to the released tag and its SHA256SUMS

What

IosDevice gets an hmi: option that builds an hmi child. IosDeviceHmi implements the HMI interface with scrios, a separate Apache-2.0 program (a static binary for Linux; built on the MIT idevice crate) that streams an iOS device's display and takes input through Apple's CoreDevice services over usbmux. The driver runs scrios stream --format framed --udid <udid> --video-codec h264|h265 [--no-control] ..., reads its framed stdout (u32 len | u64 pts_us | u8 flags | payload, JSON control events) on the exporter and serves it through the shared hub. Touch, keyboard keys, hard keys and text from the client become scrios's stdin commands. The pure-Python driver only runs the binary, the way fls is used; the binary comes from executable: or PATH.

export:
  iphone:
    type: jumpstarter_driver_iosdevice.driver.IosDevice
    config:
      udid: "00008140-..."
      appium: {wda_port: 8100}
      hmi: {}               # hmi: {control: false} for a picture and nothing else; codec: hevc for HEVC
  • The HMI takes the UDID and the usbmux address from the device driver. scrios --version is checked before use, and the first config event must say protocol: 1.
  • What it declares (the subset scrios genuinely takes): codecs H.264 and HEVC; a touch pointer with one contact; keyboard keys as the W3C KeyboardEvent.code names that map to a USB HID usage (letters, digits, function keys, arrows, editing keys, punctuation, modifiers, the keypad), sent as kb down|up, and text in a US layout (kb type); both need scrios's keyboard service, and until scrios says it is ready they are answered with its reason. Hard keys: Home, App Switcher, Volume up/down, Mute, Siri. Lock and power are opt-in (accepted only when hard_keys lists them). There is no back button on iOS. max_size, bit_rate and max_fps are a ConfigurationError (scrios streams at the display's own size, rate and bit rate); quality is not adjustable.
  • Every write to scrios goes through a bounded command queue on its own thread, and the process is supervised (stop command, then SIGTERM, then SIGKILL, within a deadline): a scrios that stops reading cannot freeze the exporter or the end of a lease. A stall asks for a key picture first and restarts scrios if the silence goes on; restarts back off.
  • The HMI package is an extra: pip install jumpstarter-driver-iosdevice[hmi]; IosDevice imports nothing from it unless hmi: is set.
  • python/Containerfile installs a pinned scrios release (ARG SCRIOS_VERSION, ARG SCRIOS_BASE_URL) the way fls is installed, but verifies it: the build downloads scrios-<arch>-linux-musl and the release's SHA256SUMS and runs sha256sum -c. It fails closed: a missing release, an asset that is not in SHA256SUMS and a wrong checksum all stop the build. Nothing is downloaded when the image runs, and SCRIOS_BASE_URL can point at a mirror.
  • Docs: an "HMI (scrios)" section in the package README, a row in the HMI README's table of what each implementation declares and in the guide's table, an example exporter config.

Waits for a scrios release

SCRIOS_VERSION in python/Containerfile is a placeholder (v0.1.0): scrios has no release or tag yet. Until it has, the image build step fails (closed, as above). The first scrios release must exist, and SCRIOS_VERSION and its SHA256SUMS must be pinned to it, before this PR leaves draft and is merged. Everything else (the driver, the tests, the docs) is independent of the release.

How to test

cd python
make pkg-test-jumpstarter-driver-iosdevice
make pkg-ty-jumpstarter-driver-iosdevice

The tests run the HMI package's HmiConformance suite against IosDeviceHmi with fake_scrios, a small program that speaks scrios's framed protocol (H.264 and HEVC, hid readiness, kb commands) and can misbehave on request (never read stdin, send no config, die mid-stream, send a key picture without parameter sets, ignore stop and SIGTERM, flood stderr); it is started as a real child process through a wrapper used as executable:. Two named tests cover a scrios that never reads stdin (the exporter keeps serving, input calls return at once, close() returns within its deadline) and a final close that has to escalate to SIGKILL.

Notes on what was checked: the package's tests (876 pass and 6 skip on Python 3.13 and 3.14; 874 pass and 8 skip on 3.12, where the two existing Device Hub TLS-PSK tests that need Python 3.13 skip) pass; ty (on 3.14), ruff, typos and uv lock --check pass; the docs build with -W -n; changed-line coverage is 97%.

Not covered by CI: a real scrios and a real device (it needs Developer Mode and a mounted developer disk image; scrios doctor checks them). With one: add hmi: {} to an IosDevice export and run jmp shell --exporter-config ... -- j <device> hmi view (add control: false first for a safe first run).

Stack

3 of 3: [HMI interface] [scrcpy backend in adb] this PR

scrios is the separate program that streams an iPhone's screen and takes input. Its
framed output (`u32 len | u64 pts_us | u8 flags | payload`, with JSON control records) is
read by ScriosStream: chunking-safe, a limit on the length, units before the first
`config` dropped and counted, a protocol other than 1 or a codec other than H.264 or HEVC
fatal.

fake_scrios is a runnable program that speaks the same protocol, with the failures the
driver has to survive (never reading stdin, no config, dying mid-stream, a key picture
without parameter sets, ignoring stop and SIGTERM, flooding stderr), for tests that run it
as the real child process.
`hmi:` on IosDevice builds an `hmi` child, the way `appium:` builds its child:
IosDeviceHmi(HmiDriver) runs `scrios stream --format framed ...` for the device, reads its
H.264 or HEVC on the exporter and serves it through the shared hub. Touch, keyboard codes
(through USB HID usages), hard keys (HOME, LOCK, VOLUME_UP/DOWN, MUTE, ASSIST, APP_SWITCH)
and text from the client are translated here into scrios's stdin commands; the device's
protocol never reaches the client. `hmi: {control: false}` runs scrios with --no-control.

Every write to scrios goes through a bounded command queue on its own thread, so a scrios
that stops reading cannot freeze the exporter, and the process is supervised with a
bounded stop (stop command, SIGTERM, SIGKILL). The binary comes from `executable:` or PATH
and its version and protocol are checked. The HMI package is an extra
(`jumpstarter-driver-iosdevice[hmi]`), imported only when the option is used.
…st its SHA256SUMS

Like fls: a static musl binary per architecture, SCRIOS_VERSION pinned, SCRIOS_BASE_URL a
build argument (default: the project's GitHub releases) so a mirror can be used. The build
fails if the asset is not listed in SHA256SUMS or does not match it. Nothing is downloaded
when the image runs.
…ter lists it; ceilings are refused

LOCK and POWER (scrios's one lock button) leave the default keys and buttons; the HMI
says them as optional_keys and accepts them only when `hard_keys` lists them, because
pressing it can lock the user out of a device whose only way in is its screen.

max_size, bit_rate and max_fps are now a ConfigurationError on the scrios HMI instead of
a logged warning: scrios streams at the display's own size, rate and bit rate, and an
option that cannot work should fail loudly.
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kirkbrauer
kirkbrauer added this pull request to stack #1360 November 19, 2026 21:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant