Repository navigation
feat(iosdevice): stream and control an iPhone's display with scrios (hmi: option) - #1359
Draft
kirkbrauer wants to merge 4 commits into
Draft
kirkbrauer wants to merge 4 commits into
kirkbrauer wants to merge 4 commits into
Conversation
scrios is the separate program that streams an iPhone's screen and takes input. Its framed output (`u32 len | u64 pts_us | u8 flags | payload`, with JSON control records) is read by ScriosStream: chunking-safe, a limit on the length, units before the first `config` dropped and counted, a protocol other than 1 or a codec other than H.264 or HEVC fatal. fake_scrios is a runnable program that speaks the same protocol, with the failures the driver has to survive (never reading stdin, no config, dying mid-stream, a key picture without parameter sets, ignoring stop and SIGTERM, flooding stderr), for tests that run it as the real child process.
`hmi:` on IosDevice builds an `hmi` child, the way `appium:` builds its child:
IosDeviceHmi(HmiDriver) runs `scrios stream --format framed ...` for the device, reads its
H.264 or HEVC on the exporter and serves it through the shared hub. Touch, keyboard codes
(through USB HID usages), hard keys (HOME, LOCK, VOLUME_UP/DOWN, MUTE, ASSIST, APP_SWITCH)
and text from the client are translated here into scrios's stdin commands; the device's
protocol never reaches the client. `hmi: {control: false}` runs scrios with --no-control.
Every write to scrios goes through a bounded command queue on its own thread, so a scrios
that stops reading cannot freeze the exporter, and the process is supervised with a
bounded stop (stop command, SIGTERM, SIGKILL). The binary comes from `executable:` or PATH
and its version and protocol are checked. The HMI package is an extra
(`jumpstarter-driver-iosdevice[hmi]`), imported only when the option is used.
…st its SHA256SUMS Like fls: a static musl binary per architecture, SCRIOS_VERSION pinned, SCRIOS_BASE_URL a build argument (default: the project's GitHub releases) so a mirror can be used. The build fails if the asset is not listed in SHA256SUMS or does not match it. Nothing is downloaded when the image runs.
…ter lists it; ceilings are refused LOCK and POWER (scrios's one lock button) leave the default keys and buttons; the HMI says them as optional_keys and accepts them only when `hard_keys` lists them, because pressing it can lock the user out of a device whose only way in is its screen. max_size, bit_rate and max_fps are now a ConfigurationError on the scrios HMI instead of a logged warning: scrios streams at the display's own size, rate and bit rate, and an option that cannot work should fail loudly.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft until scrios 0.1.0 is released (testing on Linux, macOS and Windows in progress).
Stacked on the Android backend PR, which is stacked on the HMI interface PR; the base of this PR is that branch.
SCRIOS_VERSIONto the released tag and its SHA256SUMSWhat
IosDevicegets anhmi:option that builds anhmichild.IosDeviceHmiimplements the HMI interface with scrios, a separate Apache-2.0 program (a static binary for Linux; built on the MITidevicecrate) that streams an iOS device's display and takes input through Apple's CoreDevice services over usbmux. The driver runsscrios stream --format framed --udid <udid> --video-codec h264|h265 [--no-control] ..., reads its framed stdout (u32 len | u64 pts_us | u8 flags | payload, JSON control events) on the exporter and serves it through the shared hub. Touch, keyboard keys, hard keys and text from the client become scrios's stdin commands. The pure-Python driver only runs the binary, the wayflsis used; the binary comes fromexecutable:orPATH.scrios --versionis checked before use, and the firstconfigevent must sayprotocol: 1.KeyboardEvent.codenames that map to a USB HID usage (letters, digits, function keys, arrows, editing keys, punctuation, modifiers, the keypad), sent askb down|up, and text in a US layout (kb type); both need scrios's keyboard service, and until scrios says it is ready they are answered with its reason. Hard keys: Home, App Switcher, Volume up/down, Mute, Siri. Lock and power are opt-in (accepted only whenhard_keyslists them). There is no back button on iOS.max_size,bit_rateandmax_fpsare aConfigurationError(scrios streams at the display's own size, rate and bit rate); quality is not adjustable.pip install jumpstarter-driver-iosdevice[hmi];IosDeviceimports nothing from it unlesshmi:is set.python/Containerfileinstalls a pinned scrios release (ARG SCRIOS_VERSION,ARG SCRIOS_BASE_URL) the wayflsis installed, but verifies it: the build downloadsscrios-<arch>-linux-musland the release'sSHA256SUMSand runssha256sum -c. It fails closed: a missing release, an asset that is not inSHA256SUMSand a wrong checksum all stop the build. Nothing is downloaded when the image runs, andSCRIOS_BASE_URLcan point at a mirror.Waits for a scrios release
SCRIOS_VERSIONinpython/Containerfileis a placeholder (v0.1.0): scrios has no release or tag yet. Until it has, the image build step fails (closed, as above). The first scrios release must exist, andSCRIOS_VERSIONand itsSHA256SUMSmust be pinned to it, before this PR leaves draft and is merged. Everything else (the driver, the tests, the docs) is independent of the release.How to test
The tests run the HMI package's
HmiConformancesuite againstIosDeviceHmiwithfake_scrios, a small program that speaks scrios's framed protocol (H.264 and HEVC,hidreadiness,kbcommands) and can misbehave on request (never read stdin, send noconfig, die mid-stream, send a key picture without parameter sets, ignore stop and SIGTERM, flood stderr); it is started as a real child process through a wrapper used asexecutable:. Two named tests cover a scrios that never reads stdin (the exporter keeps serving, input calls return at once,close()returns within its deadline) and a final close that has to escalate to SIGKILL.Notes on what was checked: the package's tests (876 pass and 6 skip on Python 3.13 and 3.14; 874 pass and 8 skip on 3.12, where the two existing Device Hub TLS-PSK tests that need Python 3.13 skip) pass;
ty(on 3.14),ruff,typosanduv lock --checkpass; the docs build with-W -n; changed-line coverage is 97%.Not covered by CI: a real scrios and a real device (it needs Developer Mode and a mounted developer disk image;
scrios doctorchecks them). With one: addhmi: {}to anIosDeviceexport and runjmp shell --exporter-config ... -- j <device> hmi view(addcontrol: falsefirst for a safe first run).Stack
3 of 3: [HMI interface] [scrcpy backend in adb] this PR