Repository navigation
feat(adb): stream and control an Android device's display with scrcpy (hmi: option) - #1358
Draft
kirkbrauer wants to merge 4 commits into
Draft
kirkbrauer wants to merge 4 commits into
kirkbrauer wants to merge 4 commits into
Conversation
…er for the HMI The first part of the scrcpy backend of the HMI interface: the control and video protocol (protocol), the pinned scrcpy-server with its SHA-256 check (server_binary) and the helper that fetches it at image build, with nothing downloaded at run time (fetch: python -m jumpstarter_driver_adb.hmi.fetch), and a thin adb command helper built on the device driver's own adb path, environment and shared server (adb).
ScrcpySession pushes and starts the server, reads the H.264 or HEVC stream into the hub's sink and writes the control socket only from a command-queue thread, so a device that stops reading cannot block the exporter; its stop is bounded and leaves no forward, process or file behind even when it overtakes a slow start. ScrcpySource is the HmiSource on top of it, declaring the subset scrcpy genuinely takes (H.264 and HEVC, one touch contact, the keyboard codes that have an Android key code, text, hard keys, quality levels), the hardware encoder choice, a stall probe that asks adb whether the server is still there, and an input sink that maps touch, key codes with their modifiers, keys (a long press on a timer) and text to scrcpy control messages. FakeAdb and FakeScrcpyServer make it testable without a device.
`hmi:` on AdbDevice builds an `hmi` child the way `appium:` builds its child: AdbDeviceHmi(HmiDriver) takes the serial, the shared ADB server and the adb path from the device driver (a public AdbDevice.current_serial() now serves it and Appium), pushes the pinned scrcpy-server, and serves the device's picture to any number of viewers from one encode, following the network. Touch, keyboard codes, hard keys (BACK, HOME, APP_SWITCH, VOLUME_UP/DOWN) and text from the client are translated on the exporter into scrcpy's control messages; the device's protocol never reaches the client. AdbDeviceHmiClient adds the Android extras (press_key, type_text, start_app, clipboard, rotate, panel, list_encoders). The HMI package is an extra (`jumpstarter-driver-adb[hmi]`), imported only when the option is used. The device driver closes the HMI before it releases the shared server.
…sts it POWER, LOCK and SLEEP leave the default keys and the default buttons; the HMI says them as optional_keys, and accepts them only when `hard_keys` lists them. Pressing one can lock the user out of a device whose only way in is its screen.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the HMI interface PR (
jumpstarter-driver-hmi); the base of this PR is that branch. Until it merges, the first six commits are that PR's. This PR stands alone: it needs only the interface PR, and has no dependency on any other program besides the pinnedscrcpy-server.What
AdbDevicegets anhmi:option that builds anhmichild, the wayappium:builds its child.AdbDeviceHmiimplements the HMI interface for Android with scrcpy's server: the exporter pushes the pinnedscrcpy-server, starts it as theshelluser (no root, no app on the device), reads the stream the device's own encoder makes (H.264, or HEVC withcodec: hevc), and serves it to any number of viewers from that one encode. Touch, keyboard keys, hard keys and text from the client are translated on the exporter into scrcpy's control messages; scrcpy's protocol never reaches the client.adb_pathfrom the device driver, so the config does not repeat them.AdbDevice.current_serial()is new and public (the serial now, only if the shared server lists the device as ready); Appium uses it too (its behavior and messages are unchanged).scrcpy-serveris Apache-2.0 and pinned to a version by SHA-256. It is not vendored and not downloaded at run time: an image build fetches it withpython -m jumpstarter_driver_adb.hmi.fetch --dest /opt/scrcpy, and the file is verified before every push.KeyboardEvent.codenames that have an Android key code (letters, digits, function keys, arrows, editing keys, punctuation, modifiers with their meta state, the numpad), and text in a US layout; hard keys Back, Home, Recents, Volume up and down. Power, lock and sleep are opt-in (accepted only whenhard_keyslists them). A key code outside that table is never sent: the exporter answers it with an error line.minimaltohigh, orautofollowing the network); a key-picture request restarts scrcpy's capture, so the hub waits for the periodic key picture and never asks.AdbDeviceHmiClient(j <device>.adb hmi ...):key(works without the stream, asinput keyevent),text,app,clipboard get|set,rotate,panel,encoders.stopis bounded and leaves no forward, process or pushed file behind, even when it overtakes a slow start.pip install jumpstarter-driver-adb[hmi].AdbDeviceimports nothing from it unlesshmi:is set, and a config withhmi:and no extra gets aConfigurationErrorthat says how to install it.How to test
The tests use a fake adb and a fake scrcpy-server (
jumpstarter_driver_adb.hmi.fakes), and run the HMI package'sHmiConformancesuite againstAdbDeviceHmi(also view-only and capped variants). They cover the wire protocol (H.264 and H.265 streams, the latter from the HEVC test pattern), the server push/cache/SHA check, the forward lifecycle, a stop at every phase of a start, input mapping (a long press on a timer that never blocks the caller, key down/up with modifier state, held keys let go), and thehmi:option (child built, hand-written child refused, unknown option, no import without it, closed before the shared server is released).Notes on what was checked: the package's tests (677 pass, 32 skip) run on Python 3.12, 3.13 and 3.14;
ty,ruff,typosanduv lock --checkpass; the docs build with-W -n; changed-line coverage is 98%.Not covered by CI: a real device and a real scrcpy-server. With one: add
hmi: {}to anAdbDeviceexport and runjmp shell --exporter-config ... -- j <device>.adb hmi view.Stack
2 of 3: [HMI interface] this PR [iOS backend in iosdevice]