Skip to content

feat(adb): stream and control an Android device's display with scrcpy (hmi: option) - #1358

Draft
kirkbrauer wants to merge 4 commits into
feat/hmi-interfacefrom
feat/hmi-adb
Draft

kirkbrauer wants to merge 4 commits into
feat/hmi-interfacefrom
feat/hmi-adb

Conversation

@kirkbrauer

Copy link
Copy Markdown
Member

Stacked on the HMI interface PR (jumpstarter-driver-hmi); the base of this PR is that branch. Until it merges, the first six commits are that PR's. This PR stands alone: it needs only the interface PR, and has no dependency on any other program besides the pinned scrcpy-server.

What

AdbDevice gets an hmi: option that builds an hmi child, the way appium: builds its child. AdbDeviceHmi implements the HMI interface for Android with scrcpy's server: the exporter pushes the pinned scrcpy-server, starts it as the shell user (no root, no app on the device), reads the stream the device's own encoder makes (H.264, or HEVC with codec: hevc), and serves it to any number of viewers from that one encode. Touch, keyboard keys, hard keys and text from the client are translated on the exporter into scrcpy's control messages; scrcpy's protocol never reaches the client.

export:
  phone:
    type: jumpstarter_driver_adb.driver.AdbDevice
    config:
      usb_port: "1-4.2"
      appium: {}
      hmi: {}             # native size, follows the network; max_size/bit_rate/max_fps only to cap the stream on purpose
  • The HMI takes the serial, the shared ADB server and adb_path from the device driver, so the config does not repeat them. AdbDevice.current_serial() is new and public (the serial now, only if the shared server lists the device as ready); Appium uses it too (its behavior and messages are unchanged).
  • scrcpy-server is Apache-2.0 and pinned to a version by SHA-256. It is not vendored and not downloaded at run time: an image build fetches it with python -m jumpstarter_driver_adb.hmi.fetch --dest /opt/scrcpy, and the file is verified before every push.
  • What it declares (the subset scrcpy genuinely takes): codecs H.264 and HEVC; a touch pointer with one contact (no mouse, pen, buttons or wheel); keyboard keys as the W3C KeyboardEvent.code names that have an Android key code (letters, digits, function keys, arrows, editing keys, punctuation, modifiers with their meta state, the numpad), and text in a US layout; hard keys Back, Home, Recents, Volume up and down. Power, lock and sleep are opt-in (accepted only when hard_keys lists them). A key code outside that table is never sent: the exporter answers it with an error line.
  • Quality levels (minimal to high, or auto following the network); a key-picture request restarts scrcpy's capture, so the hub waits for the periodic key picture and never asks.
  • Android extras on AdbDeviceHmiClient (j <device>.adb hmi ...): key (works without the stream, as input keyevent), text, app, clipboard get|set, rotate, panel, encoders.
  • The control socket is written only by a command-queue thread, so a device that stops reading cannot block the exporter; stop is bounded and leaves no forward, process or pushed file behind, even when it overtakes a slow start.
  • The HMI package is an extra: pip install jumpstarter-driver-adb[hmi]. AdbDevice imports nothing from it unless hmi: is set, and a config with hmi: and no extra gets a ConfigurationError that says how to install it.
  • Docs: an "HMI" section in the package README, a row in the HMI README's table of what each implementation declares and in the guide's table, an example exporter config.

How to test

cd python
make pkg-test-jumpstarter-driver-adb
make pkg-ty-jumpstarter-driver-adb

The tests use a fake adb and a fake scrcpy-server (jumpstarter_driver_adb.hmi.fakes), and run the HMI package's HmiConformance suite against AdbDeviceHmi (also view-only and capped variants). They cover the wire protocol (H.264 and H.265 streams, the latter from the HEVC test pattern), the server push/cache/SHA check, the forward lifecycle, a stop at every phase of a start, input mapping (a long press on a timer that never blocks the caller, key down/up with modifier state, held keys let go), and the hmi: option (child built, hand-written child refused, unknown option, no import without it, closed before the shared server is released).

Notes on what was checked: the package's tests (677 pass, 32 skip) run on Python 3.12, 3.13 and 3.14; ty, ruff, typos and uv lock --check pass; the docs build with -W -n; changed-line coverage is 98%.

Not covered by CI: a real device and a real scrcpy-server. With one: add hmi: {} to an AdbDevice export and run jmp shell --exporter-config ... -- j <device>.adb hmi view.

Stack

2 of 3: [HMI interface] this PR [iOS backend in iosdevice]

…er for the HMI

The first part of the scrcpy backend of the HMI interface: the control and video protocol
(protocol), the pinned scrcpy-server with its SHA-256 check (server_binary) and the helper
that fetches it at image build, with nothing downloaded at run time (fetch:
python -m jumpstarter_driver_adb.hmi.fetch), and a thin adb command helper built on the
device driver's own adb path, environment and shared server (adb).
ScrcpySession pushes and starts the server, reads the H.264 or HEVC stream into the hub's
sink and writes the control socket only from a command-queue thread, so a device that stops
reading cannot block the exporter; its stop is bounded and leaves no forward, process or
file behind even when it overtakes a slow start. ScrcpySource is the HmiSource on top of it,
declaring the subset scrcpy genuinely takes (H.264 and HEVC, one touch contact, the keyboard
codes that have an Android key code, text, hard keys, quality levels), the hardware encoder
choice, a stall probe that asks adb whether the server is still there, and an input sink
that maps touch, key codes with their modifiers, keys (a long press on a timer) and text to
scrcpy control messages. FakeAdb and FakeScrcpyServer make it testable without a device.
`hmi:` on AdbDevice builds an `hmi` child the way `appium:` builds its child:
AdbDeviceHmi(HmiDriver) takes the serial, the shared ADB server and the adb path from the
device driver (a public AdbDevice.current_serial() now serves it and Appium), pushes the
pinned scrcpy-server, and serves the device's picture to any number of viewers from one
encode, following the network. Touch, keyboard codes, hard keys (BACK, HOME, APP_SWITCH,
VOLUME_UP/DOWN) and text from the client are translated on the exporter into scrcpy's
control messages; the device's protocol never reaches the client. AdbDeviceHmiClient adds
the Android extras (press_key, type_text, start_app, clipboard, rotate, panel,
list_encoders).

The HMI package is an extra (`jumpstarter-driver-adb[hmi]`), imported only when the option
is used. The device driver closes the HMI before it releases the shared server.
…sts it

POWER, LOCK and SLEEP leave the default keys and the default buttons; the HMI says them
as optional_keys, and accepts them only when `hard_keys` lists them. Pressing one can lock
the user out of a device whose only way in is its screen.
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kirkbrauer
kirkbrauer added this pull request to stack #1360 November 19, 2026 21:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant