Skip to content

feat(hmi): add jumpstarter-driver-hmi, a generic remote display with declared input (H.264, touch/mouse/keyboard) - #1357

Draft
kirkbrauer wants to merge 6 commits into
mainfrom
feat/hmi-interface
Draft

kirkbrauer wants to merge 6 commits into
mainfrom
feat/hmi-interface

Conversation

@kirkbrauer

@kirkbrauer kirkbrauer commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Overview

A new package, jumpstarter-driver-hmi: one interface for an HMI, a remote display with declared input. The device's picture is streamed (H.264 by default; HEVC and MJPEG are optional) and whatever the device takes back is sent to it: a pointer (touch, mouse, pen), a keyboard, hard keys and typed text.

The device declares what it takes, and nothing else is ever sent to it; a device that declares no input is a valid, view-only HMI. (HMI here means the device's interface, not an HMI application that runs on it.) It covers embedded touch screens and mobile devices today and is shaped for remote desktops (VNC) and KVM devices later.

It follows the interface pattern of jumpstarter-driver-power: the package declares HmiInterface and ships the machinery that implements it once; device drivers add an hmi: child, the way they add appium: (the next PRs do this for jumpstarter-driver-adb and jumpstarter-driver-iosdevice). Other drivers can implement it too.

This PR has no device code. It contains:

  • HmiInterface / HmiDriver: the exports (video(), input(), hmi_info(), status(), start/stop, set_quality, recording), implemented on a shared hub for any HmiSource (the small contract a device implements: start, stop(deadline), capabilities, input_sink, request_keyframe, ...). The exporter checks every input event against what the device declared before it reaches the device.
  • The shared hub: one encode, any number of viewers. A joining viewer starts at once from the cached group of pictures (age- and size-gated); a slow viewer is dropped to its next key picture without slowing the others; a stuck viewer is let go; parameter sets (SPS/PPS, and VPS for HEVC) are repaired; key-picture requests are coalesced and rate limited. Quality adapts to the network between the client and the exporter (a ladder of rungs; one slow viewer among several never lowers the shared encode).
  • A fragmented-MP4 recorder (muxed on the exporter from the stream's own access units, no transcode, no ffmpeg) for H.264 and HEVC.
  • proc/: a command queue that cannot block its callers (one writer thread, bounded, moves coalesced) and a supervised child process with a bounded stop (stop command, SIGTERM, SIGKILL), for sources that run a program.
  • HmiClient and the j <hmi> ... commands (status, info, start, stop, quality, key, text, tap, swipe, click, scroll, press, recording, record, view). The client's helpers check what the HMI declares and send nothing it does not take.
  • j <hmi> view: a small local web page (plain ES module, no build step, no network) that decodes H.264 and HEVC with WebCodecs and MJPEG as pictures, and shows only the input the HMI declares (pointer, wheel, keyboard, hard keys as buttons, text). It binds 127.0.0.1 only, checks Host and Origin, sends a CSP, and its address carries a per-session token that the stream and input endpoints require.
  • MockHmi (synthetic pictures, as a touch screen or as a mouse and keyboard, in any codec; pattern: true serves a real test pattern so a browser can decode it) and jumpstarter_driver_hmi.testing with a FakeSource and an HmiConformance suite that an implementation runs against a fake of its device; the suite checks what the implementation declares.
  • A guide (Getting started > Guides > HMI Streaming), the package README (with a table of what each implementation declares), and the docs index entries.

Why

Streaming a display well is mostly not about the device: fan-out to several viewers, latency under a slow link, key-picture handling, quality adaptation, recording and input handling are the same for every device, and they are easy to get subtly wrong (a viewer that blocks the encode, a writer that stalls the exporter when the device stops reading). Putting them in one place lets each device driver supply only what is device-specific, and gives clients and UIs one interface for all of them.

Design notes

  • The receiver always runs on the exporter. Whatever the device speaks, the exporter receives it and turns it into this interface: one framing, one hub. Input from the client is translated on the exporter into the device's protocol. Device protocols never reach the client.
  • The device declares what it takes in hmi_info()["input"]: pointer (kinds, absolute, relative, buttons, wheel, max_contacts), keyboard (codes: "all", a list of W3C KeyboardEvent.code names or none; text; layout), keys and hard_keys, clipboard (reserved), plus the codecs it can produce and the codec in use. A UI shows only what is declared. A source implements the subset it genuinely supports.
  • Wire format v1: frames are u32 length | u8 kind | body; kind 0 is one JSON codec line (codec: avc1.*, hev1.*/hvc1.* or mjpeg; width, height, fps, format, timestamp, v; optional coded_*, rotation, point_scale, input, viewer), kind 1 is an 8-byte microsecond timestamp plus one access unit (Annex-B with the parameter sets before every key picture, or one JPEG); kinds 2 to 15 are reserved and skipped. Input is one JSON object per line: pointer (touch is a shorthand for a pointer of kind touch; x/y in pixels of the shown picture, buttons as PointerEvent.buttons, wheel dx/dy in pixels), keyboard (W3C KeyboardEvent.code, down/up), key (a named hard key) and text; errors come back as {"error": ...} lines the exporter never waits on. Back-channel lines on video(): {"resync": true} and {"lag": seconds}. The full text is the module docstring of jumpstarter_driver_hmi.common.
  • One codec per stream, chosen by the codec option or start(codec=...). H.264 is the default. MJPEG is not recorded (the recorder muxes H.264 and HEVC).
  • Power and lock keys are opt-in. They are in no default list and are not accepted unless the exporter lists them in hard_keys, because pressing one can lock the user out of a device whose only way in is its screen.
  • stop() and the end of a lease are bounded in time, and a stop that overtakes a slow start leaves nothing running.

How to test

cd python
make pkg-test-jumpstarter-driver-hmi
make pkg-ty-jumpstarter-driver-hmi

Without a device, try the web view with the mock HMI (add mode: mouse for a mouse and keyboard, codec: mjpeg or codec: hevc for another codec):

# hmi-demo.yaml
apiVersion: jumpstarter.dev/v1alpha1
kind: ExporterConfig
metadata: {namespace: default, name: hmi-demo}
endpoint: ""
token: ""
export:
  hmi:
    type: jumpstarter_driver_hmi.driver.MockHmi
    config: {pattern: true}
jmp shell --exporter-config hmi-demo.yaml -- j hmi view

Notes on what was checked: the package's tests (1579 pass, 129 skip: conformance checks that do not apply to what a given mock declares) run on Python 3.12, 3.13 and 3.14; ty, ruff, typos and uv lock --check pass; the docs build with -W -n; changed-line coverage is 95%. The MP4 recorder's output was decoded with ffmpeg for H.264 and HEVC. The page was driven in Chrome 157 (decoding and painting the H.264, HEVC and MJPEG test patterns; touch, mouse (buttons, wheel, hover, a chord of buttons), keyboard, hard keys, a rotary dial and a view-only HMI produce the expected lines; the token, Host, Origin and CSP rules hold): 183 checks pass. Other browsers were not tried. The 129 skips are not hiding failures: they are the conformance cases that need an input or a codec the mock under test does not declare.

Stack

This is PR 1 of 3, and the base is main:

  1. this PR: jumpstarter-driver-hmi
  2. the scrcpy backend in jumpstarter-driver-adb (hmi: option on AdbDevice)
  3. the iOS backend in jumpstarter-driver-iosdevice (hmi: option on IosDevice)

PRs 2 and 3 are opened against the previous branch so their diffs show only their own change.

… shared hub, recorder, process helpers

jumpstarter-driver-hmi is the interface for an HMI, a remote display with declared input:
a picture (H.264 by default, HEVC and MJPEG optional) and whatever the device takes back,
which it declares: a pointer (touch, mouse, pen), a keyboard, hard keys, rotary knobs and typed text. It
is for embedded touch screens and mobile devices, and shaped for remote desktops and KVM
devices later. This is its pure library, with no device code:

* common: the wire format (framing, the codec line, pointer, keyboard, key, rotary and text
  input events), the key vocabulary and the capabilities an HMI declares;
* source: HmiSource, HubSink and InputSink, the contract a device implements, taking the
  subset of the input it genuinely supports;
* hub: one encode, any number of viewers (group-of-pictures cache, age- and size-gated
  joins, per-viewer drops, key-pictures-only mode, parameter-set repair, coalesced key
  picture requests, the lag back-channel) for H.264, HEVC and MJPEG, and the quality ladder
  with its adaptation;
* recording: a fragmented-MP4 muxer (H.264 and HEVC) and recorder fed by the hub as a
  viewer;
* proc: a command queue that cannot block its callers and a supervised child process
  with a bounded stop (SIGTERM, then SIGKILL).
HmiDriver implements the HMI interface once, on the shared hub, for any HmiSource:
video() with the resync and lag back-channel, input() with pointer, keyboard, key, rotary and
text events checked against what the device declares (a kind, a button or a contact it does
not have is ignored, a wheel, key code, key press, rotary or text it cannot take is answered
with an error line the exporter never waits on, and the device is told small contact numbers), the codec option and start(codec=...), quality with adaptation to
the network, recording, a stall watchdog, an idle stop and a stop that is bounded in time.
HmiClient carries the lag meter and reporter (a cancelled portal task or a failing open no
longer escapes or leaks a thread), helpers that check what the HMI declares (tap, swipe,
click, scroll, press, turn, text...) and the `j <hmi> ...` commands.

MockHmi serves synthetic pictures for tests and demos, as a touch screen or as a mouse and
keyboard, in any codec. jumpstarter_driver_hmi.testing has a FakeSource and HmiConformance:
the suite an implementation runs against its own fake to show it implements what it declares.
Serves a page on 127.0.0.1 (never beyond it; the Host and Origin headers are checked) and
opens the browser. The page needs no network: it decodes the stream with WebCodecs (H.264,
HEVC) or as pictures (MJPEG), shows only the input the HMI declares (pointer, wheel,
keyboard, the hard keys as buttons, a held button being a long press, a dial for each
rotary, text), honours the
codec line's rotation and reports its lag. /ws relays the stream and the input lines one to
one; /info is hmi_info().
A source can say which keys it can send but does not offer by default (optional_keys). They
are accepted only when the exporter lists them in `hard_keys`: pressing the power or lock
button can lock the user out of a device whose only way in is its screen. The mock HMI
no longer offers POWER by default.
`MockHmi(pattern=True)` loops a small real stream in place of synthetic units (60 pictures
of a 240x400 moving test chart, a key picture with its parameter sets every 15 pictures, in
H.264 or HEVC, or 15 JPEG pictures for MJPEG), so that a browser or ffmpeg can decode it: the
local web view can be tried without a device. The tests run the conformance suite on it and
decode its recording with ffmpeg.
Power and lock are opt-in keys, so the `key` command's help and the client's docstring
point to hmi_info() for the keys an HMI has.
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kirkbrauer
kirkbrauer added this pull request to stack #1360 November 19, 2026 21:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant