Repository navigation
feat(hmi): add jumpstarter-driver-hmi, a generic remote display with declared input (H.264, touch/mouse/keyboard) - #1357
Draft
kirkbrauer wants to merge 6 commits into
Draft
kirkbrauer wants to merge 6 commits into
kirkbrauer wants to merge 6 commits into
Conversation
… shared hub, recorder, process helpers jumpstarter-driver-hmi is the interface for an HMI, a remote display with declared input: a picture (H.264 by default, HEVC and MJPEG optional) and whatever the device takes back, which it declares: a pointer (touch, mouse, pen), a keyboard, hard keys, rotary knobs and typed text. It is for embedded touch screens and mobile devices, and shaped for remote desktops and KVM devices later. This is its pure library, with no device code: * common: the wire format (framing, the codec line, pointer, keyboard, key, rotary and text input events), the key vocabulary and the capabilities an HMI declares; * source: HmiSource, HubSink and InputSink, the contract a device implements, taking the subset of the input it genuinely supports; * hub: one encode, any number of viewers (group-of-pictures cache, age- and size-gated joins, per-viewer drops, key-pictures-only mode, parameter-set repair, coalesced key picture requests, the lag back-channel) for H.264, HEVC and MJPEG, and the quality ladder with its adaptation; * recording: a fragmented-MP4 muxer (H.264 and HEVC) and recorder fed by the hub as a viewer; * proc: a command queue that cannot block its callers and a supervised child process with a bounded stop (SIGTERM, then SIGKILL).
HmiDriver implements the HMI interface once, on the shared hub, for any HmiSource: video() with the resync and lag back-channel, input() with pointer, keyboard, key, rotary and text events checked against what the device declares (a kind, a button or a contact it does not have is ignored, a wheel, key code, key press, rotary or text it cannot take is answered with an error line the exporter never waits on, and the device is told small contact numbers), the codec option and start(codec=...), quality with adaptation to the network, recording, a stall watchdog, an idle stop and a stop that is bounded in time. HmiClient carries the lag meter and reporter (a cancelled portal task or a failing open no longer escapes or leaks a thread), helpers that check what the HMI declares (tap, swipe, click, scroll, press, turn, text...) and the `j <hmi> ...` commands. MockHmi serves synthetic pictures for tests and demos, as a touch screen or as a mouse and keyboard, in any codec. jumpstarter_driver_hmi.testing has a FakeSource and HmiConformance: the suite an implementation runs against its own fake to show it implements what it declares.
Serves a page on 127.0.0.1 (never beyond it; the Host and Origin headers are checked) and opens the browser. The page needs no network: it decodes the stream with WebCodecs (H.264, HEVC) or as pictures (MJPEG), shows only the input the HMI declares (pointer, wheel, keyboard, the hard keys as buttons, a held button being a long press, a dial for each rotary, text), honours the codec line's rotation and reports its lag. /ws relays the stream and the input lines one to one; /info is hmi_info().
A source can say which keys it can send but does not offer by default (optional_keys). They are accepted only when the exporter lists them in `hard_keys`: pressing the power or lock button can lock the user out of a device whose only way in is its screen. The mock HMI no longer offers POWER by default.
`MockHmi(pattern=True)` loops a small real stream in place of synthetic units (60 pictures of a 240x400 moving test chart, a key picture with its parameter sets every 15 pictures, in H.264 or HEVC, or 15 JPEG pictures for MJPEG), so that a browser or ffmpeg can decode it: the local web view can be tried without a device. The tests run the conformance suite on it and decode its recording with ffmpeg.
Power and lock are opt-in keys, so the `key` command's help and the client's docstring point to hmi_info() for the keys an HMI has.
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
A new package,
jumpstarter-driver-hmi: one interface for an HMI, a remote display with declared input. The device's picture is streamed (H.264 by default; HEVC and MJPEG are optional) and whatever the device takes back is sent to it: a pointer (touch, mouse, pen), a keyboard, hard keys and typed text.The device declares what it takes, and nothing else is ever sent to it; a device that declares no input is a valid, view-only HMI. (HMI here means the device's interface, not an HMI application that runs on it.) It covers embedded touch screens and mobile devices today and is shaped for remote desktops (VNC) and KVM devices later.
It follows the interface pattern of
jumpstarter-driver-power: the package declaresHmiInterfaceand ships the machinery that implements it once; device drivers add anhmi:child, the way they addappium:(the next PRs do this forjumpstarter-driver-adbandjumpstarter-driver-iosdevice). Other drivers can implement it too.This PR has no device code. It contains:
HmiInterface/HmiDriver: the exports (video(),input(),hmi_info(),status(),start/stop,set_quality, recording), implemented on a shared hub for anyHmiSource(the small contract a device implements:start,stop(deadline),capabilities,input_sink,request_keyframe, ...). The exporter checks every input event against what the device declared before it reaches the device.proc/: a command queue that cannot block its callers (one writer thread, bounded, moves coalesced) and a supervised child process with a bounded stop (stop command, SIGTERM, SIGKILL), for sources that run a program.HmiClientand thej <hmi> ...commands (status,info,start,stop,quality,key,text,tap,swipe,click,scroll,press,recording,record,view). The client's helpers check what the HMI declares and send nothing it does not take.j <hmi> view: a small local web page (plain ES module, no build step, no network) that decodes H.264 and HEVC with WebCodecs and MJPEG as pictures, and shows only the input the HMI declares (pointer, wheel, keyboard, hard keys as buttons, text). It binds127.0.0.1only, checksHostandOrigin, sends a CSP, and its address carries a per-session token that the stream and input endpoints require.MockHmi(synthetic pictures, as a touch screen or as a mouse and keyboard, in any codec;pattern: trueserves a real test pattern so a browser can decode it) andjumpstarter_driver_hmi.testingwith aFakeSourceand anHmiConformancesuite that an implementation runs against a fake of its device; the suite checks what the implementation declares.Getting started > Guides > HMI Streaming), the package README (with a table of what each implementation declares), and the docs index entries.Why
Streaming a display well is mostly not about the device: fan-out to several viewers, latency under a slow link, key-picture handling, quality adaptation, recording and input handling are the same for every device, and they are easy to get subtly wrong (a viewer that blocks the encode, a writer that stalls the exporter when the device stops reading). Putting them in one place lets each device driver supply only what is device-specific, and gives clients and UIs one interface for all of them.
Design notes
hmi_info()["input"]:pointer(kinds,absolute,relative,buttons,wheel,max_contacts),keyboard(codes:"all", a list of W3CKeyboardEvent.codenames or none;text;layout),keysandhard_keys,clipboard(reserved), plus thecodecsit can produce and thecodecin use. A UI shows only what is declared. A source implements the subset it genuinely supports.u32 length | u8 kind | body; kind 0 is one JSON codec line (codec:avc1.*,hev1.*/hvc1.*ormjpeg;width,height,fps,format,timestamp,v; optionalcoded_*,rotation,point_scale,input,viewer), kind 1 is an 8-byte microsecond timestamp plus one access unit (Annex-B with the parameter sets before every key picture, or one JPEG); kinds 2 to 15 are reserved and skipped. Input is one JSON object per line:pointer(touchis a shorthand for a pointer of kind touch;x/yin pixels of the shown picture,buttonsasPointerEvent.buttons, wheeldx/dyin pixels),keyboard(W3CKeyboardEvent.code,down/up),key(a named hard key) andtext; errors come back as{"error": ...}lines the exporter never waits on. Back-channel lines onvideo():{"resync": true}and{"lag": seconds}. The full text is the module docstring ofjumpstarter_driver_hmi.common.codecoption orstart(codec=...). H.264 is the default. MJPEG is not recorded (the recorder muxes H.264 and HEVC).hard_keys, because pressing one can lock the user out of a device whose only way in is its screen.stop()and the end of a lease are bounded in time, and a stop that overtakes a slow start leaves nothing running.How to test
Without a device, try the web view with the mock HMI (add
mode: mousefor a mouse and keyboard,codec: mjpegorcodec: hevcfor another codec):Notes on what was checked: the package's tests (1579 pass, 129 skip: conformance checks that do not apply to what a given mock declares) run on Python 3.12, 3.13 and 3.14;
ty,ruff,typosanduv lock --checkpass; the docs build with-W -n; changed-line coverage is 95%. The MP4 recorder's output was decoded with ffmpeg for H.264 and HEVC. The page was driven in Chrome 157 (decoding and painting the H.264, HEVC and MJPEG test patterns; touch, mouse (buttons, wheel, hover, a chord of buttons), keyboard, hard keys, a rotary dial and a view-only HMI produce the expected lines; the token,Host,Originand CSP rules hold): 183 checks pass. Other browsers were not tried. The 129 skips are not hiding failures: they are the conformance cases that need an input or a codec the mock under test does not declare.Stack
This is PR 1 of 3, and the base is
main:jumpstarter-driver-hmijumpstarter-driver-adb(hmi:option onAdbDevice)jumpstarter-driver-iosdevice(hmi:option onIosDevice)PRs 2 and 3 are opened against the previous branch so their diffs show only their own change.