Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 26 additions & 14 deletions src/agent_env/drivers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,32 +81,44 @@ cua-driver's UI Automation tree, whose walk of it fails.
On a Hyprland desktop (Wayland), as Omarchy's is, cua-driver runs its Wayland backend (`CUA_DRIVER_RS_ENABLE_WAYLAND=1`)
and sends input through the seats of Hyprland's `cua-hyprland-plugin`. Hyprland's session turns this on
(`HYPRLAND_INSTANCE_SIGNATURE`), not any Wayland one, since the driver reaches Hyprland's IPC; `supports` names no
Wayland desktop yet, so gate 1 doesn't match it to one. The daemon starts once the compositor's socket answers, on a
socket in the session user's runtime folder. What the driver asks Hyprland (`hyprctl -j`) is in `hyprland.py`, beside
`driver.py`. Keys and text go to the window Hyprland has focused, in the foreground, and to the desktop with nothing
focused; clicks, drags and scrolls stay desktop-scope. `computer-ready` also checks that the plugin is loaded, enabled,
Wayland desktop yet, so gate 1 doesn't match it to one. It runs cua-driver 0.44.0, its own pin, since 0.38.0 refuses
background input there, the delivery cua-driver's own tools default to: `computer-ready` checks its wheel's executable
(`hyprland/amd64` in `EXECUTABLES`) and the tools `mcp-hyprland.json` pins, while `requirements.txt` keeps 0.38.0 for
the other desktops, so a Hyprland computer installs 0.44.0 itself. The daemon starts once the compositor's socket
answers, on a socket in the session user's runtime folder. What the driver asks Hyprland (`hyprctl -j`) is in
`hyprland.py`, beside `driver.py`. Text goes to the desktop in the foreground, where Hyprland's shortcuts, a menu
holding the keyboard and non-ASCII text all work. A key goes to the window Hyprland has focused, in the foreground,
where Backspace, the arrows and Home/End edit text as a person's keys do, which at the desktop they don't, pressed as on
a US keyboard, the only layout the plugin takes there, so a shifted character goes as Shift and the key it's on. Super
and keys held with it, Hyprland's own shortcuts, go to the desktop, as does a key with nothing focused and one the
window path refuses because a layer such as Omarchy's menu holds the keyboard, punctuation there by its keysym name:
given the character, cua-driver falls back to libei, which can hang its input worker. Clicks, drags and scrolls stay
desktop-scope. Keys aimed at a window need the seat's keymap to be the US layout's, and an input method such as
Omarchy's fcitx5 takes on the keymap that desktop text uploads, after which cua-driver refuses them
(`unsupported_layout`): the driver then puts the layout back with a chord of the layout's own keys at the desktop,
Super+Ctrl+Alt+Shift+F12, which it binds to nothing in Hyprland first (`hyprctl eval`), so Hyprland keeps it from every
window, and sends the key again. It presses the chord only while its own bind alone has it: with the chord bound to
anything else, the key fails as cua-driver refused it. `computer-ready` also checks that the plugin is loaded, enabled,
built for this Hyprland and taking input (`transport.ready`), and every input, a `computer_*` operation or one of
cua-driver's own tools that isn't read-only, checks it again first and fails as `device_unavailable`, sending nothing,
when it isn't. The runtime runs ready checks only until they first pass, and a Hyprland restarted after a crash runs in
safe mode with no plugins, where input is accepted but never arrives. A plugin loaded again into a running Hyprland has
no input seat until the desktop restarts, and a driver that outlives its Hyprland can't reach the next one, which has a
new instance signature, so the driver's services have to restart with the graphical session. Keys aimed at a window
skip Hyprland's own shortcuts, such as Super+2, and are refused while a layer such as Omarchy's menu holds the keyboard,
and non-ASCII text aimed at one is refused; sent to the desktop in the foreground, all three work, but Backspace, the
arrows and Home/End do nothing in a text box, so neither way alone covers both.
new instance signature, so the driver's services have to restart with the graphical session.

To move it to another cua-driver version:

1. Pin the version in `requirements.txt`.
2. Regenerate `mcp.json` from what that version's MCP server, `cua-driver mcp`, says at connect and lists: its
`instructions`, each tool's `name`, `description`, `inputSchema`, `annotations` and `outputSchema`, and each
resource's `uri`, `name`, `description` and `mimeType`. That takes a running daemon, so do it on a Linux test
machine, `mcp-macos.json` the same way in a macOS computer, and `mcp-windows.json` in a Windows computer's signed-in
session, on the driver's venv, since `cua-driver mcp` refuses Session 0, where ssh lands.
3. Set `EXECUTABLES` in `driver.py` to the sha256 of `cua_driver/bin/cua-driver` in each arch's wheel, and of
CuaDriver.app's executable for `macos/arm64`, and the release tarball's URL and sha256 in `env.toml`'s
`[install.macos]` and its folder in `install-app.sh`; for `windows/amd64`, the zip's `cua-driver.exe`, its URL and
sha256 in `[install.windows]` and its folder in `install-windows.ps1`.
machine, `mcp-hyprland.json` the same way on a Hyprland desktop at its own pin, `mcp-macos.json` in a macOS
computer, and `mcp-windows.json` in a Windows computer's signed-in session, on the driver's venv, since
`cua-driver mcp` refuses Session 0, where ssh lands.
3. Set `EXECUTABLES` in `driver.py` to the sha256 of `cua_driver/bin/cua-driver` in each arch's wheel, and in the wheel
of Hyprland's own pin for `hyprland/amd64`, and of CuaDriver.app's executable for `macos/arm64`, and the release
tarball's URL and sha256 in `env.toml`'s `[install.macos]` and its folder in `install-app.sh`; for `windows/amd64`,
the zip's `cua-driver.exe`, its URL and sha256 in `[install.windows]` and its folder in `install-windows.ps1`.
4. Run `tst/unit/drivers/cua_driver_test.py`, then `tst/integration/computer/cua_driver_test.py` on Linux, where
`computer-ready` names any tool or executable that doesn't match its pin.

Expand Down
63 changes: 54 additions & 9 deletions src/agent_env/drivers/envs/cua-driver/driver.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@
cua-driver starts again after ending it for idling, where a named one stays ended. A call the daemon can't take, as
while it restarts, fails as unavailable; input whose reply is lost fails with its outcome unknown, since it may have
arrived. It needs ``agentenv-framework-protocol[computer]`` and ``cua-driver`` where it runs. On a Hyprland desktop
(Wayland) it runs cua-driver's Wayland backend, sends keys and text to the focused window in the foreground, and is
ready, and takes input, only while Hyprland has cua-driver's plugin loaded, asking Hyprland through hyprland.py."""
(Wayland) it runs cua-driver's Wayland backend at its own pin, sends text to the desktop and keys to the focused
window in the foreground, Super and a key a menu holds the keyboard from going to the desktop, and is ready, and takes
input, only while Hyprland has cua-driver's plugin loaded, asking Hyprland through hyprland.py."""
from __future__ import annotations

import asyncio
Expand Down Expand Up @@ -77,18 +78,20 @@
SOCKET = "/run/agentenv/cua-driver.sock"
#: What cua-driver's own MCP server (``cua-driver mcp``) says at connect and lists at the pinned version on each OS: its
#: instructions, its tools, which the running daemon must list the same, and its skill resources.
MANIFEST = Path(__file__).with_name("mcp-macos.json" if MACOS else "mcp-windows.json" if WINDOWS else "mcp.json")
MANIFEST = Path(__file__).with_name("mcp-macos.json" if MACOS else "mcp-windows.json" if WINDOWS else
"mcp-hyprland.json" if HYPRLAND else "mcp.json")
#: Where the release's zip is unpacked on Windows, beside this file.
WINDOWS_EXECUTABLE = Path(__file__).with_name("cua-driver") / "cua-driver.exe"
#: CuaDriver.app's executable, which macOS runs the daemon as.
APP_EXECUTABLE = Path("/Applications/CuaDriver.app/Contents/MacOS/cua-driver")
#: The sha256 of the cua-driver executable each arch's pinned wheel installs, on macOS CuaDriver.app's, and on Windows
#: the release zip's.
#: The sha256 of the cua-driver executable each arch's pinned wheel installs, on macOS CuaDriver.app's, on Windows the
#: release zip's, and on Hyprland its own pin's wheel's, 0.44, which delivers background input there.
EXECUTABLES = {
"amd64": "bf57b2f3778a0a7daa9388e241f20ce948b32a7a2f4dd102e497f34e54137934",
"arm64": "26e9eee8165e9ce7b614b5e3d68e5d963231385a82b65236406718b5ce1ff936",
"macos/arm64": "b4d974a9c1c92a700cd7be7ede987aea4a1e6835e37039ce3d628b71bd5f0ca8",
"windows/amd64": "ffa58d88779ee219304b0c307f4cfa970fe822f830c6ecf9254eda5c29e22bf2",
"hyprland/amd64": "f4259d5135d23d4d03968e074d144a10ece6305ca28ef48edfb3ef5c9e724228",
}
#: How the daemon delivers the pointer's actions: in the foreground on macOS, where it doesn't by default.
FOREGROUND = {"delivery_mode": "foreground"} if MACOS else {}
Expand Down Expand Up @@ -140,6 +143,7 @@ def __init__(self) -> None:
self._skill: tuple[str, str, dict[str, str]] | None = None
self._texts: dict[str, str] | None = None
self._keymap: _Keymap | None = None
self._reset_bind: dict | None = None

def forwarded_tools(self) -> list[ForwardedTool]:
return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]),
Expand Down Expand Up @@ -209,13 +213,53 @@ async def scroll_at(self, x, y, direction, amount, modifiers) -> None:
async def type_in(self, text) -> None:
if not MACOS and not WINDOWS and not HYPRLAND and any(ord(ch) >= LATIN_1 for ch in text):
await asyncio.to_thread(self._hold, text)
await self._send("type_text", text=text, **(await hyprland.focused() if HYPRLAND else {}))
await self._send("type_text", text=text, **(hyprland.DESKTOP if HYPRLAND else {}))

@press
async def press_keys(self, keys) -> None:
*held, key = keys
await self._send("press_key", modifiers=_modifiers(held), key=MODIFIERS.get(key) or NAMED.get(key, key),
**(await hyprland.focused() if HYPRLAND else {}))
modifiers, name = _modifiers(held), MODIFIERS.get(key) or NAMED.get(key, key)
if HYPRLAND:
await self._press_on_hyprland(modifiers, name)
else:
await self._send("press_key", modifiers=modifiers, key=name)

async def _press_on_hyprland(self, modifiers: list[str], key: str) -> None:
"""A key on Hyprland: to the focused window as a person presses it on a US keyboard, where it edits text as a
person's does, unless it's Super or held with it, which is Hyprland's own, or nothing is focused, or a layer
such as a menu holds the keyboard and refuses it there; then to the desktop, punctuation by its keysym name."""
if key != "super" and "super" not in modifiers and (window := await hyprland.focused()):
try:
return await self._to_window({**hyprland.on_us_keyboard(modifiers, key), **window})
except DeliveryFailed as e:
if hyprland.GRABBED not in e.message:
raise
await self._send("press_key", modifiers=modifiers, key=hyprland.PUNCTUATION.get(key, key),
**hyprland.DESKTOP)

async def _to_window(self, pressed: dict) -> None:
"""A key aimed at a window, the US layout's keymap put back first when cua-driver refuses it for the seat's
keymap, which an input method changes after desktop text, by a chord bound to nothing but this driver's bind,
which Hyprland keeps from every window."""
try:
return await self._send("press_key", **pressed)
except DeliveryFailed as e:
if hyprland.UNLAID not in e.message:
raise
if (bind := await hyprland.reset_bind(self._reset_bind)) is None:
raise DeliveryFailed(e.message, hint=f"Hyprland has {hyprland.RESET_BIND}, which puts the US keymap back, "
"bound to something this driver didn't bind, or it couldn't bind "
"it, so the driver didn't press it") from e
self._reset_bind = bind
await self._send("press_key", **hyprland.RESET, **hyprland.DESKTOP)
deadline = time.monotonic() + hyprland.RESET_S
while True:
try:
return await self._send("press_key", **pressed)
except DeliveryFailed as e:
if hyprland.UNLAID not in e.message or time.monotonic() > deadline:
raise
await asyncio.sleep(hyprland.RESET_POLL_S)

async def _forward(self, tool: str, acts: bool, /, **arguments: Any) -> CallToolResult:
"""A call to one of cua-driver's own tools in its caller's session, answered as cua-driver answers it; one that
Expand Down Expand Up @@ -275,7 +319,8 @@ async def _pinned_checks(self) -> list[ReadyCheck]:
return [ReadyCheck("cua-driver", False, "its daemon isn't answering yet")]
arch = self.platform.arch or ""
digest = await asyncio.to_thread(_executable_sha256)
expected = EXECUTABLES.get(f"macos/{arch}" if MACOS else f"windows/{arch}" if WINDOWS else arch)
expected = EXECUTABLES.get(f"macos/{arch}" if MACOS else f"windows/{arch}" if WINDOWS else
f"hyprland/{arch}" if HYPRLAND else arch)
running = {t["name"]: t["input_schema"] for t in listed}
pinned = {t["name"]: t["inputSchema"] for t in _manifest()["tools"]}
differ = sorted(name for name in running.keys() | pinned.keys() if running.get(name) != pinned.get(name))
Expand Down
Loading
Loading