diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index 1e9678acd..df95be372 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -81,19 +81,30 @@ cua-driver's UI Automation tree, whose walk of it fails. On a Hyprland desktop (Wayland), as Omarchy's is, cua-driver runs its Wayland backend (`CUA_DRIVER_RS_ENABLE_WAYLAND=1`) and sends input through the seats of Hyprland's `cua-hyprland-plugin`. Hyprland's session turns this on (`HYPRLAND_INSTANCE_SIGNATURE`), not any Wayland one, since the driver reaches Hyprland's IPC; `supports` names no -Wayland desktop yet, so gate 1 doesn't match it to one. The daemon starts once the compositor's socket answers, on a -socket in the session user's runtime folder. What the driver asks Hyprland (`hyprctl -j`) is in `hyprland.py`, beside -`driver.py`. Keys and text go to the window Hyprland has focused, in the foreground, and to the desktop with nothing -focused; clicks, drags and scrolls stay desktop-scope. `computer-ready` also checks that the plugin is loaded, enabled, +Wayland desktop yet, so gate 1 doesn't match it to one. It runs cua-driver 0.44.0, its own pin, since 0.38.0 refuses +background input there, the delivery cua-driver's own tools default to: `computer-ready` checks its wheel's executable +(`hyprland/amd64` in `EXECUTABLES`) and the tools `mcp-hyprland.json` pins, while `requirements.txt` keeps 0.38.0 for +the other desktops, so a Hyprland computer installs 0.44.0 itself. The daemon starts once the compositor's socket +answers, on a socket in the session user's runtime folder. What the driver asks Hyprland (`hyprctl -j`) is in +`hyprland.py`, beside `driver.py`. Text goes to the desktop in the foreground, where Hyprland's shortcuts, a menu +holding the keyboard and non-ASCII text all work. A key goes to the window Hyprland has focused, in the foreground, +where Backspace, the arrows and Home/End edit text as a person's keys do, which at the desktop they don't, pressed as on +a US keyboard, the only layout the plugin takes there, so a shifted character goes as Shift and the key it's on. Super +and keys held with it, Hyprland's own shortcuts, go to the desktop, as does a key with nothing focused and one the +window path refuses because a layer such as Omarchy's menu holds the keyboard, punctuation there by its keysym name: +given the character, cua-driver falls back to libei, which can hang its input worker. Clicks, drags and scrolls stay +desktop-scope. Keys aimed at a window need the seat's keymap to be the US layout's, and an input method such as +Omarchy's fcitx5 takes on the keymap that desktop text uploads, after which cua-driver refuses them +(`unsupported_layout`): the driver then puts the layout back with a chord of the layout's own keys at the desktop, +Super+Ctrl+Alt+Shift+F12, which it binds to nothing in Hyprland first (`hyprctl eval`), so Hyprland keeps it from every +window, and sends the key again. It presses the chord only while its own bind alone has it: with the chord bound to +anything else, the key fails as cua-driver refused it. `computer-ready` also checks that the plugin is loaded, enabled, built for this Hyprland and taking input (`transport.ready`), and every input, a `computer_*` operation or one of cua-driver's own tools that isn't read-only, checks it again first and fails as `device_unavailable`, sending nothing, when it isn't. The runtime runs ready checks only until they first pass, and a Hyprland restarted after a crash runs in safe mode with no plugins, where input is accepted but never arrives. A plugin loaded again into a running Hyprland has no input seat until the desktop restarts, and a driver that outlives its Hyprland can't reach the next one, which has a -new instance signature, so the driver's services have to restart with the graphical session. Keys aimed at a window -skip Hyprland's own shortcuts, such as Super+2, and are refused while a layer such as Omarchy's menu holds the keyboard, -and non-ASCII text aimed at one is refused; sent to the desktop in the foreground, all three work, but Backspace, the -arrows and Home/End do nothing in a text box, so neither way alone covers both. +new instance signature, so the driver's services have to restart with the graphical session. To move it to another cua-driver version: @@ -101,12 +112,13 @@ To move it to another cua-driver version: 2. Regenerate `mcp.json` from what that version's MCP server, `cua-driver mcp`, says at connect and lists: its `instructions`, each tool's `name`, `description`, `inputSchema`, `annotations` and `outputSchema`, and each resource's `uri`, `name`, `description` and `mimeType`. That takes a running daemon, so do it on a Linux test - machine, `mcp-macos.json` the same way in a macOS computer, and `mcp-windows.json` in a Windows computer's signed-in - session, on the driver's venv, since `cua-driver mcp` refuses Session 0, where ssh lands. -3. Set `EXECUTABLES` in `driver.py` to the sha256 of `cua_driver/bin/cua-driver` in each arch's wheel, and of - CuaDriver.app's executable for `macos/arm64`, and the release tarball's URL and sha256 in `env.toml`'s - `[install.macos]` and its folder in `install-app.sh`; for `windows/amd64`, the zip's `cua-driver.exe`, its URL and - sha256 in `[install.windows]` and its folder in `install-windows.ps1`. + machine, `mcp-hyprland.json` the same way on a Hyprland desktop at its own pin, `mcp-macos.json` in a macOS + computer, and `mcp-windows.json` in a Windows computer's signed-in session, on the driver's venv, since + `cua-driver mcp` refuses Session 0, where ssh lands. +3. Set `EXECUTABLES` in `driver.py` to the sha256 of `cua_driver/bin/cua-driver` in each arch's wheel, and in the wheel + of Hyprland's own pin for `hyprland/amd64`, and of CuaDriver.app's executable for `macos/arm64`, and the release + tarball's URL and sha256 in `env.toml`'s `[install.macos]` and its folder in `install-app.sh`; for `windows/amd64`, + the zip's `cua-driver.exe`, its URL and sha256 in `[install.windows]` and its folder in `install-windows.ps1`. 4. Run `tst/unit/drivers/cua_driver_test.py`, then `tst/integration/computer/cua_driver_test.py` on Linux, where `computer-ready` names any tool or executable that doesn't match its pin. diff --git a/src/agent_env/drivers/envs/cua-driver/driver.py b/src/agent_env/drivers/envs/cua-driver/driver.py index 9f0f5a826..1d6cd3baa 100644 --- a/src/agent_env/drivers/envs/cua-driver/driver.py +++ b/src/agent_env/drivers/envs/cua-driver/driver.py @@ -9,8 +9,9 @@ cua-driver starts again after ending it for idling, where a named one stays ended. A call the daemon can't take, as while it restarts, fails as unavailable; input whose reply is lost fails with its outcome unknown, since it may have arrived. It needs ``agentenv-framework-protocol[computer]`` and ``cua-driver`` where it runs. On a Hyprland desktop -(Wayland) it runs cua-driver's Wayland backend, sends keys and text to the focused window in the foreground, and is -ready, and takes input, only while Hyprland has cua-driver's plugin loaded, asking Hyprland through hyprland.py.""" +(Wayland) it runs cua-driver's Wayland backend at its own pin, sends text to the desktop and keys to the focused +window in the foreground, Super and a key a menu holds the keyboard from going to the desktop, and is ready, and takes +input, only while Hyprland has cua-driver's plugin loaded, asking Hyprland through hyprland.py.""" from __future__ import annotations import asyncio @@ -77,18 +78,20 @@ SOCKET = "/run/agentenv/cua-driver.sock" #: What cua-driver's own MCP server (``cua-driver mcp``) says at connect and lists at the pinned version on each OS: its #: instructions, its tools, which the running daemon must list the same, and its skill resources. -MANIFEST = Path(__file__).with_name("mcp-macos.json" if MACOS else "mcp-windows.json" if WINDOWS else "mcp.json") +MANIFEST = Path(__file__).with_name("mcp-macos.json" if MACOS else "mcp-windows.json" if WINDOWS else + "mcp-hyprland.json" if HYPRLAND else "mcp.json") #: Where the release's zip is unpacked on Windows, beside this file. WINDOWS_EXECUTABLE = Path(__file__).with_name("cua-driver") / "cua-driver.exe" #: CuaDriver.app's executable, which macOS runs the daemon as. APP_EXECUTABLE = Path("/Applications/CuaDriver.app/Contents/MacOS/cua-driver") -#: The sha256 of the cua-driver executable each arch's pinned wheel installs, on macOS CuaDriver.app's, and on Windows -#: the release zip's. +#: The sha256 of the cua-driver executable each arch's pinned wheel installs, on macOS CuaDriver.app's, on Windows the +#: release zip's, and on Hyprland its own pin's wheel's, 0.44, which delivers background input there. EXECUTABLES = { "amd64": "bf57b2f3778a0a7daa9388e241f20ce948b32a7a2f4dd102e497f34e54137934", "arm64": "26e9eee8165e9ce7b614b5e3d68e5d963231385a82b65236406718b5ce1ff936", "macos/arm64": "b4d974a9c1c92a700cd7be7ede987aea4a1e6835e37039ce3d628b71bd5f0ca8", "windows/amd64": "ffa58d88779ee219304b0c307f4cfa970fe822f830c6ecf9254eda5c29e22bf2", + "hyprland/amd64": "f4259d5135d23d4d03968e074d144a10ece6305ca28ef48edfb3ef5c9e724228", } #: How the daemon delivers the pointer's actions: in the foreground on macOS, where it doesn't by default. FOREGROUND = {"delivery_mode": "foreground"} if MACOS else {} @@ -140,6 +143,7 @@ def __init__(self) -> None: self._skill: tuple[str, str, dict[str, str]] | None = None self._texts: dict[str, str] | None = None self._keymap: _Keymap | None = None + self._reset_bind: dict | None = None def forwarded_tools(self) -> list[ForwardedTool]: return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]), @@ -209,13 +213,53 @@ async def scroll_at(self, x, y, direction, amount, modifiers) -> None: async def type_in(self, text) -> None: if not MACOS and not WINDOWS and not HYPRLAND and any(ord(ch) >= LATIN_1 for ch in text): await asyncio.to_thread(self._hold, text) - await self._send("type_text", text=text, **(await hyprland.focused() if HYPRLAND else {})) + await self._send("type_text", text=text, **(hyprland.DESKTOP if HYPRLAND else {})) @press async def press_keys(self, keys) -> None: *held, key = keys - await self._send("press_key", modifiers=_modifiers(held), key=MODIFIERS.get(key) or NAMED.get(key, key), - **(await hyprland.focused() if HYPRLAND else {})) + modifiers, name = _modifiers(held), MODIFIERS.get(key) or NAMED.get(key, key) + if HYPRLAND: + await self._press_on_hyprland(modifiers, name) + else: + await self._send("press_key", modifiers=modifiers, key=name) + + async def _press_on_hyprland(self, modifiers: list[str], key: str) -> None: + """A key on Hyprland: to the focused window as a person presses it on a US keyboard, where it edits text as a + person's does, unless it's Super or held with it, which is Hyprland's own, or nothing is focused, or a layer + such as a menu holds the keyboard and refuses it there; then to the desktop, punctuation by its keysym name.""" + if key != "super" and "super" not in modifiers and (window := await hyprland.focused()): + try: + return await self._to_window({**hyprland.on_us_keyboard(modifiers, key), **window}) + except DeliveryFailed as e: + if hyprland.GRABBED not in e.message: + raise + await self._send("press_key", modifiers=modifiers, key=hyprland.PUNCTUATION.get(key, key), + **hyprland.DESKTOP) + + async def _to_window(self, pressed: dict) -> None: + """A key aimed at a window, the US layout's keymap put back first when cua-driver refuses it for the seat's + keymap, which an input method changes after desktop text, by a chord bound to nothing but this driver's bind, + which Hyprland keeps from every window.""" + try: + return await self._send("press_key", **pressed) + except DeliveryFailed as e: + if hyprland.UNLAID not in e.message: + raise + if (bind := await hyprland.reset_bind(self._reset_bind)) is None: + raise DeliveryFailed(e.message, hint=f"Hyprland has {hyprland.RESET_BIND}, which puts the US keymap back, " + "bound to something this driver didn't bind, or it couldn't bind " + "it, so the driver didn't press it") from e + self._reset_bind = bind + await self._send("press_key", **hyprland.RESET, **hyprland.DESKTOP) + deadline = time.monotonic() + hyprland.RESET_S + while True: + try: + return await self._send("press_key", **pressed) + except DeliveryFailed as e: + if hyprland.UNLAID not in e.message or time.monotonic() > deadline: + raise + await asyncio.sleep(hyprland.RESET_POLL_S) async def _forward(self, tool: str, acts: bool, /, **arguments: Any) -> CallToolResult: """A call to one of cua-driver's own tools in its caller's session, answered as cua-driver answers it; one that @@ -275,7 +319,8 @@ async def _pinned_checks(self) -> list[ReadyCheck]: return [ReadyCheck("cua-driver", False, "its daemon isn't answering yet")] arch = self.platform.arch or "" digest = await asyncio.to_thread(_executable_sha256) - expected = EXECUTABLES.get(f"macos/{arch}" if MACOS else f"windows/{arch}" if WINDOWS else arch) + expected = EXECUTABLES.get(f"macos/{arch}" if MACOS else f"windows/{arch}" if WINDOWS else + f"hyprland/{arch}" if HYPRLAND else arch) running = {t["name"]: t["input_schema"] for t in listed} pinned = {t["name"]: t["inputSchema"] for t in _manifest()["tools"]} differ = sorted(name for name in running.keys() | pinned.keys() if running.get(name) != pinned.get(name)) diff --git a/src/agent_env/drivers/envs/cua-driver/hyprland.py b/src/agent_env/drivers/envs/cua-driver/hyprland.py index b940a3440..2c981e0ab 100644 --- a/src/agent_env/drivers/envs/cua-driver/hyprland.py +++ b/src/agent_env/drivers/envs/cua-driver/hyprland.py @@ -4,25 +4,89 @@ from __future__ import annotations import asyncio +import contextlib import json +import string import subprocess from agentenv_protocol.computer import DeviceUnavailable, ReadyCheck #: How long a request to Hyprland's IPC waits for its answer. TIMEOUT_S = 5 +#: How keys and text go to the desktop: in the foreground, through the plugin's seat, so Hyprland's own shortcuts see +#: them first and then whatever has the keyboard gets them, a window or a layer such as a menu. Without it a key with +#: modifiers goes through libei, which Hyprland doesn't serve. +DESKTOP = {"delivery_mode": "foreground"} +#: X's keysym names for ASCII's punctuation, by which cua-driver presses those keys at the desktop: given the character +#: there, it has no evdev key for it and falls back to libei, which can hang its input worker. +PUNCTUATION = { + "!": "exclam", '"': "quotedbl", "#": "numbersign", "$": "dollar", "%": "percent", "&": "ampersand", + "'": "apostrophe", "(": "parenleft", ")": "parenright", "*": "asterisk", "+": "plus", ",": "comma", "-": "minus", + ".": "period", "/": "slash", ":": "colon", ";": "semicolon", "<": "less", "=": "equal", ">": "greater", + "?": "question", "@": "at", "[": "bracketleft", "\\": "backslash", "]": "bracketright", "^": "asciicircum", + "_": "underscore", "`": "grave", "{": "braceleft", "|": "bar", "}": "braceright", "~": "asciitilde", +} +#: The key each character a US keyboard types with Shift is on: a key aimed at a window is one of the seat's US layout, +#: the only layout cua-driver's plugin takes, and a character it has no key for, or an uppercase letter's, it refuses or +#: types unshifted. +SHIFTED = { + "!": "1", "@": "2", "#": "3", "$": "4", "%": "5", "^": "6", "&": "7", "*": "8", "(": "9", ")": "0", "_": "-", + "+": "=", "{": "[", "}": "]", "|": "\\", ":": ";", '"': "'", "<": ",", ">": ".", "?": "/", "~": "`", + **{letter: letter.lower() for letter in string.ascii_uppercase}, +} +#: What cua-driver says when a key aimed at a window is refused because a layer, such as a menu, holds the keyboard. +GRABBED = "foreground_grab" +#: What it says when one is refused because the seat's keymap isn't the US layout's: an input method such as Omarchy's +#: fcitx5 takes on the keymap that desktop text, or a desktop key the layout has no key for, uploads. +UNLAID = "unsupported_layout" +#: A chord at the desktop that puts the US layout's keymap back: its keys are all the layout's, so cua-driver sends it +#: with that keymap, which an input method takes on in turn, and it's bound to nothing (RESET_BIND), so Hyprland keeps +#: it from every window. +RESET = {"modifiers": ["super", "ctrl", "alt", "shift"], "key": "f12"} +RESET_BIND = "SUPER + CTRL + ALT + SHIFT + F12" +#: RESET_BIND as Hyprland lists its binds: Super, Alt, Ctrl and Shift's mask, and its key. +RESET_LISTED = (64 | 8 | 4 | 1, "F12") +#: How long a key aimed at a window waits for the keymap RESET puts back, and how often it's sent again meanwhile. +RESET_S, RESET_POLL_S = 1.0, 0.05 + + +def on_us_keyboard(modifiers: list[str], key: str) -> dict: + """A key as a person presses it on a US keyboard, a character typed with Shift as Shift and the key it's on.""" + if base := SHIFTED.get(key): + return {"modifiers": modifiers if "shift" in modifiers else [*modifiers, "shift"], "key": base} + return {"modifiers": modifiers, "key": key} async def focused() -> dict: - """Where keys go: the focused window, whose Hyprland address is its cua-driver window id, in the foreground through - the plugin's seat, since a desktop-scope key with modifiers goes through libei, which Hyprland doesn't serve; with - nothing focused, the desktop.""" + """The focused window as a key's target: its pid, its Hyprland address, which is its cua-driver window id, in the + foreground through the plugin's seat, where a key edits text as a person's does; nothing with nothing focused.""" window = await asyncio.to_thread(hyprctl, "activewindow") if not window or not window.get("address"): return {} return {"pid": window["pid"], "window_id": int(window["address"], 16), "delivery_mode": "foreground"} +async def reset_bind(made: dict | None) -> dict | None: + """The bind of RESET_BIND this driver may press, as Hyprland lists it: ``made``, the one it bound to nothing, while + that bind alone has the chord; one it binds to nothing now when the chord has none, as once Hyprland's config + reloads; None while any other bind has the chord, since one anybody else made could do anything.""" + return await asyncio.to_thread(_reset_bind, made) + + +def _reset_bind(made: dict | None) -> dict | None: + if binds := _chord_binds(): + return made if made is not None and binds == [made] else None + lua(f'hl.bind("{RESET_BIND}", function() end)') + return binds[0] if len(binds := _chord_binds()) == 1 else None + + +def _chord_binds() -> list[dict]: + """RESET_BIND's binds as Hyprland lists them, each by its mask, key, dispatcher and argument, which tell them apart: + a bind made in Lua has the dispatcher ``__lua`` and its function's number as its argument.""" + return [{field: bind.get(field) for field in ("modmask", "key", "dispatcher", "arg")} + for bind in hyprctl("binds") or [] if (bind.get("modmask"), bind.get("key")) == RESET_LISTED] + + async def plugin() -> ReadyCheck: """That Hyprland has cua-driver's plugin loaded, enabled, built for it and taking input; one restarted after a crash runs in safe mode, with no plugins, where input is sent but never arrives, and a plugin loaded again into a @@ -53,12 +117,18 @@ async def require_plugin() -> None: raise DeviceUnavailable(check.detail) -def hyprctl(request: str) -> dict | None: - """Hyprland's JSON answer to ``request``, or None when it gives none, as for a plugin's request it doesn't know or - when Hyprland doesn't answer.""" +def hyprctl(request: str) -> dict | list | None: + """Hyprland's JSON answer to ``request``, a list for one such as ``binds``, or None when it gives none, as for a + plugin's request it doesn't know or when Hyprland doesn't answer.""" try: out = subprocess.run(["hyprctl", "-j", request], capture_output=True, text=True, timeout=TIMEOUT_S) answer = json.loads(out.stdout) except (OSError, subprocess.TimeoutExpired, ValueError): return None - return answer if isinstance(answer, dict) else None + return answer if isinstance(answer, (dict, list)) else None + + +def lua(code: str) -> None: + """Runs ``code`` in Hyprland's Lua config (``hyprctl eval``); what it did is for the caller to ask Hyprland.""" + with contextlib.suppress(OSError, subprocess.TimeoutExpired): + subprocess.run(["hyprctl", "eval", code], capture_output=True, text=True, timeout=TIMEOUT_S) diff --git a/src/agent_env/drivers/envs/cua-driver/mcp-hyprland.json b/src/agent_env/drivers/envs/cua-driver/mcp-hyprland.json new file mode 100644 index 000000000..e6a4d3232 --- /dev/null +++ b/src/agent_env/drivers/envs/cua-driver/mcp-hyprland.json @@ -0,0 +1,10367 @@ +{ + "instructions": "cua-driver: background GUI automation. No shell: for non-GUI outcomes use an app API/SDK, CLI or filesystem and read the result back there.\n\n1. `list_windows({app:\"Name\"})` or `launch_app` for `pid` and `window_id`.\n2. Read narrowly: `get_window_state` with `query:\"Save\"`. Re-read with `since:\"latest\"`.\n3. Act by `element_token` (AT-SPI): row `[N]` is `:N`. Pixels only for surfaces missing from the tree.\n4. Act through `run_steps({steps:[{tool,args}],observe:{}})`, even for one step: stops at the first failure, returns what changed. Loops, branches or retries: one `run_script` call.\n5. Verify at checkpoints, not after every action: `verify_state(pid, window_id, expect)`. `unknown` and `effect:\"unverifiable\"` are not success.\n\nUse the narrowest semantic route first: `set_window_frame` plus `list_windows` for geometry, typed browser tools for pages. Stay in the background; `delivery_mode:\"foreground\"` only if refused or unverifiable.\n\nChrome/Edge/Brave page: `get_browser_state({pid, window_id})`, its `detail.next_call` (`browser_prepare`) if refused, then `browser_click`/`browser_type`/`browser_pointer` (hover, drag); never desktop pointer tools.\n\n`start_session` is optional; repeat a short `session` label on every call that accepts it. On continuation/recent-work, call `history_status`; if ready, make one bounded initial `history_query` before broad discovery.\n\nGuides: `get_guide({topic})` (drag, browser, permissions, batching, menus), `skill://cua-driver/SKILL.md`, LINUX.md (X11/Wayland status, AT-SPI bus, BETA-level support).", + "tools": [ + { + "name": "list_apps", + "description": "List Linux apps — both currently running and installed-but-not-running — with per-app state flags:\n\n- running: is a process for this app live? (pid is 0 when false)\n- active: reserved (Linux X11/Wayland focus model differs from frontmost-app); always false.\n- kind: `\"desktop\"` for XDG `.desktop` launcher entries.\n- launch_path: the launcher command from `Exec=` (field codes stripped). Pass to `launch_app(launch_path=...)`.\n- bundle_id: the XDG \"desktop file id\" — the `.desktop` file's path relative to its XDG `applications/` root with the `.desktop` suffix stripped and path separators replaced with `-` (e.g. `kde4/konqbrowser.desktop` → `kde4-konqbrowser`).\n- last_used: RFC3339 mtime of the `.desktop` file, when readable.\n- windows: the app's current top-level windows (same records as list_windows); empty for a process without one.\n\nRunning apps come from `/proc`. Installed apps come from XDG Desktop Entry files in $XDG_DATA_HOME/applications and each $XDG_DATA_DIRS entry's applications/ subdir. Entries with `NoDisplay=true` or `Hidden=true` are filtered. A `.desktop` file whose launcher matches a running process (by basename), or whose WM_CLASS a top-level window carries, is merged into a single entry with `running: true`.\n\nUse this for \"is X installed?\" as well as \"is X running?\". For per-window state — visibility, geometry, titles — call list_windows instead.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "apps": { + "items": { + "properties": { + "active": { + "type": "boolean" + }, + "bundle_id": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "type": [ + "string", + "null" + ] + }, + "last_used": { + "type": [ + "string", + "null" + ] + }, + "launch_path": { + "type": [ + "string", + "null" + ] + }, + "name": { + "type": "string" + }, + "pid": { + "minimum": 0, + "type": "integer" + }, + "running": { + "type": "boolean" + } + }, + "required": [ + "pid", + "name", + "running", + "active" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "apps" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "list_windows", + "description": "List top-level windows. Each record includes z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one). To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order. Filter with `app` or `pid`: an unfiltered call lists every window on the host.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "app": { + "description": "Only windows of apps whose name contains this text (case-insensitive), or whose bundle id equals it. Prefer this or `pid` over an unfiltered call, which returns every window on the host.", + "type": "string" + }, + "on_screen_only": { + "description": "When true, filter to visible windows only. Default false.", + "type": "boolean" + }, + "pid": { + "description": "Only list windows owned by this process ID.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "current_space_id": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "windows": { + "items": { + "properties": { + "app_name": { + "type": "string" + }, + "bounds": { + "properties": { + "height": { + "type": "number" + }, + "width": { + "type": "number" + }, + "x": { + "type": "number" + }, + "y": { + "type": "number" + } + }, + "required": [ + "x", + "y", + "width", + "height" + ], + "type": "object" + }, + "current_space_id": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "is_on_screen": { + "type": "boolean" + }, + "layer": { + "format": "int32", + "type": [ + "integer", + "null" + ] + }, + "minimized": { + "type": [ + "boolean", + "null" + ] + }, + "on_current_space": { + "type": [ + "boolean", + "null" + ] + }, + "pid": { + "maximum": 4294967295, + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "space_ids": { + "items": { + "minimum": 0, + "type": "integer" + }, + "type": [ + "array", + "null" + ] + }, + "title": { + "type": "string" + }, + "window_id": { + "minimum": 0, + "type": "integer" + }, + "z_index": { + "description": "Higher values are closer to the front. Null means the provider cannot observe stacking order; callers must not infer an order from array position or treat null as zero.", + "type": [ + "integer", + "null" + ] + } + }, + "required": [ + "window_id", + "pid", + "app_name", + "title", + "bounds", + "is_on_screen", + "z_index" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "windows" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_window_state", + "description": "Walk a running app's AT-SPI tree and return it ONCE, as compact Markdown by default (`tree_format:\"markdown\"`). Every actionable element is tagged `[N]` (an element_index); its element_token is `:N`, where `snapshot_id` is printed in the response header and returned in structuredContent. Pass `tree_format:\"elements\"` for the structured `elements` array instead (explicit `element_token`, `role`, `label`, `value`, `enabled`, `selected`, `actions`, `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`), or `\"both\"` for both (about twice the size). Set `query` to project the tree to matching rows plus their ancestor chain while preserving original indices; `total_element_count` reports the complete snapshot.\n\nSize: a read is bounded by default (`max_elements` 250 actionable nodes). When the walk stops at that budget the response says `Tree truncated at max_elements=…` and `truncated: true`; pass a larger `max_elements`, or `query` / `max_depth`, to reach the rest. `_note` is omitted unless `verbose:true`. `full_output:true` restores the previous full response: both representations, all metadata, and the walker's own limits (≤5 000 nodes).\n\nDIFF READS: pass `since: ` from an earlier read of the same window to get only what changed: `+` added rows, `~` changed rows, `-` removed rows (their ids are the old snapshot's), or `no change since …`. Rows not listed keep their previous `[N]` unless a `reindexed:` line says otherwise. The response carries a NEW snapshot_id: use it in element_tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) `since` falls back to a full read and `since_status` says why. The focused element is not reported on Linux yet.\n\nAlways returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.\n\nThe mirror image: pass `include_accessibility_tree:false` to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error. Optional `max_dimension` caps the returned screenshot's long edge in pixels for a cheap thumbnail.\n\nOptional `max_elements` / `max_depth` bound the AT-SPI walk (Electron / large web apps produce 10k+ element trees). Markdown and structured elements are truncated identically. Default: `max_elements` 250.\n\nTIME BUDGET: `timeout_ms` (default 1000) bounds the whole AT-SPI walk. When omitted, a window's first snapshot gets a 2000 ms cold-start budget so Chromium/Electron can finish publishing its initial tree. Explicit values are always honored. Large apps (LibreOffice, GIMP, file managers) can exceed the budget; the call then returns the PARTIAL tree with `truncated: true`, `truncation_reason`, `nodes_visited`/`nodes_pending` and `elements_complete: false`. Every element listed is real and clickable; elements after the cut are simply missing. Retry with a larger `timeout_ms` (e.g. 5000) or narrow with `query`/`max_depth` when the element you need is absent.\n\nSCREENSHOT SCALE: the screenshot is delivered at or below 1.15 megapixels (long edge <= max_image_dimension, 1568 by default), because larger images are downsized before a model reads them and its pixel coordinates would then be uniformly short. Element `screenshot_frame`s and x/y for the pointer tools are pixels of the delivered screenshot (`frame` stays in screen coordinates, as on every platform); `frame_scale` < 1 reports the downsizing. An explicit per-call `max_image_dimension` (0 = native) replaces this cap.\n\nPOPUP MENUS: a context menu / popover / combo list is an override-redirect window that list_windows never shows. A click or right_click that opened one names it in its result (`popup: {window_id, bounds, title}`); pass that window_id here to walk the popup's own AT-SPI toplevel so its menu items get element indices (then click them by element_token). Omitting window_id while a popup of this pid is open walks that popup.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "capture_mode": { + "description": "DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old \"som\"/\"screenshot\" aliases) is accepted but has no effect.", + "enum": [ + "ax", + "vision" + ], + "type": "string" + }, + "display_only": { + "description": "Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions.", + "type": "boolean" + }, + "full_output": { + "description": "Default false. Set true to restore the previous full response: both `elements` and `tree_markdown`, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with `since`.", + "type": "boolean" + }, + "include_accessibility_tree": { + "description": "Default true — walk the AT-SPI tree and return it (per `tree_format`) alongside the screenshot. Set false to SKIP the AT-SPI walk entirely and return just the screenshot plus window metadata (window_bounds, app_name, window_title) — the capture-only path for a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return).", + "type": "boolean" + }, + "include_screenshot": { + "description": "Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action).", + "type": "boolean" + }, + "max_depth": { + "description": "Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps.", + "minimum": 1, + "type": "integer" + }, + "max_dimension": { + "description": "Legacy optional cap on the returned screenshot's long edge. Applied on top of the configured max_image_dimension ceiling when max_image_dimension is omitted.", + "minimum": 1, + "type": "integer" + }, + "max_elements": { + "description": "Cap on total AT-SPI nodes walked. Default 250 (5 000 with full_output:true); the response states when the tree was cut. Raise it to read further into a large window.", + "minimum": 1, + "type": "integer" + }, + "max_image_dimension": { + "description": "Per-call long-edge override. This value wins over configured and legacy limits; 0 returns native-resolution PNG bytes. Omit to preserve configured behavior.", + "minimum": 0, + "type": "integer" + }, + "pid": { + "description": "Process ID that owns the window.", + "type": "integer" + }, + "query": { + "description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.", + "type": "string" + }, + "screenshot_out_file": { + "description": "When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "since": { + "description": "A `snapshot_id` from an earlier get_window_state of the same window, or \"latest\" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: `+` added rows, `~` changed rows, `-` removed rows (removed ids are the old snapshot's), or `no change`. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; `since_status` says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a `reindexed:` line says otherwise.", + "type": "string" + }, + "timeout_ms": { + "default": 1000, + "description": "Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with `truncated: true`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `elements_complete: false`; retry with a larger value (e.g. 5000) or narrow with `query` / `max_depth`.", + "maximum": 120000, + "minimum": 100, + "type": "integer" + }, + "tree_format": { + "description": "Which tree representation to return. Default \"markdown\": the compact `tree_markdown` only, where a row `[N]` is addressed with element_token `:N`. \"elements\": the structured `elements` array only (per-element element_token, frame, parent_index, value). \"both\": the two together, as before; roughly doubles the size.", + "enum": [ + "markdown", + "elements", + "both" + ], + "type": "string" + }, + "verbose": { + "description": "Default false. Set true to include the `_note` string and the full `background_input` capability report (macOS), which are omitted by default to save context.", + "type": "boolean" + }, + "window_id": { + "description": "Native window identifier from list_windows, or the `popup.window_id` a click / right_click result named (an open context menu / popover; its menu items then get element indices). Omitted: the pid's open popup menu when one is mapped, else its focused / active / largest window.", + "type": "integer" + } + }, + "required": [ + "pid" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "app_name": { + "type": [ + "string", + "null" + ] + }, + "degraded": { + "type": [ + "boolean", + "null" + ] + }, + "degraded_reason": { + "type": [ + "string", + "null" + ] + }, + "element_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "elements": { + "items": { + "properties": { + "actions": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ] + }, + "depth": { + "minimum": 0, + "type": "integer" + }, + "element_index": { + "minimum": 0, + "type": "integer" + }, + "element_token": { + "type": [ + "string", + "null" + ] + }, + "enabled": { + "type": [ + "boolean", + "null" + ] + }, + "frame": { + "properties": { + "h": { + "type": "number" + }, + "w": { + "type": "number" + }, + "x": { + "type": "number" + }, + "y": { + "type": "number" + } + }, + "required": [ + "x", + "y", + "w", + "h" + ], + "type": [ + "object", + "null" + ] + }, + "in_web_content": { + "type": [ + "boolean", + "null" + ] + }, + "label": { + "type": [ + "string", + "null" + ] + }, + "max": { + "type": [ + "number", + "null" + ] + }, + "min": { + "type": [ + "number", + "null" + ] + }, + "parent_index": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "role": { + "type": "string" + }, + "selected": { + "type": [ + "boolean", + "null" + ] + }, + "value": { + "type": [ + "string", + "null" + ] + }, + "value_description": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "element_index", + "role", + "depth" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "elements_complete": { + "type": [ + "boolean", + "null" + ] + }, + "filtered_element_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "pid": { + "minimum": 0, + "type": "integer" + }, + "returned_element_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "screenshot_file_path": { + "type": [ + "string", + "null" + ] + }, + "screenshot_frame_valid": { + "type": [ + "boolean", + "null" + ] + }, + "screenshot_height": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "screenshot_mime_type": { + "type": [ + "string", + "null" + ] + }, + "screenshot_scale": { + "type": [ + "number", + "null" + ] + }, + "screenshot_width": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "since": { + "type": [ + "string", + "null" + ] + }, + "since_status": { + "type": [ + "string", + "null" + ] + }, + "snapshot_id": { + "type": [ + "string", + "null" + ] + }, + "total_element_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "tree_diff": { + "type": [ + "string", + "null" + ] + }, + "tree_format": { + "type": [ + "string", + "null" + ] + }, + "tree_markdown": { + "type": [ + "string", + "null" + ] + }, + "truncated": { + "type": [ + "boolean", + "null" + ] + }, + "truncation_hint": { + "type": [ + "string", + "null" + ] + }, + "truncation_reason": { + "type": [ + "string", + "null" + ] + }, + "window_bounds": { + "properties": { + "height": { + "type": "number" + }, + "width": { + "type": "number" + }, + "x": { + "type": "number" + }, + "y": { + "type": "number" + } + }, + "required": [ + "x", + "y", + "width", + "height" + ], + "type": [ + "object", + "null" + ] + }, + "window_id": { + "minimum": 0, + "type": "integer" + }, + "window_title": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "pid", + "window_id" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "verify_state", + "description": "Deterministically verify bounded predicates against one exact window. The driver evaluates structured window/accessibility state and may return the final screenshot as uninterpreted visual evidence for a multimodal caller. Predicate results are satisfied, unsatisfied, or unknown; unknown never implies success. Accessibility projections are conservative: absence remains unknown unless the observed search domain is proven exhaustive.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "expect": { + "description": "One to eight predicates, combined with logical AND.", + "items": { + "additionalProperties": false, + "properties": { + "element": { + "additionalProperties": false, + "properties": { + "enabled": { + "type": "boolean" + }, + "exists": { + "description": "Assert that at least one trusted element matches the selector.\n\nElement walks are not yet exhaustive on every platform, so absence\ncannot be proven. `false` is rejected instead of returning an\nindefinitely-unknown predicate.", + "type": "boolean" + }, + "selected": { + "type": "boolean" + }, + "selector": { + "additionalProperties": false, + "properties": { + "label_contains": { + "minLength": 1, + "type": "string" + }, + "role": { + "minLength": 1, + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "value_equals": { + "type": "string" + } + }, + "required": [ + "selector" + ], + "type": "object" + }, + "window": { + "additionalProperties": false, + "properties": { + "bounds": { + "additionalProperties": false, + "properties": { + "height": { + "type": "number" + }, + "tolerance_px": { + "maximum": 100, + "minimum": 0, + "type": "number" + }, + "width": { + "type": "number" + }, + "x": { + "type": "number" + }, + "y": { + "type": "number" + } + }, + "required": [ + "x", + "y", + "width", + "height" + ], + "type": "object" + }, + "exists": { + "type": "boolean" + } + }, + "required": [], + "type": "object" + } + }, + "required": [], + "type": "object" + }, + "maxItems": 8, + "minItems": 1, + "type": "array" + }, + "include_screenshot": { + "description": "Return the final window screenshot as image content for a multimodal\ncaller. The driver does not interpret that image.", + "type": "boolean" + }, + "pid": { + "description": "Exact process whose window may be observed.", + "minimum": 1, + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session. This\nfield never selects capture modality or authorization.", + "type": "string" + }, + "stable_samples": { + "default": 2, + "description": "Consecutive satisfied samples required before returning success.", + "maximum": 5, + "minimum": 1, + "type": "integer" + }, + "timeout_ms": { + "default": 5000, + "description": "Bounded wait. Zero performs one sample.", + "maximum": 10000, + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Exact native window identifier.", + "type": "integer" + } + }, + "required": [ + "pid", + "window_id", + "expect" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "elapsed_ms": { + "minimum": 0, + "type": "integer" + }, + "predicates": { + "items": { + "properties": { + "index": { + "minimum": 0, + "type": "integer" + }, + "observed_json": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "status": { + "enum": [ + "satisfied", + "unsatisfied", + "unknown" + ], + "type": "string" + }, + "unknown_reason": { + "anyOf": [ + { + "enum": [ + "invalid_predicate", + "unsupported_predicate", + "untrusted_source", + "multi_match", + "target_missing", + "observation_unavailable", + "stability_unproven" + ], + "type": "string" + }, + { + "type": "null" + } + ] + } + }, + "required": [ + "index", + "status", + "unknown_reason", + "observed_json" + ], + "type": "object" + }, + "type": "array" + }, + "samples": { + "minimum": 0, + "type": "integer" + }, + "stable": { + "type": "boolean" + }, + "status": { + "enum": [ + "satisfied", + "unsatisfied", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "status", + "stable", + "elapsed_ms", + "samples", + "predicates" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "launch_app", + "description": "Launch a Linux app in the background. Provide launch_path (preferred — round-trip the value from list_apps), name (tried as a direct command, then matched against installed .desktop applications, then handed to xdg-open if it is a URL or existing file path), bundle_id (ignored on Linux), or urls (list of URLs to open). Resolution precedence: launch_path > name > bundle_id. Errors when the name resolves to nothing launchable.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "additional_arguments": { + "description": "Extra command-line arguments passed to the launched process.", + "items": { + "type": "string" + }, + "type": "array" + }, + "bundle_id": { + "description": "Ignored on Linux (macOS/Windows concept).", + "type": "string" + }, + "launch_path": { + "description": "Round-trip the `launch_path` returned by `list_apps` — the Exec= command from the .desktop file with XDG field codes already stripped. Highest precedence on Linux; spawned directly via the system shell.", + "type": "string" + }, + "name": { + "description": "App name or command to launch. Tried as a direct command first, then matched against installed .desktop applications (exact display name, desktop-file id, or Exec basename; else an unambiguous display-name substring).", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "urls": { + "description": "URLs to open via xdg-open.", + "items": { + "type": "string" + }, + "type": "array" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "kill_app", + "description": "Force-terminate a process by pid (kill -9 equivalent on Linux). Use as escalation when the cooperative close path failed to make the process exit. Unsaved state is lost — prefer the cooperative path first.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "pid": { + "description": "PID of the process to terminate.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "required": [ + "pid" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + } + }, + { + "name": "bring_to_front", + "description": "Persistently activate a window so subsequent input lands on it. This deliberately breaks the no-foreground contract and is not part of the normal input ladder. For an ordinary `background_unavailable` response, retry only the refused action with `delivery_mode:\"foreground\"`; the input tool performs its own activate, act, and restore sequence. Use `bring_to_front` only for a focus-proxy surface that must remain foreground across multiple calls, such as a remote desktop session, or when repeated action-scoped activation prevents the remote surface from accepting input. X11: EWMH _NET_ACTIVE_WINDOW activation (the `wmctrl -a` equivalent, with proper timestamp handling to beat focus-stealing prevention). Wayland: activates through a target-addressable compositor adapter (wlroots foreign-toplevel or the GNOME Shell helper) and refuses when the compositor offers no safe adapter. Matches the macOS / Windows bring_to_front rung.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "pid": { + "description": "Process ID of the app to activate.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "window_id": { + "description": "X11 window id (xid) to activate. If omitted, the first window of `pid` is used.", + "type": "integer" + } + }, + "required": [ + "pid" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + } + }, + { + "name": "set_window_frame", + "description": "Set one exact top-level window's frame in the desktop-coordinate space reported by list_windows and verify the resulting geometry through an independent readback.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "height": { + "description": "New height, in the same units as list_windows bounds.", + "minimum": 1, + "type": "number" + }, + "pid": { + "description": "Process ID that owns the window.", + "minimum": 1, + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "width": { + "description": "New width, in the same units as list_windows bounds.", + "minimum": 1, + "type": "number" + }, + "window_id": { + "description": "Window ID from list_windows.", + "minimum": 1, + "type": "integer" + }, + "x": { + "description": "New left edge in the desktop coordinate space reported by list_windows.", + "type": "number" + }, + "y": { + "description": "New top edge in the desktop coordinate space reported by list_windows.", + "type": "number" + } + }, + "required": [ + "pid", + "window_id", + "x", + "y", + "width", + "height" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "invoke_menu", + "description": "Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels. Menu paths and refusals: `get_guide({topic:\"menus\"})`.", + "inputSchema": { + "additionalProperties": false, + "description": "Exact, immediate-child application menu path to resolve and invoke through\nthe operating system's accessibility API. Path labels are matched after\ntrimming surrounding whitespace and otherwise remain case-sensitive. On\nmacOS, three periods in a label also match the ellipsis character that\nnative menu titles use (`Save As...` finds `Save As…`).", + "properties": { + "path": { + "description": "Menu labels from the top-level menu to the item, e.g. `[\"File\", \"Save As...\"]`\n(1 to 16 labels).", + "items": { + "maxLength": 200, + "minLength": 1, + "type": "string" + }, + "maxItems": 16, + "minItems": 1, + "type": "array" + }, + "pid": { + "description": "Process ID of the application that owns the menu.", + "minimum": 1, + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "window_id": { + "description": "Window ID from list_windows whose menu is invoked.", + "minimum": 1, + "type": "integer" + } + }, + "required": [ + "pid", + "window_id", + "path" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "click", + "description": "Click against a target pid. **Prefer `element_token` over pixel coordinates** — element_token works on backgrounded / hidden windows, surfaces a stable handle, and tells you what you're clicking via the cached AT-SPI element's role + label. Reach for `x, y` only when the target is a canvas / custom-drawn surface that doesn't appear in the AT-SPI tree.\n\nProvide either (window_id + x/y) or (pid + element_token). x/y are WINDOW-LOCAL pixels of window_id (the get_window_state screenshot frame); a point outside the window is refused (point_outside_window) and every pixel result reports the resolved window_point / screen_point. Background delivery first resolves the accessible under the point and fires its action (the result says `hit: \"\" action=...`; a file / list item is selected through its container: `selected: ...`), else presses the real virtual pointer there. A click that opens a popup menu names it (`popup: {window_id, bounds}`): pass that window_id to get_window_state to index its items. The token is stale once the next get_window_state of the same window replaces its snapshot — re-snapshot every turn before clicking.\n\nAfter a zoom call, pass from_zoom=true to auto-translate zoom-image coords back to full-window space.\n\n`count`: 1 (default), 2 for a double-click, 3 for a triple-click (selects a line / paragraph in editors) — one press train with real double-click cadence, in background and foreground alike; there is no separate triple_click tool.\n\nbutton: \"left\" (default), \"right\", or \"middle\". Defaults to left so the field is fully back-compat. X11: routes through XSendEvent ButtonPress/Release with the matching button code. Native Wayland: only left-button is supported via the virtual-pointer protocol — right/middle return an error rather than silently degrading to left. `modifier` holds ctrl/shift/alt/super for the click on X11. Native Wayland refuses modified pointer clicks until its input protocol can carry keyboard modifier state.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "button": { + "description": "Mouse button. Default: \"left\" (legacy back-compat). X11: routed via ButtonPress/Release with the matching evdev code. Native Wayland: only left-button is supported via the virtual-pointer protocol; right/middle return an error.", + "enum": [ + "left", + "right", + "middle" + ], + "type": "string" + }, + "capture_id": { + "description": "Optional one-shot binding to the exact PNG returned by get_window_state or get_desktop_state. When present, x/y are interpreted in that capture and admitted before native dispatch.", + "type": "string" + }, + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "count": { + "description": "Number of clicks in one press train: 1 (default), 2 = double-click, 3 = triple-click (line/paragraph selection). There is no separate triple_click tool — pass count: 3 here.", + "maximum": 3, + "minimum": 1, + "type": "integer" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "modifier": { + "description": "Modifier keys held during the action: cmd, shift, option/alt, ctrl.", + "items": { + "type": "string" + }, + "type": "array" + }, + "pid": { + "description": "Target process ID. Required unless scope is \"desktop\" or element_token is supplied (the token carries the pid).", + "type": "integer" + }, + "scope": { + "default": "window", + "description": "Coordinate frame (default \"window\"). Pass \"desktop\" with x,y and no pid/window_id for a screen-absolute click in get_desktop_state coordinates.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target": { + "additionalProperties": true, + "description": "Exact window (kind \"window\", with pid and window_id) or desktop (kind \"desktop\", with display_id). \"primary\" is the portable desktop target.", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.", + "type": "number" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "double_click", + "description": "Double-click at (x,y) or an element_token (AT-SPI bounds). Background delivery on X11 sends real button events from the session's virtual master pointer (path mpx_pointer); a point covered by another application's window is refused (target_occluded), one covered by this app's own dialog is retargeted to it (retargeted_to). No focus steal. Provide either (window_id + x/y) or (pid + element_token). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "pid": { + "description": "Target process ID. Optional with element_token (the token carries the pid).", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.", + "type": "number" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "right_click", + "description": "Right-click at (x,y) or an element_token (AT-SPI bounds). Background delivery on X11 sends real button events from the session's virtual master pointer (path mpx_pointer); a point covered by another application's window is refused (target_occluded), one covered by this app's own dialog is retargeted to it (retargeted_to), one outside the window is refused (point_outside_window; x/y are window-local pixels, the result reports window_point / screen_point). No focus steal. When a context menu opens, the result names it (`popup: {window_id, bounds, title}`): call get_window_state(pid, window_id=) to index its menu items and click them by element_token. Provide either (window_id + x/y) or (pid + element_token). After a zoom call, pass from_zoom=true to auto-translate zoom-image coords.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "modifier": { + "description": "Modifier keys held during the action: cmd, shift, option/alt, ctrl.", + "items": { + "type": "string" + }, + "type": "array" + }, + "pid": { + "description": "Target process ID. Optional with element_token (the token carries the pid).", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.", + "type": "number" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "drag", + "description": "Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels. Background delivery on X11 is one real held gesture on the session's virtual master pointer (path mpx_pointer: press, interpolated motion, release) — it lands on GTK/VCL/Qt/Chromium; a point covered by another application's window is refused (target_occluded), one covered by this app's own dialog is retargeted to it. Headless servers fall back to XSendEvent. duration_ms (default 500), steps (default 20). No focus steal. Limits and fallbacks: `get_guide({topic:\"drag\"})`.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "button": { + "description": "Mouse button. Default \"left\".", + "enum": [ + "left", + "right", + "middle" + ], + "type": "string" + }, + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "duration_ms": { + "description": "Total drag duration. Default: 500.", + "maximum": 10000, + "minimum": 0, + "type": "integer" + }, + "from_x": { + "description": "Drag start X: window-local pixels of the target window's own get_window_state screenshot; pass scope:\"desktop\" for get_desktop_state pixels.", + "type": "number" + }, + "from_y": { + "description": "Drag start Y (same frame as from_x).", + "type": "number" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "modifier": { + "description": "Modifier keys held during the action: cmd, shift, option/alt, ctrl.", + "items": { + "type": "string" + }, + "type": "array" + }, + "pid": { + "description": "Target process ID. Required unless scope is \"desktop\".", + "type": "integer" + }, + "scope": { + "default": "window", + "description": "Use \"desktop\" with no pid/window_id for get_desktop_state screen coordinates. Default \"window\".", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "steps": { + "description": "Intermediate MotionNotify events. Default: 20.", + "maximum": 200, + "minimum": 1, + "type": "integer" + }, + "target": { + "additionalProperties": true, + "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`).", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "to_x": { + "description": "Drag end X (same frame as from_x).", + "type": "number" + }, + "to_y": { + "description": "Drag end Y (same frame as from_x).", + "type": "number" + }, + "window_id": { + "description": "Target window XID. Required.", + "type": "integer" + } + }, + "required": [ + "from_x", + "from_y", + "to_x", + "to_y" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "mouse_button_down", + "description": "Press and hold a mouse button at (x,y) via background X11 delivery. Does not release the button; pair with mouse_drag / mouse_button_up. Returns the current held-button state.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "button": { + "description": "Mouse button. Default \"left\".", + "enum": [ + "left", + "right", + "middle" + ], + "type": "string" + }, + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "pid": { + "description": "Target process ID.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.", + "type": "string" + }, + "window_id": { + "description": "Window id from list_windows.", + "type": "integer" + }, + "x": { + "description": "Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.", + "type": "number" + } + }, + "required": [ + "pid", + "window_id", + "x", + "y" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "mouse_drag", + "description": "Move a previously-held mouse button to a new point via background X11 delivery. Requires an active mouse_button_down state; does not release the button. Returns the updated held-button state.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "duration_ms": { + "description": "Total drag duration. Default: 500.", + "maximum": 10000, + "minimum": 0, + "type": "integer" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "pid": { + "description": "Optional process ID; must match the held button's target.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.", + "type": "string" + }, + "steps": { + "description": "Intermediate MotionNotify events. Default: 20.", + "maximum": 200, + "minimum": 1, + "type": "integer" + }, + "window_id": { + "description": "Optional window id; must match the held button's target.", + "type": "integer" + }, + "x": { + "description": "Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.", + "type": "number" + } + }, + "required": [ + "x", + "y" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "mouse_button_up", + "description": "Release a previously-held mouse button via background X11 delivery. If x/y are omitted, releases at the last held position. Returns the current held-button state.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "from_zoom": { + "description": "Set true after a zoom call to auto-translate zoom-image pixel coordinates back to full-window space.", + "type": "boolean" + }, + "pid": { + "description": "Optional process ID; must match the held button's target.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. When both are present, session takes precedence over cursor_id.", + "type": "string" + }, + "window_id": { + "description": "Optional window id; must match the held button's target.", + "type": "integer" + }, + "x": { + "description": "Window-local pixel X of the target window's own get_window_state screenshot (0..screenshot_width). For get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Window-local pixel Y of the target window's own get_window_state screenshot (0..screenshot_height); see x.", + "type": "number" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "parallel_mouse_drag", + "description": "Run multiple mouse drag gestures concurrently via Linux MPX/XI2 virtual master pointers. Each drag item runs on its own session-scoped master pointer (true same-window concurrent draws on X11). Each item presses once, glides continuously through its whole path, and releases once — one smooth held drag, not a chain of clicks. A path is given either as a straight segment (from_x/from_y → to_x/to_y) or as a function `fn` = y(x) sampled over [x_from, x_to] in window-local pixels (e.g. fn:\"x\" is a diagonal, fn:\"300+120*sin(x/40)\" a sine wave). Functions support + - * / ^, sin/cos/tan, sqrt, abs, exp, ln, pi, e.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "drags": { + "description": "Two or more drag gestures to run concurrently, each on its own virtual master pointer.", + "items": { + "additionalProperties": false, + "properties": { + "button": { + "description": "Mouse button. Default \"left\".", + "enum": [ + "left", + "right", + "middle" + ], + "type": "string" + }, + "duration_ms": { + "description": "Default: 1500 for fn paths, 500 for straight.", + "maximum": 10000, + "minimum": 0, + "type": "integer" + }, + "fn": { + "description": "Expression y(x) in window-local pixels; sampled over [x_from,x_to]. Mutually exclusive with from_x/to_x.", + "type": "string" + }, + "from_x": { + "description": "Straight-segment start X (window-local pixels).", + "type": "number" + }, + "from_y": { + "description": "Straight-segment start Y (window-local pixels).", + "type": "number" + }, + "path": { + "description": "Explicit window-local waypoints [[x,y],...] (>=2); pressed once, glided through, released once. Takes precedence over fn/from-to.", + "items": { + "items": { + "type": "number" + }, + "type": "array" + }, + "type": "array" + }, + "samples": { + "description": "Waypoints sampled along `fn`. Default: 80.", + "maximum": 400, + "minimum": 2, + "type": "integer" + }, + "session": { + "description": "Session/cursor id; also keys the virtual master pointer.", + "type": "string" + }, + "steps": { + "description": "Motion sub-steps along the whole path. Default: scaled to path length.", + "maximum": 300, + "minimum": 1, + "type": "integer" + }, + "to_x": { + "description": "Straight-segment end X (window-local pixels).", + "type": "number" + }, + "to_y": { + "description": "Straight-segment end Y (window-local pixels).", + "type": "number" + }, + "window_id": { + "description": "Window id from list_windows.", + "type": "integer" + }, + "x_from": { + "description": "Domain start (window-local x) when `fn` is used.", + "type": "number" + }, + "x_to": { + "description": "Domain end (window-local x) when `fn` is used.", + "type": "number" + } + }, + "required": [ + "session", + "window_id" + ], + "type": "object" + }, + "minItems": 2, + "type": "array" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "required": [ + "drags" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "type_text", + "description": "Type text to a window via XSendEvent (KeyPress/KeyRelease). No focus steal.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "delay_ms": { + "description": "Milliseconds between characters. Default 0 sends batched key events (up to 16 per batch; Return and Tab stay real key presses). A positive value sends one key per character with this pause, for debounced search boxes and autocomplete.", + "maximum": 200, + "minimum": 0, + "type": "integer" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "pid": { + "description": "Target process ID. Omit with scope \"desktop\" to type into the focused application.", + "type": "integer" + }, + "scope": { + "default": "window", + "description": "Use \"desktop\" with no pid to type into the focused application. Default \"window\".", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target": { + "additionalProperties": true, + "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`).", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "text": { + "description": "Text to type.", + "type": "string" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Pixel X of the field to type into — the element px action form. Pass x,y (no element_token) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Window-local pixels of the target window's own get_window_state screenshot by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\") — a bare pid+x/y is otherwise misinterpreted as window-local and can focus the wrong widget.", + "type": "number" + }, + "y": { + "description": "Pixel Y of the field (see x).", + "type": "number" + } + }, + "required": [ + "text" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "press_key", + "description": "Press a key via XSendEvent to a window. No focus steal.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "key": { + "description": "Key name: enter/return, tab, escape, space, backspace, delete, insert, home, end, pageup, pagedown, up, down, left, right, f1-f12, or any single ASCII character.", + "type": "string" + }, + "modifiers": { + "description": "Modifier keys held while the key is pressed: ctrl, shift, alt, super.", + "items": { + "type": "string" + }, + "type": "array" + }, + "pid": { + "description": "Target process ID. Omit with scope \"desktop\" to send the key to the focused application.", + "type": "integer" + }, + "scope": { + "default": "window", + "description": "Use \"desktop\" with no pid to send the key to the focused application. Default \"window\".", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target": { + "additionalProperties": true, + "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`).", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_token. Window-local pixels by default (same convention as click); for get_desktop_state pixels pass scope:\"desktop\" (or coordinate_frame:\"desktop\").", + "type": "number" + }, + "y": { + "description": "Pixel Y (see x).", + "type": "number" + } + }, + "required": [ + "key" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "hotkey", + "description": "Press a combination of keys simultaneously, e.g. [\"ctrl\",\"c\"] for Copy. Delivered to the target window without focus steal (real key events from a virtual master keyboard on X11); target does NOT need to be frontmost. Window-manager chords cannot reach the WM that way: alt+F4 on a window target closes that window through the WM (_NET_CLOSE_WINDOW, effect confirmed when it unmaps); super+*, alt+tab, ctrl+alt+* and other WM bindings are refused with code wm_chord_unavailable — use bring_to_front / list_windows / kill_app or delivery_mode:\"foreground\" instead.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "keys": { + "description": "Modifier(s) + one non-modifier key, e.g. [\"ctrl\",\"c\"].", + "items": { + "type": "string" + }, + "minItems": 2, + "type": "array" + }, + "pid": { + "description": "Target process ID. Omit with scope \"desktop\" to send the chord to the focused application.", + "type": "integer" + }, + "scope": { + "default": "window", + "description": "Use \"desktop\" with no pid to send the chord to the focused application. Default \"window\".", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target": { + "additionalProperties": true, + "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`).", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y or element_index; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Ctrl+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach.", + "type": "number" + }, + "y": { + "description": "Screenshot-pixel Y (see x).", + "type": "number" + } + }, + "required": [ + "keys" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "set_value", + "description": "Set value of an AT-SPI element via SetValue action.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "pid": { + "description": "Target process ID. Optional with element_token (the token carries the pid).", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "value": { + "description": "New value. Written through AT-SPI EditableText/Value when the element exposes them; otherwise (GTK2 spin scales, VCL spin buttons) the field is clicked with the session's real pointer, its text selected and replaced through the virtual keyboard and committed with Tab, then read back.", + "type": "string" + }, + "window_id": { + "description": "Omit when element_token is supplied (the token carries it).", + "type": "integer" + } + }, + "required": [ + "value" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "scroll", + "description": "Scroll the target pid's focused region via XSendEvent Button4/5. direction required; by defaults to line, amount defaults to 3.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "amount": { + "description": "Number of scroll steps. Default: 3.", + "maximum": 50, + "minimum": 1, + "type": "integer" + }, + "by": { + "description": "Scroll granularity. Default: line.", + "enum": [ + "line", + "page" + ], + "type": "string" + }, + "coordinate_frame": { + "description": "Frame of x/y (and from_x/from_y/to_x/to_y). Default \"window\": window-local screenshot pixels as returned by get_window_state. \"desktop\": full-screen pixels as returned by get_desktop_state, translated to the target window. Passing scope:\"desktop\" together with pid/window_id means the same thing.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "cursor_id": { + "description": "Optional multi-cursor instance id. Default: 'default'.", + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "Input delivery mode. 'background' (default) never activates or raises the target window. On X11 it injects via XTEST / the XInput2 master pointer (no focus steal). On Wayland it goes through libei + xdg-desktop-portal, which injects to the compositor's input focus — Wayland's security model has no per-window background targeting, so a specific non-focused window cannot be aimed at; when no libei backend is available the tool returns a structured background_unavailable error. 'foreground' is the explicit escalation: activate the target (X11 _NET_ACTIVE_WINDOW; Wayland compositor activate), inject, then restore the prior active window — a brief focus swap unless the target was already active. Matches the macOS / Windows delivery_mode surface.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "direction": { + "description": "Scroll direction.", + "enum": [ + "up", + "down", + "left", + "right" + ], + "type": "string" + }, + "element_token": { + "description": "Element handle `:` from get_window_state: row `[11]` of a read whose snapshot_id is s0000002a is \"s0000002a:11\" (also in `structuredContent.elements[].element_token`). Returns an explicit stale error naming the current snapshots once a newer read supersedes it.", + "pattern": "^s[0-9a-f]{8}:[0-9]+$", + "type": "string" + }, + "pid": { + "description": "Target process ID. Required unless scope is \"desktop\" or element_token is supplied (the token carries the pid).", + "type": "integer" + }, + "scope": { + "default": "window", + "description": "Use \"desktop\" with x,y and no pid/window_id for get_desktop_state screen coordinates. Default \"window\".", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target": { + "additionalProperties": true, + "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`).", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "window_id": { + "description": "Window id from list_windows. Required with x/y; optional with element_token (the token carries it).", + "type": "integer" + }, + "x": { + "description": "Window-local screenshot-pixel X of the scroll target. Pass with y and without element_token.", + "type": "number" + }, + "y": { + "description": "Window-local screenshot-pixel Y of the scroll target. Pass with x and without element_token.", + "type": "number" + } + }, + "required": [ + "direction" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "clipboard_read", + "description": "List available system clipboard types and optionally return privacy-sensitive plain text. Clipboard content is never retained in telemetry.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "include_text": { + "default": false, + "description": "Return plain-text clipboard content in addition to the available types.\nClipboard content is privacy-sensitive and is never retained in telemetry.", + "type": "boolean" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "content_redacted_from_telemetry": { + "type": "boolean" + }, + "privacy_sensitive": { + "type": "boolean" + }, + "supported": { + "type": "boolean" + }, + "text": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "types": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "supported", + "types", + "text", + "privacy_sensitive", + "content_redacted_from_telemetry" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "clipboard_write", + "description": "Replace the system clipboard with exactly one value: plain text, an image from an absolute local path, or a file URL from an absolute local path. Returns the available types for read-back before paste.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "file_path": { + "description": "Absolute path to a local file to place on the clipboard as a file URL.", + "type": "string" + }, + "image_path": { + "description": "Absolute path to a local image to place on the clipboard.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that\naccepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "text": { + "description": "Plain text to place on the clipboard.", + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "content_redacted_from_telemetry": { + "type": "boolean" + }, + "privacy_sensitive": { + "type": "boolean" + }, + "supported": { + "type": "boolean" + }, + "types": { + "items": { + "type": "string" + }, + "type": "array" + }, + "written_type": { + "type": "string" + } + }, + "required": [ + "supported", + "written_type", + "types", + "privacy_sensitive", + "content_redacted_from_telemetry" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_screen_size", + "description": "Return the logical size of the main display in points plus its backing scale factor. Agents click in points; Retina displays have scale_factor 2.0. Requires no TCC permissions.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "height": { + "type": "number" + }, + "scale_factor": { + "type": "number" + }, + "width": { + "type": "number" + } + }, + "required": [ + "width", + "height", + "scale_factor" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_desktop_state", + "description": "Capture the full display in the desktop action coordinate frame. Use the returned PNG directly as the coordinate source for actions whose target is {kind:\"desktop\",display_id:\"primary\"}. No AT-SPI walk.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "max_image_dimension": { + "description": "Optional long-edge cap for the returned PNG, in pixels (aspect ratio preserved). Omitted or 0 returns the full-size capture. When the cap downsizes the image, the response reports `screenshot_original_width/height`, and x/y read off it for this session's later scope:\"desktop\" actions (or passed with its `capture_id`) are mapped back to the full-size frame automatically.", + "minimum": 0, + "type": "integer" + }, + "screenshot_out_file": { + "description": "Write PNG here instead of base64.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "agent_overlay_capture": { + "additionalProperties": false, + "properties": { + "method": { + "type": [ + "string", + "null" + ] + }, + "reason": { + "type": [ + "string", + "null" + ] + }, + "status": { + "oneOf": [ + { + "const": "excluded", + "type": "string" + }, + { + "const": "not_present", + "type": "string" + }, + { + "const": "not_excluded", + "type": "string" + } + ] + } + }, + "required": [ + "status" + ], + "type": [ + "object", + "null" + ] + }, + "display": { + "type": "string" + }, + "platform": { + "enum": [ + "macos", + "linux", + "windows" + ], + "type": "string" + }, + "scale_factor": { + "type": "number" + }, + "screen_height": { + "type": "integer" + }, + "screen_width": { + "type": "integer" + }, + "screenshot_file_path": { + "type": "string" + }, + "screenshot_height": { + "type": "integer" + }, + "screenshot_mime_type": { + "const": "image/png" + }, + "screenshot_width": { + "type": "integer" + } + }, + "required": [ + "platform", + "display", + "screenshot_width", + "screenshot_height", + "screen_width", + "screen_height", + "scale_factor", + "screenshot_mime_type" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_cursor_position", + "description": "Return the current mouse cursor position in screen points (origin top-left).", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "available": { + "type": "boolean" + }, + "source": { + "type": "string" + }, + "x": { + "type": "number" + }, + "y": { + "type": "number" + } + }, + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "move_cursor", + "description": "Move the synthetic agent cursor without changing the user's pointer. Only an explicit scope=desktop request moves the real OS pointer in get_desktop_state coordinates.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "cursor_id": { + "description": "Cursor instance to move. Default: 'default'.", + "type": "string" + }, + "scope": { + "default": "window", + "description": "\"window\" (default) moves only the agent cursor overlay; \"desktop\" moves the real OS pointer.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target": { + "additionalProperties": true, + "description": "Preferred per-call target: an exact window (`kind=\"window\"`, `pid`, `window_id`) or the\nprimary desktop (`kind=\"desktop\"`, `display_id=\"primary\"`).", + "properties": { + "display_id": { + "description": "Display id. Required when kind is desktop; use \"primary\".", + "type": "string" + }, + "kind": { + "description": "window requires pid and window_id; desktop requires display_id.", + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "pid": { + "description": "Process id. Required when kind is window.", + "minimum": 0, + "type": "integer" + }, + "window_id": { + "description": "Window id from list_windows. Required when kind is window.", + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "x": { + "description": "Destination X. Window scope: screen coordinates of the agent cursor overlay. Desktop scope: get_desktop_state screenshot pixels.", + "type": "number" + }, + "y": { + "description": "Destination Y, in the same space as x.", + "type": "number" + } + }, + "required": [ + "x", + "y" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "set_agent_cursor_enabled", + "description": "Show or hide the agent cursor owned by a session.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "enabled": { + "description": "`true` shows the session's agent cursor overlay; `false` hides it.", + "type": "boolean" + }, + "session": { + "description": "Public label of the session that owns the cursor.", + "type": "string" + } + }, + "required": [ + "session", + "enabled" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "enabled": { + "type": "boolean" + }, + "session": { + "type": "string" + } + }, + "required": [ + "session", + "enabled" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "set_agent_cursor_motion", + "description": "Configure the movement style, timing, effects and visibility timing for a session cursor.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "arc_flow": { + "description": "Added to the arc asymmetry of `signature_arc`, `spring_settle` and `comet_swoop`:\npositive moves the apex toward the destination. Clamped to -1..1 (default 0).", + "type": "number" + }, + "arc_size": { + "description": "Scales the arc of `signature_arc`, `spring_settle` and `comet_swoop`: 0.25 (default)\nkeeps the style's arc, 0 is a straight line, 0.5 doubles it. Clamped to 0..1.", + "type": "number" + }, + "dwell_after_click_ms": { + "description": "Pause after a click animation, in milliseconds. Clamped to 0..5000 (default 80).", + "type": "number" + }, + "effects": { + "additionalProperties": false, + "description": "Turn effects on or off. Omit a field to keep it; `\"default\"` restores the style default.", + "properties": { + "glow": { + "description": "Soft glow around the cursor that grows with speed.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "magnet": { + "description": "Target glow when the `magnetic` style locks on.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "ripple": { + "description": "Ring that expands from the hotspot on click.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "squish": { + "description": "Brief scale-down of the cursor on click.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "trail": { + "description": "Short fading trail behind the cursor.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "end_handle": { + "description": "Arc control-point offset from the end, as a fraction of the distance, for\n`signature_arc`, `spring_settle` and `comet_swoop`. Clamped to 0..1 (default 0.3).", + "type": "number" + }, + "glide_duration_ms": { + "description": "Move duration in milliseconds for `fixed` timing (1430 ms when 0, the default). A\nnonzero value with `native` or the default `steady` timing also fixes the duration.\nClamped to 0..5000.", + "type": "number" + }, + "idle_hide_ms": { + "description": "Hide the cursor after this many idle milliseconds; 0 never hides it. Clamped to\n0..60000 (default 15000).", + "type": "number" + }, + "session": { + "description": "Public label of the session that owns the cursor. Omitted or null motion fields keep\ntheir current value.", + "type": "string" + }, + "spring": { + "description": "Arrival spring damping for `classic`: 1 is critically damped, 0.3 is bouncy. Clamped\nto 0.3..1 (default 0.72).", + "type": "number" + }, + "start_handle": { + "description": "Arc control-point offset from the start, as a fraction of the distance, for\n`signature_arc`, `spring_settle` and `comet_swoop`. Clamped to 0..1 (default 0.3).", + "type": "number" + }, + "style": { + "description": "Trajectory style. `signature_arc` (default) is one arc with a small follow-through;\n`spring_settle` lands with one soft bounce; `magnetic` is pulled into the target;\n`comet_swoop` is a wide arc with a short trail; `adaptive` picks a careful approach for\nsmall targets and a swoop for long moves; `classic` is the previous Dubins glide. When\nthe theme's reduced motion is on, every move is a short straight glide with no effects.", + "enum": [ + "signature_arc", + "spring_settle", + "magnetic", + "comet_swoop", + "adaptive", + "classic" + ], + "type": "string" + }, + "timing": { + "description": "`steady` (default) keeps the style's timing but makes every move take at least 600 ms,\nwith a gentler ease; `relaxed` and `calm` do the same with a 450 or 750 ms floor.\n`native` uses the style's own timing; `fitts` scales the move time with distance and\ntarget size; `deliberate` is a slower Fitts curve (450..1300 ms); `fixed` uses\nglide_duration_ms (1430 ms when 0).", + "enum": [ + "native", + "fitts", + "fixed", + "relaxed", + "steady", + "calm", + "deliberate" + ], + "type": "string" + }, + "turn_radius": { + "description": "Minimum turning radius of the `classic` glide path, in points; smaller turns tighter.\nClamped to 1..1000 (default 80).", + "type": "number" + } + }, + "required": [ + "session" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "motion": { + "properties": { + "arc_flow": { + "type": "number" + }, + "arc_size": { + "type": "number" + }, + "dwell_after_click_ms": { + "type": "number" + }, + "effects": { + "properties": { + "glow": { + "type": "boolean" + }, + "magnet": { + "type": "boolean" + }, + "ripple": { + "type": "boolean" + }, + "squish": { + "type": "boolean" + }, + "trail": { + "type": "boolean" + } + }, + "required": [ + "trail", + "glow", + "magnet", + "ripple", + "squish" + ], + "type": [ + "object", + "null" + ] + }, + "end_handle": { + "type": "number" + }, + "glide_duration_ms": { + "type": "number" + }, + "idle_hide_ms": { + "type": "number" + }, + "spring": { + "type": "number" + }, + "start_handle": { + "type": "number" + }, + "style": { + "enum": [ + "signature_arc", + "spring_settle", + "magnetic", + "comet_swoop", + "adaptive", + "classic" + ], + "type": "string" + }, + "timing": { + "enum": [ + "native", + "fitts", + "fixed", + "relaxed", + "steady", + "calm", + "deliberate" + ], + "type": "string" + }, + "turn_radius": { + "type": "number" + } + }, + "required": [ + "start_handle", + "end_handle", + "arc_size", + "arc_flow", + "spring", + "glide_duration_ms", + "dwell_after_click_ms", + "idle_hide_ms", + "turn_radius" + ], + "type": "object" + }, + "session": { + "type": "string" + } + }, + "required": [ + "session", + "motion" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_agent_cursor_state", + "description": "Return the session cursor's theme, semantic playback, position, visibility, and motion.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "Public label of the session whose cursor to inspect.", + "type": "string" + } + }, + "required": [ + "session" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "enabled": { + "type": "boolean" + }, + "motion": { + "properties": { + "arc_flow": { + "type": "number" + }, + "arc_size": { + "type": "number" + }, + "dwell_after_click_ms": { + "type": "number" + }, + "effects": { + "properties": { + "glow": { + "type": "boolean" + }, + "magnet": { + "type": "boolean" + }, + "ripple": { + "type": "boolean" + }, + "squish": { + "type": "boolean" + }, + "trail": { + "type": "boolean" + } + }, + "required": [ + "trail", + "glow", + "magnet", + "ripple", + "squish" + ], + "type": [ + "object", + "null" + ] + }, + "end_handle": { + "type": "number" + }, + "glide_duration_ms": { + "type": "number" + }, + "idle_hide_ms": { + "type": "number" + }, + "spring": { + "type": "number" + }, + "start_handle": { + "type": "number" + }, + "style": { + "enum": [ + "signature_arc", + "spring_settle", + "magnetic", + "comet_swoop", + "adaptive", + "classic" + ], + "type": "string" + }, + "timing": { + "enum": [ + "native", + "fitts", + "fixed", + "relaxed", + "steady", + "calm", + "deliberate" + ], + "type": "string" + }, + "turn_radius": { + "type": "number" + } + }, + "required": [ + "start_handle", + "end_handle", + "arc_size", + "arc_flow", + "spring", + "glide_duration_ms", + "dwell_after_click_ms", + "idle_hide_ms", + "turn_radius" + ], + "type": "object" + }, + "position": { + "anyOf": [ + { + "properties": { + "x": { + "type": "number" + }, + "y": { + "type": "number" + } + }, + "required": [ + "x", + "y" + ], + "type": "object" + }, + { + "type": "null" + } + ] + }, + "session": { + "type": "string" + }, + "theme": { + "properties": { + "fallback": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "id": { + "type": "string" + }, + "profile": { + "type": "string" + }, + "reduced_motion": { + "enum": [ + "auto", + "on", + "off" + ], + "type": "string" + }, + "version": { + "type": "string" + } + }, + "required": [ + "id", + "version", + "profile", + "reduced_motion", + "fallback" + ], + "type": "object" + }, + "visual_state": { + "properties": { + "frame": { + "minimum": 0, + "type": "integer" + }, + "modifiers": { + "items": { + "type": "string" + }, + "type": "array" + }, + "phase": { + "type": "string" + }, + "preempted_count": { + "minimum": 0, + "type": "integer" + }, + "requested_action": { + "enum": [ + "idle", + "observe", + "click", + "drag", + "scroll", + "text", + "key", + "navigate", + "app", + "transfer", + "record", + "system" + ], + "type": "string" + }, + "resolved_action": { + "enum": [ + "idle", + "observe", + "click", + "drag", + "scroll", + "text", + "key", + "navigate", + "app", + "transfer", + "record", + "system" + ], + "type": "string" + } + }, + "required": [ + "requested_action", + "resolved_action", + "modifiers", + "phase", + "frame", + "preempted_count" + ], + "type": "object" + } + }, + "required": [ + "session", + "enabled", + "position", + "theme", + "visual_state", + "motion" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "set_agent_cursor_theme", + "description": "Select an already-installed cursor theme for a session.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "reduced_motion": { + "default": "auto", + "description": "Theme animation policy: `on` uses the theme's reduced-motion frames, `off` always\nanimates, `auto` (default) leaves the choice to the host.", + "enum": [ + "auto", + "on", + "off" + ], + "type": "string" + }, + "session": { + "description": "Public label of the session that owns the cursor.", + "type": "string" + }, + "theme_id": { + "description": "Id of an installed cursor theme (see `cua-driver cursor-theme list`).", + "maxLength": 200, + "minLength": 1, + "type": "string" + } + }, + "required": [ + "session", + "theme_id" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "session": { + "type": "string" + }, + "theme": { + "properties": { + "fallback": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "id": { + "type": "string" + }, + "profile": { + "type": "string" + }, + "reduced_motion": { + "enum": [ + "auto", + "on", + "off" + ], + "type": "string" + }, + "version": { + "type": "string" + } + }, + "required": [ + "id", + "version", + "profile", + "reduced_motion", + "fallback" + ], + "type": "object" + } + }, + "required": [ + "session", + "theme" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "check_permissions", + "description": "Check required permissions for cua-driver-rs on Linux. How to get them granted: `get_guide({topic:\"permissions\"})`.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "prompt": { + "description": "Ignored: Linux has no permission prompts.", + "type": "boolean" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "health_report", + "description": "Single-call end-to-end driver diagnostics. Designed to let downstream consumers ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and platform capability status. On macOS, prompt-capable direct capture is deliberately skipped. For standalone CuaDriver, use `cua-driver permissions grant` to verify it explicitly. For an embedded or linked driver, use the host's permission flow for the app or executable macOS attributes this process's permission request to; the standalone command does not verify the host's capture readiness. cua-driver owns the health model; consumers stay thin.\n\nInput — all optional:\n {\n \"include\": [\"\", ...], // run only these\n \"skip\": [\"\", ...] // skip these\n }\nIf both are given, `include` wins.\n\nCanonical check names:\n macOS : binary_version, platform_supported, session_active,\n bundle_identity, tcc_accessibility, tcc_screen_recording,\n ax_capability, screen_capture_capability, windowserver_connection\n Windows: binary_version, platform_supported, session_active,\n ax_capability (via UIA), screen_capture_capability (via DXGI)\n Linux : binary_version, platform_supported, session_active,\n ax_capability (via AT-SPI), screen_capture_capability (via X11)\n\nOutput — stable contract, schema_version=\"1\":\n {\n \"schema_version\": \"1\",\n \"platform\": \"darwin\" | \"win32\" | \"linux\",\n \"driver_version\": \"\",\n \"overall\": \"ok\" | \"degraded\" | \"failed\",\n \"checks\": [\n {\n \"name\": \"\",\n \"status\": \"pass\" | \"fail\" | \"skip\",\n \"message\": \"\",\n \"hint\": \"\",\n \"data\": { /* check-specific structured fields */ }\n },\n ...\n ]\n }\n\n`overall` rules:\n - `ok` — every non-skipped check passes\n - `degraded` — at least one non-core check fails (binary is still usable)\n - `failed` — any core check fails (binary_version, platform_supported, session_active)\n\nStability: schema_version=\"1\" is the contract. Future breaking changes will be `\"2\"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "include": { + "description": "Only run these checks (canonical names). Wins over `skip`.", + "items": { + "type": "string" + }, + "type": "array" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "skip": { + "description": "Skip these checks (canonical names). Ignored when `include` is set.", + "items": { + "type": "string" + }, + "type": "array" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "get_config", + "description": "Return current cua-driver-rs configuration.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "set_config", + "description": "Update cua-driver-rs configuration. capture_mode / max_image_dimension take effect immediately.\n\nPass each setting as its own field: `{\"capture_mode\": \"som\", \"max_image_dimension\": 0}`.\n\nThe experimental_pip keys persist to ~/.cua-driver/config.json and apply on next daemon restart (the PiP backend is initialised once at startup; Linux ships only the trait stub today — see issue #1729).\n\n`cursor.motion.style`, `cursor.motion.timing` and `cursor.motion.effects.` (`on`, `off`, or `default`) save the default cursor motion for sessions started afterwards. `cursor.motion: \"default\"` clears every saved cursor motion default.\n\n`cursor.badges` (`minimal` default, `verbose`, or `off`) sets the agent cursor badge for every cursor at once.\n\n`cursor.float` (float at rest and while moving) and `cursor.halo` (soft glow) are `on` (default), `off` or `default`; `cursor.marks` (action marks) is `on`, `off` (default) or `default`; `cursor.rotate` is `heading` (default), `lean`, `off` or `default`. All apply to every cursor at once.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "capture_mode": { + "description": "Default capture mode for get_window_state. (\"som\"/\"screenshot\" still decode as deprecated aliases.)", + "enum": [ + "ax", + "vision" + ], + "type": "string" + }, + "cursor.badges": { + "description": "Agent cursor badge. minimal (default): no badge at rest; a brief chip for a foreground moment or a key combo, and the session name on hover or when two or more agent cursors are on screen. verbose: the earlier badge with the session name and delivery/target chips on every action. off: no badge. Applies immediately to every cursor.", + "enum": [ + "minimal", + "verbose", + "off", + "default" + ], + "type": "string" + }, + "cursor.float": { + "description": "Agent cursor float: a slow four-second sway, bob and slight tilt, at rest and while moving, that eases to zero on every arrival and holds there for the click, drag, drop, scroll or typing, so they land exactly upright on the hotspot. on (default). off: the cursor never floats and a resting overlay stops redrawing. Applies immediately to every cursor.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.halo": { + "description": "Agent cursor halo: a soft glow in the cursor's colour behind the arrow, so it is easier to follow on any background. on (default). off: no halo. Applies immediately to every cursor.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.marks": { + "description": "Agent cursor action marks: the small cue next to the arrow for observe, click, type, keys, scroll, drag, apps, files, recording and system calls, plus the focus box around a clicked element on macOS. off (default): a plain arrow. Applies immediately to every cursor.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.motion": { + "description": "Pass `default` to clear every saved cursor motion default.", + "enum": [ + "default" + ], + "type": "string" + }, + "cursor.motion.effects.glow": { + "description": "Saved default for the `glow` cursor effect: on, off, or default (the style's own). Omit the key to leave it unchanged.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.motion.effects.magnet": { + "description": "Saved default for the `magnet` cursor effect: on, off, or default (the style's own). Omit the key to leave it unchanged.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.motion.effects.ripple": { + "description": "Saved default for the `ripple` cursor effect: on, off, or default (the style's own). Omit the key to leave it unchanged.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.motion.effects.squish": { + "description": "Saved default for the `squish` cursor effect: on, off, or default (the style's own). Omit the key to leave it unchanged.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.motion.effects.trail": { + "description": "Saved default for the `trail` cursor effect: on, off, or default (the style's own). Omit the key to leave it unchanged.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "cursor.motion.style": { + "description": "Saved default cursor motion style for new sessions. Overridden by start_session cursor_motion and set_agent_cursor_motion; reduced motion always wins. Built-in default: signature_arc.", + "enum": [ + "signature_arc", + "spring_settle", + "magnetic", + "comet_swoop", + "adaptive", + "classic" + ], + "type": "string" + }, + "cursor.motion.timing": { + "description": "Saved default cursor move timing for new sessions. Built-in default: steady (a 600 ms floor on every move).", + "enum": [ + "native", + "fitts", + "fixed", + "relaxed", + "steady", + "calm", + "deliberate" + ], + "type": "string" + }, + "cursor.rotate": { + "description": "Agent cursor rotation while it moves: heading (default) points the tip along the path; lean tilts the arrow at most 30 degrees toward where it is going (on is the same as heading). Both return upright before arrival, so clicks, typing and scrolling stay upright. off: the arrow always stays upright. Applies immediately to every cursor.", + "enum": [ + "off", + "lean", + "heading", + "on", + "default" + ], + "type": "string" + }, + "experimental_pip": { + "description": "Enable the experimental PiP preview window (applies next restart; Linux backend stubbed).", + "type": "boolean" + }, + "experimental_pip_geometry": { + "description": "PiP window size + optional position in `WxH` or `WxH+X+Y` form.", + "type": "string" + }, + "fast_path": { + "description": "Experimental fast path (default off). shadow: check cached targets at act time and report would-refuse verdicts; guard: refuse an action whose target changed since it was read (target_changed); on: guard, plus a small delta in each action result and tree-only re-reads. Runtime only, not saved; env CUA_DRIVER_FAST_PATH seeds it at start.", + "enum": [ + "off", + "shadow", + "guard", + "on" + ], + "type": "string" + }, + "max_image_dimension": { + "description": "Max dimension for screenshot resizing (0 = no limit).", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + } + }, + { + "name": "get_accessibility_tree", + "description": "Return a lightweight snapshot of the desktop: running processes and on-screen visible windows with their bounds and owner pid.\n\nFor the full AT-SPI subtree of a single window (with interactive element indices you can click by), use get_window_state instead — this is a fast discovery read.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "zoom", + "description": "Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixels, with 20% padding. Output is at most 500 px wide. Corners, not a size: x1,y1 is the top-left and x2,y2 the bottom-right (x,y,width,height is also accepted and converted).\n\nAfter a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space. Coordinate actions return `screenshot_context_missing` when no current snapshot contains a screenshot owned by this session. `from_zoom` actions return `zoom_context_missing` when the zoom was never created or was replaced; call `get_window_state`, then `zoom`, again on the same connection.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "pid": { + "description": "Optional target pid. When omitted, the driver resolves the unique current snapshot for this session and window.", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "window_id": { + "description": "Window id of the window captured by get_window_state.", + "type": "integer" + }, + "x1": { + "description": "Left edge of the region in window-local screenshot pixels.", + "type": "number" + }, + "x2": { + "description": "Right edge of the region in window-local screenshot pixels.", + "type": "number" + }, + "y1": { + "description": "Top edge of the region in window-local screenshot pixels.", + "type": "number" + }, + "y2": { + "description": "Bottom edge of the region in window-local screenshot pixels.", + "type": "number" + } + }, + "required": [ + "window_id", + "x1", + "y1", + "x2", + "y2" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "page", + "description": "Legacy browser compatibility tool. Prefer get_browser_state and the typed browser_* tools for exact targeting, endpoint ownership, and consent. Read-only get_text and query_dom remain available by default. Mutating actions require the daemon operator to set CUA_DRIVER_ENABLE_LEGACY_PAGE_MUTATIONS=1 before daemon startup (restart the daemon after changing it); this escape hatch does not provide the typed browser surface's exact binding or existing-profile grant guarantees. Supports Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron apps (via CDP), Chromium/Firefox on Windows (via UIA for read; CDP for execute_javascript when --remote-debugging-port is set), and WKWebView/Tauri/AT-SPI fallbacks.\n\nActions:\n- execute_javascript: Run JS and return the result.\n- get_text: Extract visible text from the page. On macOS, structuredContent.source identifies javascript, cdp, or ax_fallback; a successful AX fallback preserves the triggering error and truncation state in structuredContent.\n- query_dom: Find elements matching a CSS selector, with the same macOS fallback metadata contract as get_text.\n- click_element: Click a CSS-selected element AND animate the agent cursor to its on-screen center first (so the user sees what the agent is doing). Prefer over `execute_javascript('el.click()')` whenever you want visible cursor feedback.\n- insert_text: Insert `text` at whatever currently holds DOM focus in one native operation (CDP Input.insertText) — no synthesized key events, but more durable than a one-shot execute_javascript write since rich-text editors already have to treat it like an IME commit. Try this before type_keystrokes on a contenteditable that discarded an execute_javascript write. Click/focus the target field first.\n- type_keystrokes: Type `text` via real per-character keystroke events into whatever currently holds DOM focus. Slower than insert_text but the most durable rung — use it when insert_text also gets discarded, or the editor's own keydown/keyup handlers need to see real keys. Click/focus the target field first.\n- enable_javascript_apple_events: macOS-only — patch the browser's Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user confirmation and a browser restart).", + "inputSchema": { + "additionalProperties": false, + "properties": { + "action": { + "description": "Action to perform.", + "enum": [ + "execute_javascript", + "get_text", + "query_dom", + "click_element", + "insert_text", + "type_keystrokes", + "enable_javascript_apple_events" + ], + "type": "string" + }, + "attributes": { + "description": "Element attributes to include in query_dom results.", + "items": { + "type": "string" + }, + "type": "array" + }, + "bundle_id": { + "description": "Bundle ID of the browser. Required for enable_javascript_apple_events (macOS only).", + "type": "string" + }, + "cdp_port": { + "description": "Optional, for execute_javascript/insert_text/type_keystrokes: use this exact CDP port instead of auto-discovering one from pid. Needed when the port was opened via the browser's own remote-debugging toggle rather than a launch-time flag, since that path may not answer the auto-discovery probe.", + "maximum": 65535, + "minimum": 1, + "type": "integer" + }, + "css_selector": { + "description": "CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*').", + "type": "string" + }, + "javascript": { + "description": "JavaScript to execute. Required for execute_javascript.", + "type": "string" + }, + "pid": { + "description": "Target process ID.", + "type": "integer" + }, + "selector": { + "description": "CSS selector for click_element (e.g. 'button.submit', '#login a').", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "target_url_contains": { + "description": "Optional, for execute_javascript/insert_text/type_keystrokes: require exactly one browser tab whose URL contains this substring. Use this on a multi-tab browser — there's no built-in link between window_id and which tab a CDP call reaches.", + "type": "string" + }, + "text": { + "description": "Text to insert or type. Required for insert_text and type_keystrokes. The target field must already have DOM focus (click/focus it first).", + "type": "string" + }, + "user_has_confirmed_enabling": { + "description": "Must be true to proceed with enable_javascript_apple_events. This will quit and relaunch the browser.", + "type": "boolean" + }, + "window_id": { + "description": "Target window ID from list_windows.", + "type": "integer" + } + }, + "required": [ + "action" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "get_browser_state", + "description": "Read-only browser inspection. Mode 1 (bind): pass pid + window_id of a native browser window to classify it, correlate it to a CDP target (exact-or-refuse), and mint a session-scoped target id plus tab ids. Mode 2 (snapshot): pass target_id + tab_id. The dom_refs_v1 compatibility format returns composed DOM refs. semantic_v2 joins accessibility, DOM, layout, and viewport state; ranks visible content before retained/offscreen state; and returns a semantic outline, typed action refs, content refs, scoped reads, and opaque continuation. Never performs setup: a browser that is not attached yet refuses with browser_consent_required or browser_requires_setup, and detail.next_call names the browser_prepare call that attaches it. Use this route for any web page in Chrome, Edge or Brave; desktop pointer tools move the user's pointer. Setup and recovery: `get_guide({topic:\"browser\"})`.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "continuation": { + "description": "Opaque continuation minted by an earlier semantic_v2 response.", + "type": "string" + }, + "include_screenshot": { + "default": false, + "description": "Capture the exact tab viewport as PNG through CDP without selecting the tab or foregrounding its native window. The request refuses if capture cannot be completed.", + "type": "boolean" + }, + "pid": { + "description": "Native browser process id (bind mode).", + "type": "integer" + }, + "query": { + "description": "Read-only semantic match over role, accessible name, and visible text.", + "type": "string" + }, + "scope_ref": { + "description": "Current semantic/content ref whose subtree should be observed.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.", + "type": "string" + }, + "snapshot_format": { + "description": "Versioned snapshot contract. dom_refs_v1 remains the compatibility default.", + "enum": [ + "dom_refs_v1", + "semantic_v2" + ], + "type": "string" + }, + "tab_id": { + "description": "Opaque tab id from get_browser_state (session-scoped).", + "type": "string" + }, + "target_id": { + "description": "Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).", + "type": "string" + }, + "window_id": { + "description": "Native window id owned by pid (bind mode).", + "type": "integer" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "browser_prepare", + "description": "Attach the browser tools to a browser. For a page in the user's running Chrome, Edge or Brave, call it once with pid, window_id, session and strategy {\"kind\":\"existing_profile\"} when get_browser_state refuses with browser_consent_required or browser_requires_setup; then call get_browser_state again and act with browser_click, browser_type or browser_pointer. It attaches, or refuses with nothing changed and says why. Explicitly prepare an owned DevTools endpoint for a browser. pid is required for an existing process or existing-profile attachment, and optional only for allow_launch=true with an isolated profile. Existing endpoints are detected without side effects. Acting setup for an isolated profile follows the runtime permission mode and optional capability manifest. It requires allow_launch=true, launches a separate browser, and never copies, modifies, or terminates the requested user profile. Without pid, only a platform-attested system Chrome/Edge installation (or a root-owned package payload on Linux) is eligible; redirects and user-controlled locations fail closed. Existing-profile attachment is explicit and follows the runtime's immutable permission mode: standard requires an explicit --grant existing-profile launch grant or an embedding authorization host, bounded requires a launch-approved exact resource manifest, and unrestricted requires explicit trusted startup risk acceptance. Ordinary MCP transport approval never proves profile authorization. On proven platforms, an authorized request also permits one bounded exact-window setup: open the recognized browser product's fixed remote-debugging page, toggle its uniquely matched per-instance checkbox, prove the PID-owned loopback endpoint, and close the temporary tab. Every visible effect is reported; ambiguity is refused. Where attaching would bring the browser in front of the user's app (macOS Chrome shows its own remote-debugging prompt in front whenever the driver opens a new browser connection; sessions for the same browser share one approved connection, kept for up to 30 minutes idle and 8 hours in total, so later sessions do not prompt again), existing-profile attachment refuses with background_unavailable unless delivery_mode is \"foreground\" (the app that was in front is brought back afterwards) or the browser window is on another Space than the user's, where it attaches in the background while the driver reverts the browser's own brief activations. A driver-owned isolated launch result carries an advisory lifecycle object: the browser process ends with the calling session after idle_expires_in_seconds without session activity, an isolated_named profile persists after the session and an isolated_new profile does not, and explicit end, disconnect, revocation or shutdown end it earlier. Setup and recovery: `get_guide({topic:\"browser\"})`.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "allow_launch": { + "description": "Allow a separate driver-owned isolated Chromium process to be launched (default false).", + "type": "boolean" + }, + "delivery_mode": { + "default": "background", + "description": "Existing-profile attachment only. background (default) attaches in the background when the browser window is on another Space than the user's, and otherwise refuses with background_unavailable, before any visible effect, where attaching would bring the browser in front of the user's app. foreground accepts that, for a screen nobody else is using or when the user agreed; the app that was in front is brought back afterwards.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "pid": { + "description": "Browser process id to prepare. Required except for a driver-owned isolated_new/isolated_named launch with allow_launch=true.", + "type": "integer" + }, + "profile": { + "additionalProperties": false, + "description": "Driver-owned isolated Chromium profile to launch with allow_launch=true. mode=isolated_new creates a fresh throwaway profile; mode=isolated_named reuses the named driver-owned profile. Never an existing user profile.", + "properties": { + "mode": { + "enum": [ + "isolated_new", + "isolated_named" + ], + "type": "string" + }, + "name": { + "description": "Required only for isolated_named; 1-64 path-safe ASCII characters.", + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": "object" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.", + "type": "string" + }, + "strategy": { + "additionalProperties": false, + "description": "Attach to an already-running browser instead of launching one. Pass an object, {\"kind\":\"existing_profile\"}, not a string. kind=existing_profile attaches to the user's running profile at pid/window_id under the runtime's profile authorization.", + "properties": { + "kind": { + "enum": [ + "existing_profile" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "window_id": { + "description": "Exact native window approval anchor; required for strategy.kind=existing_profile.", + "type": "integer" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "browser_navigate", + "description": "Navigate one tab of an exactly-bound browser target to a new URL (http/https/about only). Refused for heuristic bindings. Navigation invalidates all p: refs for the tab.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.", + "type": "string" + }, + "tab_id": { + "description": "Opaque tab id from get_browser_state (session-scoped).", + "type": "string" + }, + "target_id": { + "description": "Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).", + "type": "string" + }, + "url": { + "description": "Destination URL (http:, https:, or about:).", + "type": "string" + } + }, + "required": [ + "target_id", + "tab_id", + "url" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "browser_click", + "description": "Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture unless delivery_mode=\"foreground\" accepts that the browser window may activate (including Linux Chromium configurations whose trusted route cannot stay in the background; for example a browser inside a sandbox). input_route=\"dom_event\" (synthetic pointer and mouse down/up/click sequence at the element's center, ref required) is used only when explicitly requested; it proves dispatch, not control activation, because trust-gated controls may ignore synthetic events. If a click's handler opens a page-owned JavaScript dialog, on either route, the call returns at once with a dialog block (present, kind, dialog_id) to pass to browser_dialog. Refused while a page-owned dialog is open on the tab, and for heuristic bindings.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "delivery_mode": { + "default": "background", + "description": "background (default) refuses trusted input where it would activate the browser window, including Linux Chromium and macOS Chromium configurations whose trusted route cannot stay in the background. foreground accepts that activation, for a browser whose window nobody else is using (for example inside a sandbox).", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "input_route": { + "description": "\"trusted\" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. \"dom_event\": synthetic full-background DOM click, only when explicitly requested. Dispatch does not prove the control activated; refresh page state and verify the expected postcondition.", + "enum": [ + "trusted", + "dom_event" + ], + "type": "string" + }, + "ref": { + "description": "Page element ref in the p: namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.", + "type": "string" + }, + "tab_id": { + "description": "Opaque tab id from get_browser_state (session-scoped).", + "type": "string" + }, + "target_id": { + "description": "Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).", + "type": "string" + }, + "x": { + "description": "Viewport x (CSS px) — alternative to ref.", + "type": "number" + }, + "y": { + "description": "Viewport y (CSS px) — alternative to ref.", + "type": "number" + } + }, + "required": [ + "target_id", + "tab_id" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "browser_type", + "description": "Type text into an exactly-bound tab via the Input domain. mode=\"insert_text\" (default) uses Input.insertText; mode=\"keystrokes\" dispatches per-character key events. Both insert at the caret, so typing into a field that already holds text appends to it; pass replace=true to set the field instead, or to clear it by typing an empty string. Pass a ref to an editable element from the latest snapshot. A ref is required; heuristic bindings are refused.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "mode": { + "description": "insert_text (default): bulk Input.insertText. keystrokes: per-character Input.dispatchKeyEvent.", + "enum": [ + "insert_text", + "keystrokes" + ], + "type": "string" + }, + "ref": { + "description": "Page element ref in the p: namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.", + "type": "string" + }, + "replace": { + "description": "false (default): insert at the caret, appending to whatever the field already holds. true: select the element's whole content first so the text replaces it — with an empty text this clears the field. Replacement goes through the selection, so beforeinput/input still fire and framework state stays consistent.", + "type": "boolean" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.", + "type": "string" + }, + "tab_id": { + "description": "Opaque tab id from get_browser_state (session-scoped).", + "type": "string" + }, + "target_id": { + "description": "Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).", + "type": "string" + }, + "text": { + "description": "Text to type.", + "type": "string" + } + }, + "required": [ + "target_id", + "tab_id", + "ref", + "text" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "browser_select", + "description": "Select one option of a native HTML ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "files": { + "description": "Absolute paths of 1 to 32 local regular files to assign to the input.", + "items": { + "description": "Absolute path to one local regular file.", + "type": "string" + }, + "maxItems": 32, + "minItems": 1, + "type": "array" + }, + "ref": { + "description": "Page element ref in the p: namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session.", + "type": "string" + }, + "tab_id": { + "description": "Opaque tab id from get_browser_state (session-scoped).", + "type": "string" + }, + "target_id": { + "description": "Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id).", + "type": "string" + } + }, + "required": [ + "target_id", + "tab_id", + "ref", + "files" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "browser_download", + "description": "Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "destination_root": { + "description": "Absolute, existing, canonical directory approved to receive the download.", + "type": "string" + }, + "ref": { + "description": "Live page ref whose activation initiates the download.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. This tool requires the label that owns its browser target, tab, and refs.", + "type": "string" + }, + "tab_id": { + "description": "Opaque exact tab id from get_browser_state.", + "type": "string" + }, + "target_id": { + "description": "Opaque exact browser target id from get_browser_state.", + "type": "string" + } + }, + "required": [ + "session", + "target_id", + "tab_id", + "ref", + "destination_root" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "browser_pointer", + "description": "Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. Semantic refs must declare pointer for hover, right-click, double-click, and drag; scroll accepts a scroll or pointer capability. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "action": { + "description": "Pointer gesture. scroll needs delta_x or delta_y; drag needs destination_ref or to_x/to_y.", + "enum": [ + "hover", + "right_click", + "double_click", + "scroll", + "drag" + ], + "type": "string" + }, + "delivery_mode": { + "default": "background", + "description": "background (default) refuses trusted input where it would activate the browser window, including Linux Chromium and macOS Chromium configurations whose trusted route cannot stay in the background. foreground accepts that activation, for a browser whose window nobody else is using (for example inside a sandbox).", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "delta_x": { + "description": "Horizontal scroll delta in CSS pixels.", + "type": "number" + }, + "delta_y": { + "description": "Vertical scroll delta in CSS pixels.", + "type": "number" + }, + "destination_ref": { + "description": "Drag destination page ref in the exact same frame.", + "type": "string" + }, + "input_route": { + "default": "trusted", + "description": "trusted sends CDP Input events; dom_event synthesizes DOM events in the page and requires ref.", + "enum": [ + "trusted", + "dom_event" + ], + "type": "string" + }, + "ref": { + "description": "Origin page ref. Alternative to x/y.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. This tool requires the label that owns its browser target, tab, and refs.", + "type": "string" + }, + "tab_id": { + "description": "Opaque tab id minted by get_browser_state.", + "type": "string" + }, + "target_id": { + "description": "Opaque target id minted by get_browser_state.", + "type": "string" + }, + "to_x": { + "description": "Drag destination viewport x in CSS pixels.", + "type": "number" + }, + "to_y": { + "description": "Drag destination viewport y in CSS pixels.", + "type": "number" + }, + "x": { + "description": "Origin viewport x in CSS pixels.", + "type": "number" + }, + "y": { + "description": "Origin viewport y in CSS pixels.", + "type": "number" + } + }, + "required": [ + "target_id", + "tab_id", + "session", + "action" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "delivery": { + "additionalProperties": false, + "properties": { + "delivered_count": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "mode": { + "enum": [ + "background", + "foreground", + "not_applicable", + "unknown" + ], + "type": "string" + } + }, + "required": [ + "mode" + ], + "type": [ + "object", + "null" + ] + }, + "effect": { + "enum": [ + "confirmed", + "partial", + "unverifiable", + "suspected_noop", + "refused" + ], + "type": "string" + }, + "error": { + "additionalProperties": false, + "properties": { + "code": { + "type": "string" + }, + "hint": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "code" + ], + "type": [ + "object", + "null" + ] + }, + "escalation": { + "additionalProperties": false, + "properties": { + "reason": { + "enum": [ + "route_unavailable", + "delivery_failed", + "effect_unconfirmed", + "suspected_noop", + "permission_required" + ], + "type": "string" + }, + "target": { + "enum": [ + "pixel", + "foreground", + "page", + "session" + ], + "type": "string" + } + }, + "required": [ + "target", + "reason" + ], + "type": [ + "object", + "null" + ] + }, + "evidence": { + "items": { + "additionalProperties": false, + "properties": { + "detail": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "enum": [ + "value_readback", + "window_change" + ], + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + "type": [ + "array", + "null" + ] + }, + "fast_path": { + "type": "object" + }, + "hint": { + "type": [ + "string", + "null" + ] + }, + "next_call": { + "additionalProperties": false, + "properties": { + "arguments": { + "type": "object" + }, + "tool": { + "type": "string" + } + }, + "required": [ + "tool", + "arguments" + ], + "type": [ + "object", + "null" + ] + }, + "route": { + "enum": [ + "accessibility", + "synthetic_events", + "global_input", + "system_api", + "dom", + "trusted_input" + ], + "type": "string" + }, + "summary": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "effect", + "route" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "start_recording", + "description": "Start trajectory recording for the calling session. Each action-tool invocation (click, right_click, scroll, type_text, press_key, hotkey, set_value) from that session writes a turn folder under `output_dir`. Without `session`, every call on the same connection is recorded, including named-session calls. Other connections and session lifecycle calls (`start_session` / `end_session`) are not recorded. CLI recordings started with `cua-driver recording start` are daemon-wide.\n\nEach turn folder holds:\n\n- `before_state.json` / `after_state.json` — application AX/UIA/AT-SPI state immediately before and after the action.\n- `before.png` / `after.png` — target-window screenshots immediately before and after the action.\n- `evidence.json` — capture status and a stable classification when an expected artifact could not be captured.\n- `app_state.json` — post-action AX/UIA snapshot for the target pid.\n- `screenshot.png` — compatibility alias of `after.png`.\n- `action.json` — tool name, full input arguments, result summary, result-error flag, pid, click point (when applicable), ISO-8601 timestamp.\n- `click.png` — for dispatched click-family actions only, `before.png` with a red marker at the click point. A call refused before target resolution is explicitly not applicable instead.\n\nThe per-turn accessibility walk is bounded like `get_window_state`: `state_timeout_ms` (default 1000) caps each before/after walk, a walk that runs out of budget records the PARTIAL tree, and `evidence.json` carries `truncated`, `truncation_reason`, `nodes_visited`, `nodes_pending` and `timeout_ms` for that phase. A provider that stops answering is abandoned after the budget plus a short grace and the phase is classified `state_capture_timeout`. Actions refused before dispatch (for example an unknown or expired `capture_id`) skip the state walk; their state is classified `not_applicable` / `action_refused_before_dispatch`. Pass `include_accessibility_tree: false` to record screenshots and actions without state.\n\nPass `redact_values` with any secret the session will type (for example a password): each occurrence in a turn's text files is replaced with `[redacted]`, and such turns are not replayed.\n\nTurn folders are named `turn-00001/`, `turn-00002/`, etc. Turn numbering restarts at 1 each time recording is (re-)started.\n\n**Video is off by default.** Pass `record_video: true` to also capture the main display to `/recording.mp4` (H.264 / 30 fps) for the lifetime of the session. The recording is torn down automatically when the MCP client disconnects.\n\n**macOS uses native ScreenCaptureKit** (daemon-owned SCStream + SCRecordingOutput) under the daemon's Screen Recording grant, with no ffmpeg subprocess. Requires macOS 15.0+. On macOS 26 (Tahoe), the first direct capture can also show a one-time consent asking to let Cua Driver bypass the system private window picker and directly access your screen and audio; choose Allow, or run `cua-driver permissions grant` beforehand to answer it up front. The recorder captures screen video only and does not enable system-audio capture.\n\n**Windows + Linux X11 use an ffmpeg subprocess** (`gdigrab` / `x11grab` + libx264). Requires ffmpeg on PATH (winget install Gyan.FFmpeg / apt install ffmpeg); when ffmpeg is missing or fails on startup the per-turn capture (screenshots + action.json) still runs and the session's `last_error` field carries the diagnostic.\n\n**Linux Wayland routes by compositor**: GNOME uses the trusted Shell helper's frames, wlroots compositors use `wf-recorder`, and compositors without wlr-screencopy (KDE/KWin) record the monitor the user grants through the xdg-desktop-portal ScreenCast (one consent dialog per install; requires a build with the `portal-capture` feature and ffmpeg on PATH). Unsupported combinations report the exact limitation in `last_error`.\n\nState persists for the life of the daemon; a restart resets to disabled with no on-disk state. Call `stop_recording` to disable + finalize the mp4.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "include_accessibility_tree": { + "default": true, + "description": "Default true. Set false to skip the per-turn before/after accessibility walks entirely; screenshots, click markers and action.json are still recorded and state is classified `state_capture_disabled`.", + "type": "boolean" + }, + "output_dir": { + "description": "Absolute or ~-rooted directory where turn folders and (when enabled) the video file are written.", + "type": "string" + }, + "record_video": { + "description": "Capture the main display to /recording.mp4. Default: false. Set to true to also capture the main display to recording.mp4 (otherwise only the per-turn screenshots + JSON are recorded). On macOS this uses native ScreenCaptureKit (macOS 15.0+); macOS 26 can show a one-time direct screen-capture consent on first use (see `cua-driver permissions grant`). On Windows + Linux it requires ffmpeg on PATH.", + "type": "boolean" + }, + "redact_values": { + "description": "Secret values, such as passwords or tokens, that must never be written to the recording. Every occurrence in a turn's action.json and state files is replaced with `[redacted]`, and the turn is marked `redacted: true` so replay_trajectory skips it. The values are held in memory only, never written, and cleared when recording stops. Screenshots and video are not altered.", + "items": { + "minLength": 1, + "type": "string" + }, + "maxItems": 64, + "minItems": 1, + "type": "array" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "state_timeout_ms": { + "default": 1000, + "description": "Wall-clock budget in milliseconds for EACH per-turn before/after accessibility walk (same default and bounds as get_window_state's timeout_ms). A walk that runs out records the partial tree and marks it truncated in evidence.json.", + "maximum": 120000, + "minimum": 100, + "type": "integer" + } + }, + "required": [ + "output_dir" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + } + }, + { + "name": "stop_recording", + "description": "Stop trajectory recording. Disables further per-turn capture and, when video was enabled, gracefully terminates the ffmpeg subprocess so the mp4's moov atom is finalized (the file is playable). Calling stop on an already-stopped session is a no-op. The response carries `last_video_path` pointing at the finalized mp4 (when video was on).\n\nA manual `stop_recording` is **unconditional** — it stops whatever recording is active regardless of which session started it. Ownership-scoped teardown (so one client disconnecting can't stop a recording a later client started) is handled by the registry's `session_end` lifecycle hook, not by this tool.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + } + }, + { + "name": "get_recording_state", + "description": "Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled.\n\nPure read-only.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "replay_trajectory", + "description": "Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. `dir` must point at a directory previously written by `start_recording`. Each `turn-NNNNN/` is parsed for `action.json`, and the recorded tool is called with its recorded `arguments` via the same dispatch path an MCP / CLI call uses.\n\nCaveats:\n- Element-token actions (`click({pid, element_token})` etc.) will fail because element tokens are per-snapshot and don't survive across sessions. Pixel clicks (`click({pid, x, y})`) and all keyboard tools replay cleanly. Failures are reported but don't stop replay unless `stop_on_error` is true.\n- `get_window_state` and other read-only tools are NOT currently recorded, so replays do not re-populate the per-(pid, window_id) element cache.\n- If recording is ENABLED while replay runs, the replay itself is recorded into the currently configured output directory. That's deliberate: recording a replay against a new build and diffing the two trajectories is the regression-test workflow.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "delay_ms": { + "description": "Milliseconds to sleep between turns, for human-observable pacing. Default 500.", + "maximum": 10000, + "minimum": 0, + "type": "integer" + }, + "dir": { + "description": "Trajectory directory previously written by `start_recording`. Absolute or ~-rooted.", + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "stop_on_error": { + "description": "Stop replay on the first tool-call error. Default true — set false to best-effort through the full trajectory.", + "type": "boolean" + } + }, + "required": [ + "dir" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": false, + "readOnlyHint": false + } + }, + { + "name": "install_ffmpeg", + "description": "Install the ffmpeg binary used by start_recording's video capture (Linux/Windows; macOS records natively and needs no ffmpeg). Two-step and confirmed: called without `confirm` it only REPORTS the exact install command for this platform's package manager; pass `confirm: true` to actually run it. No-op if ffmpeg is already on PATH. ffmpeg is run as a separate process, never linked into the driver.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "confirm": { + "description": "Run the install command. Without it, only the planned command is reported.", + "type": "boolean" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "start_session", + "description": "Optionally create or return a lifecycle session before acting. For multi-call work, prefer a short public `session` label and repeat it on every call that accepts it; an omitted value uses the authenticated transport lease's implicit session instead. This tool is optional because an ordinary action can create or reuse a named run directly. Use it to set the initial cursor theme before acting or to revive a public name after it has ended; ordinary actions never revive ended names. `capture_scope` is deprecated compatibility input; new callers select window or desktop modality per action. Idempotent.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "capture_scope": { + "description": "Deprecated compatibility policy. New callers select window or desktop\nmodality on each action instead of storing it on the session.", + "enum": [ + "auto", + "window", + "desktop" + ], + "type": "string" + }, + "cursor_motion": { + "additionalProperties": false, + "description": "Optional initial cursor motion (style, timing, effects and tuning). The host applies it\nbefore the cursor is first made visible. A later `set_agent_cursor_motion` call wins;\nthis wins over the saved default (`cursor.motion.*` in the driver config) and the\nbuilt-in `signature_arc`. Reduced motion always wins.", + "properties": { + "arc_flow": { + "description": "Added to the arc asymmetry of `signature_arc`, `spring_settle` and `comet_swoop`:\npositive moves the apex toward the destination. Clamped to -1..1 (default 0).", + "type": "number" + }, + "arc_size": { + "description": "Scales the arc of `signature_arc`, `spring_settle` and `comet_swoop`: 0.25 (default)\nkeeps the style's arc, 0 is a straight line, 0.5 doubles it. Clamped to 0..1.", + "type": "number" + }, + "dwell_after_click_ms": { + "description": "Pause after a click animation, in milliseconds. Clamped to 0..5000 (default 80).", + "type": "number" + }, + "effects": { + "additionalProperties": false, + "description": "Turn effects on or off. Omit a field to keep it; `\"default\"` restores the style default.", + "properties": { + "glow": { + "description": "Soft glow around the cursor that grows with speed.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "magnet": { + "description": "Target glow when the `magnetic` style locks on.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "ripple": { + "description": "Ring that expands from the hotspot on click.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "squish": { + "description": "Brief scale-down of the cursor on click.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "trail": { + "description": "Short fading trail behind the cursor.", + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "end_handle": { + "description": "Arc control-point offset from the end, as a fraction of the distance, for\n`signature_arc`, `spring_settle` and `comet_swoop`. Clamped to 0..1 (default 0.3).", + "type": "number" + }, + "glide_duration_ms": { + "description": "Move duration in milliseconds for `fixed` timing (1430 ms when 0, the default). A\nnonzero value with `native` or the default `steady` timing also fixes the duration.\nClamped to 0..5000.", + "type": "number" + }, + "idle_hide_ms": { + "description": "Hide the cursor after this many idle milliseconds; 0 never hides it. Clamped to\n0..60000 (default 15000).", + "type": "number" + }, + "spring": { + "description": "Arrival spring damping for `classic`: 1 is critically damped, 0.3 is bouncy. Clamped\nto 0.3..1 (default 0.72).", + "type": "number" + }, + "start_handle": { + "description": "Arc control-point offset from the start, as a fraction of the distance, for\n`signature_arc`, `spring_settle` and `comet_swoop`. Clamped to 0..1 (default 0.3).", + "type": "number" + }, + "style": { + "description": "Trajectory style. `signature_arc` (default) is one arc with a small follow-through;\n`spring_settle` lands with one soft bounce; `magnetic` is pulled into the target;\n`comet_swoop` is a wide arc with a short trail; `adaptive` picks a careful approach for\nsmall targets and a swoop for long moves; `classic` is the previous Dubins glide. When\nthe theme's reduced motion is on, every move is a short straight glide with no effects.", + "enum": [ + "signature_arc", + "spring_settle", + "magnetic", + "comet_swoop", + "adaptive", + "classic" + ], + "type": "string" + }, + "timing": { + "description": "`steady` (default) keeps the style's timing but makes every move take at least 600 ms,\nwith a gentler ease; `relaxed` and `calm` do the same with a 450 or 750 ms floor.\n`native` uses the style's own timing; `fitts` scales the move time with distance and\ntarget size; `deliberate` is a slower Fitts curve (450..1300 ms); `fixed` uses\nglide_duration_ms (1430 ms when 0).", + "enum": [ + "native", + "fitts", + "fixed", + "relaxed", + "steady", + "calm", + "deliberate" + ], + "type": "string" + }, + "turn_radius": { + "description": "Minimum turning radius of the `classic` glide path, in points; smaller turns tighter.\nClamped to 1..1000 (default 80).", + "type": "number" + } + }, + "required": [], + "type": "object" + }, + "cursor_theme": { + "additionalProperties": true, + "description": "Optional initial cursor theme. The host applies it before the cursor is\nfirst made visible, avoiding a flash of the default theme.", + "properties": { + "reduced_motion": { + "default": "auto", + "enum": [ + "auto", + "on", + "off" + ], + "type": "string" + }, + "theme_id": { + "type": "string" + } + }, + "required": [ + "theme_id" + ], + "type": "object" + }, + "session": { + "description": "Optional stable public label for this run (e.g. \"research-run-1\").\nWhen omitted, the authenticated transport lease's implicit session is\ncreated or returned.", + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "active": { + "type": "boolean" + }, + "capture_scope": { + "enum": [ + "auto", + "window", + "desktop" + ], + "type": "string" + }, + "cursor_motion": { + "additionalProperties": false, + "properties": { + "arc_flow": { + "type": [ + "number", + "null" + ] + }, + "arc_size": { + "type": [ + "number", + "null" + ] + }, + "dwell_after_click_ms": { + "type": [ + "number", + "null" + ] + }, + "effects": { + "additionalProperties": false, + "properties": { + "glow": { + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "magnet": { + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "ripple": { + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "squish": { + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + }, + "trail": { + "enum": [ + "on", + "off", + "default" + ], + "type": "string" + } + }, + "type": [ + "object", + "null" + ] + }, + "end_handle": { + "type": [ + "number", + "null" + ] + }, + "glide_duration_ms": { + "type": [ + "number", + "null" + ] + }, + "idle_hide_ms": { + "type": [ + "number", + "null" + ] + }, + "spring": { + "type": [ + "number", + "null" + ] + }, + "start_handle": { + "type": [ + "number", + "null" + ] + }, + "style": { + "enum": [ + "signature_arc", + "spring_settle", + "magnetic", + "comet_swoop", + "adaptive", + "classic" + ], + "type": "string" + }, + "timing": { + "enum": [ + "native", + "fitts", + "fixed", + "relaxed", + "steady", + "calm", + "deliberate" + ], + "type": "string" + }, + "turn_radius": { + "type": [ + "number", + "null" + ] + } + }, + "type": [ + "object", + "null" + ] + }, + "desktop_capture_authorized": { + "type": "boolean" + }, + "desktop_unlocked": { + "type": "boolean" + }, + "effective_scope": { + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "escalation_detail": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "escalation_reason": { + "anyOf": [ + { + "enum": [ + "ax_tree_pixel_mismatch", + "background_delivery_failed", + "foreground_ineffective", + "no_window_target", + "other" + ], + "type": "string" + }, + { + "type": "null" + } + ] + }, + "revived": { + "type": "boolean" + }, + "session": { + "type": "string" + } + }, + "required": [ + "session", + "capture_scope", + "effective_scope", + "desktop_capture_authorized", + "desktop_unlocked", + "escalation_reason", + "escalation_detail", + "active", + "revived" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "escalate_session", + "description": "Deprecated compatibility tool for legacy capture-scope sessions. New callers select window or desktop modality on each action. No deescalate_session tool exists.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "detail": { + "description": "Optional bounded diagnostic detail. Never use secrets or page content.", + "maxLength": 200, + "type": "string" + }, + "reason": { + "description": "Why the window-scoped attempt failed and desktop capture is needed.", + "enum": [ + "ax_tree_pixel_mismatch", + "background_delivery_failed", + "foreground_ineffective", + "no_window_target", + "other" + ], + "type": "string" + }, + "session": { + "description": "Public label of the legacy capture-scope session to escalate.", + "type": "string" + } + }, + "required": [ + "session", + "reason" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": false, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "capture_scope": { + "enum": [ + "auto", + "window", + "desktop" + ], + "type": "string" + }, + "desktop_capture_authorized": { + "type": "boolean" + }, + "desktop_unlocked": { + "type": "boolean" + }, + "effective_scope": { + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "escalation_detail": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "escalation_reason": { + "anyOf": [ + { + "enum": [ + "ax_tree_pixel_mismatch", + "background_delivery_failed", + "foreground_ineffective", + "no_window_target", + "other" + ], + "type": "string" + }, + { + "type": "null" + } + ] + }, + "session": { + "type": "string" + } + }, + "required": [ + "session", + "capture_scope", + "effective_scope", + "desktop_capture_authorized", + "desktop_unlocked", + "escalation_reason", + "escalation_detail" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_session", + "description": "Read content-free lifecycle, cursor, recording, and idle status for one session visible to this authenticated transport. Omit `session` to inspect its implicit session.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "session": { + "description": "Optional public label. When omitted, inspect the caller's attached\nimplicit session.", + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "client_kind": { + "enum": [ + "cli", + "direct", + "mcp", + "python_sdk", + "typescript_sdk" + ], + "type": "string" + }, + "cursor_visible": { + "type": "boolean" + }, + "expires_in_seconds": { + "minimum": 0, + "type": "integer" + }, + "idle_seconds": { + "minimum": 0, + "type": "integer" + }, + "implicit": { + "type": "boolean" + }, + "recording_active": { + "type": "boolean" + }, + "session": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "state": { + "enum": [ + "active", + "ending" + ], + "type": "string" + }, + "transport": { + "enum": [ + "cli", + "daemon", + "mcp_stdio", + "mcp_http" + ], + "type": "string" + } + }, + "required": [ + "session", + "implicit", + "state", + "client_kind", + "transport", + "cursor_visible", + "recording_active", + "idle_seconds", + "expires_in_seconds" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "list_sessions", + "description": "List content-free lifecycle summaries attached to this authenticated transport lease. It does not enumerate other callers' sessions.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "cursor": { + "description": "Opaque continuation cursor returned by a previous call.", + "type": "string" + }, + "limit": { + "description": "Maximum number of content-free summaries to return (default 50, max\n100). Ordinary agent transports are scoped to their own lease.", + "minimum": 0, + "type": "integer" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "next_cursor": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "sessions": { + "items": { + "properties": { + "client_kind": { + "enum": [ + "cli", + "direct", + "mcp", + "python_sdk", + "typescript_sdk" + ], + "type": "string" + }, + "cursor_visible": { + "type": "boolean" + }, + "expires_in_seconds": { + "minimum": 0, + "type": "integer" + }, + "idle_seconds": { + "minimum": 0, + "type": "integer" + }, + "implicit": { + "type": "boolean" + }, + "recording_active": { + "type": "boolean" + }, + "session": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "state": { + "enum": [ + "active", + "ending" + ], + "type": "string" + }, + "transport": { + "enum": [ + "cli", + "daemon", + "mcp_stdio", + "mcp_http" + ], + "type": "string" + } + }, + "required": [ + "session", + "implicit", + "state", + "client_kind", + "transport", + "cursor_visible", + "recording_active", + "idle_seconds", + "expires_in_seconds" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "sessions", + "next_cursor" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_session_state", + "description": "Deprecated compatibility alias that reads a live legacy session's capture policy. Use get_session for lifecycle state.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "session": { + "description": "Optional public label. When omitted, inspect the caller's attached\nimplicit session.", + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "capture_scope": { + "enum": [ + "auto", + "window", + "desktop" + ], + "type": "string" + }, + "desktop_capture_authorized": { + "type": "boolean" + }, + "desktop_unlocked": { + "type": "boolean" + }, + "effective_scope": { + "enum": [ + "window", + "desktop" + ], + "type": "string" + }, + "escalation_detail": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ] + }, + "escalation_reason": { + "anyOf": [ + { + "enum": [ + "ax_tree_pixel_mismatch", + "background_delivery_failed", + "foreground_ineffective", + "no_window_target", + "other" + ], + "type": "string" + }, + { + "type": "null" + } + ] + }, + "session": { + "type": "string" + } + }, + "required": [ + "session", + "capture_scope", + "effective_scope", + "desktop_capture_authorized", + "desktop_unlocked", + "escalation_reason", + "escalation_detail" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "end_session", + "description": "End one visible lifecycle session and run its cursor, recording, configuration, and other cleanup hooks exactly once. Omit `session` to end the authenticated transport's implicit session. Idempotent.", + "inputSchema": { + "additionalProperties": true, + "properties": { + "session": { + "description": "Optional public label to end. When omitted, end the caller's attached\nimplicit session.", + "type": "string" + } + }, + "required": [], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": false + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "active": { + "const": false + }, + "session": { + "type": "string" + } + }, + "required": [ + "session", + "active" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + }, + { + "name": "get_guide", + "description": "Read Cua Driver guidance on demand, taken from the guide bundled with this driver (no skill install needed). Topics: overview, drag, browser, permissions, batching, menus. Call with no topic to list them. Read-only.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "topic": { + "description": "One of: overview, drag, browser, permissions, batching, menus. Omit to list the topics.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + } + }, + { + "name": "run_steps", + "description": "The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with `observe`) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus `observe:{}` replaces an action call followed by a get_window_state call. Each step is `{tool, args}` where `tool` is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and `args` are exactly that tool's arguments. `pid` and `window_id` (or `app`/`window`) on run_steps itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the `observe`, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.\n\nReturns per-step status (`ok` or the error message) and, when `observe` is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without `observe`). A step refused because its app ignores background input says how to re-send it; `foreground_fallback:true` retries such steps in the foreground once (the window is activated). That read is a `since:\"latest\"` diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without `observe` nothing is read. The batch uses one session: steps may omit `session` or repeat the batch's. `delay_ms` pauses between steps (max 2000). At most 64 steps; `delivery_mode` on run_steps is the default for its steps.\n\nTARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written `{\"click\": {...args}}` as well as `{tool, args}`, and its args may name the target instead of carrying a token: `role` (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), `name` (the element's label; exact match wins over substring, case-insensitive), `nth` (0-based, when several match), and `app` (app name or bundle id) and/or `window` (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step `timeout_ms`, max 10000) for the element to appear, and acts on it; with only `app` it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name `app` once. The first lookup in a window also reads it once for `observe`, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add `nth` or a fuller name.\n\nWAITS AND CHECKS: a step may add `wait_for` (checked before its action) and `expect` (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: `text` matches any row of the tree, static text included; `gone:true` waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.\n\nExample (fill a form, submit, confirm the dialog, check the result):\n{\"steps\":[{\"set_value\":{\"app\":\"Safari\",\"role\":\"textfield\",\"name\":\"Email\",\"value\":\"ada@example.com\"}},{\"click\":{\"role\":\"button\",\"name\":\"Submit\"},\"expect\":[{\"role\":\"button\",\"name\":\"Confirm\"}]},{\"click\":{\"role\":\"button\",\"name\":\"Confirm\"},\"expect\":[{\"name\":\"Confirm\",\"gone\":true},{\"text\":\"Thanks\"}]}],\"observe\":{}} Batching rules: `get_guide({topic:\"batching\"})`.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "app": { + "description": "Default app (name or bundle id) for steps that name no window.", + "type": "string" + }, + "delay_ms": { + "description": "Pause between steps in milliseconds (not after the last). Default 0.", + "maximum": 2000, + "minimum": 0, + "type": "integer" + }, + "delivery_mode": { + "description": "Default delivery_mode for steps whose tool takes one and that set none.", + "enum": [ + "background", + "foreground" + ], + "type": "string" + }, + "foreground_fallback": { + "description": "When a step is refused because its target ignores background input (Electron, LibreOffice and other toolkits), retry it once with delivery_mode:\"foreground\": the window is activated and the pointer may move. Default false.", + "type": "boolean" + }, + "observe": { + "description": "Optional end-of-batch observation: arguments for ONE get_window_state call. `pid` and `window_id` default to those of the last step that names both. Defaults to since=\"latest\" (only what changed since your last read of that window with the same query/max_elements/max_depth; a full read when there is none), include_screenshot=false and max_elements=250; pass include_screenshot=true to see the window, or since=null for a full read. Omit to read nothing. {} observes the last window.", + "type": "object" + }, + "pid": { + "description": "Default pid for steps that name no window (a step's own pid/window_id/app/window wins).", + "type": "integer" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "steps": { + "description": "Ordered actions. Execution stops at the first failing step. Each step is {tool, args} or {: args}, plus optional wait_for / expect / timeout_ms; a step may also be only a wait_for or expect check.", + "items": { + "additionalProperties": true, + "properties": { + "args": { + "description": "Arguments for that tool, as in a direct call, optionally with role/name/nth/app/window to name the target instead of pid/window_id/element_token.", + "type": "object" + }, + "expect": { + "description": "Up to 4 checks that must hold after the action: {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}. Default timeout 2000 ms.", + "items": { + "type": "object" + }, + "maxItems": 4, + "type": "array" + }, + "timeout_ms": { + "description": "How long a named target may take to appear. Default 3000.", + "maximum": 10000, + "minimum": 0, + "type": "integer" + }, + "tool": { + "description": "Action tool to run.", + "enum": [ + "click", + "double_click", + "right_click", + "set_value", + "type_text", + "press_key", + "hotkey", + "scroll", + "drag", + "move_cursor" + ], + "type": "string" + }, + "wait_for": { + "description": "Check that must hold before the action: {role?, name?, text?, gone?, app?, window?, timeout_ms?}. Default timeout 5000 ms.", + "type": "object" + } + }, + "type": "object" + }, + "maxItems": 64, + "minItems": 1, + "type": "array" + }, + "window": { + "description": "Default window title substring for steps that name no window.", + "type": "string" + }, + "window_id": { + "description": "Default window_id for steps that name no window, with `pid`.", + "type": "integer" + } + }, + "required": [ + "steps" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "run_script", + "description": "Run a JavaScript program that drives apps through Cua Driver, in ONE call: loops, conditions, reads and checks between actions, and only the result comes back. Use it when the flow needs logic (a loop over rows, a branch on what the app shows, a retry); use run_steps for a fixed list of steps.\n\nThe script is the body of an async function: use `await`, and `return` a JSON value. It runs in a sandbox with no filesystem, network, process, timers or environment; the only API is `cua`:\n- `await cua.getApp(\"TextEdit\" | \"com.apple.TextEdit\" | {pid, windowId})` → app (follows the app's frontmost window unless a windowId is given)\n- `await cua.launch(nameOrBundleId)` → app; `cua.listApps()`, `cua.listWindows({pid?})`, `cua.sleep(ms)`, `cua.computer.target`\n- app.getState(opts?) → get_window_state result (`tree_markdown` or `tree_diff`, `snapshot_id`; pass get_window_state options such as {since:\"latest\"})\n- app.query({role?, name?, text?}) → matching elements across the app's windows {element_token, role, label, value, enabled, selected, frame}, then matching static-text rows {role, label, value, text, display_only} (read text such as a status line from their `value`)\n- app.waitFor({role?, name?, text?, gone?, value?, timeout_ms?}) and app.verify(predicate | predicates) (verify_state)\n- app.click(t), doubleClick(t), rightClick(t), typeText(text, t?), pressKey(\"return\" | \"cmd+s\", t?), hotkey([\"cmd\",\"s\"]), scroll(t, direction, amount?), setValue(t, value), drag([x,y], [x,y]), zoom([x1,y1,x2,y2])\nwhere a target `t` is {role, name, nth?} (found fresh, waits up to 3 s), an element_token string, a row number from this app's last getState(), or [x, y] window pixels.\n- `cua.call(tool, args)` for any other allowed tool: click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, launch_app, list_apps, list_windows, get_window_state, verify_state, zoom.\nA failed driver call throws a CuaError (catch it to recover). Every call goes through the same session, permission and approval checks as a direct call.\n\nLimits: `timeout_ms` wall time (default 30000, max 120000), `max_calls` driver calls (default 100, max 500), 32 MiB heap. Returns the return value, console output, and a per-call log; an error names the script line, the call and the reason.\n\nExample:\nconst app = await cua.getApp(\"Calculator\");\nfor (const key of [\"7\", \"multiply\", \"6\", \"equals\"]) await app.click({role: \"button\", name: key});\nconst [display] = await app.query({role: \"statictext\"});\nreturn display.value; Batching rules: `get_guide({topic:\"batching\"})`.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "max_calls": { + "description": "Most driver calls the script may make. Default 100.", + "maximum": 500, + "minimum": 1, + "type": "integer" + }, + "script": { + "description": "JavaScript: the body of an async function. `return` the result.", + "maxLength": 65536, + "minLength": 1, + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + }, + "timeout_ms": { + "description": "Wall-time limit. Default 30000.", + "maximum": 120000, + "minimum": 100, + "type": "integer" + } + }, + "required": [ + "script" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "check_for_update", + "description": "Check the saved stable/nightly Cua Driver channel for a release on GitHub. Returns current and selected channels, current and latest versions, an `update_available` boolean, the install one-liner, and the release notes URL. Read-only — never installs. Pacman-owned Linux executables return package-manager guidance without checking GitHub. Mirror of `cua-driver check-update --json`.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": true, + "readOnlyHint": true + } + }, + { + "name": "install_extension", + "description": "Preview or install one Driver-managed optional extension. The first call without confirm returns the exact signed artifact, destination, license, source, and trust plan without mutation. Re-call with confirm=true to perform that exact verified installation.", + "inputSchema": { + "additionalProperties": false, + "properties": { + "confirm": { + "description": "Install the previewed extension. Omit or false for a read-only plan.", + "type": "boolean" + }, + "name": { + "description": "Extension to preview or install. Only perception (the local visual-region parser used by parse_visual_regions) is available.", + "enum": [ + "perception" + ], + "type": "string" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "required": [ + "name" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": true, + "idempotentHint": false, + "openWorldHint": true, + "readOnlyHint": false + } + }, + { + "name": "parse_visual_regions", + "description": "Parse one immutable registered capture into model-neutral text and icon regions.", + "inputSchema": { + "additionalProperties": false, + "description": "Parse one immutable screenshot capture. Pass the `capture_id` returned by\n`get_window_state` or `get_desktop_state`; the Driver capture registry\nresolves it to the exact pixels and action-coordinate transform.", + "properties": { + "capture_id": { + "description": "`capture_id` of a screenshot returned by get_window_state or get_desktop_state.", + "minLength": 1, + "type": "string" + }, + "options": { + "additionalProperties": false, + "default": {}, + "description": "Optional, model-neutral controls for one bounded parse.", + "properties": { + "kinds": { + "description": "Region kinds to return (`text`, `icon`). Omit for all kinds.", + "items": { + "enum": [ + "text", + "icon" + ], + "type": "string" + }, + "maxItems": 2, + "minItems": 1, + "type": "array" + }, + "max_regions": { + "description": "Return at most this many regions.", + "minimum": 1, + "type": "integer" + }, + "min_confidence": { + "description": "Drop regions below this confidence (0 to 1).", + "maximum": 1, + "minimum": 0, + "type": "number" + } + }, + "required": [], + "type": "object" + }, + "session": { + "description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.", + "type": "string" + } + }, + "required": [ + "capture_id" + ], + "type": "object" + }, + "annotations": { + "destructiveHint": false, + "idempotentHint": true, + "openWorldHint": false, + "readOnlyHint": true + }, + "outputSchema": { + "anyOf": [ + { + "additionalProperties": true, + "properties": { + "capture": { + "properties": { + "action_coordinate_space": { + "oneOf": [ + { + "properties": { + "kind": { + "const": "screenshot_pixels", + "type": "string" + } + }, + "required": [ + "kind" + ], + "type": "object" + }, + { + "properties": { + "kind": { + "const": "affine", + "type": "string" + }, + "m11": { + "type": "number" + }, + "m12": { + "type": "number" + }, + "m21": { + "type": "number" + }, + "m22": { + "type": "number" + }, + "tx": { + "type": "number" + }, + "ty": { + "type": "number" + } + }, + "required": [ + "kind", + "m11", + "m12", + "m21", + "m22", + "tx", + "ty" + ], + "type": "object" + } + ] + }, + "capture_id": { + "type": "string" + }, + "captured_at": { + "type": [ + "string", + "null" + ] + }, + "screenshot": { + "properties": { + "height": { + "minimum": 1, + "type": "integer" + }, + "mime_type": { + "const": "image/png" + }, + "reference": { + "type": "string" + }, + "sha256": { + "type": [ + "string", + "null" + ] + }, + "width": { + "minimum": 1, + "type": "integer" + } + }, + "required": [ + "reference", + "width", + "height", + "mime_type" + ], + "type": "object" + }, + "source": { + "oneOf": [ + { + "properties": { + "kind": { + "const": "window", + "type": "string" + }, + "pid": { + "minimum": 0, + "type": "integer" + }, + "window_id": { + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "kind", + "pid", + "window_id" + ], + "type": "object" + }, + { + "properties": { + "display_id": { + "const": "primary" + }, + "kind": { + "const": "primary_desktop", + "type": "string" + } + }, + "required": [ + "kind", + "display_id" + ], + "type": "object" + } + ] + } + }, + "required": [ + "capture_id", + "source", + "screenshot", + "action_coordinate_space" + ], + "type": "object" + }, + "parser": { + "properties": { + "backend": { + "type": [ + "string", + "null" + ] + }, + "extension_id": { + "type": "string" + }, + "extension_version": { + "type": "string" + }, + "fixture_sha256": { + "type": [ + "string", + "null" + ] + }, + "model_id": { + "type": "string" + }, + "model_manifest_sha256": { + "type": [ + "string", + "null" + ] + }, + "model_source_revision": { + "type": [ + "string", + "null" + ] + }, + "model_version": { + "type": "string" + }, + "onnx_runtime_library_sha256": { + "type": [ + "string", + "null" + ] + }, + "onnx_runtime_version": { + "type": [ + "string", + "null" + ] + }, + "runtime": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "extension_id", + "extension_version", + "model_id", + "model_version" + ], + "type": "object" + }, + "regions": { + "items": { + "properties": { + "bounds": { + "properties": { + "height": { + "minimum": 1, + "type": "integer" + }, + "width": { + "minimum": 1, + "type": "integer" + }, + "x": { + "minimum": 0, + "type": "integer" + }, + "y": { + "minimum": 0, + "type": "integer" + } + }, + "required": [ + "x", + "y", + "width", + "height" + ], + "type": "object" + }, + "confidence": { + "maximum": 1, + "minimum": 0, + "type": "number" + }, + "group_id": { + "type": [ + "string", + "null" + ] + }, + "id": { + "type": "string" + }, + "interactive": { + "type": "boolean" + }, + "kind": { + "enum": [ + "text", + "icon" + ], + "type": "string" + }, + "label": { + "type": [ + "string", + "null" + ] + }, + "parent_id": { + "type": [ + "string", + "null" + ] + }, + "reading_order": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "text": { + "type": [ + "string", + "null" + ] + } + }, + "required": [ + "id", + "kind", + "bounds", + "confidence", + "interactive" + ], + "type": "object" + }, + "type": "array" + }, + "request_id": { + "type": [ + "string", + "null" + ] + }, + "schema": { + "const": "cua.visual_regions_v1" + }, + "timing": { + "properties": { + "duration_ms": { + "minimum": 0, + "type": "integer" + }, + "inference_ms": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + }, + "preprocess_ms": { + "minimum": 0, + "type": [ + "integer", + "null" + ] + } + }, + "required": [ + "duration_ms" + ], + "type": "object" + }, + "warnings": { + "items": { + "properties": { + "code": { + "type": "string" + }, + "detail": { + "type": [ + "string", + "null" + ] + }, + "message": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + }, + "type": "array" + } + }, + "required": [ + "schema", + "capture", + "parser", + "regions", + "timing" + ], + "type": "object" + }, + { + "additionalProperties": true, + "anyOf": [ + { + "required": [ + "refusal" + ] + }, + { + "required": [ + "status" + ] + }, + { + "required": [ + "code" + ] + } + ], + "type": "object" + } + ], + "type": "object" + } + } + ], + "resources": [ + { + "uri": "skill://cua-driver/BROWSER.md", + "name": "BROWSER.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/EMBEDDING.md", + "name": "EMBEDDING.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/LINUX.md", + "name": "LINUX.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/MACOS.md", + "name": "MACOS.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/README.md", + "name": "README.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/RECORDING.md", + "name": "RECORDING.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/RUNTIME.md", + "name": "RUNTIME.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/SKILL.md", + "name": "cua-driver", + "description": "Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server; snapshot its accessibility tree, act through snapshot-bound element tokens, native menu paths, exact window geometry, or pixel coordinates, and verify from fresh state. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host, or to continue, resume, or recall recent Cua activity.", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/VISUAL.md", + "name": "VISUAL.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/WORKFLOW.md", + "name": "WORKFLOW.md", + "mimeType": "text/markdown" + }, + { + "uri": "skill://cua-driver/WINDOWS.md", + "name": "WINDOWS.md", + "mimeType": "text/markdown" + } + ] +} diff --git a/tst/unit/drivers/cua_driver_test.py b/tst/unit/drivers/cua_driver_test.py index f174573be..a5ecec3e0 100644 --- a/tst/unit/drivers/cua_driver_test.py +++ b/tst/unit/drivers/cua_driver_test.py @@ -24,7 +24,7 @@ import agent_env from agentenv_protocol import SESSION_META_KEY -from agentenv_protocol.computer import DeliveryFailed, DeviceUnavailable, Unsupported +from agentenv_protocol.computer import DeliveryFailed, DeviceUnavailable, Platform, Unsupported DRIVER = Path(agent_env.__file__).parent / "drivers" / "envs" / "cua-driver" / "driver.py" @@ -38,7 +38,10 @@ def _png(width: int, height: int) -> str: class FakeDaemon: """cua-driver's daemon as the driver sees it, on ``path``: each request by the connection it came on, ``refuse`` naming tools whose results are errors, ``fail`` tools it can't call, ``slow`` tools it answers after a second, - ``drop`` tools whose connection it closes without answering, and ``tools`` what ``list`` lists.""" + ``drop`` tools whose connection it closes without answering, ``grab`` a layer such as a menu holding the keyboard, + from which it refuses a key aimed at a window, ``unlaid`` the seat's keymap changed, for which it refuses one too + until a chord at the desktop of the layout's keys puts it back, which ``relaid`` false keeps it from, and ``tools`` + what ``list`` lists.""" def __init__(self, manifest: list[dict], path: Path) -> None: self.path = path @@ -47,6 +50,8 @@ def __init__(self, manifest: list[dict], path: Path) -> None: self.fail: set[str] = set() self.slow: set[str] = set() self.drop: set[str] = set() + self.grab = self.unlaid = False + self.relaid = True self.tools = [{"name": t["name"], "input_schema": t["inputSchema"]} for t in manifest] self.connections = 0 self.writers: list[asyncio.StreamWriter] = [] @@ -85,6 +90,12 @@ def reply(self, request: dict) -> dict: return {"ok": False, "error": f"unknown tool {name}", "exit_code": 64} if name in self.refuse: return {"ok": True, "result": {"isError": True, "content": [{"type": "text", "text": f"{name} refused"}]}} + if name == "press_key" and "pid" in request["args"] and (self.grab or self.unlaid): + why = "(primary_target_busy): foreground_grab" if self.grab else "(unsupported_layout): unsupported_layout" + return {"ok": True, "result": {"isError": True, "content": [ + {"type": "text", "text": f"foreground_unavailable {why}"}]}} + if name == "press_key" and "super" in request["args"]["modifiers"] and self.relaid: + self.unlaid = False if name == "list_windows": return {"ok": True, "result": {"content": [{"type": "text", "text": "Found 0 windows"}], "structuredContent": {"windows": []}}} @@ -464,16 +475,25 @@ def daemon_down(path, mode, buffering): class FakeHyprland: """Hyprland's IPC as ``hyprctl -j`` answers: an answer by request, a list of them answered one per ask, None for a - request it has no answer to, and every request asked, in order.""" + request it has no answer to, and every request asked, in order; ``binds`` what it lists as bound, and ``lua`` the + Lua it was given (``hyprctl eval``), a bind in which it then lists unless ``binding`` is false.""" def __init__(self, answers: dict): - self.answers, self.asked = answers, [] + self.answers, self.asked, self.binds, self.lua, self.binding = answers, [], [], [], True - def __call__(self, request: str) -> dict | None: + def __call__(self, request: str) -> dict | list | None: self.asked.append(request) + if request == "binds": + return self.binds answer = self.answers.get(request) return answer.pop(0) if isinstance(answer, list) else answer + def run_lua(self, code: str) -> None: + self.lua.append(code) + if self.binding and code.startswith('hl.bind("SUPER + CTRL + ALT + SHIFT + F12"'): + self.binds.append({"modmask": 77, "key": "F12", "dispatcher": "__lua", "arg": str(300 + len(self.lua)), + "non_consuming": False}) + @pytest.fixture def hyprland(module, monkeypatch) -> FakeHyprland: @@ -483,33 +503,156 @@ def hyprland(module, monkeypatch) -> FakeHyprland: monkeypatch.setattr(module, name, value) fake = FakeHyprland({"cua:status": PLUGIN_LOADED, "activeworkspace": {"id": 1}}) monkeypatch.setattr(module.hyprland, "hyprctl", fake) + monkeypatch.setattr(module.hyprland, "lua", fake.run_lua) return fake @pytest.mark.asyncio -async def test_on_hyprland_keys_go_to_the_focused_window_in_the_foreground_and_text_skips_xs_keymap( +async def test_on_hyprland_text_goes_to_the_desktop_and_keys_to_the_focused_window_as_on_a_us_keyboard( module, daemon, hyprland): hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} driver = module.CuaDriver() await driver.type_in("Hello, Wörld") - await driver.press_keys(["Control", "a"]) + for keys in (["Control", "a"], ["Backspace"], ["Control", "+"], ["Shift", "?"], ["A"], ["/"]): + await driver.press_keys(keys) await driver.click_at(10, 20, "left", 1, []) focused = {"pid": 42, "window_id": 0x5557E0F44AD0, "delivery_mode": "foreground"} assert daemon.calls() == [ - ("type_text", {"scope": "desktop", "text": "Hello, Wörld", **focused}), + ("type_text", {"scope": "desktop", "text": "Hello, Wörld", "delivery_mode": "foreground"}), ("press_key", {"scope": "desktop", "key": "a", "modifiers": ["ctrl"], **focused}), + ("press_key", {"scope": "desktop", "key": "backspace", "modifiers": [], **focused}), + ("press_key", {"scope": "desktop", "key": "=", "modifiers": ["ctrl", "shift"], **focused}), + ("press_key", {"scope": "desktop", "key": "/", "modifiers": ["shift"], **focused}), + ("press_key", {"scope": "desktop", "key": "a", "modifiers": ["shift"], **focused}), + ("press_key", {"scope": "desktop", "key": "/", "modifiers": [], **focused}), ("click", {"scope": "desktop", "x": 10, "y": 20, "button": "left", "count": 1, "modifier": []}), ] - assert driver._keymap is None and hyprland.asked == ["activewindow", "cua:status"] * 2 + ["cua:status"] + assert driver._keymap is None + assert hyprland.asked == ["cua:status"] + ["activewindow", "cua:status"] * 6 + ["cua:status"] + + +@pytest.mark.asyncio +async def test_on_hyprland_super_and_keys_held_with_it_go_to_the_desktop_as_hyprlands_own(module, daemon, hyprland): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + driver = module.CuaDriver() + + await driver.press_keys(["Meta", "2"]) + await driver.press_keys(["Super"]) + + assert daemon.calls() == [ + ("press_key", {"scope": "desktop", "key": "2", "modifiers": ["super"], "delivery_mode": "foreground"}), + ("press_key", {"scope": "desktop", "key": "super", "modifiers": [], "delivery_mode": "foreground"})] + assert "activewindow" not in hyprland.asked + + +@pytest.mark.asyncio +async def test_on_hyprland_with_nothing_focused_keys_go_to_the_desktop_punctuation_by_its_keysym_name( + module, daemon, hyprland): + for keys in (["Escape"], ["Control", "/"], ["!"]): + await module.CuaDriver().press_keys(keys) + + assert daemon.calls() == [ + ("press_key", {"scope": "desktop", "key": "escape", "modifiers": [], "delivery_mode": "foreground"}), + ("press_key", {"scope": "desktop", "key": "slash", "modifiers": ["ctrl"], "delivery_mode": "foreground"}), + ("press_key", {"scope": "desktop", "key": "exclam", "modifiers": [], "delivery_mode": "foreground"})] + + +@pytest.mark.asyncio +async def test_on_hyprland_a_key_refused_while_a_menu_holds_the_keyboard_goes_to_the_desktop(module, daemon, hyprland): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + daemon.grab = True + + for keys in (["?"], ["Escape"]): + await module.CuaDriver().press_keys(keys) + + focused = {"pid": 42, "window_id": 0x5557E0F44AD0, "delivery_mode": "foreground"} + assert daemon.calls() == [ + ("press_key", {"scope": "desktop", "key": "/", "modifiers": ["shift"], **focused}), + ("press_key", {"scope": "desktop", "key": "question", "modifiers": [], "delivery_mode": "foreground"}), + ("press_key", {"scope": "desktop", "key": "escape", "modifiers": [], **focused}), + ("press_key", {"scope": "desktop", "key": "escape", "modifiers": [], "delivery_mode": "foreground"})] + + +RESET = ("press_key", {"scope": "desktop", "key": "f12", "modifiers": ["super", "ctrl", "alt", "shift"], + "delivery_mode": "foreground"}) + + +@pytest.mark.asyncio +async def test_on_hyprland_a_key_refused_for_the_seats_keymap_goes_again_once_a_chord_bound_to_nothing_puts_it_back( + module, daemon, hyprland): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + driver = module.CuaDriver() + + for keys in (["Backspace"], ["Control", "a"]): + daemon.unlaid = True + await driver.press_keys(keys) + hyprland.binds.clear() + daemon.unlaid = True + await driver.press_keys(["Backspace"]) + + window = {"scope": "desktop", "pid": 42, "window_id": 0x5557E0F44AD0, "delivery_mode": "foreground"} + backspace = ("press_key", {"key": "backspace", "modifiers": [], **window}) + select_all = ("press_key", {"key": "a", "modifiers": ["ctrl"], **window}) + assert daemon.calls() == [backspace, RESET, backspace, select_all, RESET, select_all, backspace, RESET, backspace] + assert hyprland.lua == ['hl.bind("SUPER + CTRL + ALT + SHIFT + F12", function() end)'] * 2 @pytest.mark.asyncio -async def test_on_hyprland_with_nothing_focused_keys_go_to_the_desktop(module, daemon, hyprland): - await module.CuaDriver().press_keys(["Escape"]) +async def test_on_hyprland_a_key_refused_for_the_seats_keymap_fails_without_the_chord_while_another_bind_has_it( + module, daemon, hyprland): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + driver = module.CuaDriver() + daemon.unlaid = True + await driver.press_keys(["Backspace"]) + for theirs in ([{"modmask": 77, "key": "F12", "dispatcher": "__lua", "arg": "7", "non_consuming": False}], + [*hyprland.binds, {"modmask": 77, "key": "F12", "dispatcher": "exec", "arg": "reboot"}]): + hyprland.binds[:] = theirs + daemon.unlaid = True + with pytest.raises(DeliveryFailed, match="unsupported_layout") as refused: + await driver.press_keys(["Backspace"]) + assert "bound to something this driver didn't bind" in refused.value.hint - assert daemon.calls() == [("press_key", {"scope": "desktop", "key": "escape", "modifiers": []})] + assert [args.get("pid") for _, args in daemon.calls()] == [42, None, 42, 42, 42] and len(hyprland.lua) == 1 + + +@pytest.mark.asyncio +async def test_on_hyprland_a_key_refused_for_the_seats_keymap_fails_without_the_chord_when_hyprland_wont_bind_it( + module, daemon, hyprland): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + hyprland.binding, daemon.unlaid = False, True + + with pytest.raises(DeliveryFailed, match="unsupported_layout"): + await module.CuaDriver().press_keys(["Backspace"]) + + assert [args.get("pid") for _, args in daemon.calls()] == [42] and len(hyprland.lua) == 1 + + +@pytest.mark.asyncio +async def test_on_hyprland_a_key_still_refused_for_the_seats_keymap_after_the_chord_fails_in_time( + module, daemon, hyprland, monkeypatch): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + daemon.unlaid, daemon.relaid = True, False + monkeypatch.setattr(module.hyprland, "RESET_S", 0.2) + + with pytest.raises(DeliveryFailed, match="unsupported_layout"): + await module.CuaDriver().press_keys(["Backspace"]) + + calls = daemon.calls() + assert calls[1] == RESET and len(calls) > 3 and all(args.get("pid") == 42 for _, args in calls[:1] + calls[2:]) + + +@pytest.mark.asyncio +async def test_on_hyprland_a_key_the_focused_window_refuses_otherwise_fails_without_going_to_the_desktop( + module, daemon, hyprland): + hyprland.answers["activewindow"] = {"address": "0x5557e0f44ad0", "pid": 42} + daemon.refuse.add("press_key") + + with pytest.raises(DeliveryFailed, match="cua-driver press_key: press_key refused"): + await module.CuaDriver().press_keys(["Backspace"]) + + assert [args.get("pid") for _, args in daemon.calls()] == [42] @pytest.mark.asyncio @@ -532,10 +675,13 @@ async def test_on_hyprland_input_is_refused_and_not_sent_once_hyprland_runs_with @pytest.mark.asyncio -async def test_on_hyprland_readiness_checks_every_time_that_hyprland_has_cua_drivers_plugin( +async def test_on_hyprland_readiness_checks_its_own_pin_and_every_time_that_hyprland_has_cua_drivers_plugin( module, daemon, hyprland, monkeypatch): driver = module.CuaDriver() - monkeypatch.setattr(module, "_executable_sha256", lambda: module.EXECUTABLES[driver.platform.arch]) + driver.platform = Platform(os="linux", arch="amd64", display="wayland") + digests = iter([module.EXECUTABLES["amd64"], module.EXECUTABLES["hyprland/amd64"]]) + monkeypatch.setattr(module, "_executable_sha256", lambda: next(digests)) + x11 = await driver.ready_checks() hyprland.answers.update({ "cua:status": [PLUGIN_LOADED, None, {**PLUGIN_LOADED, "configured": False}, {**PLUGIN_LOADED, "abi": {"match": False}}, PLUGIN_RELOADED, None], @@ -543,6 +689,7 @@ async def test_on_hyprland_readiness_checks_every_time_that_hyprland_has_cua_dri checks = [await driver.ready_checks() for _ in range(6)] + assert (x11[0].name, x11[0].ok) == ("cua-driver", False) assert [(c.name, c.ok) for c in checks[0]] == [("cua-driver", True), ("cua-driver tools", True), ("cua-driver plugin", True)] assert [(c[-1].ok, c[-1].detail) for c in checks[1:]] == [ @@ -553,7 +700,7 @@ async def test_on_hyprland_readiness_checks_every_time_that_hyprland_has_cua_dri "loading a replacement plugin"), (False, "Hyprland doesn't answer on its IPC; if it restarted after this driver started, restart the driver " "with it")] - assert sum(1 for _, r in daemon.requests if r["method"] == "list") == 1 + assert sum(1 for _, r in daemon.requests if r["method"] == "list") == 2 @pytest.mark.usefixtures("hyprland")