Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/scripts/infra_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@
# Each image's check, beyond its build: the gateway imports its server, Postgres turns healthy, the sidecars and the
# browser answer their own commands, Claude Code's, Codex's, Gemini's, Hermes', OpenCode's and pi's agents run their
# CLIs and import their servers, OpenHands' imports its server, which loads the OpenHands SDK, Pydantic AI's imports its
# server, which loads Pydantic AI, Browser Use's loads Browser Use Pi and imports its server, the proxy imports its
# server, which loads Pydantic AI, Browser Use's loads browser-use in its own environment and imports its server, the proxy imports its
# server, the mail example's env imports its server, and the base computer's desktop comes up, every program running.
_SMOKES = {
"agentenv-gateway": ["docker", "run", "--rm", "--entrypoint", "python", "{image}", "-c", "import gateway.entrypoint"],
Expand All @@ -103,7 +103,7 @@
"agentenv-claude-code": ["docker", "run", "--rm", "--entrypoint", "sh", "{image}", "-c",
'claude --version && python -c "import a2a_server"'],
"agentenv-browser-use": ["docker", "run", "--rm", "--entrypoint", "sh", "{image}", "-c",
'node -e "import(\'@browser_use/pi\')" && python -c "import a2a_server"'],
'/opt/browser-use/bin/python -c "import agent" && python -c "import a2a_server"'],
"agentenv-codex": ["docker", "run", "--rm", "--entrypoint", "sh", "{image}", "-c",
'codex --version && mcp-remote --help >/dev/null && python -c "import a2a_server"'],
"agentenv-gemini": ["docker", "run", "--rm", "--entrypoint", "sh", "{image}", "-c",
Expand Down
13 changes: 7 additions & 6 deletions src/agent_env/agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,13 @@ a relay that adds task attribution, and takes its model from the prompt, else `o
`install/v1` too, into a container or onto a Linux host with glibc. Its folder's `README.md` covers the extensions
it implements. Run its smoke task with `agent-env run agents --task letta-code-smoke`.

`agents/browser-use/` runs [Browser Use Pi](https://github.com/browser-use/browser-use-pi) behind the protocol's A2A
agent framework: each turn is one `node agent.mjs` run that drives the browser the task gives it through
`cdp-config/v1`, with the task env's MCP tools beside it, and a context's turns continue one conversation. Name it as
`browser-use`, its alias, or by its id, `@local/agentenv-framework/agents/browser-use`. It calls the `[model]`
endpoint as an OpenAI-compatible one and takes its model from the prompt. Its folder's `README.md` covers the extensions it implements. Run its smoke task with
`agent-env run agents --task browser-use-smoke`.
`agents/browser-use/` runs [browser-use](https://github.com/browser-use/browser-use) 1.0 behind the protocol's A2A
agent framework: each turn is one `agent.py` run, in browser-use's own environment, that drives the browser the task
gives it through `cdp-config/v1`, with the task env's MCP tools beside it, and a context's turns continue one
conversation. Name it as `browser-use`, its alias, or by its id, `@local/agentenv-framework/agents/browser-use`. It
takes its model from the prompt and calls the `[model]` endpoint through its Anthropic Messages route for a Claude
model, as browser-use runs Claude, and as an OpenAI-compatible one for any other. Its folder's `README.md` covers the
extensions it implements. Run its smoke task with `agent-env run agents --task browser-use-smoke`.

`agents/openhands/` runs the [OpenHands](https://github.com/OpenHands/software-agent-sdk) agent behind the
protocol's A2A agent framework, driving the OpenHands SDK in process rather than a CLI: each turn is one run of a
Expand Down
17 changes: 7 additions & 10 deletions src/agent_env/agents/agents/browser-use/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,16 +1,13 @@
# docker build -t browser-use-a2a src/agent_env/agents/agents/browser-use
FROM mirror.gcr.io/library/node:22-bookworm-slim@sha256:4b150b48b4116e2a22f2c39cc27025fbf4f153e2078fd3b14d159fdb5c7279da AS node
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --ignore-scripts --omit=dev --no-audit --no-fund --loglevel=error

FROM mirror.gcr.io/library/python:3.12-slim-bookworm@sha256:e697c81897302a4c4333bade000d98038fbb21eb064f8ee1fd42d02ebcd9d337
COPY --from=node /usr/local/bin/node /usr/local/bin/node
WORKDIR /app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt && useradd --create-home agent
COPY --from=node /app/node_modules node_modules
COPY a2a_server.py agent.mjs package.json ./
COPY requirements.txt browser-use-requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt \
&& python -m venv /opt/browser-use \
&& /opt/browser-use/bin/pip install --no-cache-dir -r browser-use-requirements.txt \
&& useradd --create-home agent
COPY a2a_server.py agent.py ./
USER agent
ENV ANONYMIZED_TELEMETRY=false BH_TELEMETRY=0
EXPOSE 8000
CMD ["python3", "a2a_server.py"]
54 changes: 37 additions & 17 deletions src/agent_env/agents/agents/browser-use/README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,36 @@
# Browser Use A2A agent

[Browser Use Pi](https://github.com/browser-use/browser-use-pi) wrapped with `agentenv_protocol.a2a_agent`, shipped in
the `agents` bundle as `@local/agentenv-framework/agents/browser-use`, with alias `browser-use`. Browser Use Pi drives
a real browser over CDP from a persistent JavaScript REPL. `package-lock.json` pins `@browser_use/pi` to 0.1.0, and
`package.json` overrides the `undici` and `brace-expansion` it pins with their patched releases.
[browser-use](https://github.com/browser-use/browser-use) 1.0 wrapped with `agentenv_protocol.a2a_agent`, shipped in
the `agents` bundle as `@local/agentenv-framework/agents/browser-use`, with alias `browser-use`. browser-use drives a
real browser over CDP from a persistent Python REPL. `browser-use-requirements.txt` pins it to 1.0.3, which pins every
dependency of its own exactly, `mcp` 2 among them, so the image installs it in an environment of its own,
`/opt/browser-use`, beside the A2A server's.

Each turn runs `node agent.mjs` once. It calls the `[model]` endpoint as an OpenAI-compatible one
(`LITELLM_BASE_URL`, `LITELLM_API_KEY`, `LITELLM_EXTRA_HEADERS`), and the task env's MCP tools are the agent's tools
next to the browser, named `<server>__<tool>`.
- **The browser** is the one the task gives it through `cdp-config/v1`, as the `browser_connect_cdp` step does: the endpoint `browser` names, else the turn's only one.
- **Connecting:** it connects by the endpoint's `ws_url`, or by `http_url` with `connect_by = "http"`.
- **Its tab:** when the browser has one tab open, as a browser computer does, the agent drives that tab, which is the computer's screen, so its work shows there and in a recording. With several, Browser Use opens a tab of its own.
- **With no endpoint,** or several and none named, the turn fails with `cdp.no_endpoint` or `cdp.ambiguous`, an `infra_error`. It never drives a browser of its own.
- **An endpoint with headers** fails the turn with `browser_use.cdp_headers`, an `infra_error`, since Browser Use Pi connects by the address alone.
- **Across turns:** a context's turns continue one conversation through a saved history file, and each turn connects afresh.
Each turn runs `agent.py` once in that environment.
- **The model** is the prompt's, on the `[model]` endpoint (`LITELLM_BASE_URL`, `LITELLM_API_KEY`,
`LITELLM_EXTRA_HEADERS`). A Claude model goes through the endpoint's Anthropic Messages route, as browser-use runs
Claude: `effort` is its adaptive thinking's, medium unset, browser-use's default, and its prompt is cached. Its
profile comes from the bare model name, since a provider prefix (`anthropic/claude-...`) hides the model's adaptive
thinking from Pydantic AI. Any other model goes through the OpenAI-compatible route, with `effort` sent as
`reasoning_effort` and none sent unset.
- **The browser** is the one the task gives it through `cdp-config/v1`, as the `browser_connect_cdp` step does: the
endpoint `browser` names, else the turn's only one. It connects by the endpoint's `ws_url`, or by `http_url` with
`connect_by = "http"`.
- **Its tab:** when the browser has one tab open, as a browser computer does, the agent drives that tab, which is the
computer's screen, so its work shows there and in a recording, and stays there when the turn ends. With several,
browser-use opens a tab of its own. browser-use closes a tab of its own when it stops, so its last page would be out
of reach after the turn. Either way the browser keeps running.
- **With no endpoint,** or several and none named, the turn fails with `cdp.no_endpoint` or `cdp.ambiguous`, an
`infra_error`. It never drives a browser of its own.
- **An endpoint with headers** fails the turn with `browser_use.cdp_headers`, an `infra_error`, since browser-use
connects by the address alone.
- **The task env's MCP tools** are the agent's tools beside its REPL, named `<server>__<tool>`.
- **Across turns:** a context's turns continue one conversation through a saved history file, and each turn connects
afresh.
- **Limits:** a turn makes at most `max_steps` model requests (browser-use's 100 unset), its delivery included, and
browser-use's run ends `DELIVERY_SECONDS` before the turn's `timeout_seconds`, which leaves it time to deliver.
- **Telemetry:** off. The image and each turn set `ANONYMIZED_TELEMETRY=false` and `BH_TELEMETRY=0`, so neither
browser-use nor its browser daemon reports anything.

It takes its model from the prompt, so a run needs `[model] default` (or the `agent` role) set. The smoke task deploys
`chromium` from the `browsers` bundle beside it, connects the agent to it, and has it read a synthetic page's title:
Expand All @@ -24,10 +41,13 @@ agent-env run agents --task browser-use-smoke

| Extension | Implementation |
|---|---|
| `agent-config/v1` | `model` (else `BROWSER_USE_A2A_MODEL`), `effort`, `system_prompt` (added to Browser Use Pi's instructions), `max_steps`, `browser`, `connect_by` (`ws` or `http`), `timeout_seconds` |
| `agent-config/v1` | `model` (else `BROWSER_USE_A2A_MODEL`), `effort` (`low`, `medium` or `high`), `mode` (`default` or `ultrafast`, browser-use's accessibility-tree helpers), `system_prompt` (added to browser-use's instructions), `max_steps`, `browser`, `connect_by` (`ws` or `http`), `timeout_seconds` |
| `cdp-config/v1` | SDK default; each turn reads its endpoint with `request.cdp_endpoint(browser)` |
| `mcp-config/v1` | SDK default; each server's tools are listed per turn and called over streamable HTTP |
| `trajectory/v1` | Browser Use Pi's agent events, live, format `browser-use-pi-events/1`, without the streamed `message_update` / `tool_execution_update` deltas |
| `trajectory/v1` | browser-use's messages, as Pydantic AI writes them, each turn's when it ends, format `browser-use-messages/1` |

A run that stops early (`max_steps`, `timeout`, `cost_limit`, ...) fails with code `browser_use.<status>` and its
last text. Skills, snapshots, peers and `install/v1` are not implemented, so the validator reports them unsupported.
A turn's usage reports browser-use's tokens, cached ones among its input, with its model requests, cache reads and
writes, and its cost when browser-use knows the model's price. A run that ends without delivering (`max_steps`,
`timeout`, `error`) fails with code `browser_use.<status>` and its error. One that reaches `max_steps` or its timeout
gets a forced delivery turn first, as browser-use does, and completes with that answer. Skills, snapshots, peers and
`install/v1` are not implemented, so the validator reports them unsupported.
Loading
Loading