Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,16 @@ A package of your own can add env types, providers, task steps and stores to age

Every task in a bundle is an [OpenEnv](https://github.com/openenv/OpenEnv) env, with the `openenv` extra installed (`pip install 'agentenv-framework[openenv]'`). `agent_env.openenv.build_app(bundle)` returns the OpenEnv server for a bundle, a folder or an installed bundle's name, or for a mapping of split names to bundles. Any OpenEnv client resets it to a task, which deploys the task's envs and opens its agent's turn, acts through the envs' tools and an added `end_turn` tool, and gets the task's verifier score as its reward.

An agent a task installs into a container (`install_agent` with `container_name`) is played through AgentEnv's proxy, which an episode installs there in its place. A task image can carry the proxy, so an episode only starts it: copy the installed `agent_env` package's `agents/agents/proxy` folder into the image's build context as `agentenv-proxy`, and end the Dockerfile with

```dockerfile
COPY --from=ghcr.io/astral-sh/uv:0.13.1@sha256:4a1f3666816dafadd20ceb71eac9a7d56d7ef30279f7d64048519ab901ada336 /uv /opt/agentenv-proxy/bin/uv
COPY agentenv-proxy /opt/agentenv-proxy/src
RUN sh /opt/agentenv-proxy/src/bake.sh
```

A container whose baked proxy holds other requirements than this agent-env's proxy gets the proxy installed afresh. Name the image as `run_docker_container`'s `image_artifact_id`, so it's built once, not every episode.

## Time zones

Servers and the env clock work in UTC. To run an agent in a user's time zone, the task sets it on the agent's `deploy_agent` step as `env_vars: {"TZ": "America/New_York"}`, which reaches the agent's sandbox, and names the zone in the agent's prompt. While the env clock is armed, the gateway's `get_time` tool takes that IANA name as `time_zone` and adds the `local_time` in that zone, and a trigger executor's working `hours` can take a `time_zone` instead of a fixed `utc_offset`, so they keep their local hours across daylight-saving changes.
Expand Down
21 changes: 21 additions & 0 deletions src/agent_env/agents/agents/proxy/bake.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/sh
# Bakes the proxy into an image as it's built, so an install into one of its containers only starts it: a private
# Python and a venv holding requirements.txt in /opt/agentenv-proxy, and a marker naming the requirements it holds,
# which install.sh checks. Plain sh, for images with no bash. A task image's Dockerfile ends with
#
# COPY --from=ghcr.io/astral-sh/uv:0.13.1@sha256:4a1f3666816dafadd20ceb71eac9a7d56d7ef30279f7d64048519ab901ada336 /uv /opt/agentenv-proxy/bin/uv
# COPY agentenv-proxy /opt/agentenv-proxy/src
# RUN sh /opt/agentenv-proxy/src/bake.sh
#
# where agentenv-proxy is this folder, copied into the image's build context.
set -eu

here=$(cd "$(dirname "$0")" && pwd)
. "$here/versions.sh"

export UV_PYTHON_PREFERENCE=only-managed UV_PYTHON_INSTALL_DIR="$BAKED/python" UV_CACHE_DIR="$BAKED/cache"
"$BAKED/bin/uv" venv --quiet --python "$PYTHON_VERSION" "$BAKED/venv"
"$BAKED/bin/uv" pip install --quiet --python "$BAKED/venv/bin/python" -r "$here/requirements.txt"
rm -rf "$BAKED/cache"
"$BAKED/venv/bin/python" -c 'import hashlib, sys; print(hashlib.sha256(open(sys.argv[1], "rb").read()).hexdigest())' \
"$here/requirements.txt" > "$BAKED/baked"
55 changes: 31 additions & 24 deletions src/agent_env/agents/agents/proxy/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,14 @@
# install.sh container NAME PORT into the running container NAME, as root, in its working directory
#
# On a host it runs in a venv of the host's own Python, which apt installs when the host has none. In a container it
# brings uv, checked against its published checksum, and a private Python, so the container's own are never touched.
# brings uv, checked against its published checksum and kept on the host, and a private Python, so the container's own
# are never touched; in one whose image bake.sh baked the proxy into, for these requirements, it only starts it.
set -euo pipefail

UV_VERSION=0.13.1
UV_SHA256_X86_64=6c86040a8f045ab9adc9e3ec4cdafa03fd00e9a988ed2bf28971c7fc2d0debd0
UV_SHA256_AARCH64=345a8308e4d1835cdfaadf692836fd3137a6d45864773f05ad7bf59442b1ba16
PYTHON_VERSION=3.12

mode=$1 name=$2 port=$3
here=$(cd "$(dirname "$0")" && pwd)
. "$here/versions.sh"
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
sha256() { if command -v sha256sum >/dev/null; then sha256sum "$1"; else shasum -a 256 "$1"; fi | cut -d' ' -f1; }

case $mode in
host)
Expand All @@ -40,28 +38,37 @@ case $mode in
target=$name
root=/opt/agentenv-proxy-$port # one per port: two agents in a container listen on two
on() { docker exec -u 0 "$target" sh -c "$1"; }
case $(on 'uname -m') in
x86_64|amd64) uv_cpu=x86_64 uv_sha256=$UV_SHA256_X86_64 ;;
aarch64|arm64) uv_cpu=aarch64 uv_sha256=$UV_SHA256_AARCH64 ;;
*) echo "uv has no build for $(on 'uname -m')" >&2; exit 1 ;;
esac
stage=$(mktemp -d "$here/../stage.XXXXXX")
trap 'rm -rf "$stage"' EXIT
uv_archive=uv-$uv_cpu-unknown-linux-musl.tar.gz
curl -fsSL --retry 3 "https://github.com/astral-sh/uv/releases/download/$UV_VERSION/$uv_archive" -o "$stage/$uv_archive"
sum=$(if command -v sha256sum >/dev/null; then sha256sum "$stage/$uv_archive"; else shasum -a 256 "$stage/$uv_archive"; fi)
[ "${sum%% *}" = "$uv_sha256" ] || { echo "$uv_archive doesn't match its published checksum" >&2; exit 1; }
tar -xzf "$stage/$uv_archive" -C "$stage"
# An earlier install's proxy, a retried step's say, stops first, so its port is free.
on "if [ -f $root/proxy.pid ]; then kill \$(cat $root/proxy.pid) 2>/dev/null || true; fi; rm -rf $root; mkdir -p $root/bin"
docker cp "$stage/uv-$uv_cpu-unknown-linux-musl/uv" "$target:$root/bin/uv" >/dev/null
if [ "$(on "cat $BAKED/baked 2>/dev/null" || true)" = "$(sha256 "$here/requirements.txt")" ]; then
python=$BAKED/venv/bin/python
else
case $(on 'uname -m') in
x86_64|amd64) uv_cpu=x86_64 uv_sha256=$UV_SHA256_X86_64 ;;
aarch64|arm64) uv_cpu=aarch64 uv_sha256=$UV_SHA256_AARCH64 ;;
*) echo "uv has no build for $(on 'uname -m')" >&2; exit 1 ;;
esac
stage=$(mktemp -d "$here/../stage.XXXXXX")
trap 'rm -rf "$stage"' EXIT
uv_archive=uv-$uv_cpu-unknown-linux-musl.tar.gz
cached=${XDG_CACHE_HOME:-${HOME:-/tmp}/.cache}/agentenv-proxy/$UV_VERSION-$uv_archive
if [ ! -f "$cached" ] || [ "$(sha256 "$cached")" != "$uv_sha256" ]; then
mkdir -p "$(dirname "$cached")"
curl -fsSL --retry 3 "https://github.com/astral-sh/uv/releases/download/$UV_VERSION/$uv_archive" -o "$stage/$uv_archive"
[ "$(sha256 "$stage/$uv_archive")" = "$uv_sha256" ] || { echo "$uv_archive doesn't match its published checksum" >&2; exit 1; }
mv -f "$stage/$uv_archive" "$cached.$$" && mv -f "$cached.$$" "$cached"
fi
tar -xzf "$cached" -C "$stage"
docker cp "$stage/uv-$uv_cpu-unknown-linux-musl/uv" "$target:$root/bin/uv" >/dev/null
docker cp "$here/requirements.txt" "$target:$root/requirements.txt" >/dev/null
on "cd $root && export UV_PYTHON_PREFERENCE=only-managed UV_PYTHON_INSTALL_DIR=$root/python UV_CACHE_DIR=$root/cache \
&& bin/uv venv --quiet --python $PYTHON_VERSION venv && bin/uv pip install --quiet --python venv/bin/python \
-r requirements.txt && rm -rf cache"
python=$root/venv/bin/python
fi
docker cp "$here/a2a_server.py" "$target:$root/a2a_server.py" >/dev/null
docker cp "$here/requirements.txt" "$target:$root/requirements.txt" >/dev/null
on "cd $root && export UV_PYTHON_PREFERENCE=only-managed UV_PYTHON_INSTALL_DIR=$root/python UV_CACHE_DIR=$root/cache \
&& bin/uv venv --quiet --python $PYTHON_VERSION venv && bin/uv pip install --quiet --python venv/bin/python \
-r requirements.txt && rm -rf cache"
docker exec -d -u 0 -e A2A_PORT="$port" "$target" \
sh -c "echo \$\$ > $root/proxy.pid; exec $root/venv/bin/python $root/a2a_server.py > $root/proxy.log 2>&1"
sh -c "echo \$\$ > $root/proxy.pid; exec $python $root/a2a_server.py > $root/proxy.log 2>&1"
;;
*) echo "usage: install.sh host NAME PORT WORKDIR USER | install.sh container NAME PORT" >&2; exit 2 ;;
esac
8 changes: 8 additions & 0 deletions src/agent_env/agents/agents/proxy/versions.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# What install.sh and bake.sh share, sourced by both. A uv bump changes UV_VERSION, both checksums and UV_IMAGE_DIGEST,
# the digest of ghcr.io/astral-sh/uv:$UV_VERSION's image index, which the bake recipe pins.
UV_VERSION=0.13.1
UV_SHA256_X86_64=6c86040a8f045ab9adc9e3ec4cdafa03fd00e9a988ed2bf28971c7fc2d0debd0
UV_SHA256_AARCH64=345a8308e4d1835cdfaadf692836fd3137a6d45864773f05ad7bf59442b1ba16
UV_IMAGE_DIGEST=sha256:4a1f3666816dafadd20ceb71eac9a7d56d7ef30279f7d64048519ab901ada336
PYTHON_VERSION=3.12
BAKED=/opt/agentenv-proxy # where bake.sh puts the proxy in an image
26 changes: 20 additions & 6 deletions src/agent_env/task_step/task_steps/install_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
import httpx

from agent_env.a2a_agent import A2AAgent
from agent_env.artifact.artifacts.docker_image import DockerImageArtifact
from agent_env.config import get_config
from agent_env.entity.scopes import FieldScope
from agent_env.providers.sandbox_providers.local_docker.provider import (
Expand All @@ -47,6 +48,9 @@
USER_HOST_KEY = "user_host"
# The per-step param that installs this agent in place of the step's own, as an episode plays it through a proxy.
PROXY_PARAM = "proxy"
# Each agent image's read of its card's install/v1 extension, by the image artifact's id and version, under which the
# image never changes.
_INSTALL_EXTENSIONS: dict[tuple[str, Optional[int]], asyncio.Future] = {}


class HostPlace(NamedTuple):
Expand Down Expand Up @@ -266,7 +270,7 @@ async def execute(self, context: TaskStepContext) -> TaskStepContext:
logger.info(f"Loading agent image {agent.docker_image_artifact.image_name} onto VM (needed to read agent card)...")
await sandbox.load_docker_images([agent.docker_image_artifact])

install_ext = await self._extract_install_extension(sandbox, agent.docker_image_artifact.image_name)
install_ext = await self._extract_install_extension(sandbox, agent.docker_image_artifact)
params = install_ext.get("params") or {}
host_mode = self.container_name is None
commands_key = INSTALL_COMMANDS_HOST_KEY if host_mode else INSTALL_COMMANDS_KEY
Expand Down Expand Up @@ -406,7 +410,7 @@ async def _host_place_of(self, sandbox, agent: A2AAgent) -> HostPlace:
)
logger.info(f"Loading agent image {agent.docker_image_artifact.image_name} onto VM (to read where it works)...")
await sandbox.load_docker_images([agent.docker_image_artifact])
install_ext = await self._extract_install_extension(sandbox, agent.docker_image_artifact.image_name)
install_ext = await self._extract_install_extension(sandbox, agent.docker_image_artifact)
params = install_ext.get("params") or {}
workdir, user = params.get(WORKDIR_HOST_KEY), params.get(USER_HOST_KEY)
if not all(isinstance(value, str) and value for value in (workdir, user)):
Expand Down Expand Up @@ -448,10 +452,20 @@ def _instance_ttl_seconds(ds) -> int:
pass
return TaskStep.DEFAULT_TTL_SECONDS

async def _extract_install_extension(self, sandbox, agent_image_name: str) -> dict:
"""Run the agent image briefly with --entrypoint python3 to dump its AGENT_CARD and find the install/v1 extension."""
from agent_env.a2a_agent import A2AAgent
async def _extract_install_extension(self, sandbox, image: DockerImageArtifact) -> dict:
"""The install/v1 extension of the card ``image`` serves, read once in this process for each image artifact
version: installs at once share one read, and one that failed, or ran on another event loop, is made again."""
key = (image.id, image.version)
read = _INSTALL_EXTENSIONS.get(key)
if (read is None or read.get_loop() is not asyncio.get_running_loop()
or (read.done() and (read.cancelled() or read.exception() is not None))):
read = asyncio.ensure_future(self._read_install_extension(sandbox, image.image_name))
_INSTALL_EXTENSIONS[key] = read
return await asyncio.shield(read)

@staticmethod
async def _read_install_extension(sandbox, agent_image_name: str) -> dict:
"""Run the agent image briefly with --entrypoint python3 to dump its AGENT_CARD and find the install/v1 extension."""
script = (
"import sys, json; sys.path.insert(0, '/app'); "
"from a2a_server import AGENT_CARD; "
Expand Down Expand Up @@ -481,7 +495,7 @@ def _check_tunnel_collision(self, context: TaskStepContext, sandbox_id: Optional
if other.sandbox_id == sandbox_id and other.api_url == a2a_url:
raise RuntimeError(f"Tunnel for port {a2a_port} ({a2a_url}) already serves deployed agent '{other.agent_name}' on sandbox {sandbox_id}; pass a distinct a2a_port on this step")

async def _wait_for_agent_card(self, a2a_url: str, timeout_s: int = 300, poll_s: int = 5) -> dict:
async def _wait_for_agent_card(self, a2a_url: str, timeout_s: int = 300, poll_s: float = 0.5) -> dict:
deadline = asyncio.get_event_loop().time() + timeout_s
last_err: Exception | None = None
async with httpx.AsyncClient() as client:
Expand Down
2 changes: 1 addition & 1 deletion tst/integration/agent/proxy_container_install_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ def test_installed_into_a_container_the_proxy_serves_and_its_bash_runs_the_shell
assert has_bash == (shell == "bash")
folder = tmp_path / "agent"
folder.mkdir()
for file in ("install.sh", "a2a_server.py"):
for file in ("install.sh", "versions.sh", "a2a_server.py"):
shutil.copy(AGENT_DIR / file, folder / file)
(folder / "requirements.txt").write_text("agentenv-framework-protocol[agent] @ file:///tmp/protocol\n")
subprocess.run(["docker", "cp", f"{PROTOCOL_DIR}/.", f"{name}:/tmp/protocol"], check=True, capture_output=True)
Expand Down
98 changes: 98 additions & 0 deletions tst/integration/agent/proxy_local_container_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
"""An agent installed into a task container on the local sandbox, played from this machine through the proxy installed
in its place: an image with the proxy baked in only starts it, one without gets uv and a venv, and both play the turn
and pass the task's tests. Needs a Docker daemon, and network access for the images, uv, Python and the proxy's
requirements."""

import shutil
import subprocess
import tempfile
import uuid
from pathlib import Path

import pytest

from agent_env.a2a_agent.a2a_agent import A2AAgent
from agent_env.artifact import DockerImageArtifact
from agent_env.artifact.store import reset_artifact_store
from agent_env.config import configure, reset_config
from agent_env.task import EpisodeSpec, Task
from tst.util.proxy_bake import BAKE_RECIPE, PROXY_DIR

pytestmark = [pytest.mark.integration, pytest.mark.int_test_slow]

TASK_DOCKERFILE = """FROM mirror.gcr.io/library/debian:bookworm-slim
RUN mkdir -p /work && echo hello > /work/greeting.txt
WORKDIR /work
"""
# An agent the proxy replaces in a container needs no more than a build context.
REPLACED_DOCKERFILE = "FROM mirror.gcr.io/library/python:3.12-slim\n"


@pytest.fixture
def local_backends(monkeypatch, tmp_path):
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "state"))
monkeypatch.setenv("AGENT_ENV_DOCUMENT_STORE", "local")
monkeypatch.setenv("AGENT_ENV_OBJECT_STORE", "local")
monkeypatch.setenv("AGENT_ENV_LOCAL_SANDBOX_DIR", str(tmp_path / "sandboxes"))
configure()
reset_artifact_store()
try:
yield tmp_path
finally:
reset_artifact_store()
reset_config()


@pytest.mark.asyncio
@pytest.mark.parametrize(("baked", "proxy"), [(True, "baked"), (False, "installed")], ids=["baked", "installed"])
async def test_an_agent_installed_into_a_task_container_is_played_through_the_proxy_in_its_place(local_backends, baked,
proxy):
uid = uuid.uuid4().hex[:8]
image = _task_image(local_backends / "task-image", uid, baked)
task = Task.from_dict({"id": f"proxy-local-{uid}", "steps": [
{"id": "box", "type": "deploy_sandbox", "sandbox_name": "box", "sandbox_mode": "vm", "sandbox_type": "local",
"exposed_ports": [8000]},
{"id": "task", "type": "run_docker_container", "sandbox_name": "box", "container_name": "task",
"image_artifact_id": image.id, "ports": [8000], "command_override": "sleep infinity"},
{"id": "install", "type": "install_agent", "sandbox_name": "box", "container_name": "task",
"a2a_agent_id": _replaced(uid).id, "agent_name": "solver"},
{"id": "solve", "type": "prompt_agent", "agent_name": "solver", "prompt_id": "solve",
"prompt": "Copy greeting.txt to answer.txt."},
{"id": "check", "type": "run_container_unit_tests_verifier", "sandbox_name": "box", "container_name": "task",
"command": "grep -qx hello /work/answer.txt"},
]})

try:
episode = await EpisodeSpec.from_task(task).start()
try:
turn = await episode.next_turn()
async with turn.tools() as tools:
where = "test -e /opt/agentenv-proxy-8000/bin/uv && echo installed || echo baked"
assert (await tools.call("bash", {"command": f"cat greeting.txt; {where}"})).text == f"hello\n{proxy}\n"
await tools.call("bash", {"command": "cp greeting.txt answer.txt"})
await turn.end("Copied.")
assert await episode.result() == {"check": 1.0}
finally:
await episode.abandon("the test ended")
finally:
subprocess.run(["docker", "rmi", "-f", image.image_name], capture_output=True)


def _task_image(context: Path, uid: str, baked: bool) -> DockerImageArtifact:
context.mkdir()
bake = "".join(f"{line}\n" for line in BAKE_RECIPE) if baked else ""
(context / "Dockerfile").write_text(TASK_DOCKERFILE + bake)
if baked:
shutil.copytree(PROXY_DIR, context / "agentenv-proxy", ignore=shutil.ignore_patterns("__pycache__"))
return DockerImageArtifact.put_context(f"proxy-task-image-{uid}", description="a task container",
context_path=str(context), dockerfile_path="Dockerfile", platform=None)


def _replaced(uid: str) -> A2AAgent:
with tempfile.TemporaryDirectory() as build_dir:
context = Path(build_dir)
(context / "Dockerfile").write_text(REPLACED_DOCKERFILE)
image = DockerImageArtifact.put_context(f"replaced-agent-image-{uid}", description="a replaced agent",
context_path=str(context), dockerfile_path="Dockerfile", platform=None)
return A2AAgent.put(id=f"replaced-agent-{uid}", docker_image_artifact=image)
16 changes: 10 additions & 6 deletions tst/integration/cli/run_bundle_task_image_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import json
import logging
import subprocess

import pytest
from click.testing import CliRunner
Expand Down Expand Up @@ -58,12 +59,15 @@ def test_a_task_container_from_an_image_folder_is_built_by_the_first_run_alone(s
builds, build = [], host_images._build
monkeypatch.setattr(host_images, "_build", lambda *args: builds.append(args[2]) or build(*args))

first = CliRunner().invoke(cli, ["run", str(root)])
try:
first = CliRunner().invoke(cli, ["run", str(root)])

assert first.exit_code == 0 and "tasks/grade.json v1: passed" in first.output, first.output
assert len(builds) == 1, (first.output, builds)
assert first.exit_code == 0 and "tasks/grade.json v1: passed" in first.output, first.output
assert len(builds) == 1, (first.output, builds)

again = CliRunner().invoke(cli, ["run", str(root)])
again = CliRunner().invoke(cli, ["run", str(root)])

assert again.exit_code == 0 and "tasks/grade.json v1: passed" in again.output, again.output
assert "building with docker" not in again.output and len(builds) == 1, (again.output, builds)
assert again.exit_code == 0 and "tasks/grade.json v1: passed" in again.output, again.output
assert "building with docker" not in again.output and len(builds) == 1, (again.output, builds)
finally:
subprocess.run(["docker", "rmi", "-f", *builds], capture_output=True) # teardown keeps it, for a next run
Loading
Loading