diff --git a/AGENTS.md b/AGENTS.md index a1875d1b9..051f6745c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -117,7 +117,7 @@ and deserialized through a registry. | Package | Contents | |---|---| -| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`, one site: a server image, `backend_image` in its env.toml, serving the pages and API, ready when its card's `website/v1` `ready` answers a `WebsiteReadiness` that says so, as `agentenv_protocol.website` (`website_readiness`, `env_readiness`) defines a website env's health for the deploy and `DeployedEnv.readiness()` alike, or, for a card without `ready`, when its `health` path answers 200, which is deprecated; the gateway container's `env/gateway/constants.py` mirrors the URI and op, a test keeping them equal; a separate frontend image is deprecated, warned of at put and optional in a bundle, and a stored `backend_docker_image_artifact` reads as the server image), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]` (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch on a builder of the put's sandbox provider, by default the env's (`utils/builder.py`; `--sandbox` on `env build`, `bundle register` and `run`): this machine's Docker for `local`, else a VM of the provider, which fetches the put's stored files by signed URL, takes the rest inline and pushes to the image store itself, so it builds only with stores a remote VM reaches; each arch goes to the first link that runs it natively, else one that emulates it (a VM registers QEMU; `--no-emulate` leaves it `skipped` instead, for a later put), and its pair records the provider that built it (`builder`); a run or register that reuses a computer env composes what it has left (`pending`), a run only the forms its sandbox's deploys boot (`booted_forms`: on each link, the first form it boots natively, by the deploy's own rule, `boot.first_native`; a compose of a disk adds its container), and a deploy composes first what it boots that no compose started, and a container it runs natively that was built but not probed, as one whose probe a setting that didn't resolve stopped, booting it only once ready (`ComputerEnv.compose_unstarted`); tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 (`utils/probing.py`) then boots each native build on the put's sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference, over its env.toml's `[settings]` and a computer env's `settings` beside it in `apps`, each by top-level field (`params.app_settings`; checked at put, a required one left to the deploy; the deploy record keeps the merged values as `app_settings`, secrets withheld); its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. Its site route, `/website//`, never forwards a site's data plane or card (`/agentenv/…`, `/.well-known/agent-env.json`), which agentenv reaches through `/svc//`; a site with no frontend image is served by its server image there. Its CDP route, `/browser//cdp//` (`env/gateway/browser_proxy.py`), reaches the website browser behind the `/svc` child ``, found from that child's card (it, or an app among its children, offering `browser/v1` `cdp` with `/cdp/` in its `computer-app/v1` `agent_routes`): it relays the browser's discovery and WebSocket both ways, each leg pinged every 20 s, and every CDP address the browser hands out, in `json/version`, `json/list` and its `cdp` answer through `/svc//`, comes back at the gateway, everything before `/cdp//` becoming the address the caller reached the gateway at and `/browser/`; each connection is a `browser_connection` trajectory event, without its token. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | +| `env/` | `Env` base class: a subclass declares `type`, implements `from_dict` and `async deploy(...) -> DeployedEnv`, and may override `reset`, `close` and `load_file_artifact_universe`; teardown closes each env a run deployed before terminating its sandboxes by id. Built-ins in `env/envs/`: `MCPServerEnv` (`mcp_server`), `MultiEnv` (`multi`, MCP servers plus websites), `ServiceDBEnv` (`service_db`, PostgreSQL), `WebsiteEnv` (`website`, one site: a server image, `backend_image` in its env.toml, serving the pages and API, ready when its card's `website/v1` `ready` answers a `WebsiteReadiness` that says so, as `agentenv_protocol.website` (`website_readiness`, `env_readiness`) defines a website env's health for the deploy and `DeployedEnv.readiness()` alike, or, for a card without `ready`, when its `health` path answers 200, which is deprecated; the gateway container's `env/gateway/constants.py` mirrors the URI and op, a test keeping them equal; a separate frontend image is deprecated, warned of at put and optional in a bundle, and a stored `backend_docker_image_artifact` reads as the server image), `GatewayEnv` (`gateway_server`), `ComputerDriverEnv` (`computer_driver`, `env/envs/computer/driver_env.py`: a computer driver's files and what its class declares for each platform it supports, read at put in a process of its own, `agentenv_protocol.computer._declare`, where the driver's own dependencies needn't be installed, with the `port` it serves on and its install by OS, `[install.]`, or for one distribution `[install."/"]`, which replaces its OS type's there, as its `known_failures` do (`env/envs/computer/utils/install.py`): files, apt `packages` with extra `sources`, a `python` requirements file for a venv, `downloads` by URL and sha256, `run` scripts, the services the base's init runs, and `ports`; it deploys only as a computer env's driver), `ComputerEnv` (`computer`, in `env/envs/computer/`: a base computer and a driver, matched at put by gate 1 before anything is stored, then composed by `composer.py` into an image for each form and arch on a builder of the put's sandbox provider, by default the env's (`utils/builder.py`; `--sandbox` on `env build`, `bundle register` and `run`): this machine's Docker for `local`, else a VM of the provider, which fetches the put's stored files by signed URL, takes the rest inline and pushes to the image store itself, so it builds only with stores a remote VM reaches; each arch goes to the first link that runs it natively, else one that emulates it (a VM registers QEMU; `--no-emulate` leaves it `skipped` instead, for a later put), and its pair records the provider that built it (`builder`); a run or register that reuses a computer env composes what it has left (`pending`), a run only the forms its sandbox's deploys boot (`booted_forms`: on each link, the first form it boots natively, by the deploy's own rule, `boot.first_native`; a compose of a disk adds its container), and a deploy composes first what it boots that no compose started, and a container it runs natively that was built but not probed, as one whose probe a setting that didn't resolve stopped, booting it only once ready (`ComputerEnv.compose_unstarted`); tagged `agentenv-computers:` in the image store by a key of the image's Dockerfile, build context and lock so any env with the same inputs reuses it; every layer's packages, requirements and downloads are resolved at put in a container of the base for each arch (`utils/packages.py`, `utils/resolve_in_base.py`), stored in the object store by sha256 under `agentenv-packages/`, and listed in a lock stored by a digest of its inputs, which any env with the same inputs reuses, and builds install only the lock's files, every install step `RUN --network=none`; each pair's status goes into its metadata, `composed`, the arches built are written as its computer, a `computer` artifact over one index, and `agent-env env build` resumes what a put left; gate 3 (`utils/probing.py`) then boots each native build on the put's sandbox provider and probes it at `full`, making it `ready` or `failed`, and a build no provider runs natively stays `unverified`. Its deploy, through the `computer` env provider (`providers/env_providers/env_computer_provider.py`), boots its first `ready` build on a sandbox provider that runs its arch natively (`native_platforms()`), with no gateway in front and the driver's card as the env card, then checks its `display`; `deploy_env` `env_params` take a `form`, `computer_settings` (deploy-phase values of the base's settings; a secret one, here or on the env, only as a bare `secret:`/`env:` reference) and `driver_settings` (the driver's `ComputerSettings`, such as its default coordinate space); a `network` that allows nothing runs it on local Docker's internal network behind a relay (`providers/sandbox_providers/local_docker/relay/`); `reset` and `close` have the driver end its recordings first (`computer-recording/v1` `end_all`, each uploading what's left and its manifest under the grant it started with), and `reset` then boots a fresh container in its place (`reset_container`), keeping the computer's address but not an agent's MCP session to the old driver; its `load_file_artifact_universe` writes files under the session user's home through the driver's `data/add`, `Downloads` by default, each file's bytes through a read grant when the store's grants reach it and its network allows everything, else inline (`env/envs/computer/utils/inside.py`)), `ComputerAppEnv` (`computer_app`, `env/envs/computer/app_env.py`: an app on a computer, a `ComputerAppEnvironment` class its env.toml's `module` names, read at put by the same `_declare`, which declares each OS with its own decorator (`@linux(...)`, or `@install(os_type, spec)` for any OS type; `@linux(distribution=...)` or `@install("/", spec)` for one distribution, which replaces its OS type's there) holding its install, its launch and, optionally, a skill and settings and build settings that add to the class's there, and install steps and ready checks for every OS or those their `os=` names; everything that reads its settings, launch or steps asks for the computer's OS type; deployed alone it is a one-app computer env, a `ComputerEnv` of the computer and driver its env.toml names, the composer installing it under `/opt/agentenv/apps//` with its services as supervisord programs beside its own server (`agentenv_protocol.computer.serve_app`, in a venv with the protocol's `app` extra, on `.agentenv/server.sock` in its folder), which adds its launcher, launches a `boot` app once per boot and says through `computer-app/v1` whether it's ready, as the driver's `computer-ready` asks, its card named after it, and `env_params` `app_settings` handing it values of its `Settings` in `AGENTENV_APP_SETTINGS`, a secret one only as a bare reference, over its env.toml's `[settings]` and a computer env's `settings` beside it in `apps`, each by top-level field (`params.app_settings`; checked at put, a required one left to the deploy; the deploy record keeps the merged values as `app_settings`, secrets withheld); its env.toml's `build_settings`, values of its `BuildSettings` checked at put, are written to `.agentenv/build_settings.json` in its folder, so they're part of the image's key, its install steps read them, and a deploy can't change them). An env type whose services take environment data defines `load_environment_artifact` and `load_environment_universe_artifact`; `load_artifact` refuses an environment artifact aimed at a type that doesn't, at save and before it loads. An env built on the protocol SDK can serve agents a skill (`@agent_skill`, `skill/v1`), which `add_skills` `env_ids` installs under its environment name (behind a gateway, each server's under that server's name); a skill with files beside its SKILL.md goes as a bundle stored once per content under `agentenv-env-skills/` in the object store. `env/gateway/gateway.py` is the env-side gateway server, container-only code. Its site route, `/website//`, never forwards a site's data plane or card (`/agentenv/…`, `/.well-known/agent-env.json`), which agentenv reaches through `/svc//`; a site with no frontend image is served by its server image there. Its CDP route, `/browser//cdp//` (`env/gateway/browser_proxy.py`), reaches the website browser behind the `/svc` child ``, found from that child's card (it, or an app among its children, offering `browser/v1` `cdp` with `/cdp/` in its `computer-app/v1` `agent_routes`): it relays the browser's discovery and WebSocket both ways, each leg pinged every 20 s, and every CDP address the browser hands out, in `json/version`, `json/list` and its `cdp` answer through `/svc//`, comes back at the gateway, everything before `/cdp//` becoming the address the caller reached the gateway at and `/browser/`; each connection is a `browser_connection` trajectory event, without its token. While its clock is armed it serves agents `get_time` (`env/gateway/get_time.py`), which answers in UTC and, given an IANA `time_zone`, also in that zone; executors' working `hours` (`env/gateway/triggers.py`) take a fixed `utc_offset` or an IANA `time_zone`. | | `artifact/` | Immutable versioned artifacts; every `put()` creates a new version. The document goes to the document store, binary payloads to the object store. Every artifact but an environment universe, whose `metadata` key holds its legacy metadata file ids, carries a `metadata` dict outside its versioned fields, changed by compare-and-swap (`merge_metadata`). Built-ins: `CliArtifact` (`cli`), `ComputerArtifact` (`computer`, a base computer: its architectures, OS, session user, settings and container form, whose image `put` reads from its registry into `metadata.validated_images`), `FileArtifact` (`file`), `FileArtifactUniverse` (`file_artifact_universe`), `DockerImageArtifact` (`docker_image`), `EnvironmentArtifact` (`environment`: its environment's files by path, stored as the one wrapped `file_artifact_ref` every environment artifact had when it has one file at its filename, else as `file_artifact_refs`; `get_file_artifacts()` reads either, and a load hands each file to the env's `data/add` with its path, which a pre-v1 load, taking one file, refuses), `EnvironmentUniverseArtifact` (`environment_universe`), `SkillArtifact` (`skill`); `[artifacts] type_aliases` maps another stored spelling of a type onto one of them. A `DockerImageArtifact` holds a tar.gz of its image, or, written by `put_ref` (`--image-ref` on `env mcp-server put` and `a2a-agent put`), only a registry reference with its tag pinned to a digest, which sandboxes pull, or a published image's tag with its `image_digest` and its build context, which sandboxes pull by the digest and build from the context when they can't (the infra images, and a bundle agent built from a shipped agent's sources, `Plan.published`), or, written by `put_context`, only its build context, which a VM sandbox builds before running it (`VmSandbox.load_docker_images`) and Modal builds before creating its container (`ModalSandboxProvider.prepare_image`, outside a fallback chain's deadline); anything else that runs images by name (a container on the local provider, a lone server on any provider but Modal) refuses such an image. A build context (`utils/docker/context.py`) is what `docker build` would send, `.dockerignore` applied, written so equal files give equal bytes and an equal `source_digest`. | | `task/`, `task_step/` | A `Task` holds its `TaskStep`s inline; `Task.run()` executes them as a DAG. `depends_on` (None means all prior steps) gates a step, independent steps run concurrently, `fail_task_on_error` makes a failure fatal or tolerated, `retry_config` rolls a failed span back through the step journal and re-dispatches it. Built-in steps live in `task_step/task_steps/` (`deploy_env`, `deploy_agent`, `prompt_agent`, the verifiers under `verifiers/`, among them `start_computer_recording` and `end_computer_recording`, and more). On an env whose card serves computer-exec/v1 (a computer), `run_code` with `env_id` and `run_container_unit_tests_verifier` with `env_id` run inside it as its session user through the driver: files staged with `data/add` and read with `data/get`, commands run with `run`; `env_outcome_verifier` hands a `verify` that takes an `env_url` keyword the env's address. A computer's framework-only routes (everything but MCP and the card) take the per-deploy control token its provider boots it with (`DeployedEnv.control_token`, `AGENTENV_CONTROL_TOKEN`); framework calls send `deployed.control_headers`, `Task.run` runs every step under `protocol_v1.authorized` with each deployed env's (a `verify` included), and a `run_code` script inside the computer gets `input["env"]["headers"]`. `teardown_run` and the `teardown_sandboxes` step share one core (`task/teardown.py` `tear_down`): the recordings the run left open end with fresh grants, each env still up closes (`Env.close`; a computer env ends any recording still open), what stayed open is registered from its namespace, then every sandbox is terminated by id. `task/episode/` runs a task with some of its agents played from outside, as a training loop does (`EpisodeSpec`). | | `store/` | Four store ABCs with local and cloud implementations: `DocumentStore` (SQLite, MongoDB, DynamoDB, Firestore with MongoDB compatibility), `ObjectStore` (filesystem, S3, Cloud Storage), `ImageStore` (local OCI registry, ECR), `SecretStore` (env vars or file, AWS Secrets Manager, Google Cloud Secret Manager). `VersionedEntityStore` implements the shared versioned get/put logic, `QueryBuilder` is the immutable chained query API, `store/base.py` holds the error types. `store/routing.py` is the local namespaces, `@local` and `@hf` (`is_local_id`), each with a documents file of its own: while the CLI or a `build_app` server runs, a local id and what a local task's run records go to local stores beside the configured ones, and that run's writes to the configured stores are refused. A new backend must pass the conformance kits in `tst/store/`. | diff --git a/packages/agentenv-protocol/README.md b/packages/agentenv-protocol/README.md index 0feaec16d..cd9b29f3b 100644 --- a/packages/agentenv-protocol/README.md +++ b/packages/agentenv-protocol/README.md @@ -789,8 +789,13 @@ class MailApp(ComputerAppEnvironment): models whose fields add to the class's `Settings` and `BuildSettings` on that OS. What's on the class applies to every OS. `@install(os_type, spec, launch=..., ...)` is the generic form, for an OS type a plugin adds, with that OS type's own install model as `spec`; `@linux(...)` is - `@install("linux", Linux(...), ...)`. On the computer the class reads what's declared for the OS - it runs on (`platform.detect()`). + `@install("linux", Linux(...), ...)`. `@linux(distribution="omarchy", ...)`, or + `@install("linux/omarchy", ...)`, declares for one distribution, by its /etc/os-release ID, what + replaces the OS type's declaration there. On the computer the class reads what's declared for the + platform its computer was composed for, which the composer sets as `AGENTENV_PLATFORM` for an app + that declares anything for a distribution, else the one it runs on (`platform.detect()`): its + distribution's, else its OS type's. An install step or ready check's `os=` names OS types or + distributions, and one for an OS type runs on all of it. - **`launch=Launch(...)`** says how it starts in the desktop session: with it (`at="boot"`) or when the agent opens it (`"demand"`). `ready=Ready(...)` says when it's ready, within `timeout_s` of its launch: a window matching `launch.window` by its class, instance or title (on Hyprland, a Wayland diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py index 9243d2a05..004703ccd 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/app.py @@ -29,7 +29,7 @@ from .app_data import AppDataPlane from .app_runtime import AppRuntime from .errors import InvalidArgument -from .platform import detect +from .platform import PLATFORM_ENV, applies, detect, entry_for, platform_key #: The environment variable a computer delivers its apps' settings in: JSON of each app's values by its name. APP_SETTINGS_ENV = "AGENTENV_APP_SETTINGS" @@ -327,7 +327,9 @@ def install(os_type: str, spec: BaseModel, *, launch: Optional[Launch] = None, p the app's own there: its ``launch``; the ``paths`` it keeps its data in, relative to the session user's home, which serve its data plane when it has no data hooks; its ``skill``, a file of its folder, instead of the class's; and ``settings`` and ``build_settings``, models whose fields add to the class's ``Settings`` and ``BuildSettings`` there. - ``@linux`` is this for Linux; an OS type a plugin adds is declared with it.""" + ``os_type`` may name one of its distributions as ``/``, /etc/os-release's ID (``linux/omarchy``), + whose declaration replaces the OS type's there. ``@linux`` is this for Linux; an OS type a plugin adds is declared + with it.""" try: owned = tuple(_PATHS.validate_python(list(paths or []))) except ValidationError: @@ -344,16 +346,18 @@ def deco(cls: type) -> type: return deco -def linux(*, files: str = ".", packages: Optional[Packages] = None, python: Optional[str] = None, - sources: Optional[list[AptSource]] = None, downloads: Optional[list[Download]] = None, - run: Optional[list[RunScript]] = None, services: Optional[list[Service]] = None, - ports: Optional[list[int]] = None, launch: Optional[Launch] = None, paths: Optional[list[str]] = None, - skill: Optional[str] = None, settings: Optional[type[BaseModel]] = None, +def linux(*, distribution: Optional[str] = None, files: str = ".", packages: Optional[Packages] = None, + python: Optional[str] = None, sources: Optional[list[AptSource]] = None, + downloads: Optional[list[Download]] = None, run: Optional[list[RunScript]] = None, + services: Optional[list[Service]] = None, ports: Optional[list[int]] = None, launch: Optional[Launch] = None, + paths: Optional[list[str]] = None, skill: Optional[str] = None, settings: Optional[type[BaseModel]] = None, build_settings: Optional[type[BaseModel]] = None) -> Callable: - """``@install`` for Linux: how a Linux computer installs the app, ``Linux``'s fields, and what is its own there.""" - return install("linux", Linux(files=files, packages=packages, python=python, sources=sources, downloads=downloads, - run=run, services=services or [], ports=ports), - launch=launch, paths=paths, skill=skill, settings=settings, build_settings=build_settings) + """``@install`` for Linux, or with ``distribution`` for that distribution alone: how such a computer installs the + app, ``Linux``'s fields, and what is its own there.""" + spec = Linux(files=files, packages=packages, python=python, sources=sources, downloads=downloads, run=run, + services=services or [], ports=ports) + return install(platform_key("linux", distribution), spec, launch=launch, paths=paths, skill=skill, + settings=settings, build_settings=build_settings) def windows(*, files: str = ".", python: Optional[str] = None, downloads: Optional[list[Download]] = None, @@ -459,18 +463,18 @@ class BuildSettings(BaseModel): app declares its own, which its install steps read too. None is a secret, since the image keeps them.""" def __init__(self) -> None: - cls, on = type(self), detect().os + cls, on = type(self), _here() values = json.loads(os.environ.get(APP_SETTINGS_ENV) or "{}").get(os.environ.get("ENVIRONMENT_NAME"), {}) self.settings = cls.settings_on(on).model_validate(values) self.build_settings = _built_settings(cls, on) - declared = cls._installs.get(on) + declared = entry_for(cls._installs, on) launch = declared.launch if declared is not None else None self.launch = self._launch_for(launch) if launch is not None else None self.paths = list(declared.paths) if declared is not None else [] ready = launch.ready if launch is not None else None self._computer_app = AppRuntime( self.launch, {fn.__name__: getattr(self, fn.__name__) for fn in _methods(cls) - if hasattr(fn, _READY_CHECK_ATTR) and _runs_on(getattr(fn, _READY_CHECK_ATTR), on)}, + if hasattr(fn, _READY_CHECK_ATTR) and applies(getattr(fn, _READY_CHECK_ATTR), on)}, ready.timeout_s if ready is not None else DEFAULT_TIMEOUT_S, desktop=self._on_desktop(), launch_fds=self._launch_fds, relaunch_on_start=self._relaunch_on_start(), launch_running=self._launch_running()) @@ -509,14 +513,15 @@ async def _relaunched(self, op: str) -> None: @classmethod def settings_on(cls, os_type: Optional[str]) -> type[BaseModel]: - """Its ``Settings`` on ``os_type``, with the fields its OS adds.""" - added = declared.settings if (declared := cls._installs.get(os_type)) is not None else None + """Its ``Settings`` on ``os_type``, an OS type or ``/``, with the fields it adds there.""" + added = declared.settings if (declared := entry_for(cls._installs, os_type)) is not None else None return _added(added, cls.Settings) if added is not None else cls.Settings @classmethod def build_settings_on(cls, os_type: Optional[str]) -> type[BaseModel]: - """Its ``BuildSettings`` on ``os_type``, with the fields its OS adds.""" - added = declared.build_settings if (declared := cls._installs.get(os_type)) is not None else None + """Its ``BuildSettings`` on ``os_type``, an OS type or ``/``, with the fields it adds + there.""" + added = declared.build_settings if (declared := entry_for(cls._installs, os_type)) is not None else None return _added(added, cls.BuildSettings) if added is not None else cls.BuildSettings @extension(COMPUTER_APP, op="launch", description=( @@ -588,7 +593,7 @@ def _computer_app_skill(self) -> tuple[str, str]: def _computer_app_skill_file(self) -> Optional[str]: """The file of the app's folder holding its skill on this OS, or the class's.""" - declared = type(self)._installs.get(detect().os) + declared = entry_for(type(self)._installs, _here()) return declared.skill if declared is not None and declared.skill is not None else type(self).skill @classmethod @@ -618,8 +623,10 @@ def declaration(cls) -> AppDeclaration: if hasattr(fn, _INSTALL_STEP_ATTR) for scope, os_types in [getattr(fn, _INSTALL_STEP_ATTR)]] checks = [AppReadyCheck(name=fn.__name__, os=getattr(fn, _READY_CHECK_ATTR)) for fn in methods if hasattr(fn, _READY_CHECK_ATTR)] + installed_os = {key.partition("/")[0] for key in cls._installs} for method in [*steps, *checks]: - if unknown := [os_type for os_type in method.os or [] if os_type not in cls._installs]: + if unknown := [os_type for os_type in method.os or [] + if entry_for(cls._installs, os_type) is None and os_type not in installed_os]: raise ValueError(f"{cls.__name__}.{method.name} runs on {unknown[0]!r}, which {cls.__name__} declares " f"no install for; it declares {', '.join(cls._installs) or 'none'}") return AppDeclaration( @@ -678,8 +685,13 @@ def _built_settings(cls: type[ComputerAppEnvironment], os_type: Optional[str]) - return cls.build_settings_on(os_type).model_validate(values) -def _runs_on(os_types: Optional[list[str]], os_type: Optional[str]) -> bool: - return os_types is None or os_type in os_types +def _here() -> Optional[str]: + """The platform this runs on, by the key an app's parts for it are under (``platform_key``): the one its computer + was composed for (``PLATFORM_ENV``), else what it detects.""" + if composed := os.environ.get(PLATFORM_ENV): + return composed + running = detect() + return platform_key(running.os, running.distribution) @functools.cache diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/platform.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/platform.py index 334801d0a..3b0f5d02b 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/platform.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/platform.py @@ -7,10 +7,15 @@ import sys from dataclasses import dataclass from pathlib import Path +from typing import Iterable, Mapping, Optional, TypeVar PREDICATE_KEYS = ("os", "distribution", "version", "arch", "display") _ARCH = {"x86_64": "amd64", "amd64": "amd64", "aarch64": "arm64", "arm64": "arm64"} +_Entry = TypeVar("_Entry") +#: The platform (``platform_key``) the computer's composer built an app for, which its server and install steps read in +#: place of what /etc/os-release says, so they act on the install the image holds; unset outside a composed computer. +PLATFORM_ENV = "AGENTENV_PLATFORM" _OS = {"linux": "linux", "darwin": "macos", "win32": "windows"} @@ -52,6 +57,30 @@ def detect(environ: dict | None = None, os_release: Path = Path("/etc/os-release arch=_ARCH.get(machine, machine or None), display=display) +def platform_key(os_type: Optional[str], distribution: Optional[str] = None) -> Optional[str]: + """What a platform's own install, launch or known failures are keyed by: ``/``, its distribution + being /etc/os-release's ID, or its OS type when it has none.""" + if os_type is None: + return None + return f"{os_type}/{distribution}" if distribution else os_type + + +def entry_for(by_platform: Mapping[str, _Entry], key: Optional[str]) -> Optional[_Entry]: + """``by_platform``'s entry for the platform ``key`` names: its distribution's, which replaces its OS type's there, + else its OS type's.""" + if key is None: + return None + return by_platform[key] if key in by_platform else by_platform.get(key.partition("/")[0]) + + +def applies(keys: Optional[Iterable[str]], key: Optional[str]) -> bool: + """Whether what's declared for ``keys``, OS types or distributions, or for every OS when None, applies on the + platform ``key`` names: one names its OS type or its distribution.""" + if keys is None: + return True + return key is not None and (key in keys or key.partition("/")[0] in keys) + + def supported(entry: dict) -> list[Platform]: """The platforms a ``supports`` entry names: its ``os`` type, with each of its displays and arches, and its distribution and version when it names one. ``{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64"]}`` diff --git a/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py b/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py index bf78fae3a..e9aed87a1 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/computer/run_install_step.py @@ -11,8 +11,7 @@ from typing import NoReturn from ._load import load_class -from .app import ComputerAppEnvironment, _built_settings -from .platform import detect +from .app import ComputerAppEnvironment, _built_settings, _here USAGE = "usage: python -m agentenv_protocol.computer.run_install_step : " @@ -33,7 +32,7 @@ def main(argv: list[str]) -> None: if argv[1] not in {step.name for step in cls.declaration().install_steps}: raise SystemExit(f"{cls.__name__} has no @install_step method {argv[1]!r}") app = cls.__new__(cls) - app.settings, app.build_settings = _NoDeploy(), _built_settings(cls, detect().os) + app.settings, app.build_settings = _NoDeploy(), _built_settings(cls, _here()) done = getattr(app, argv[1])() if inspect.isawaitable(done): asyncio.run(done) diff --git a/packages/agentenv-protocol/tests/test_computer_declare.py b/packages/agentenv-protocol/tests/test_computer_declare.py index 6edc7afc5..d948bf7b1 100644 --- a/packages/agentenv-protocol/tests/test_computer_declare.py +++ b/packages/agentenv-protocol/tests/test_computer_declare.py @@ -18,11 +18,11 @@ from agentenv_protocol.computer import AppDeclaration # noqa: E402 from agentenv_protocol.computer._declare import _Stub, _StubModule, declare, main # noqa: E402 from agentenv_protocol.computer import (APP_SETTINGS_ENV, ComputerAppEnvironment, Launch, Linux, # noqa: E402 - Windows, install, linux, windows) + Windows, install, install_step, linux, ready_check, windows) from agentenv_protocol.computer import app as app_module # noqa: E402 from agentenv_protocol.computer.app import BUILD_SETTINGS # noqa: E402 from pydantic import BaseModel # noqa: E402 -from agentenv_protocol.computer.platform import Platform # noqa: E402 +from agentenv_protocol.computer.platform import PLATFORM_ENV, Platform # noqa: E402 DRIVER = '''import not_installed_here from not_installed_here.input import Base, Keyboard @@ -337,6 +337,106 @@ class BuildSettings(BaseModel): assert (type(notes.settings), notes.build_settings.edition, notes.launch) == (Notes.Settings, "home", None) +@pytest.mark.parametrize(("distribution", "launched", "checked"), [ + ("omarchy", ["notes", "--wayland"], ["anywhere", "here"]), ("ubuntu", ["notes"], ["anywhere"]), + (None, ["notes"], ["anywhere"])]) +def test_an_app_on_a_distribution_it_declares_for_takes_that_part_and_elsewhere_its_os_types( + monkeypatch, tmp_path, distribution, launched, checked): + class WaylandSettings(BaseModel): + portal: bool = True + + @linux(distribution="omarchy", launch=Launch(command=["notes", "--wayland"]), settings=WaylandSettings) + @linux(launch=Launch(command=["notes"])) + class Notes(ComputerAppEnvironment): + @ready_check(os="linux") + def anywhere(self): + return True + + @ready_check(os="linux/omarchy") + def here(self): + return True + + monkeypatch.setattr(app_module, "detect", lambda: Platform(os="linux", distribution=distribution)) + monkeypatch.setenv("ENVIRONMENT_NAME", "notes") + monkeypatch.delenv(APP_SETTINGS_ENV, raising=False) + monkeypatch.chdir(tmp_path) + + notes = Notes() + + assert set(Notes.declaration().os) == {"linux", "linux/omarchy"} + assert notes.launch.command == launched and hasattr(notes.settings, "portal") == (distribution == "omarchy") + assert sorted(notes._computer_app.ready_checks) == checked + + +def test_an_install_step_or_ready_check_on_no_os_type_the_app_installs_on_is_refused(): + @linux(distribution="omarchy") + class Notes(ComputerAppEnvironment): + @ready_check(os="linux") + def anywhere(self): + return True + + @ready_check(os="macos") + def mac(self): + return True + + with pytest.raises(ValueError, match="Notes.mac runs on 'macos', which Notes declares no install for"): + Notes.declaration() + + +@pytest.mark.parametrize(("distribution", "checked"), [("ubuntu", ["anywhere", "on_ubuntu"]), + ("omarchy", ["anywhere"])]) +def test_an_install_step_or_ready_check_for_a_distribution_runs_there_on_its_os_types_install( + monkeypatch, tmp_path, distribution, checked): + @linux(launch=Launch(command=["notes"])) + class Notes(ComputerAppEnvironment): + @install_step(os="linux/ubuntu") + def fonts(self): + pass + + @ready_check(os="linux") + def anywhere(self): + return True + + @ready_check(os="linux/ubuntu") + def on_ubuntu(self): + return True + + monkeypatch.setattr(app_module, "detect", lambda: Platform(os="linux", distribution=distribution)) + monkeypatch.setenv("ENVIRONMENT_NAME", "notes") + monkeypatch.delenv(APP_SETTINGS_ENV, raising=False) + monkeypatch.chdir(tmp_path) + + assert [step.os for step in Notes.declaration().install_steps] == [["linux/ubuntu"]] + assert sorted(Notes()._computer_app.ready_checks) == checked + + +def test_an_app_reads_the_platform_its_computer_was_composed_for_over_what_it_detects(monkeypatch, tmp_path): + @linux(distribution="omarchy", launch=Launch(command=["notes", "--wayland"])) + @linux(launch=Launch(command=["notes"])) + class Notes(ComputerAppEnvironment): + pass + + monkeypatch.setattr(app_module, "detect", lambda: Platform(os="linux", distribution="arch")) + monkeypatch.setenv("ENVIRONMENT_NAME", "notes") + monkeypatch.delenv(APP_SETTINGS_ENV, raising=False) + monkeypatch.chdir(tmp_path) + detected = Notes().launch.command + monkeypatch.setenv(PLATFORM_ENV, "linux/omarchy") + + assert (detected, Notes().launch.command) == (["notes"], ["notes", "--wayland"]) + + +def test_an_install_step_or_ready_check_for_a_distribution_with_no_install_of_its_own_or_its_os_types_is_refused(): + @linux(distribution="omarchy") + class Notes(ComputerAppEnvironment): + @install_step(os="linux/ubuntu") + def fonts(self): + pass + + with pytest.raises(ValueError, match="Notes.fonts runs on 'linux/ubuntu', which Notes declares no install for"): + Notes.declaration() + + def test_a_declaration_stored_before_each_os_had_its_own_part_reads_as_its_linux_part(): stored = {"supports": LINUX, "permissions": [], "settings": {"type": "object"}, "data": [], "install_steps": [], "install": {"linux": {"files": ".", "services": [{"name": "imap", "command": "bin/imap"}]}}, @@ -353,7 +453,7 @@ def test_a_declaration_stored_before_each_os_had_its_own_part_reads_as_its_linux def test_linux_and_windows_take_every_field_of_their_install_and_what_install_adds_for_an_os(): added = set(inspect.signature(install).parameters) - {"os_type", "spec"} - assert set(inspect.signature(linux).parameters) == set(Linux.model_fields) | added + assert set(inspect.signature(linux).parameters) == set(Linux.model_fields) | added | {"distribution"} assert set(inspect.signature(windows).parameters) == set(Windows.model_fields) | added diff --git a/src/agent_env/artifact/artifacts/computer.py b/src/agent_env/artifact/artifacts/computer.py index 879511ec6..b193393ff 100644 --- a/src/agent_env/artifact/artifacts/computer.py +++ b/src/agent_env/artifact/artifacts/computer.py @@ -46,6 +46,10 @@ SettingValue = str | int | float | bool +#: A distribution, as os-release's ID names it here: lowercase letters, digits and "-", from a letter. +DISTRIBUTION = r"^[a-z][a-z0-9-]*$" + + class LinuxOS(BaseModel): """A Linux base: its distribution and version (``rolling`` for one with no versions, such as Arch), and the display server its desktop runs, ``none`` for one with no desktop, whose driver's screen is something else, a page say.""" @@ -53,7 +57,7 @@ class LinuxOS(BaseModel): model_config = ConfigDict(extra="forbid") type: Literal["linux"] = "linux" - distribution: str = Field(pattern=r"^[a-z][a-z0-9-]*$", description="os-release's ID: ubuntu, debian, fedora, arch") + distribution: str = Field(pattern=DISTRIBUTION, description="os-release's ID: ubuntu, debian, fedora, arch") version: str = Field(min_length=1, description="os-release's VERSION_ID, or rolling") display: Literal["x11", "wayland", "none"] build: str | None = None diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index ce16eb470..1e9678acd 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -61,7 +61,10 @@ pointer has moved along its path (P7). `env.toml`'s `[[known_failures.macos]]` n records them in the build's probe (`known_failures`) and doesn't fail the build for them; any other failure still does. A driver's `known_failures`, by OS type, names a check by its id and name, with why. One that a probe didn't see fail is recorded in its probe (`unused_known_failures`, with what it saw) and warned of, so a waiver doesn't outlive its -defect; P7's is intermittent, so a probe it passes doesn't show it fixed. +defect; P7's is intermittent, so a probe it passes doesn't show it fixed. An install or known failures can also be for +one distribution, keyed `"/"` by its /etc/os-release ID, as `[install."linux/omarchy"]` and +`[[known_failures."linux/omarchy"]]`: on that distribution they replace the OS type's, never merge with them, and +elsewhere the OS type's apply. A distribution no `supports` entry names is refused at put. On Windows (`amd64`, composed onto a disk such as `windows-11`), cua-driver is the cua-driver-rs release's zip, checked by sha256 and unpacked beside the driver (`install-windows.ps1`). Its daemon runs in the session user's diff --git a/src/agent_env/env/envs/computer/app_env.py b/src/agent_env/env/envs/computer/app_env.py index cdbb9d540..ad3708e65 100644 --- a/src/agent_env/env/envs/computer/app_env.py +++ b/src/agent_env/env/envs/computer/app_env.py @@ -11,6 +11,7 @@ from agentenv_protocol.computer import AppDeclaration, Launch from agentenv_protocol.computer.app import SOCKET +from agentenv_protocol.computer.platform import entry_for from agent_env.artifact import Artifact from agent_env.artifact.artifacts.computer import ComputerArtifact @@ -19,7 +20,8 @@ from agent_env.env.envs.computer.utils import composer, gate from agent_env.env.envs.computer.utils.composer import Composing from agent_env.env.envs.computer.env import APP_TYPE, ComputerEnv -from agent_env.env.envs.computer.utils.install import APP_INSTALLS, AppInstall, folder_problems, installs +from agent_env.env.envs.computer.utils.install import (APP_INSTALLS, AppInstall, folder_problems, installs, platform_of, + unsupported) from agent_env.env.envs.computer.driver_env import ComputerDriverEnv, declared from agent_env.env.store import get_env_store from agent_env.store.ids import derive_id @@ -64,13 +66,13 @@ def composed_apps(self) -> list[ComputerAppEnv]: def settings_schema(self, os_type: str | None) -> dict[str, Any]: """The JSON schema of its settings on ``os_type``: its class's, with the fields that OS adds; with no OS type, its class's.""" - on = self.declaration.os.get(os_type) + on = entry_for(self.declaration.os, os_type) own = on.settings if on is not None else None return own if own is not None else self.declaration.settings def build_settings_schema(self, os_type: str | None) -> dict[str, Any] | None: """The JSON schema of its build settings on ``os_type``, as ``settings_schema``; None when it declares none.""" - on = self.declaration.os.get(os_type) + on = entry_for(self.declaration.os, os_type) own = on.build_settings if on is not None else None return own if own is not None else self.declaration.build_settings @@ -82,7 +84,7 @@ def secret_settings(self, os_type: str | None) -> set[str]: def launch(self, os_type: str) -> Launch | None: """How it launches on ``os_type``.""" - on = self.declaration.os.get(os_type) + on = entry_for(self.declaration.os, os_type) return on.launch if on is not None else None def to_dict(self) -> dict: @@ -121,8 +123,9 @@ def put(cls, *, id: str, environment_name: str, module: str, files: FileArtifact app = cls(id=id, version=None, environment_name=environment_name, module=module, files=files, declaration=declaration, computer=computer, driver=driver, build_settings=build_settings, settings=settings) - os_type = computer.os.type if computer is not None else None + os_type = platform_of(computer.os) if computer is not None else None _, problems = _installs(declaration) + problems += unsupported(declaration.os, declaration.supports, "install") problems += _class_beyond_reach(module, declaration) problems += _in_server_folder(declaration, files.get_file_artifacts()) problems += gate.check_app_build_settings(app, app.build_settings, "build_settings", os_type) diff --git a/src/agent_env/env/envs/computer/driver_env.py b/src/agent_env/env/envs/computer/driver_env.py index 9e5b98252..5a8b1478e 100644 --- a/src/agent_env/env/envs/computer/driver_env.py +++ b/src/agent_env/env/envs/computer/driver_env.py @@ -13,10 +13,10 @@ from pydantic import BaseModel, ConfigDict, Field, ValidationError from agent_env.artifact import Artifact -from agent_env.artifact.artifacts.computer import OS_TYPES from agent_env.artifact.artifacts.file_artifact_universe import FileArtifactUniverse from agent_env.env.env import Env -from agent_env.env.envs.computer.utils.install import DRIVER_INSTALLS, Install, folder_problems, installs +from agent_env.env.envs.computer.utils.install import (DRIVER_INSTALLS, Install, folder_problems, installs, os_type_of, + platform_problem, unsupported) from agent_env.store.ids import derive_id if TYPE_CHECKING: @@ -70,8 +70,8 @@ def to_dict(self) -> dict: @classmethod def from_dict(cls, data: dict) -> ComputerDriverEnv: files = Artifact.get(data["files"]["id"], version=data["files"]["version"]) - install = {os_type: DRIVER_INSTALLS[os_type].model_validate(table) - for os_type, table in (data.get("install") or {}).items()} + install = {key: DRIVER_INSTALLS[os_type_of(key)].model_validate(table) + for key, table in (data.get("install") or {}).items()} return cls(id=data["id"], version=data.get("version"), module=data["module"], files=files, declaration=DriverDeclaration.model_validate(data["declaration"]), install=install, port=data.get("port", DEFAULT_PORT), metadata=data.get("metadata") or {}, @@ -82,6 +82,9 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerDriverEnv: """The driver's class is read in a process of its own (``declared``).""" fields = cls.accept_toml(data, ctx) declaration = DriverDeclaration.model_validate(declared(ctx, fields["module"], cls.type)) + if problems := [*unsupported(fields["install"], declaration.supports, "install"), + *unsupported(fields.get("known_failures") or {}, declaration.supports, "known_failures")]: + ctx.refuse(list(map(ctx.config_problem, problems))) universe = FileArtifactUniverse.put_bundled(derive_id(ctx.id, "files"), files=ctx.files()) return cls.put(id=ctx.id, module=fields["module"], files=universe, declaration=declaration, install=fields["install"], port=fields.get("port", DEFAULT_PORT), @@ -91,7 +94,8 @@ def from_toml(cls, data: dict, ctx: AuthoringContext) -> ComputerDriverEnv: def accept_toml(cls, data: dict, ctx: AuthoringContext) -> dict: """The keys of ``data``, an env.toml, a computer driver env takes: ``module = ":"``, naming a file of its folder; ``port``, where it serves; ``install``, by OS type, whose files and requirements are in its - folder; and ``known_failures``, by OS type, the probe checks it fails there. Raises BundleError listing every + folder; and ``known_failures``, by OS type, the probe checks it fails there. An install or known failures for + a distribution, ``/``, replace its OS type's there. Raises BundleError listing every problem.""" fields, refused = ctx.accepted(data, module=str, port=int, install=dict, known_failures=dict) files = ctx.files() @@ -140,14 +144,15 @@ def _refuse(ctx: AuthoringContext, problem: str) -> NoReturn: def _known_failures(data: dict) -> tuple[dict[str, list[KnownFailure]], list[str]]: - """The known failures ``data`` declares by OS type, and the problems with it, each named by its key.""" + """The known failures ``data`` declares by OS type or distribution, and the problems with it, each named by its + key.""" found, problems = {}, [] - for os_type, failures in data.items(): - if os_type not in OS_TYPES: - problems.append(f"known_failures.{os_type}: a known failure is for an OS type: {', '.join(OS_TYPES)}") + for key, failures in data.items(): + if problem := platform_problem(key): + problems.append(f"known_failures.{key}: a known failure is for {problem}") continue try: - found[os_type] = [KnownFailure.model_validate(failure) for failure in failures] + found[key] = [KnownFailure.model_validate(failure) for failure in failures] except (ValidationError, TypeError) as e: - problems.append(f"known_failures.{os_type}: {e}") + problems.append(f"known_failures.{key}: {e}") return found, problems diff --git a/src/agent_env/env/envs/computer/env.py b/src/agent_env/env/envs/computer/env.py index 5abca9a85..c530ba54d 100644 --- a/src/agent_env/env/envs/computer/env.py +++ b/src/agent_env/env/envs/computer/env.py @@ -34,6 +34,7 @@ from agent_env.env.envs.computer import COMPUTER_ENV_TYPE from agent_env.env.envs.computer.utils import boot, composer, gate, inside, sites from agent_env.env.envs.computer.utils.composed import stale +from agent_env.env.envs.computer.utils.install import platform_of from agent_env.env.envs.computer.utils.composer import Composing from agent_env.env.envs.computer.utils import params as computer_params from agent_env.env.envs.computer.driver_env import ComputerDriverEnv @@ -142,8 +143,8 @@ def put(cls, *, id: str, computer: ComputerArtifact, driver: ComputerDriverEnv, by_name = {app.environment_name: app for app in apps} for name, defaults in app_settings.items(): problems += ([f"apps.{name}.settings: {id} has no app {name!r}"] if name not in by_name else - gate.check_app_settings(by_name[name], defaults, f"apps.{name}.settings", computer.os.type, - partial=True)) + gate.check_app_settings(by_name[name], defaults, f"apps.{name}.settings", + platform_of(computer.os), partial=True)) if problems: raise ValueError(f"computer env {id!r}: " + "; ".join(problems)) env = get_env_store().put_document(cls( diff --git a/src/agent_env/env/envs/computer/utils/boot.py b/src/agent_env/env/envs/computer/utils/boot.py index f106949fe..6b8f50c02 100644 --- a/src/agent_env/env/envs/computer/utils/boot.py +++ b/src/agent_env/env/envs/computer/utils/boot.py @@ -110,23 +110,24 @@ def settings_environment(computer: ComputerArtifact, values: dict[str, Any], *, def app_settings_environment(apps: list[ComputerAppEnv], app_settings: dict[str, dict], - os_type: str) -> dict[str, str]: - """The environment that hands each of ``apps``, on a computer of ``os_type``, the values ``app_settings`` gives it, - by its name, in ``APP_SETTINGS_ENV``, a secret's resolved through the secret store; nothing when none has any.""" + platform: str) -> dict[str, str]: + """The environment that hands each of ``apps``, on a computer of ``platform`` (``install.platform_of``), the values + ``app_settings`` gives it, by its name, in ``APP_SETTINGS_ENV``, a secret's resolved through the secret store; + nothing when none has any.""" given = [(app, app_settings[app.environment_name]) for app in apps if app_settings.get(app.environment_name)] if not given: return {} return {APP_SETTINGS_ENV: json.dumps({app.environment_name: { - name: get_config().resolve_reference(value) if name in app.secret_settings(os_type) and value is not None + name: get_config().resolve_reference(value) if name in app.secret_settings(platform) and value is not None else value for name, value in values.items()} for app, values in given})} def recorded_app_settings(apps: list[ComputerAppEnv], app_settings: dict[str, dict], - os_type: str) -> dict[str, dict[str, Any]]: - """The values ``app_settings`` hands each of ``apps`` on a computer of ``os_type``, by its name, as a record keeps - them, a secret's withheld; its class's defaults fill in the rest.""" - return {app.environment_name: {name: WITHHELD if name in app.secret_settings(os_type) else value + platform: str) -> dict[str, dict[str, Any]]: + """The values ``app_settings`` hands each of ``apps`` on a computer of ``platform`` (``install.platform_of``), by + its name, as a record keeps them, a secret's withheld; its class's defaults fill in the rest.""" + return {app.environment_name: {name: WITHHELD if name in app.secret_settings(platform) else value for name, value in values.items()} for app in apps if (values := app_settings.get(app.environment_name))} @@ -193,11 +194,11 @@ async def discard(sandbox: Sandbox) -> None: await asyncio.to_thread(remove_local_work_dir, sandbox.sandbox_id) -def ready_timeout(apps: list[ComputerAppEnv], os_type: str) -> float: - """How long a computer of ``os_type`` and ``apps`` may take to be ready: ``READY_TIMEOUT_S``, or longer when an app - that starts with it there gives itself longer.""" +def ready_timeout(apps: list[ComputerAppEnv], platform: str) -> float: + """How long a computer of ``platform`` (``install.platform_of``) and ``apps`` may take to be ready: + ``READY_TIMEOUT_S``, or longer when an app that starts with it there gives itself longer.""" return max([READY_TIMEOUT_S, *(launch.ready.timeout_s + APP_READY_MARGIN_S for app in apps - if (launch := app.launch(os_type)) and launch.at == "boot" and launch.ready)]) + if (launch := app.launch(platform)) and launch.at == "boot" and launch.ready)]) async def ready(url: str, timeout: float, token: Optional[str] = None, *, answering: bool = False) -> dict: diff --git a/src/agent_env/env/envs/computer/utils/composer.py b/src/agent_env/env/envs/computer/utils/composer.py index ae804017e..5d8f6fc97 100644 --- a/src/agent_env/env/envs/computer/utils/composer.py +++ b/src/agent_env/env/envs/computer/utils/composer.py @@ -31,6 +31,7 @@ from agentenv_protocol.computer.apps import apps_folder from agentenv_protocol.computer.app import BUILD_SETTINGS, SOCKET +from agentenv_protocol.computer.platform import PLATFORM_ENV, applies, entry_for from agent_env.artifact.artifacts.computer import LOCAL_IMAGE_PREFIX, ComputerArtifact from agent_env.artifact.artifacts.docker_image import DockerImageArtifact @@ -48,7 +49,7 @@ windows_disk, ) from agent_env.env.envs.computer.utils.composed import BUILT, REPOSITORY, Record, failed, now, stale -from agent_env.env.envs.computer.utils.install import Install +from agent_env.env.envs.computer.utils.install import Install, for_os, platform_of from agent_env.env.published_images import DISK, DISK_TOOLS_IMAGE, published_disk, published_pinned from agent_env.providers.sandbox_providers.host_images import platform_tag from agent_env.store.ids import derive_id @@ -312,7 +313,7 @@ def _composed_vm_bundle(env: ComputerEnv, base: ComputerArtifact, driver: Comput apps = [(app, _stage_app(context, app, base, {})) for app in env.composed_apps()] parts = [_part(layer, _services(f"driver-{gate.folder_name(driver.id, 'driver')}", layer.install), {"MCP_PORT": driver.port}, protocol, False)] - parts += [_part(staged, dict(zip(gate.programs(app, base.os.type), [ + parts += [_part(staged, dict(zip(gate.programs(app, platform_of(base.os)), [ f"venv/bin/python -m agentenv_protocol.computer.serve_app {shlex.quote(staged.served)}", *(service.command for service in staged.install.services)])), {"ENVIRONMENT_NAME": app.environment_name}, protocol, True) for app, staged in apps] @@ -383,7 +384,7 @@ def _composed_windows_disk(env: ComputerEnv, base: ComputerArtifact, driver: Com server = [str(prefix / "venv" / "Scripts" / "python.exe"), "-m", "agentenv_protocol.computer.serve_app", staged.served] commands = [server, *(windows_disk.argv(service.command, prefix) for service in staged.install.services)] - parts.append(_windows_part(staged, prefix, dict(zip(gate.programs(app, base.os.type), commands)), + parts.append(_windows_part(staged, prefix, dict(zip(gate.programs(app, platform_of(base.os)), commands)), {"ENVIRONMENT_NAME": app.environment_name}, protocol)) values = env.computer_values() settings = {name: values.get(name, setting.default) for name, setting in base.config.items() @@ -775,6 +776,7 @@ class _Layer: app: bool = False served: str | None = None steps: tuple[tuple[str, str], ...] = () + platform: str | None = None def _stage_driver(context: Path, driver: ComputerDriverEnv, base: ComputerArtifact, @@ -782,31 +784,41 @@ def _stage_driver(context: Path, driver: ComputerDriverEnv, base: ComputerArtifa """Write the driver's install files under ``context/driver/`` and its supervisord programs beside them, served on its port, its folder under ``DRIVERS`` named for its id's last part, its programs and their file under ``driver-`` as an app's are under ``app-``, so neither ever takes the other's.""" - install, name = driver.install[base.os.type], gate.folder_name(driver.id, "driver") + install, name = for_os(driver.install, base.os), gate.folder_name(driver.id, "driver") layer = _Layer("driver", f"{DRIVERS}/{name}", install, f"agentenv-driver-{name}.conf", driver.id) _stage(context, layer, f"driver {driver.id}", driver.files, _services(f"driver-{name}", install), {"MCP_PORT": driver.port}, from_store) return layer +def _platform_environment(app: ComputerAppEnv, base: ComputerArtifact) -> dict[str, str]: + """The platform the app is composed for (``PLATFORM_ENV``), for an app that declares an install, an install step or + a ready check for a distribution, so its server and install steps act on the install its image holds whatever its + os-release says; nothing for one that doesn't, which reads the same either way, so its image stays as it was.""" + declaration = app.declaration + scoped = [key for part in [*declaration.install_steps, *declaration.ready_checks] for key in part.os or []] + return {PLATFORM_ENV: platform_of(base.os)} if any("/" in key for key in [*declaration.os, *scoped]) else {} + + def _stage_app(context: Path, app: ComputerAppEnv, base: ComputerArtifact, from_store: dict[str, tuple[str, str]]) -> _Layer: """Write the app's install files under ``context/apps//``, with the values of its build settings the env fixes, and its supervisord programs beside them, run under its name: its services and its own server, which serves its class from its venv and launches the app. Its folder under its OS's apps folder is named for its environment name.""" - install, name = app.install[base.os.type], gate.app_folder(app) + platform, composed_for = platform_of(base.os), _platform_environment(app, base) + install, name = entry_for(app.install, platform), gate.app_folder(app) file, _, cls = app.module.partition(":") served = f"{PurePosixPath(file).relative_to(install.files)}:{cls}" layer = _Layer(f"apps/{name}", str(apps_folder(base.os.type) / name), install, f"agentenv-app-{name}.conf", app.id, app=True, served=served, steps=tuple((step.name, step.scope) for step in app.declaration.install_steps - if step.os is None or base.os.type in step.os)) - programs = dict(zip(gate.programs(app, base.os.type), [ + if applies(step.os, platform)), platform=composed_for.get(PLATFORM_ENV)) + programs = dict(zip(gate.programs(app, platform), [ f"venv/bin/python -m agentenv_protocol.computer.serve_app {shlex.quote(served)}", *(service.command for service in install.services)])) - _stage(context, layer, f"app {app.id}", app.files, programs, {"ENVIRONMENT_NAME": app.environment_name}, - from_store) + _stage(context, layer, f"app {app.id}", app.files, programs, + {"ENVIRONMENT_NAME": app.environment_name, **composed_for}, from_store) if app.build_settings: (context / layer.folder / BUILD_SETTINGS).parent.mkdir(exist_ok=True) (context / layer.folder / BUILD_SETTINGS).write_text(json.dumps(app.build_settings, sort_keys=True)) @@ -932,8 +944,9 @@ def _mounted(source: str, target: str) -> str: def _runs(layer: _Layer) -> list[tuple[str, str]]: """The commands of ``layer``'s run scripts and its install steps, each with the scope it runs as: the only steps of a build that write outside the layer's folder.""" + platform = f"{PLATFORM_ENV}={shlex.quote(layer.platform)} " if layer.platform else "" return [*((run.scope, f"cd {layer.prefix} && {shlex.quote('./' + run.script)}") for run in layer.install.run or []), - *((scope, f"cd {layer.prefix} && venv/bin/python -m agentenv_protocol.computer.run_install_step " + *((scope, f"cd {layer.prefix} && {platform}venv/bin/python -m agentenv_protocol.computer.run_install_step " f"{shlex.quote(layer.served)} {step}") for step, scope in layer.steps)] diff --git a/src/agent_env/env/envs/computer/utils/gate.py b/src/agent_env/env/envs/computer/utils/gate.py index 744584c8c..1987b2dc3 100644 --- a/src/agent_env/env/envs/computer/utils/gate.py +++ b/src/agent_env/env/envs/computer/utils/gate.py @@ -10,12 +10,13 @@ from typing import TYPE_CHECKING, Any from agentenv_protocol.computer.apps import apps_folder -from agentenv_protocol.computer.platform import Platform, supported +from agentenv_protocol.computer.platform import Platform, entry_for, supported from jsonschema import Draft202012Validator from agent_env.artifact.artifacts.computer import FORMS, ComputerArtifact, SettingValue from agent_env.config.loader import ENV_REF_PREFIX, SECRET_REF_PREFIX from agent_env.providers.sandbox_providers.sandbox import NetworkPolicy +from agent_env.env.envs.computer.utils.install import for_os, platform_of from agent_env.store.image_store.oci_registry_credentials import names_registry if TYPE_CHECKING: @@ -75,9 +76,9 @@ def check_app(base: ComputerArtifact, driver: ComputerDriverEnv, app: ComputerAp problems = [] if not arches(base, driver, [app]): problems.append(f"{at}: no architecture {base.id}, {driver.id} and {app.id} all run") - if os.type not in app.install: - problems.append(f"{at}: {app.id} declares no install for {os.type}") - elif taken := sorted(set(_ports(app, os.type)) & set(_driver_ports(driver, os.type))): + if for_os(app.install, os) is None: + problems.append(f"{at}: {app.id} declares no install for {_os_shown(base)}") + elif taken := sorted(set(_ports(app, platform_of(os))) & set(_driver_ports(driver, os))): problems.append(f"{at}: {app.id} listens on {', '.join(map(str, taken))}, as {driver.id} does") if os.type == "windows" and (session := [step.name for step in app.declaration.install_steps if step.scope == "session" and (step.os is None or os.type in step.os)]): @@ -114,8 +115,8 @@ def _apps(base: ComputerArtifact, driver: ComputerDriverEnv, apps: list[Computer "own files" for app in distinct if app.environment_name == environment_name] for kind, keys in (("environment name", lambda app: [app.environment_name]), ("folder", lambda app: [str(apps_folder(base.os.type) / app_folder(app))]), - ("program", lambda app: programs(app, base.os.type)), - ("port", lambda app: _ports(app, base.os.type))): + ("program", lambda app: programs(app, platform_of(base.os))), + ("port", lambda app: _ports(app, platform_of(base.os)))): claims: dict[object, list[str]] = {} for app in distinct: for key in keys(app): @@ -124,32 +125,33 @@ def _apps(base: ComputerArtifact, driver: ComputerDriverEnv, apps: list[Computer if len(owners) > 1 and frozenset(owners) not in named: named.add(frozenset(owners)) problems.append(f"apps: {' and '.join(owners)} would share the {kind} {key!r}") - owned = [(app, PurePosixPath(path)) for app in distinct for path in _paths(app, base.os.type)] + owned = [(app, PurePosixPath(path)) for app in distinct for path in _paths(app, platform_of(base.os))] problems += [f"apps: {one.id} keeps its data in {str(a)!r} and {other.id} in {str(b)!r}, so they'd share it" for (one, a), (other, b) in itertools.combinations(owned, 2) if one is not other and (a.is_relative_to(b) or b.is_relative_to(a))] return problems -def programs(app: ComputerAppEnv, os_type: str) -> list[str]: - """The names of the supervisord programs ``app`` runs on ``os_type``: its own server's, then its services'.""" +def programs(app: ComputerAppEnv, platform: str) -> list[str]: + """The names of the supervisord programs ``app`` runs on ``platform``, an OS type or ``/`` + (``install.platform_of``): its own server's, then its services'.""" server = f"app-{app_folder(app)}" - install = app.install.get(os_type) + install = entry_for(app.install, platform) return [server, *(f"{server}-{service.name}" for service in (install.services if install is not None else []))] -def _paths(app: ComputerAppEnv, os_type: str) -> list[str]: - declared = app.declaration.os.get(os_type) +def _paths(app: ComputerAppEnv, platform: str) -> list[str]: + declared = entry_for(app.declaration.os, platform) return list(declared.paths or []) if declared is not None else [] -def _ports(app: ComputerAppEnv, os_type: str) -> list[int]: - install = app.install.get(os_type) +def _ports(app: ComputerAppEnv, platform: str) -> list[int]: + install = entry_for(app.install, platform) return list(install.ports or []) if install is not None else [] -def _driver_ports(driver: ComputerDriverEnv, os_type: str) -> list[int]: - install = driver.install.get(os_type) +def _driver_ports(driver: ComputerDriverEnv, os: Any) -> list[int]: + install = for_os(driver.install, os) return [driver.port, *(install.ports or [] if install is not None else [])] @@ -236,8 +238,8 @@ def _platform(base: ComputerArtifact, driver: ComputerDriverEnv) -> list[str]: if not arches(base, driver): problems.append(f"driver: no architecture both run: {base.id} runs {', '.join(base.arch)}, and {driver.id} " f"{', '.join(sorted({p.arch for p in _matching(base, driver.declaration.supports)}))}") - if (install := driver.install.get(os.type)) is None: - problems.append(f"driver: {driver.id} declares no install for {os.type}") + if (install := for_os(driver.install, os)) is None: + problems.append(f"driver: {driver.id} declares no install for {_os_shown(base)}") else: init = base.forms.container.init if base.forms.container is not None else "launchd" problems += [f"driver: its service {service.name!r} runs as root, and {base.id}'s {init} runs every program as " diff --git a/src/agent_env/env/envs/computer/utils/install.py b/src/agent_env/env/envs/computer/utils/install.py index 933ca69f4..b6cbb02a2 100644 --- a/src/agent_env/env/envs/computer/utils/install.py +++ b/src/agent_env/env/envs/computer/utils/install.py @@ -14,12 +14,13 @@ class with ``@linux(...)``: ``files``, a folder of its own copied under the laye import re from pathlib import Path, PurePosixPath -from typing import Any, Literal +from typing import Any, Iterable, Literal, Optional, TypeVar from agentenv_protocol.computer.app import AptSource, Download, LayerPath, Packages, RunScript +from agentenv_protocol.computer.platform import entry_for, platform_key, supported from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator, model_validator -from agent_env.artifact.artifacts.computer import OS_TYPES +from agent_env.artifact.artifacts.computer import DISTRIBUTION, OS_TYPES #: The highest port a layer can listen on. MAX_PORT = 65535 @@ -170,20 +171,57 @@ class WindowsAppInstall(WindowsInstall): APP_INSTALLS: dict[str, type[Install]] = {"linux": AppInstall, "macos": MacAppInstall, "windows": WindowsAppInstall} +_Entry = TypeVar("_Entry") + + +def platform_problem(key: str) -> Optional[str]: + """What's wrong with ``key`` as what an install or a known failure is for: an OS type agent-env composes, or one + of its distributions as ``/`` (``linux/omarchy``), whose entry replaces its OS type's there.""" + os_type, slash, distribution = key.partition("/") + if os_type in OS_TYPES and (not slash or re.fullmatch(DISTRIBUTION, distribution)): + return None + return (f"an OS type agent-env composes: {', '.join(OS_TYPES)}, or one of its distributions as " + "/, such as linux/omarchy") + + +def os_type_of(key: str) -> str: + return key.partition("/")[0] + + +def platform_of(os: Any) -> str: + """The key a base's OS, ``os``, takes its installs, launches and known failures by: ``/``, or its + OS type when it names no distribution.""" + return platform_key(os.type, getattr(os, "distribution", None)) + + +def unsupported(keys: Iterable[str], supports: list[dict], at: str) -> list[str]: + """Each distribution of ``keys``, named from ``at``, that no ``supports`` entry names: an install or known failures + that would never apply, as for a misspelled distribution.""" + named = {platform_key(platform.os, platform.distribution) for entry in supports for platform in supported(entry) + if platform.distribution} + return [f"{at}.{key}: no supports entry names {key.partition('/')[2]!r} on {os_type_of(key)}, so this never " + "applies" for key in keys if "/" in key and key not in named] + + +def for_os(by_platform: dict[str, _Entry], os: Any) -> Optional[_Entry]: + """``by_platform``'s entry for ``os``, a base's OS: its distribution's when there's one, else its OS type's.""" + return entry_for(by_platform, platform_of(os)) + + def installs(data: Any, models: dict[str, type[Install]] = DRIVER_INSTALLS) -> tuple[dict[str, Install], list[str]]: - """The installs ``data``, an ``install`` table, declares by OS type, each its type's model of ``models``, and the - problems with it, each named by its key.""" + """The installs ``data``, an ``install`` table, declares by OS type or distribution (``platform_problem``), each + its OS type's model of ``models``, and the problems with it, each named by its key.""" if not isinstance(data, dict): return {}, [f"install must be a table of OS types ({', '.join(OS_TYPES)}), not {data!r}"] found, problems = {}, [] - for os_type, table in data.items(): - if os_type not in OS_TYPES: - problems.append(f"install.{os_type}: an install is for an OS type agent-env composes: {', '.join(OS_TYPES)}") + for key, table in data.items(): + if problem := platform_problem(key): + problems.append(f"install.{key}: an install is for {problem}") continue try: - found[os_type] = models[os_type].model_validate(table) + found[key] = models[os_type_of(key)].model_validate(table) except ValidationError as e: - problems.extend(f"install.{'.'.join(map(str, (os_type, *error['loc'])))}: {error['msg']}" + problems.extend(f"install.{'.'.join(map(str, (key, *error['loc'])))}: {error['msg']}" for error in e.errors()) return found, problems diff --git a/src/agent_env/env/envs/computer/utils/params.py b/src/agent_env/env/envs/computer/utils/params.py index 3d978b412..eb6442546 100644 --- a/src/agent_env/env/envs/computer/utils/params.py +++ b/src/agent_env/env/envs/computer/utils/params.py @@ -15,6 +15,7 @@ from agent_env.artifact.artifacts.computer import SettingValue from agent_env.env.envs.computer.utils import gate, sites +from agent_env.env.envs.computer.utils.install import platform_of if TYPE_CHECKING: from agent_env.env.envs.computer.env import ComputerEnv @@ -80,7 +81,8 @@ def checked(env: ComputerEnv, env_params: Optional[dict]) -> tuple[ComputerParam + (f"; its apps are {', '.join(apps)}" if apps else "") for name in params.app_settings if name not in apps] merged = app_settings(env, params.app_settings) for name, app in apps.items(): - problems += gate.check_app_settings(app, merged[name], f"env_params.app_settings.{name}", env.computer.os.type) + problems += gate.check_app_settings(app, merged[name], f"env_params.app_settings.{name}", + platform_of(env.computer.os)) if params.sites and not params.site_routing: problems.append("env_params.sites: a computer reaches sites only with site_routing") if params.site_routing and (os_type := env.computer.os.type) not in sites.TRUST_ENVIRONMENTS: diff --git a/src/agent_env/env/envs/computer/utils/probing.py b/src/agent_env/env/envs/computer/utils/probing.py index 3989a4b10..5a26c9ad1 100644 --- a/src/agent_env/env/envs/computer/utils/probing.py +++ b/src/agent_env/env/envs/computer/utils/probing.py @@ -15,6 +15,7 @@ from agent_env.env.env import Env from agent_env.env.envs.computer.utils import boot from agent_env.env.envs.computer.utils.composed import Record, failed, now +from agent_env.env.envs.computer.utils.install import for_os, platform_of from agent_env.store.ids import derive_id from agent_env.store.local_state import awaiting_lock @@ -104,7 +105,8 @@ def _reported(env: ComputerEnv, driver: ComputerDriverEnv, form: str, arch: str, probe = {"level": report.level, "provider": provider, "passed": report.passed, "checks": report.counts(), "extensions": report.extensions, "report": {"id": stored.id, "version": stored.version}, "probed_at": now()} - known = {(failure.check, failure.name): failure for failure in driver.known_failures.get(env.computer.os.type, [])} + known = {(failure.check, failure.name): failure + for failure in for_os(driver.known_failures, env.computer.os) or []} failed_checks = [c for c in report.checks if c.status == "failed"] if waived := [c for c in failed_checks if (c.check, c.name) in known]: probe["known_failures"] = [{"check": c.check, "name": c.name, "detail": c.detail, @@ -157,7 +159,7 @@ async def _probe(links: list[SandboxProvider], env: ComputerEnv, driver: Compute try: environment = { **boot.settings_environment(env.computer, env.computer_values(), environment_name=env.computer_name), - **boot.app_settings_environment(env.composed_apps(), env.app_defaults(), env.computer.os.type)} + **boot.app_settings_environment(env.composed_apps(), env.app_defaults(), platform_of(env.computer.os))} except ConfigError as e: # an env: or secret: reference that doesn't resolve here, or no secret store to resolve it raise Unprobed(f"a setting it boots with didn't resolve here: {e}") from e problems = [] @@ -165,8 +167,8 @@ async def _probe(links: list[SandboxProvider], env: ComputerEnv, driver: Compute try: booted = await boot.boot(link, image=image, port=driver.port, environment=environment, token=token, network_policy=None, timeout=_PROBE_TTL_S, attribution=None, - ready_s=boot.ready_timeout(env.composed_apps(), env.computer.os.type), form=form, - os_type=env.computer.os.type) + ready_s=boot.ready_timeout(env.composed_apps(), platform_of(env.computer.os)), + form=form, os_type=env.computer.os.type) except Exception as e: problems.append(f"{type(link).__name__}: {e}") continue diff --git a/src/agent_env/providers/env_providers/env_computer_provider.py b/src/agent_env/providers/env_providers/env_computer_provider.py index 311d2d3da..d088f4e06 100644 --- a/src/agent_env/providers/env_providers/env_computer_provider.py +++ b/src/agent_env/providers/env_providers/env_computer_provider.py @@ -21,6 +21,7 @@ from agent_env.env.env import DeployedSandboxEnv, EnvCapabilityUnsupported from agent_env.env.envs.computer.utils import composer from agent_env.env.envs.computer.utils import params as computer_params +from agent_env.env.envs.computer.utils.install import platform_of from agent_env.env.envs.computer.utils.probing import probe_first_boot from agent_env.env.envs.computer.utils import sites from agent_env.env.envs.computer.utils.boot import ( @@ -164,7 +165,7 @@ async def _boot(self, env: ComputerEnv, builds: list[tuple[str, str, dict]], par computer, name, token = env.computer_artifact(), env.environment_name, control_token() values = _values(computer, values) app_settings = recorded_app_settings(env.composed_apps(), computer_params.app_settings(env, params.app_settings), - env.computer.os.type) + platform_of(env.computer.os)) policy = _reaching(NetworkPolicy.from_dict(env.network) if env.network else None, sandbox_provider, reach) booted = await boot( sandbox_provider, image=build["image"], port=env.driver.port, network_policy=policy, token=token, @@ -172,8 +173,8 @@ async def _boot(self, env: ComputerEnv, builds: list[tuple[str, str, dict]], par timeout=ttl_seconds, attribution=attribution, disk_size_gb=disk_size_gb, cpu=cpu, memory_mb=memory_mb, - ready_s=ready_timeout(env.composed_apps(), env.computer.os.type), form=form, held=bool(params.site_routing), - os_type=env.computer.os.type) + ready_s=ready_timeout(env.composed_apps(), platform_of(env.computer.os)), form=form, + held=bool(params.site_routing), os_type=env.computer.os.type) self._container_sandboxes.append(booted.sandbox) self._environment_sandboxes[name] = booted.sandbox try: @@ -223,7 +224,7 @@ async def reboot(self, env: ComputerEnv, deployed: DeployedSandboxEnv) -> None: env=environment) except NotImplementedError as e: # a reset the env supports, which this provider can't carry out raise RuntimeError(f"'{env.id}' can't be reset on {type(sandbox_provider).__name__}: {e}") from None - card = await ready(deployed.environment_url, ready_timeout(env.composed_apps(), env.computer.os.type), + card = await ready(deployed.environment_url, ready_timeout(env.composed_apps(), platform_of(env.computer.os)), deployed.control_token, answering=bool(params.site_routing)) _require_name(env, card) valid_for_s = (record.get("sites") or {}).get("valid_for_s", 0) @@ -267,9 +268,9 @@ async def _launch_held_apps(env: ComputerEnv, record: DeployedSandboxEnv, url: s waited to answer; the record keeps the card the computer answers then, as its boot's card may list an app by its name alone. Each is launched as the child its card lists, as a computer named for its app, a browser alone, has that name too.""" - os_type = env.computer.os.type - timeout = ready_timeout(env.composed_apps(), os_type) - held = [app for app in env.composed_apps() if (launch := app.launch(os_type)) is not None and launch.at == "boot"] + platform = platform_of(env.computer.os) + timeout = ready_timeout(env.composed_apps(), platform) + held = [app for app in env.composed_apps() if (launch := app.launch(platform)) is not None and launch.at == "boot"] launchers: dict[str, protocol_v1.ChildEnv] = {} if held: record.environment_card = await listed(url, [app.environment_name for app in held], timeout) @@ -351,7 +352,7 @@ def _environment(env: ComputerEnv, computer: ComputerArtifact, values: dict, return {**settings_environment(computer, values, environment_name=env.computer_name, driver_settings=params.driver_settings, tool_prefix=tool_prefix), **app_settings_environment(env.composed_apps(), computer_params.app_settings(env, params.app_settings), - env.computer.os.type), + platform_of(env.computer.os)), **({HOLD_BOOT_LAUNCH_ENV: "1", **sites.trust_environment(env.computer.os.type, params.sites, later=sites_later), **sites.browser_environment(env.computer.os.display)} if params.site_routing else {})} diff --git a/tst/unit/bundle/computer_app_env_test.py b/tst/unit/bundle/computer_app_env_test.py index c1b567348..59b94702a 100644 --- a/tst/unit/bundle/computer_app_env_test.py +++ b/tst/unit/bundle/computer_app_env_test.py @@ -124,8 +124,12 @@ def test_an_app_folders_settings_table_is_stored_as_the_defaults_each_deploy_sta "build_settings.colour: @local/~/apps/mail declares no build setting 'colour'"), (f'{TOML}[settings]\ncolour = "blue"\n', {}, "skill.md", "settings.colour: @local/~/apps/mail declares no setting 'colour'"), + (TOML, {"envs/mail/app.py": APP.format(supports=LINUX, skill="skill.md", linux_skill="skill-linux.md").replace( + "@linux(python=", '@linux(distribution="omarchy", services=[Service(name="imap", command="imap")])\n' + "@linux(python=", 1)}, + "skill.md", "install.linux/omarchy: no supports entry names 'omarchy' on linux, so this never applies"), ], ids=["no-module", "a-driver", "install-python", "skill", "linux-skill", "unknown-key", "build-setting", - "setting"]) + "setting", "unsupported-distribution"]) def test_an_app_folder_that_cant_be_read_is_refused_before_any_write(bundle_dir, toml, extra, skill, problem): _bundle(bundle_dir, toml=toml, skill=skill, **{path: data for path, data in extra.items() if data is not None}) for path in (path for path, data in extra.items() if data is None): diff --git a/tst/unit/bundle/computer_driver_env_test.py b/tst/unit/bundle/computer_driver_env_test.py index d76febec6..2b3eac873 100644 --- a/tst/unit/bundle/computer_driver_env_test.py +++ b/tst/unit/bundle/computer_driver_env_test.py @@ -179,9 +179,17 @@ def test_a_driver_is_written_again_only_when_one_of_its_files_changes(bundle_dir (f'{TOML}[install.linux]\ndownloads = [{{ url = "https://example.com/t", sha256 = "abc", to = "t" }}]\n{SERVE}\n', LINUX, ("left",), "install.linux.downloads.0.sha256: String should match pattern"), (f'{TOML}[install.linux]\nports = [70000]\n{SERVE}\n', LINUX, ("left",), "a port is 1 to 65535, not [70000]"), + (f'{TOML}[install."linux/Omarchy"]\n{SERVE}\n', LINUX, ("left",), + "install.linux/Omarchy: an install is for an OS type agent-env composes: linux, macos, windows, or one of its " + "distributions as /"), + (f'{TOML}[install."linux/omarchy"]\n{SERVE}\n', LINUX, ("left",), + "install.linux/omarchy: no supports entry names 'omarchy' on linux, so this never applies"), + (f'{TOML}[[known_failures."linux/omarchy"]]\ncheck = "P6"\nname = "typed text"\nreason = "its plugin"\n', LINUX, + ("left",), "known_failures.linux/omarchy: no supports entry names 'omarchy' on linux, so this never applies"), ], ids=["no-module", "module-file", "module-class", "button", "no-platform", "unknown-key", "port", "install-os", "install-python", "install-files", "install-escape", "no-service", "service-twice", "service-scope", - "run-missing", "apt-word", "sources-alone", "download-sha", "install-port"]) + "run-missing", "apt-word", "sources-alone", "download-sha", "install-port", "distribution-key", + "unsupported-distribution", "unsupported-known-failures"]) def test_a_driver_its_class_cant_be_read_from_is_refused_before_any_write(bundle_dir, toml, supports, buttons, problem): _bundle(bundle_dir, toml=toml, supports=supports, buttons=buttons) @@ -191,6 +199,22 @@ def test_a_driver_its_class_cant_be_read_from_is_refused_before_any_write(bundle Env.get(f"{ROOT}/desk") +def test_a_driver_takes_an_install_and_known_failures_for_a_distribution_it_supports(bundle_dir): + omarchy = [{"os": {"type": "linux", "display": ["x11"]}, "arch": ["amd64", "arm64"]}, + {"os": {"type": "linux", "distribution": "omarchy", "display": ["wayland"]}, "arch": ["amd64"]}] + _bundle(bundle_dir, supports=omarchy, toml=( + f'{TOML}[install.linux]\n{SERVE}\n\n[install."linux/omarchy"]\npython = "requirements.txt"\n{SERVE}\n\n' + '[[known_failures."linux/omarchy"]]\ncheck = "P6"\nname = "typed text"\nreason = "its plugin"\n'), + **{"envs/desk/requirements.txt": "mss>=9,<11\n"}) + + _run(bundle_dir) + + desk = Env.get(f"{ROOT}/desk") + assert set(desk.install) == {"linux", "linux/omarchy"} and desk.install["linux/omarchy"].python == "requirements.txt" + assert [failure.name for failure in desk.known_failures["linux/omarchy"]] == ["typed text"] + assert Env.get(desk.id, desk.version).install["linux/omarchy"].python == "requirements.txt" + + @pytest.mark.parametrize("line", ["-r base.txt", "--requirement=base.txt", "-c pins.txt", "--constraint pins.txt", "-e .", "--editable ./desk", "./vendored", "../elsewhere/pkg", "/opt/pkg", "~/pkg", "file:///opt/pkg", "desk @ file:///opt/pkg", "-f wheels/", diff --git a/tst/unit/env/envs/test_computer_env.py b/tst/unit/env/envs/test_computer_env.py index 9cc0cddbb..2e1baafd5 100644 --- a/tst/unit/env/envs/test_computer_env.py +++ b/tst/unit/env/envs/test_computer_env.py @@ -374,6 +374,25 @@ async def overran(*args): "failed", "probe", f"the probe didn't finish within {probing._PROBE_TIMEOUT_S} s") +def test_known_failures_for_a_distribution_are_waived_on_it_alone_and_replace_its_os_types_there( + base, docker, tmp_path): + docker.probe_fails["arm64"] = "the click never landed" + clicks = KnownFailure(check="P4", name="clicks show their effect", reason="its clicks lag") + elsewhere = _driver(tmp_path / "elsewhere", known_failures={"linux/omarchy": [clicks]}) + here = _driver(tmp_path / "here", known_failures={ + "linux": [KnownFailure(check="P0", name="the card", reason="its card was late")], "linux/ubuntu": [clicks]}) + + on_elsewhere = Env.get(_put(base, elsewhere, id="@local/~/kit/elsewhere", environment_name="elsewhere").id + ).metadata["composed"]["container"]["arm64"] + on_here = Env.get(_put(base, here, id="@local/~/kit/here", environment_name="here").id).metadata[ + "composed"]["container"]["arm64"] + docker.probe_fails.clear() + + assert on_elsewhere["status"] == "failed" + assert on_here["status"] == "ready" and on_here["probe"]["known_failures"][0]["reason"] == "its clicks lag" + assert "unused_known_failures" not in on_here["probe"] + + def test_a_check_the_driver_declares_it_fails_is_waived_and_recorded_and_a_declared_one_that_didnt_fail_is_flagged( base, docker, tmp_path, caplog): docker.probe_fails["arm64"] = "the click never landed" @@ -1807,6 +1826,16 @@ def test_the_locks_debs_and_wheels_are_mounted_into_the_runs_that_install_them_n build["files"]) +def test_an_app_that_declares_for_a_distribution_is_told_the_platform_it_was_composed_for(base, driver, docker, + tmp_path): + _app(base, driver, tmp_path / "mail", steps=[("index_mail", "session", "linux/ubuntu")]) + + build = {build["platform"]: build for build in docker.builds}["linux/arm64"] + assert ("RUN --network=none cd /opt/agentenv/apps/mail && AGENTENV_PLATFORM=linux/ubuntu venv/bin/python -m " + "agentenv_protocol.computer.run_install_step app.py:Mail index_mail") in build["dockerfile"] + assert 'AGENTENV_PLATFORM="linux/ubuntu"' in build["confs"]["agentenv-app-mail.conf"] + + def test_an_apps_packages_downloads_scripts_and_steps_are_fetched_at_put_and_installed_offline(base, driver, docker, tmp_path, monkeypatch): tool = b"#!/bin/sh\necho tool\n" diff --git a/tst/unit/env/envs/test_install_platform_keys.py b/tst/unit/env/envs/test_install_platform_keys.py new file mode 100644 index 000000000..49dded86f --- /dev/null +++ b/tst/unit/env/envs/test_install_platform_keys.py @@ -0,0 +1,22 @@ +from types import SimpleNamespace + +from agent_env.env.envs.computer.utils.install import Install, for_os, installs + + +def test_an_install_is_for_an_os_type_or_one_of_its_distributions(): + serves = {"services": [{"name": "driver", "command": "venv/bin/python -m driver"}]} + found, problems = installs({"linux": serves, "linux/omarchy": {**serves, "python": "requirements.txt"}, + "plan9": serves, "linux/": serves, "linux/Omarchy": serves}) + + assert set(found) == {"linux", "linux/omarchy"} and isinstance(found["linux/omarchy"], Install) + assert [problem.partition(":")[0] for problem in problems] == ["install.plan9", "install.linux/", + "install.linux/Omarchy"] + + +def test_a_distributions_entry_replaces_its_os_types_there_and_only_there(): + by_platform = {"linux": "any linux", "linux/omarchy": "omarchy"} + + assert for_os(by_platform, SimpleNamespace(type="linux", distribution="omarchy")) == "omarchy" + assert for_os(by_platform, SimpleNamespace(type="linux", distribution="ubuntu")) == "any linux" + assert for_os(by_platform, SimpleNamespace(type="linux")) == "any linux" + assert for_os({"linux/omarchy": "omarchy"}, SimpleNamespace(type="linux", distribution="ubuntu")) is None