Public agent benchmarks as agent-env bundles. Each benchmark runs
with agent-env run <name>, on agent-env's own environments, against the agents agent-env ships.
| Benchmark | Bundle | Tasks | Arms | Source |
|---|---|---|---|---|
| Online-Mind2Web | online-mind2web |
300 live-web tasks on 147 sites | Browser Use over CDP; Claude Code with Playwright MCP | osunlp/Online-Mind2Web @ eacad896 |
uv tool install agentenv-framework --with git+https://github.com/earakely-scale/agentenv-benchmarks
agent-env run online-mind2web --eval sampleagent-env run with no bundle lists the installed ones. From a checkout, a bundle also runs by its folder:
git clone https://github.com/earakely-scale/agentenv-benchmarks && cd agentenv-benchmarks
uv sync
uv run agent-env run src/agentenv_benchmarks/bundles/online-mind2web --eval sampleThe agents need a model endpoint: [model] base_url and api_key in agent-env's .agentenv/config.toml, or
LITELLM_BASE_URL and LITELLM_API_KEY. agent-env's docs cover its
configuration, sandboxes and stores.
src/agentenv_benchmarks/
bundles/<name>/ a benchmark's bundle: envs, agents, artifacts, tasks/ and evals/, as agent-env reads them
<name with underscores>/ the code that generates its tasks from its source dataset, when they're generated
tests/ checks every bundle parses and its tasks are what its generator makes
- Put its bundle in
src/agentenv_benchmarks/bundles/<name>/, with aREADME.mdwhose first line describes it and abundle.tomlnaming the agentenv-framework it needs. - If its tasks come from a dataset, add
src/agentenv_benchmarks/<name>/with the dataset pinned to a revision and a__main__that writes the tasks, asonline_mind2webdoes, and commit what it writes. - Register it in
pyproject.tomlunder[project.entry-points."agent_env.bundles"]:<name> = "agentenv_benchmarks.bundles". - Add it to the table above and its tests to
tests/.
uv sync --group dev
uv run ruff check
uv run pytestRegenerating Online-Mind2Web's tasks needs a Hugging Face login that has accepted the dataset's terms:
hf auth login
uv run --extra generate python -m agentenv_benchmarks.online_mind2webThe code is Apache-2.0. Each benchmark's tasks keep their dataset's license and attribution; see NOTICE.