A fully autonomous implement - test - fix loop for AI coding agents. Define tasks in JSON, and the loop runs for hours — planning, implementing, compiling, testing, fixing, and committing — without human intervention.
Supports Claude Code (run_claude_code.py) and OpenAI Codex CLI (run_codex.py).
We used this system to implement 11 features for a Chromium-based enterprise browser in a single overnight run:
| Metric | Value |
|---|---|
| Tasks completed | 11/11 |
| Total runtime | ~12 hours |
| Total cost | ~$181 (Claude API) |
| Human intervention | 0 |
| Features | JS dialogs, file chooser, permissions, fullscreen, downloads, toast, notifications, badges, keyboard input, etc. |
Each feature went through the full cycle: plan - implement - compile - test - (fix - retest) completely autonomously. The test agent verified goals using screenshots, accessibility tree inspection, log analysis, and unit tests.
tasks.json [pending]
|
v
Phase 0: PLAN -- planner agent reads reference code, generates Goals (test acceptance criteria)
|
v
Phase 1: IMPLEMENT -- main agent writes code based on task description + goals
|
v
Phase 2: COMPILE -- automatic build check
|
v
Phase 3: TEST -- test agent verifies goals (unit tests - protocol tests - E2E)
| |
| PASS | FAIL
v v
git commit Phase 1: FIX -- agent reads failure report, fixes code, back to compile
|
v
next task
coding-loop/
agent-loop/
run_claude_code.py # Loop runner for Claude Code CLI
run_codex.py # Loop runner for OpenAI Codex CLI
tasks.json # Your task definitions (create this)
evidence/ # Auto-created: screenshots, logs, result.md per task
loop.log # Auto-created: timestamped execution log
agents/
planner.md # Planner agent: reads reference code, generates Goals + task descriptions
test.md # Test agent: verifies goals via layered testing
examples/
tasks-example.json # Example tasks.json
Your project needs a CLAUDE.md (for Claude Code) or equivalent project context file. This file tells the AI agent about your project: structure, build commands, coding conventions, and important rules. The coding loop agents inherit this context -- without it, they won't know how to build your project.
Key CLAUDE.md sections:
- Build commands:
npm run build,make,cargo build, etc. - Test commands:
npm test,pytest, etc. - Project structure: where source/test files live
- Code style: conventions to follow
- Gotchas: e.g., "must recompile backend after C++ changes"
Copy agents/*.md to your project's .claude/agents/ directory:
cp agents/planner.md /path/to/your/project/.claude/agents/
cp agents/test.md /path/to/your/project/.claude/agents/Customize test.md for your project's testing framework and tools.
Create agent-loop/tasks.json (see examples/tasks-example.json):
{
"tasks": [
{
"id": "my-feature",
"name": "My Feature",
"description": "Detailed implementation guide -- problem, files, approach, code snippets...",
"goals_file": "Tests/Goals/my-feature.md",
"reference_files": ["path/to/reference.cc"],
"status": "pending",
"attempts": 0,
"max_attempts": 5
}
]
}The description field is critical -- it's the only guidance the implement agent gets. Write it like briefing a colleague: problem, files to change, approach, with code snippets.
# Claude Code
python3 agent-loop/run_claude_code.py
# OpenAI Codex (with build command)
python3 agent-loop/run_codex.py --build-cmd npm run build
# Options
--task my-feature # Run one task
--dry-run # Preview tasks
--reset my-feature # Reset stuck task
--safe # Require approval for each action# Watch the log
tail -f agent-loop/loop.log
# Check status
python3 -c "
import json
for t in json.load(open('agent-loop/tasks.json'))['tasks']:
print(f\" {t['id']:20s} {t.get('status','?'):12s} attempts={t.get('attempts',0)}\")
"The implement agent receives ONLY the task description as its prompt. All context must be in the description -- file paths, code snippets, API formats, implementation order.
Goals are natural-language acceptance criteria generated by the planner agent. The test agent decides HOW to verify each goal.
| Level | Method | When to use |
|---|---|---|
| 1 | Unit/integration tests | Data logic, parsing, services |
| 2 | Protocol/API injection | Event handling, request/response |
| 3 | E2E (UI automation) | Visual layout, user interaction flows |
Test failures are fed back to the implement agent for fixing, up to max_attempts cycles.
Test agents write tool tips and pitfalls to Tests/Tools/LESSONS.md. Future runs read this first, preventing repeated mistakes.
Customize compile_check() in the runner script for your build system. Defaults:
| Setting | Default | Description |
|---|---|---|
DEFAULT_MAX_ATTEMPTS |
5 | Max fix cycles per task |
DEFAULT_MAX_TURNS |
0 | Agent turns (0 = unlimited) |
DEFAULT_TIMEOUT |
0 | Seconds per run (0 = unlimited) |
- Task descriptions must be extremely detailed -- the agent has no other context
- Layered testing saves time -- unit tests are 10x faster than E2E
- LESSONS.md prevents repeated mistakes -- test agents learn from each other
- Don't mock what you can test directly -- fake servers are fragile
- Add "must recompile" reminders to CLAUDE.md for compiled backends
- Claude Code:
claudeCLI (claude.ai/code) - Codex:
codexCLI (npm install -g @openai/codex) +OPENAI_API_KEY - Python 3.9+
- A
CLAUDE.mdor equivalent project context file
MIT