Skip to content

About

Long-running autonomous coding loop: implement → test → fix, powered by AI agents (Claude Code & Codex)

Resources

Stars

15 stars

Watchers

0 watching

Forks

Latest commit

 

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Long-Running Coding Loop

A fully autonomous implement - test - fix loop for AI coding agents. Define tasks in JSON, and the loop runs for hours — planning, implementing, compiling, testing, fixing, and committing — without human intervention.

Supports Claude Code (run_claude_code.py) and OpenAI Codex CLI (run_codex.py).

Battle-Tested

We used this system to implement 11 features for a Chromium-based enterprise browser in a single overnight run:

Metric Value
Tasks completed 11/11
Total runtime ~12 hours
Total cost ~$181 (Claude API)
Human intervention 0
Features JS dialogs, file chooser, permissions, fullscreen, downloads, toast, notifications, badges, keyboard input, etc.

Each feature went through the full cycle: plan - implement - compile - test - (fix - retest) completely autonomously. The test agent verified goals using screenshots, accessibility tree inspection, log analysis, and unit tests.

How It Works

tasks.json [pending]
    |
    v
Phase 0: PLAN -- planner agent reads reference code, generates Goals (test acceptance criteria)
    |
    v
Phase 1: IMPLEMENT -- main agent writes code based on task description + goals
    |
    v
Phase 2: COMPILE -- automatic build check
    |
    v
Phase 3: TEST -- test agent verifies goals (unit tests - protocol tests - E2E)
    |           |
    | PASS      | FAIL
    v           v
git commit    Phase 1: FIX -- agent reads failure report, fixes code, back to compile
    |
    v
next task

Project Structure

coding-loop/
  agent-loop/
    run_claude_code.py    # Loop runner for Claude Code CLI
    run_codex.py          # Loop runner for OpenAI Codex CLI
    tasks.json            # Your task definitions (create this)
    evidence/             # Auto-created: screenshots, logs, result.md per task
    loop.log              # Auto-created: timestamped execution log
  agents/
    planner.md            # Planner agent: reads reference code, generates Goals + task descriptions
    test.md               # Test agent: verifies goals via layered testing
  examples/
    tasks-example.json    # Example tasks.json

Quick Start

1. Prerequisites

Your project needs a CLAUDE.md (for Claude Code) or equivalent project context file. This file tells the AI agent about your project: structure, build commands, coding conventions, and important rules. The coding loop agents inherit this context -- without it, they won't know how to build your project.

Key CLAUDE.md sections:

  • Build commands: npm run build, make, cargo build, etc.
  • Test commands: npm test, pytest, etc.
  • Project structure: where source/test files live
  • Code style: conventions to follow
  • Gotchas: e.g., "must recompile backend after C++ changes"

2. Set up agents

Copy agents/*.md to your project's .claude/agents/ directory:

cp agents/planner.md /path/to/your/project/.claude/agents/
cp agents/test.md /path/to/your/project/.claude/agents/

Customize test.md for your project's testing framework and tools.

3. Define tasks

Create agent-loop/tasks.json (see examples/tasks-example.json):

{
  "tasks": [
    {
      "id": "my-feature",
      "name": "My Feature",
      "description": "Detailed implementation guide -- problem, files, approach, code snippets...",
      "goals_file": "Tests/Goals/my-feature.md",
      "reference_files": ["path/to/reference.cc"],
      "status": "pending",
      "attempts": 0,
      "max_attempts": 5
    }
  ]
}

The description field is critical -- it's the only guidance the implement agent gets. Write it like briefing a colleague: problem, files to change, approach, with code snippets.

4. Run

# Claude Code
python3 agent-loop/run_claude_code.py

# OpenAI Codex (with build command)
python3 agent-loop/run_codex.py --build-cmd npm run build

# Options
--task my-feature    # Run one task
--dry-run            # Preview tasks
--reset my-feature   # Reset stuck task
--safe               # Require approval for each action

5. Monitor

# Watch the log
tail -f agent-loop/loop.log

# Check status
python3 -c "
import json
for t in json.load(open('agent-loop/tasks.json'))['tasks']:
    print(f\"  {t['id']:20s} {t.get('status','?'):12s} attempts={t.get('attempts',0)}\")
"

Key Design Decisions

Description-Driven Implementation

The implement agent receives ONLY the task description as its prompt. All context must be in the description -- file paths, code snippets, API formats, implementation order.

Goal-First Testing

Goals are natural-language acceptance criteria generated by the planner agent. The test agent decides HOW to verify each goal.

Layered Test Strategy

Level Method When to use
1 Unit/integration tests Data logic, parsing, services
2 Protocol/API injection Event handling, request/response
3 E2E (UI automation) Visual layout, user interaction flows

Auto-Fix Loop

Test failures are fed back to the implement agent for fixing, up to max_attempts cycles.

Shared Knowledge (LESSONS.md)

Test agents write tool tips and pitfalls to Tests/Tools/LESSONS.md. Future runs read this first, preventing repeated mistakes.

Configuration

Customize compile_check() in the runner script for your build system. Defaults:

Setting Default Description
DEFAULT_MAX_ATTEMPTS 5 Max fix cycles per task
DEFAULT_MAX_TURNS 0 Agent turns (0 = unlimited)
DEFAULT_TIMEOUT 0 Seconds per run (0 = unlimited)

Lessons Learned

  1. Task descriptions must be extremely detailed -- the agent has no other context
  2. Layered testing saves time -- unit tests are 10x faster than E2E
  3. LESSONS.md prevents repeated mistakes -- test agents learn from each other
  4. Don't mock what you can test directly -- fake servers are fragile
  5. Add "must recompile" reminders to CLAUDE.md for compiled backends

Requirements

  • Claude Code: claude CLI (claude.ai/code)
  • Codex: codex CLI (npm install -g @openai/codex) + OPENAI_API_KEY
  • Python 3.9+
  • A CLAUDE.md or equivalent project context file

License

MIT

About

Long-running autonomous coding loop: implement → test → fix, powered by AI agents (Claude Code & Codex)

Resources

Stars

15 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages