Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ Rules that must hold:
- **Agents never touch the board.** The item reaches them via `render_item_prompt`; `parse_board_update` reads an optional ` ```syncmill-board ` block, **never raises** (catches `RecursionError` too), sanitizes each field independently, drops a malformed `outputs` list whole, and agrees with `strip_board_block` on block bounds (a genuine close fence is required).
- **Load-bearing, built lazily.** Disabled by default (`build_board` → `None`). Built on first `run_item`, never in `Supervisor.__init__`, so a corrupt board cannot break a plain `run`. `Board()` raises only `BoardError` (`sqlite3.Error`, `OSError`, and `RuntimeError` from `expanduser`, which runs inside the try). Dispatch and mutation (`board` commands, `dashboard`) convert it to `ClickException` — no fallback. Read-mostly consumer surfaces degrade instead: `serve` keeps the shared-memory tools up and its `board_*` tools report the board as unavailable (not "not enabled"); `tui` opens in limited mode.
- **State machine `planned → running → (blocked | review) → done`.** `done` requires ≥1 passing `Evidence` or a human sign-off, counting evidence already on the item plus any supplied now; `board update ID --state done` with neither is rejected. An agent's advisory `done` counts only with passing verification from **this** run, else it is clamped to `review` (never raises). Humans cannot move an item into `running`.
- **Atomic claim + lease fencing.** Every claim path (`claim(id)` and `next_runnable`) is a `BEGIN IMMEDIATE` CAS `planned→running` (autocommit `isolation_level=None`, WAL, `busy_timeout`). Automatic selection (`next_runnable`) additionally requires: not `human_required`, every `depends_on` item `done` (a missing dependency is unsatisfied), FIFO `created_at, rowid`, optional exact `goal`. An explicit `claim(id)` checks only id + `planned` — it is the operator override. The lease is a fencing token: `apply_update(run_id=)` and `_block_on_crash` CAS on `state='running' AND lease_run_id=?` and drop a 0-row match. A same-state annotate keeps the lease; a no-`--state` annotate resolves its target inside the transaction. Every transactional method rolls back on **any** exception. `recover_stale_running()` runs at the top of every `run_item`.
- **Atomic claim + lease fencing.** Every claim path (`claim(id)` and `next_runnable`) is a `BEGIN IMMEDIATE` CAS `planned→running` (autocommit `isolation_level=None`, WAL, `busy_timeout`). Automatic selection (`next_runnable`) additionally requires: not `human_required`, every `depends_on` item `done` (a missing dependency is unsatisfied; an unreadable `depends_on` is unsatisfied and skipped with a warning, never blocking the rest), FIFO `created_at, rowid`, optional exact `goal`. An explicit `claim(id)` checks only id + `planned` — it is the operator override. The lease is a fencing token: `apply_update(run_id=)` and `_block_on_crash` CAS on `state='running' AND lease_run_id=?` and drop a 0-row match. A same-state annotate keeps the lease; a no-`--state` annotate resolves its target inside the transaction. Every transactional method rolls back on **any** exception. `recover_stale_running()` runs at the top of every `run_item`.
- **Memory boundary.** Items run with `SyncmillMemory.scoped(run_id)`, collapsing shared and per-agent namespaces to `board-run:{run_id}`; scoped reads also union `shared` (D4), writes never leave the scope. Afterwards one best-effort sanitized summary goes to global `shared` via `share_board_result` (output with the board block stripped, never the enriched prompt) — **only if the fenced `apply_update` applied**; a run that lost its lease publishes nothing. Decisions (`active_decisions(goal=item.goal)`: active global decisions plus that exact goal's) and the handoff baton are injected deterministically, never via top-k recall. Scratch namespaces are GC'd via `collectible_board_run_ids`, which reads ownership and liveness in one SQLite snapshot (a claim commits its lease and ledger row atomically) and excludes active leases and reclaimed runs still within one lease timeout of grace; the sweep runs under a cross-process lock, tombstones only acknowledged deletions, and never touches `shared` or `agent-runtime:*`.
- **Crash vs failure.** A crash (`BaseException`) moves the item to `blocked` + `human_required` and re-raises; a FAILED result is not a crash (`apply_update` records `blocked`).
- **Audit compaction.** `board compact` and the opt-in `board.transitions_retain_seconds` (default `0` = off) share `compact_transitions`: fold stats aggregates, then delete, in one transaction, so `stats()` is unchanged. The automatic call runs only at `run_item`'s housekeeping points (after the outcome, or on an empty claim; never on the claim or crash path) and degrades to a warning.
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture/board.md

Large diffs are not rendered by default.

42 changes: 37 additions & 5 deletions src/syncmill/board.py
Original file line number Diff line number Diff line change
Expand Up @@ -708,7 +708,9 @@ def next_runnable(self, run_id: str, goal: str | None = None) -> WorkItem | None
"""Atomically claim the next dispatchable item for ``run_id``, or ``None``.

Dispatchable = ``planned``, not ``human_required``, and every ``depends_on``
item is ``done``. Items are considered in FIFO order (``created_at``). The
item is ``done``. An unreadable ``depends_on`` (:func:`_parse_deps`) is
unsatisfied: the row is skipped with a warning, so it neither dispatches
nor blocks the items behind it. Items are considered in FIFO order (``created_at``). The
whole select-then-claim runs in one ``BEGIN IMMEDIATE`` transaction, so it
is as atomic as :meth:`claim`.

Expand Down Expand Up @@ -741,7 +743,19 @@ def next_runnable(self, run_id: str, goal: str | None = None) -> WorkItem | None
done_ids: set = set()
looked_up: set = set()

def _deps_done(deps: list) -> bool:
def _deps_done(row: sqlite3.Row) -> bool:
deps = _parse_deps(row["depends_on"])
if deps is None:
# Unsatisfied, never dispatched: a prerequisite list that
# cannot be read cannot be proven done. Skipping the row
# rather than raising keeps one damaged item from
# blocking every other item's automatic claim.
logger.warning(
"board item %s has an unreadable depends_on; automatic "
"dispatch skips it (fix the item, or claim it by id)",
_p(str(row["id"])),
)
return False
unknown = [d for d in deps if d not in looked_up]
for start in range(0, len(unknown), 500):
chunk = unknown[start : start + 500]
Expand All @@ -758,7 +772,7 @@ def _deps_done(deps: list) -> bool:
return self._deps_satisfied(deps, done_ids)

chosen = next(
(r for r in rows if _deps_done(json.loads(r["depends_on"]))),
(r for r in rows if _deps_done(r)),
None,
)
if chosen is None:
Expand All @@ -768,8 +782,7 @@ def _deps_done(deps: list) -> bool:
self._conn.execute("COMMIT")
return item
except Exception as exc:
# Broad on purpose: a non-sqlite error mid-transaction (e.g. a corrupt
# `depends_on` JSON failing `json.loads` above → ValueError) must still
# Broad on purpose: a non-sqlite error mid-transaction must still
# roll back the open BEGIN IMMEDIATE — otherwise the RESERVED lock leaks
# and the next op fails with "cannot start a transaction within a
# transaction". Surfaces as BoardError, like get()/list() already do.
Expand Down Expand Up @@ -1948,6 +1961,25 @@ def _row_to_decision(row: sqlite3.Row) -> Decision:
)


def _parse_deps(raw: object) -> list[str] | None:
"""A stored ``depends_on`` as dispatch reads it, or ``None`` when damaged.

Only a JSON list of strings is a dependency list. Anything else is
damaged, including shapes that iterate without error: an empty ``{}`` or
``""`` would make every dependency vacuously done, and ``"abc"`` would be
read as the three ids ``a``, ``b`` and ``c``.
"""
if not isinstance(raw, str):
return None
try:
value = json.loads(raw)
except (ValueError, RecursionError):
return None
if not isinstance(value, list) or not all(isinstance(d, str) for d in value):
return None
return value


def _load_list(raw: str) -> list[str]:
try:
v = json.loads(raw)
Expand Down
5 changes: 3 additions & 2 deletions src/syncmill/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@
)
from syncmill.worktree import (
WorkspaceError,
_run_git,
capture_repository_snapshot,
cleanup_orphaned_worktrees,
discard_retained_workspace,
Expand Down Expand Up @@ -853,7 +854,7 @@ def _gate_for_this_run(commands: list[str] | None):


def _dirty_count(root: Path) -> int:
proc = subprocess.run(
proc = _run_git(
["git", "status", "--porcelain"], cwd=str(root),
capture_output=True, text=True, env=isolated_git_env(),
)
Expand Down Expand Up @@ -1315,7 +1316,7 @@ def subprocess_status_dirty(root: Path) -> bool:
reported an unreadable or rejected repository ("dubious ownership") as
having nothing to preserve.
"""
proc = subprocess.run(
proc = _run_git(
["git", "status", "--porcelain"], cwd=str(root),
capture_output=True, text=True, env=isolated_git_env(),
)
Expand Down
11 changes: 11 additions & 0 deletions src/syncmill/memory_migration.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,11 +440,22 @@ def _atomic_json(path: Path, value: Any, *, mode: int = 0o600) -> None:
try:
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(json.dumps(value, ensure_ascii=False, indent=2) + "\n")
# Durable before it is published: apply, verify and rollback
# decide from these files, and a rename that reaches the disk
# ahead of the data can leave an empty state file after a crash.
fh.flush()
os.fsync(fh.fileno())
os.chmod(tmp, mode)
os.replace(tmp, path)
except BaseException:
tmp.unlink(missing_ok=True)
raise
# And the rename itself, so a crash cannot bring back the previous state.
directory = os.open(path.parent, os.O_RDONLY)
try:
os.fsync(directory)
finally:
os.close(directory)


def _artifact(migration_id: str) -> Path:
Expand Down
11 changes: 9 additions & 2 deletions src/syncmill/preflight.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@
# recognise it.
from syncmill.planning import unmapped_principal_id

#: What the preflight runner keeps of each stream. Syncmill never reads the
#: binary's stdout (the verdict is the ``--out`` artifact, which toolgraph also
#: echoes there) and shows only a short stderr snippet, so the rest is drained
#: and dropped rather than held in memory.
_OUTPUT_TAIL_CHARS = 4000

logger = logging.getLogger(__name__)

PREFLIGHT_FILE = "preflight.json"
Expand Down Expand Up @@ -199,7 +205,7 @@ def _invoke_control_toolgraph(
try:
proc = _sync_run( # owned-group runner: a timeout kills the whole tree
argv, capture_output=True, text=True, errors="replace",
timeout=cfg.timeout_seconds,
timeout=cfg.timeout_seconds, tail_chars=_OUTPUT_TAIL_CHARS,
)
except FileNotFoundError:
return (
Expand Down Expand Up @@ -859,7 +865,7 @@ def _invoke_toolgraph(
try:
proc = _sync_run( # owned-group runner: a timeout kills the whole tree
argv, capture_output=True, text=True, errors="replace",
timeout=cfg.timeout_seconds,
timeout=cfg.timeout_seconds, tail_chars=_OUTPUT_TAIL_CHARS,
)
except FileNotFoundError:
return "error", f"toolgraph binary {cfg.command[0]!r} not found in PATH"
Expand Down Expand Up @@ -978,6 +984,7 @@ def _run_preflight_inner(
# strict mode blocks on regardless of `unavailable_policy`.
errors="replace",
timeout=cfg.timeout_seconds,
tail_chars=_OUTPUT_TAIL_CHARS,
)
except FileNotFoundError:
warnings.append(f"toolgraph binary {cfg.command[0]!r} not found in PATH")
Expand Down
16 changes: 10 additions & 6 deletions src/syncmill/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
from syncmill import teamfile
from syncmill.config import SyncmillConfig
from syncmill.git_env import isolated_git_env
from syncmill.artifacts import _git_timed_out
from syncmill.worktree import _run_git


logger = logging.getLogger(__name__)
Expand Down Expand Up @@ -51,12 +53,14 @@ def git_root(path: Path) -> Path:
``--repo`` path as a configuration problem.
"""
try:
proc = subprocess.run(
proc = _run_git(
["git", "rev-parse", "--show-toplevel"],
cwd=str(path), env=isolated_git_env(), capture_output=True, text=True,
)
except OSError as exc:
raise ProjectError(f"cannot inspect as a Git repository: {path} ({exc})") from exc
if _git_timed_out(proc):
raise ProjectError(f"git timed out inspecting {path}")
if proc.returncode != 0:
raise ProjectError(f"not a Git repository: {path}")
return Path(proc.stdout.strip()).resolve()
Expand Down Expand Up @@ -249,7 +253,7 @@ def _index_supply(root: Path) -> str | None:
argv = ["git", "ls-files", "-z", "--stage", "--", f":(icase){_MARKER_DIR}"]
argv.extend(_alias_pathspecs(root, _MARKER_DIR))
try:
proc = subprocess.run(
proc = _run_git(
argv, cwd=str(root), env=isolated_git_env(), capture_output=True,
)
except OSError:
Expand Down Expand Up @@ -369,7 +373,7 @@ def _head_entries(root: Path, treeish: str) -> list[tuple[str, str]] | None:
def _git_query(root: Path, args: list[str]) -> subprocess.CompletedProcess | None:
"""Run a read-only git query, or ``None`` when it could not run at all."""
try:
return subprocess.run(
return _run_git(
["git", *args], cwd=str(root), env=isolated_git_env(), capture_output=True,
)
except OSError:
Expand Down Expand Up @@ -551,7 +555,7 @@ def _require_ignored_state_root(
relative_text = relative.as_posix().rstrip("/")
probes = (relative_text, f"{relative_text}/", f"{relative_text}/.syncmill-probe")
for probe in probes:
ignored = subprocess.run(
ignored = _run_git(
["git", "check-ignore", "-q", "--no-index", "--", probe],
cwd=str(repository),
env=env,
Expand Down Expand Up @@ -918,7 +922,7 @@ def _path_is_ignored(root: Path, relative: str) -> bool:
would append one more identical line.
"""
try:
proc = subprocess.run(
proc = _run_git(
["git", "check-ignore", "-q", "--no-index", "--", relative],
cwd=str(root), env=isolated_git_env(), capture_output=True,
)
Expand Down Expand Up @@ -1052,7 +1056,7 @@ def _write_init_artifacts(

def _extend_info_exclude(root: Path) -> None:
"""Append syncmill's rules to `.git/info/exclude` (machine-local)."""
exclude_probe = subprocess.run(
exclude_probe = _run_git(
["git", "rev-parse", "--git-path", "info/exclude"],
cwd=str(root), env=isolated_git_env(), capture_output=True, text=True,
)
Expand Down
100 changes: 83 additions & 17 deletions tests/test_board.py
Original file line number Diff line number Diff line change
Expand Up @@ -1349,27 +1349,93 @@ def test_scratch_ledger_upgrade_backfills_history_and_is_idempotent(tmp_path, cl
reopened.close()


def test_next_runnable_corrupt_depends_on_raises_board_error_no_dangling_txn(tmp_path):
# FIX: a corrupt depends_on JSON surfaces as BoardError (like get/list) AND rolls
# back the BEGIN IMMEDIATE — else the lock leaks and the next op fails with
# "cannot start a transaction within a transaction".
import sqlite3
def test_next_runnable_error_mid_scan_raises_board_error_no_dangling_txn(
tmp_path, monkeypatch
):
# A non-sqlite error inside the claim transaction surfaces as BoardError
# AND rolls back the BEGIN IMMEDIATE, else the lock leaks and the next op
# fails with "cannot start a transaction within a transaction".
import syncmill.board as board_mod

bd = Board(str(tmp_path / "b.db"))
try:
bd.add("x", depends_on=[str(uuid.uuid4())])

def boom(raw):
raise ValueError("injected")

with monkeypatch.context() as patched:
patched.setattr(board_mod, "_parse_deps", boom)
with pytest.raises(BoardError, match="injected"):
bd.next_runnable("r1")
# rolled back cleanly → a fresh BEGIN IMMEDIATE op works (no dangling txn)
assert bd.recover_stale_running() == 0
finally:
bd.close()


_DAMAGED_DEPENDS_ON = {
"invalid-json": "[oops",
"null": "null",
"int": "5",
"nested-list": '[["a"]]',
"deep": "[" * 100000 + "]" * 100000,
"empty-object": "{}",
"empty-string": '""',
"string": '"abc"',
"object": '{"x": 1}',
"list-of-int": "[1]",
}


@pytest.mark.parametrize("raw", list(_DAMAGED_DEPENDS_ON.values()), ids=list(_DAMAGED_DEPENDS_ON))
def test_next_runnable_skips_an_unreadable_depends_on_and_claims_the_rest(
tmp_path, caplog, raw
):
# One damaged row neither blocks every other item's claim nor gets claimed
# itself, including the shapes that iterate without error: an empty {} or
# "" made every dependency vacuously done, and "abc" / {"x": 1} were read
# as ids a, b, c / x, which are all done here.
db = str(tmp_path / "b.db")
bd = Board(db)
it = bd.add("x")
bd.close()
raw = sqlite3.connect(db)
raw.execute("UPDATE work_items SET depends_on='not json' WHERE id=?", (it.id,))
raw.commit()
raw.close()
bd2 = Board(db)
try:
with pytest.raises(BoardError):
bd2.next_runnable("r1")
# rolled back cleanly → a fresh BEGIN IMMEDIATE op works (no dangling txn)
assert bd2.recover_stale_running() == 0
head = bd.add("damaged head", goal="g")
tail = bd.add("healthy tail", goal="g")
for done_id in ("a", "b", "c", "x", "1"):
done = bd.add(f"done {done_id}")
bd._conn.execute(
"UPDATE work_items SET id=?, state='done' WHERE id=?", (done_id, done.id)
)
bd._conn.execute("UPDATE work_items SET depends_on=? WHERE id=?", (raw, head.id))
with caplog.at_level("WARNING", logger="syncmill.board"):
claimed = bd.next_runnable("r1")
assert claimed is not None and claimed.id == tail.id
# Read raw: get() renders depends_on through _load_list, whose repr of
# a deeply nested list overflows on 3.14 (json.loads accepts it there).
state = bd._conn.execute(
"SELECT state FROM work_items WHERE id=?", (head.id,)
).fetchone()["state"]
assert state == WorkState.PLANNED.value
warnings = [r for r in caplog.records if r.levelname == "WARNING"]
assert [head.id in r.getMessage() for r in warnings] == [True]
# The goal-scoped claim takes the same path.
assert bd.next_runnable("r2", goal="g") is None
finally:
bd2.close()
bd.close()


def test_next_runnable_still_dispatches_a_valid_dependency_list(tmp_path, caplog):
bd = Board(str(tmp_path / "b.db"))
try:
dep = bd.add("dep")
item = bd.add("item", depends_on=[dep.id])
bd._conn.execute("UPDATE work_items SET state='done' WHERE id=?", (dep.id,))
with caplog.at_level("WARNING", logger="syncmill.board"):
claimed = bd.next_runnable("r1")
assert claimed is not None and claimed.id == item.id
assert not [r for r in caplog.records if r.levelname == "WARNING"]
finally:
bd.close()


# ── __init__ closes the connection on open failure (W7) ──────────────────────
Expand Down
Loading
Loading