Repository navigation
fix(bin): select and verify the OpenCode model the installed build accepts - #11
Merged
Merged
Conversation
…aunch worker cleanup
… and effort validation
The two-second checkpoint budget INT-kills the watcher before its first poll completes fm-herdr-groups.sh sync under load, producing intermittent serial-shard failures. Validated 2/2 loaded runs at 12s.
…nts now retain the documented exit-124 assertion, and sibling checkpoint callers no longer swallow failures without checking output. `fm-watch-checkpoint`, `fm-pr-health`, `fm-fleet-ledger`, and `fm-herdr-groups` passed locally; the affected contribution test passed, but its full suite could not complete because this environment lacks `/bin/date`. Other CI failures were in unrelated supervision, follow-up, and transport tests and did not establish a code cause in this change
# Conflicts: # bin/fm-spawn.sh # tests/fm-watch-checkpoint.test.sh
…: the portable serial remainder outgrew a ten-shard split, so the modeled largest shard (1,302,306 ms) exceeded the unchanged 1,200,000 ms packing budget. tests/fm-test-run.test.sh's coverage assertion runs in portable parallel 2, which is why that job failed with the same guard error. Change (three files): - bin/fm-test-run.sh: PORTABLE_SERIAL_SHARDS 10 -> 12, the single owner of the shard count/lane names; and refreshed the grew suite's balance hint tests/fm-spawn-opencode-model.test.sh 31794 -> 122124 ms, verified against its successful timing record (122124 ms, exit 0) in the PR's CI run artifact rather than invented. - .github/workflows/ci.yml: serial shard matrix 10 -> 12 entries (lanes still derive `of<n>` from strategy.job-total, so a future mismatch is refused loudly, not silently truncated). No timeout changed; the 30-minute normal tier, 5-minute fast tier and Herdr backstop are untouched. - docs/fm-test-portable-shards.md: job inventory 15 -> 17 Linux jobs (twelve serial), and the packing account now states the 13113156 ms serial total, why ten shards could not hold it, and the measured 1092777 ms largest shard. Invariant: every portable serial shard's modeled duration must stay within the 1,200,000 ms budget, and the CI matrix, runner constant, lane names and docs must agree. It is enforced at every place that owns it: the runner constant (lane name acceptance, per-shard assignment, coverage guard), the matrix that drives job-total and artifact names, and the docs. Tests already derive the count from the runner (`--list-lanes`) and from the parsed matrix, so no test literal needed changing — verified 12 non-empty shards covering the 268-script lane exactly once. Verification: reproduced the CI failure exactly by stashing the change (`coverage guard: largest portable serial shard packs 1302306ms above the 1200000ms target`, exit 1) and re-ran with the fix: `FM_TEST_COVERAGE ok ... serial_shards=12 serial_max_ms=1092777 serial_budget_ms=1200000`, exit 0. tests/fm-test-run.test.sh: all coverage/sharding assertions pass, including 'coverage guard bounds the unmeasured share and serial packing within twenty minutes', 'portable serial shards are a deterministic disjoint cover of the serial lane', and the exact-budget boundary test. Its two remaining local failures are environmental and unrelated to this change: tests/fm-session-lock-ancestry.test.sh fails on this host at 'the pty-host was not reparented to init after the daemon ended' (that file is not touched by this PR; it is a pid-1/subreaper property of this container), and the derived `--jobs` assertion fails only because of that nested failure. tests/fm-ci-workflow.test.sh cannot run here (no ruby for its YAML parse), so I replicated its matrix-vs-runner comparison with sed/awk: matrix shard list expands to exactly portable-serial-1of12..12of12, matching --list-lanes. Scope: I reverted the timed-out agent's uncommitted edits to tests/fm-supervision-host-fixture.sh, since this round is explicitly limited to the deterministic shard correction and must not change or rerun tests/fm-supervision-host.test.sh; that handoff finding (ci-2) remains for a later focused decision, as do the four owed live OpenCode scenarios. No checks were waived, no protection weakened, no timeout raised
…both failed only at tests/fm-supervision-host.test.sh: 'return-fail: the host did not hand the wake to main'. The CI log shows that test is the sole failure in the shard (suite ran ~23 min); the Full suite check only aggregates the same lane failure. Reproduction and evidence: the exact case passes locally in ~4s (8 consecutive runs), so no behavior defect was found; the old assertion was a poll-count wait, wait_until 250 host_exited, whose real wall-clock duration can far exceed 25s on a slow runner, and upstream Firstmate already replaced these exact waits with a wall-clock wait_until_deadline 60 helper, documenting the same flake. To establish the race, I ran the exact test with a 30s engine stub: the old 25s poll-count wait necessarily expires while the host still correctly hands the wake off; with the fix the same test passes at ~35s. Fix: in tests/fm-supervision-host-fixture.sh, added wait_until_deadline (same 0.1s polling, wall-clock deadline, with hand-back diagnostics) and converted all 74 host/hook hand-back waits (host_exited and hook_exited) to a 60s deadline. This closes every sibling site in the duplicate fixture-based suite, including handback, TERM-stop, latch, mirror, and stream cases; the newer split suites already use the deadline helper. Waits that intentionally test a timing bound or a negative/trigger condition were left unchanged (park boundary 150/400/450, boundary-late 300, bounded turn 300, kill-KILL 100, negative 20, stream trigger 200). The wake-handoff assertions themselves are unchanged, as is all production behavior. Verification: the exact failing return-fail test passes; the whole tests/fm-supervision-host.test.sh suite passes (rc 0, 8m42s) after a first transient failure in an unrelated, unchanged handled_at_least wait that passed on rerun; shellcheck -x is clean; bin/fm-test-run.sh --check-coverage reports ok with 12 serial shards and serial_max_ms=1092777 under the unchanged 1200000ms budget. The pre-existing local latch-park 'tagged' failure (this session does not own supervision) reproduces on unmodified HEAD and is environmental, not caused by this change. The earlier serial-shard coverage correction (12 shards, matrix, hints, docs) is untouched; the 12-second checkpoint behavior, OpenCode model-pin behavior/tests, and the four live OpenCode scenarios remain as before, with the live scenarios still owed rather than claimed passed. No checks were waived or skipped, no timeout or full-suite guard was changed, and the PR remains open for the pipeline to push and rerun required CI
kesslerio
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Changed
bin/fm-spawn.shnow resolves the OpenCode model axis against the installed build instead of an assumed version: it probes--version/--helpand passes--modelonly when that build advertises the top-level flag, otherwise it writes.opencode/opencode.jsoncin the worker's worktree, refuses builds whose version or help surface is unreadable, rejects inlineprovider/model#variant, and keeps the effort variant beside the model (project config) or in the launch config (flag-based builds).opencode debug config, and the worker's first recorded assistant message after the launch timestamp must show the requested model or the spawn refuses and closes the endpoint.bin/fm-control-lib.shandbin/fm-teardown.shretire the pin only when this task's own ownership record names it, so a project's config is never removed and a reused pool slot cannot inherit a stale pin.tests/fm-spawn-opencode-model.test.shsuite plus an opt-intests/fm-opencode-model-live-e2e.test.shguard, updated dispatch-profile OpenCode assertions, supervision-host fixture waits converted to a wall-clockwait_until_deadlinehelper with hand-back diagnostics, timing-agnostic watcher checkpoint assertions, and portable serial shards raised from 10 to 12 inbin/fm-test-run.shwith matching CI matrix and shard-packing docs.Risk Assessment
Testing
Live-driven: the real launcher pinned and verified a requested OpenCode model, refused an inline provider/model#variant while keeping --effort as the only effort interface, mapped --effort high to the openai variant that the installed build then recorded on the worker's first message, refused and stopped a worker that ran a different model than requested, delivered the launch brief into a real OpenCode worker that started its turn on the pinned model, and passed the opt-in installed-build live guard. The portable serial repack was exercised through
bin/fm-test-run.sh --check-coverage(12 lanes, 1,092,777 ms max under the unchanged 1,200,000 ms budget). The focusedtests/fm-spawn-opencode-model.test.shregression passed;tests/fm-spawn-dispatch-profile.test.shstill stops in its inherited worker-environment case because the fixture's synthetic/usr/bin:/binPATH has no sed on this host, which is the already-recorded and declined environmental limitation rather than a product failure, so it is not re-filed. All disposable labs, secondmate homes, tmux servers, and the fake upstream were removed and the git worktree is clean.--list-lanesprints portable-serial-1of12 thro…Evidence: Live launcher run: supported OpenCode model pinned, verified, launched, and asserted
Source: Live launcher run: supported OpenCode model pinned, verified, launched, and asserted
Evidence: Spawned OpenCode worker pane: launch brief delivered, Build · Qwen3.8 Flash Next
Source: Spawned OpenCode worker pane: launch brief delivered, Build · Qwen3.8 Flash Next
Evidence: Inline provider/model#variant refused with no worker started
Source: Inline provider/model#variant refused with no worker started
Evidence: Effort high mapped to the openai variant and kept project-owned config untouched
Source: Effort high mapped to the openai variant and kept project-owned config untouched
Evidence: Worker TUI footer showing the requested model and variant (Build · Lab m1 OpenAI · high)
Source: Worker TUI footer showing the requested model and variant (Build · Lab m1 OpenAI · high)
Evidence: OpenCode's own session record for the worker's first message
Source: OpenCode's own session record for the worker's first message
Evidence: Installed-build live guard: real TUI launch ran the requested model
Source: Installed-build live guard: real TUI launch ran the requested model
Evidence: Wrong-model guard: worker on a different model refused and stopped, home intact
Source: Wrong-model guard: worker on a different model refused and stopped, home intact
Evidence: Coverage guard: 12 serial shards, 1,092,777 ms max under the 1,200,000 ms budget
Source: Coverage guard: 12 serial shards, 1,092,777 ms max under the 1,200,000 ms budget
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
🔧 **Test** - 1 issue found → auto-fixed ✅
bin/fm-spawn.sh:2696- A live model-pinned OpenCode launch printsbin/fm-spawn.sh: line 2696: json_escape: command not foundtwice on stderr.opencode_model_selection_jsoncallsjson_escape, but that helper is only defined at bin/fm-spawn.sh:3893, and the early mechanism probe at bin/fm-spawn.sh:3349 runs before the definition is evaluated, so the first validation call executes a missing command (its output is discarded, so the launch continues). I reproduced this on every real--harness opencode --model ...launch against installed OpenCode 2.2.2, including the one that succeeded, so the operator sees shell-error noise in normal spawn output and the early selection validation is silently a no-op. Definejson_escapebefore its first use (or move the OpenCode helper block after the definition) and add a behavioral assertion that a supported OpenCode launch's captured stderr contains no shell error.checkpoint: no actionable wake within 12s,state/.watch.lock/pidabsent; alsotests/fm-watch-checkpoint.test.sh...within 12s, 1s run prints...within 1s, both exit 124; the changed tests assert onlycheckpoint: no actionable wake withincheck: unacknowledged durable wake, checkpoint printedcheck: rearm-resurfaceand exited 0 in 301ms;tests/fm-pr-health.test.shand `tests/fm-watch-checkpoint…--seconds 0→ exit 2,error: --seconds must be greater than zeroprovider/model#varianton the installed OpenCode build is refused with an actionable message and no worker, pin, or pane is createdbin/fm-spawn.sh ... --harness opencode --model 'ollama-cloud/glm-5.3#high'against OpenCode 2.2.2 → exit 1, `error: inline OpenCode model variants are unsuppor…confirmed the model pin: requested=labfake/m1 observed=labfake/m1,2.2.2 model assertion passed ... observed=labfake/m1#default, spawned into a real tmux window w…<slot>/.opencode/opencode.jsonccontained the pin; after each failed launchcat <slot>/.opencode/opencode.jsoncreturned `No such file…opencode modelswith a plainopencode debug config, and on this host that command blocks ~20s and fails because the operator's residentopencode serve(pid…tests/fm-watch-checkpoint.test.sh(pass; includes the renamedtest_quiet_checkpoint_reports_completion_and_releases_lock)tests/fm-pr-health.test.sh(pass, including the timing-agnostictest_watcher_surfaces_one_wake)tests/fm-fleet-ledger.test.sh(pass, 12s checkpoint budget with exit-code tolerance)tests/fm-herdr-groups.test.sh(pass, 12s watcher poll with exit-code tolerance)tests/fm-spawn-opencode-model.test.sh(pass, full fake-build behavior matrix including pin ownership/retirement faults)tests/fm-contributions.test.sh— the modified casetest_watcher_surfaces_new_contribution_oncepassed; the file's other cases fail on this host because the fixture's fakedatehardcodes/bin/date, which NixOS lackstests/fm-spawn-dispatch-profile.test.sh— blocked on this host: the synthetic worker PATH/usr/bin:/binhas nosed, sobin/fm-git-strip-ai-trailers.shfailslive:FM_HOME=<disposable> bin/fm-watch-checkpoint.sh --seconds 12→ exit 124 at 12.07s,checkpoint: no actionable wake within 12s, lock releasedlive:bin/fm-watch-checkpoint.sh --seconds 1in a sibling disposable home → same generic line, exit 124 at 1.07slive:fm_wake_append check buried ...thenbin/fm-watch-checkpoint.sh --seconds 12→ wake surfaced (check: rearm-resurface), exit 0 in 301mslive:bin/fm-watch-checkpoint.sh --seconds 0→ exit 2, refusedlive:bin/fm-spawn.sh <task> <tmp-project> --harness opencode --model 'ollama-cloud/glm-5.3#high'against real OpenCode 2.2.2 → exit 1 with the inline-variant refusal, no launchlive:bin/fm-spawn.sh <task> <tmp-project> --harness opencode --model labfake/m1 --effort highin a marked lab home with XDG config/state/data isolated and a disposable local OpenAI-compatible endpoint → pin written{"model":{"providerID":"labfake","model":"m1"}},confirmed the model pin: requested=labfake/m1 observed=labfake/m1, session assertionobserved=labfake/m1#default, spawn succeeded, worker pane ran the real OpenCode TUI on that modellive: same launch on a host-occupied managed-service port → exit 1 refusing to launch a worker on the build's default model (fail-safe path)live: while a pinned launch ran,<slot>/.opencode/opencode.jsoncheld the pin; after each failed launchcat <slot>/.opencode/opencode.jsoncreturnedNo such file or directoryfor both slotsteardown: killed the lab worker pane, returned both treehouse slots, removed the lab home, temp project, temp XDG dirs and local model server; working tree clean🔧 Fix applied.
✅ Re-checked - no issues remain.
error: inline OpenCode model variants are unsupported: 'john-remote/qwen3.8-flash-next#high'; use provider/model with --effort instead, exit 1, zero task records writtentests/fm-fleet-ledger.test.sh,tests/fm-pr-health.test.sh,tests/fm-herdr-groups.test.shall exit 0, and the changed `watcher surfaces one newly durable contribution signal without re-ringing it…PATH=/usr/bin:/bin, and the emitted command needssedthere. On this Nix…tests/fm-spawn-opencode-model.test.sh(all cases pass, including the newcommand not foundassertion on the supported 2.x launch)tests/fm-watch-checkpoint.test.sh(all cases pass, including the renamed observable-behavior quiet checkpoint case and the real-host bound case)tests/fm-fleet-ledger.test.sh,tests/fm-pr-health.test.sh,tests/fm-herdr-groups.test.sh(all pass with the 12-second checkpoint bound and observable assertions)tests/fm-contributions.test.sh— the changed casewatcher surfaces one newly durable contribution signal without re-ringing itpasses; the suite's remaining failures are pre-existing/bin/datefakebin shims that do not exist on this NixOS host (unchanged by this diff)XDG_CONFIG_HOME=<disposable> FM_OPENCODE_MODEL_LIVE=1 tests/fm-opencode-model-live-e2e.test.sh(real installed opencode v2.2.2: pin parsed, real TUI launch ran the requested model)Livebin/fm-spawn.sh <id> <secondmate-home> --secondmate --harness opencode --model john-remote/qwen3.8-flash-next --effort highfrom a private fm-lab tmux socket in a marked lab home (spawn exit 0, pin confirmed, model assertion passed, no shell errors)Independent read of the launched worker's ownopencode session export --standalonerecord and the real TUI pane capture showing the pinned modelLive refusal:bin/fm-spawn.sh ... --scout --harness opencode --model 'john-remote/qwen3.8-flash-next#high' --effort lowLive refusal + retry: secondmate spawn against a project-owned.opencode/opencode.jsoncselecting a different model, twiceLivebin/fm-watch-checkpoint.sh --seconds 12in a fresh marked lab home (exit 124 at 12079ms, quiet line, no watch lock left)✅ No issues found.
--list-lanesprints portable-serial-1of12 thro…bin/fm-lab-home.sh create+bin/fm-home-seed.sh sm-live <home> --no-projects(disposable marked lab home and seeded secondmate)bin/fm-spawn.sh sm-live <home> --secondmate --harness opencode --model john-remote/qwen3.8-flash-next --effort highin a privatetmux -L fm-labpanebin/fm-spawn.sh sm-inline-variant <home> --secondmate --harness opencode --model 'john-remote/qwen3.8-flash-next#high' --effort lowbin/fm-spawn.sh sm-effort <home> --secondmate --harness opencode --model openai/m1 --effort highwith a disposable OpenAI-compatible endpoint at 127.0.0.1:49391 serving/v1/responsesopencode session list --standalone --format json+opencode session export <sid> --standaloneon the spawned worker directorybin/fm-spawn.sh sm-fail <home> --secondmate --harness opencode --model openai/m1 --effort highwith noopenaiprovider configured (wrong-model guard)FM_OPENCODE_MODEL_LIVE=1 bin/fm-test-run.sh tests/fm-opencode-model-live-e2e.test.sh(run twice, exit 0 both times)bin/fm-test-run.sh tests/fm-spawn-opencode-model.test.sh(exit 0)bin/fm-test-run.sh --check-coverageandbin/fm-test-run.sh --list-lanestmux -L fm-lab capture-pane -p -t driver:<window>on the live OpenCode worker panes🔧 **Document** - 1 issue found ✅
docs/configuration.md- The environment-variable reference in docs/configuration.md does not list the OpenCode launch bounds this change adds (FM_OPENCODE_PROBE_TIMEOUT, FM_OPENCODE_READ_TIMEOUT, FM_OPENCODE_MODEL_ASSERT_POLLS, FM_OPENCODE_MODEL_ASSERT_INTERVAL), and it already omits the pre-existing FM_OPENCODE_DELIVERY_POLLS/FM_OPENCODE_POLL_INTERVAL/FM_OPENCODE_SUBMIT_* knobs, so those bounds stay discoverable only from bin/fm-spawn.sh. I left the reference untouched rather than expanding this focused change into a general OpenCode tuning-knob documentation pass; a later pass could either document that whole family or state in the reference that harness launch bounds stay with the script header.✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.