Repository navigation
ci(deploy): record green machines' logs and events during the rollout - #9088
Conversation
Every production deploy since #9072 has failed the same way: one of the two bluegreen green machines never passes /ready, Fly rolls back after the 10-minute wait and deletes it, and its boot output and events go with it. The forensics step only runs after the rollback, so its 500-line log tail has never contained a line from the stuck machine. The Deploy step now, for the whole rollout: - streams `flyctl logs -a adcp-docs` to a file (stopped on every exit path by a trap); and - every 15s records the status (`flyctl machine status -d`), checks and events of each machine that did not exist before the deploy, replacing a snapshot only when a new one succeeds, so the last one taken before a rollback deletes the machine survives. On failure, the forensics step applies the existing secret redaction to all of it and adds fly-logs-deploy.txt (last 20,000 lines), one fly-logs-green-<id>.txt per green machine, green-<id>.json (state, checks, events, including any OOM exit) and green-<id>-status.txt, and lists the green machines in summary.txt. Checked with a fake flyctl: a green machine deleted mid-rollout keeps its last snapshot and boot lines, secret-shaped lines are redacted, and the log stream stops when the step exits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Escalate to human review
Escalate — gated path requires human/CODEOWNERS review.
This PR modifies .github/workflows/deploy.yml, which matches the gated glob .github/workflows/** (gated_paths: true). The required review status is REVIEW_REQUIRED, not APPROVED, so the hard approval gate (decision table row 2) is not satisfied and this cannot be auto-approved.
Escalation reasons
.github/workflows/deploy.ymlis under a hard approval gate (.github/workflows/**);review_decisionisREVIEW_REQUIRED. Human/CODEOWNERS approval is required before merge.
Medium findings
.github/workflows/deploy.yml:601— Whole-rollout log capture (up to 20k lines) widens the best-effort redaction blast radius into a repo-readable 30-day artifact; worth a conscious bound.
No Critical/High findings. The reviewer verified an explicit timeout 2400 on the background stream, bounded/reaped watcher subshell, errexit/pipefail before set +e, atomic .tmp+mv writes, [ -f ]-guarded globs, and that every uploaded file passes the redaction filter (raw logs stay out of the artifact). The single Medium does not block on its own, but the gated-path gate forces escalation.
Medium findings
- .github/workflows/deploy.yml:601 — Whole-rollout log capture (20k lines) widens best-effort redaction blast radius into repo-readable artifact
Why human review
- .github/workflows/deploy.yml is under a hard approval gate (.github/workflows/**) and review_decision is REVIEW_REQUIRED — human/CODEOWNERS approval required before merge.
- This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches
.github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
Review on #9088: uploading the whole rollout's log stream (up to 20,000 lines, including the serving fleet's request logs) through a best-effort redaction regex into a 30-day artifact widened the exposure surface far beyond what the diagnosis needs. The stream now stays on the runner. On failure the forensics step uploads only each green machine's own lines (redacted, last 3,000 per machine), which are boot output from machines that serve no traffic until they pass /ready, plus their redacted status, checks and events as before. fly-logs-deploy.txt is gone. Re-checked with a fake flyctl: no serving-fleet line reaches the artifact, green machines' lines and a deleted machine's last snapshot survive, and Bearer lines are redacted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Escalate to human review
Escalate — hard approval gate unsatisfied.
This PR modifies .github/workflows/deploy.yml, which matches the hard, path-based approval gate (.github/workflows/**). gated_paths: true and review_decision: REVIEW_REQUIRED (not APPROVED), so row 2 of the decision table fires: Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
The diff itself is in good shape. The prior Medium finding (deploy.yml:601 — whole-rollout 20k-line log capture widening redaction blast radius into the repo-readable artifact) is resolved: raw whole-rollout output now lands in /tmp/fly-live/logs-raw.txt, kept on the ephemeral runner and never uploaded, while the uploaded forensics artifact gets only per-green-machine, redacted, tail-limited lines. New code adds a 2400s timeout, an EXIT trap, glob guards for zero green machines, and a hoisted redaction pattern. No new Critical/High/Medium findings. This is strictly better than the prior pass.
Escalation reasons:
.github/workflows/deploy.yml(modified) is under a hard approval gate (.github/workflows/**) andreview_decisionisREVIEW_REQUIRED, notAPPROVED. Human/CODEOWNERS approval is required before merge.
Once a human/CODEOWNERS approval is recorded, this clean diff falls through to approve.
Why human review
- .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge.
- This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches
.github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
…step deploy-workflow.test.ts runs the Deploy step's flyctl calls by slicing the script from the first `set +e` to `deploy_exit=$?`. The green-machine watcher added a `set +e` of its own earlier in the script, so the slice started inside the watcher loop and ran it in the foreground forever: Server unit tests (2/4) hung until the 20-minute job timeout on both CI runs of #9088 (the test runs in 64ms on main). The watcher now uses `set +o errexit`, so the slice starts at the deploy's own toggle again. deploy-workflow.test.ts passes (24 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Escalate to human review
Escalate — gated workflow path + unsatisfied review status, plus one medium finding.
This PR modifies .github/workflows/deploy.yml, which matches the hard approval gate .github/workflows/** (gated_paths: true), and review_decision is REVIEW_REQUIRED, not APPROVED. Per decision rule row 2, the outcome must be escalate until a human/CODEOWNERS approval is recorded — regardless of how clean the diff is.
Escalation reasons
.github/workflows/deploy.yml(modified) matches the hard approval gate.github/workflows/**andreview_decisionisREVIEW_REQUIRED, notAPPROVED. Human/CODEOWNERS approval is required before merge.
Medium findings
.github/workflows/deploy.yml:275— Green-machine narrowing fails open: a transientflyctl machines listfailure leaves before-ids.txt empty, so every machine is classified green and serving-fleet request logs (redaction non-exhaustive) can reach the uploaded artifact. Prefer fail-closed: skip capture when the list fails.
Note: the sticky-escalation prior decision carries forward via row 2 regardless; the new medium finding is surfaced here for the human reviewer.
Medium findings
- .github/workflows/deploy.yml:275 — Green-machine narrowing fails open when initial flyctl machines list fails
Why human review
- .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge.
- Medium: .github/workflows/deploy.yml:275 green-machine narrowing fails open on flyctl machines list failure, potentially leaking serving-fleet request logs (non-exhaustive redaction) into the uploaded forensics artifact.
- This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches
.github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
Review on #9088: green machines are "every machine not listed before the deploy", and `flyctl machines list ... || true` left that list empty on a transient Machines API failure, which would have classified the serving fleet as green and uploaded its request logs on a failed deploy. The list is now retried three times and must be non-empty. Without it, the step warns and captures nothing: no log stream, no watcher, so no machine's lines can reach the artifact. stop_capture tolerates either background job being absent. Simulated with a fake flyctl: with the list available only the green machines are captured and no serving-fleet line reaches the artifact; with the list failing, nothing is captured. deploy-workflow.test.ts passes (24 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Escalate to human review
Escalate — gated path under hard approval gate, review not yet approved.
This PR modifies .github/workflows/deploy.yml, which matches the hard approval gate .github/workflows/** (gated_paths: true). The current review_decision is REVIEW_REQUIRED, not APPROVED, so the gate is NOT satisfied (decision table row 2). Human/CODEOWNERS approval must be recorded before this can merge.
Iteration credit: The prior Medium finding (green-machine narrowing fails open on initial flyctl machines list failure) is resolved — the pre-deploy list now retries 3× and skips capture fail-closed if it never succeeds, with before-ids.txt guaranteed non-empty. Net improvement.
Medium findings
- .github/workflows/deploy.yml:317 — Watch-loop flyctl list/status calls (L298/L303) lack a timeout while the log stream got
timeout 2400; stop_capture'swait "$watch_pid"can hang the Deploy step's EXIT trap if an flyctl call hangs during the exact deploy-failure the capture targets.
Only one current Medium (operational/infra path, not data-loss/schema), which on its own would not force escalate — but row 2 (gated path, review not APPROVED) fires first. Human/CODEOWNERS review required.
Escalation reasons
.github/workflows/deploy.ymlis under the hard approval gate.github/workflows/**andreview_decisionisREVIEW_REQUIRED, notAPPROVED. Human/CODEOWNERS approval required before merge.
Medium findings
- .github/workflows/deploy.yml:317 — Watch-loop flyctl calls lack timeout; EXIT-trap wait can hang the Deploy job
Why human review
- .github/workflows/deploy.yml (modified) matches the hard approval gate
.github/workflows/**and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge. - This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches
.github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
Review on #9088: the EXIT trap waited for the watcher, whose flyctl calls had no timeout, so a hung Machines API call during a failing deploy (the case this capture exists for) would have held the Deploy step until the job timeout, with no forensics uploaded. Each watcher flyctl call now runs under `timeout 30`, and stop_capture kills the watcher before reaping it. Snapshots are moved into place whole, so the last good one survives the kill. Simulated with a fake flyctl whose `machine status` hangs: the step exits as soon as the deploy work does, and the snapshot taken before the hang is kept. deploy-workflow.test.ts passes (24 tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Ladon verdict: Escalate to human review
Escalate — gated path requires human/CODEOWNERS approval.
This PR modifies .github/workflows/deploy.yml, which falls under the hard, non-overridable approval gate (.github/workflows/**). gated_paths is true and review_decision is REVIEW_REQUIRED (not APPROVED), so Row 2 of the decision table fires: the outcome must be escalate until a real human/CODEOWNERS approval is recorded.
Review quality note (not blocking): The diff is clean. It responds to the prior Medium (deploy.yml:317 watch-loop/EXIT-trap hang) and the fix holds — every new watch-loop flyctl call is now bounded (timeout 30 on machines list/machine status, timeout 2400 on the log stream), stop_capture kills logs_pid without waiting and does kill before wait, so the EXIT trap can no longer hang the Deploy job. Green-capture is fail-closed (unlistable fleet skips capture rather than uploading serving-fleet request logs). The prior Medium is resolved and not re-surfaced; no new blocking or Medium findings. Were it not for the path gate, this would be an approve — but the gate is a hard, deterministic rule and only a recorded approval lifts it.
Escalation reason: .github/workflows/deploy.yml (modified) matches .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval is required before merge.
Why human review
- .github/workflows/deploy.yml (modified) matches the hard approval gate
.github/workflows/**and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge. - This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches
.github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.
Every production deploy since #9072 has failed the same way, and the artifact the deploy uploads on failure has never captured why. This PR changes the deploy workflow so the next failure records the stuck machine.
The problem
Fly deploys the web machines blue-green. In each failed deploy, one of the two new (green) machines never passes
/readywithin the 10-minute wait. Fly rolls back and deletes that machine, along with its boot output and events. Production is still onb5ac0e1bf7.c925886c82(#9072)8577293f790038stuck,8edd003f3631e8passed in 1 min954eb9db41(#9006)812d42a94e1e18stuck,827515b7671548passed in 1.5 mind24319d922(#9015)f6ada5ff39(#9082)84ee602b450478stuck,8744420a4374d8passed in 1 minThe "Capture Fly forensics on failure" step only runs after the rollback, and its 500-line log tail has never included a line from a stuck machine. In the last run the tail reached the moment of rollback: it has 15 lines from the healthy green machine and none from the stuck one.
#9082 added an automatic docs-index retry and logs that name
/ready's blocker, so a running but unready process would have logged something. The stuck machine logged nothing. It looks like it never ran or never logged, but that can't be confirmed without its own output.What changes
Deploy step, for the whole rollout:
Log stream:
flyctl logs -a adcp-docsstreams to a file on the runner. Atrapstops it however the step exits. The full stream is never uploaded.Green-machine snapshots: every 15 seconds, for each machine that didn't exist before the deploy, it records:
flyctl machine status -d;checksandeventsfromflyctl machines list --json, which show exits, restarts and out-of-memory kills.A snapshot is replaced only when a new one succeeds, so the last one taken before Fly deletes the machine survives.
Capture Fly forensics on failure: applies the existing secret redaction to everything new, and adds:
fly-logs-green-<id>.txt: each green machine's own lines, last 3,000. These are boot output from machines that serve no traffic until they pass/ready. None of the serving fleet's request logs are uploaded (narrowed after Ladon's review).green-<id>.jsonandgreen-<id>-status.txt: each green machine's last state, checks and events;green_machines=insummary.txt.Nothing changes when a deploy succeeds: the forensics are still only uploaded on failure.
Checks
run:scripts passbash -n.tests/release-workflow-immutability.test.cjspasses.tests/ci-runner-demand-workflows.test.mjsfails the same way onmain'sdeploy.yml(aTypeErrorin the test).flyctl:Authorization: BearerandAPI_KEY=lines were filtered out, and no line from a serving machine reached the artifact;summary.txtlisted both green machines.Review: this edits
.github/workflows/**, so it needs a CODEOWNER approval or an admin merge.No changeset: CI only.
🤖 Generated with Claude Code