Skip to content

ci(deploy): record green machines' logs and events during the rollout - #9088

Merged
bokelley merged 5 commits into
mainfrom
ci/deploy-green-machine-forensics
Nov 18, 2026
Merged

bokelley merged 5 commits into
mainfrom
ci/deploy-green-machine-forensics

Conversation

@bokelley

@bokelley bokelley commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Every production deploy since #9072 has failed the same way, and the artifact the deploy uploads on failure has never captured why. This PR changes the deploy workflow so the next failure records the stuck machine.

The problem

Fly deploys the web machines blue-green. In each failed deploy, one of the two new (green) machines never passes /ready within the 10-minute wait. Fly rolls back and deletes that machine, along with its boot output and events. Production is still on b5ac0e1bf7.

Deploy Head Green machines Outcome
11:33 UTC, Nov 17 c925886c82 (#9072) 8577293f790038 stuck, 8edd003f3631e8 passed in 1 min rolled back
16:45 UTC, Nov 17 954eb9db41 (#9006) 812d42a94e1e18 stuck, 827515b7671548 passed in 1.5 min rolled back
01:23 UTC, Nov 18 d24319d922 (#9015) failed
03:41 UTC, Nov 18 f6ada5ff39 (#9082) 84ee602b450478 stuck, 8744420a4374d8 passed in 1 min rolled back

The "Capture Fly forensics on failure" step only runs after the rollback, and its 500-line log tail has never included a line from a stuck machine. In the last run the tail reached the moment of rollback: it has 15 lines from the healthy green machine and none from the stuck one.

#9082 added an automatic docs-index retry and logs that name /ready's blocker, so a running but unready process would have logged something. The stuck machine logged nothing. It looks like it never ran or never logged, but that can't be confirmed without its own output.

What changes

Deploy step, for the whole rollout:

  • Log stream: flyctl logs -a adcp-docs streams to a file on the runner. A trap stops it however the step exits. The full stream is never uploaded.

  • Green-machine snapshots: every 15 seconds, for each machine that didn't exist before the deploy, it records:

    • flyctl machine status -d;
    • the machine's checks and events from flyctl machines list --json, which show exits, restarts and out-of-memory kills.

    A snapshot is replaced only when a new one succeeds, so the last one taken before Fly deletes the machine survives.

Capture Fly forensics on failure: applies the existing secret redaction to everything new, and adds:

  • fly-logs-green-<id>.txt: each green machine's own lines, last 3,000. These are boot output from machines that serve no traffic until they pass /ready. None of the serving fleet's request logs are uploaded (narrowed after Ladon's review).
  • green-<id>.json and green-<id>-status.txt: each green machine's last state, checks and events;
  • green_machines= in summary.txt.

Nothing changes when a deploy succeeds: the forensics are still only uploaded on failure.

Checks

  • The YAML parses, and both edited run: scripts pass bash -n.
  • tests/release-workflow-immutability.test.cjs passes. tests/ci-runner-demand-workflows.test.mjs fails the same way on main's deploy.yml (a TypeError in the test).
  • Simulation with a fake flyctl:
    • a green machine deleted mid-rollout kept its last snapshot, including an out-of-memory exit event, and its boot lines;
    • Authorization: Bearer and API_KEY= lines were filtered out, and no line from a serving machine reached the artifact;
    • the log stream stopped when the step exited;
    • summary.txt listed both green machines.
  • The pre-commit and pre-push hooks passed (1,284 unit tests).

Review: this edits .github/workflows/**, so it needs a CODEOWNER approval or an admin merge.

No changeset: CI only.

🤖 Generated with Claude Code

Every production deploy since #9072 has failed the same way: one of the
two bluegreen green machines never passes /ready, Fly rolls back after the
10-minute wait and deletes it, and its boot output and events go with it.
The forensics step only runs after the rollback, so its 500-line log tail
has never contained a line from the stuck machine.

The Deploy step now, for the whole rollout:
- streams `flyctl logs -a adcp-docs` to a file (stopped on every exit path
  by a trap); and
- every 15s records the status (`flyctl machine status -d`), checks and
  events of each machine that did not exist before the deploy, replacing
  a snapshot only when a new one succeeds, so the last one taken before a
  rollback deletes the machine survives.

On failure, the forensics step applies the existing secret redaction to
all of it and adds fly-logs-deploy.txt (last 20,000 lines), one
fly-logs-green-<id>.txt per green machine, green-<id>.json (state, checks,
events, including any OOM exit) and green-<id>-status.txt, and lists the
green machines in summary.txt.

Checked with a fake flyctl: a green machine deleted mid-rollout keeps its
last snapshot and boot lines, secret-shaped lines are redacted, and the
log stream stops when the step exits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread .github/workflows/deploy.yml Resolved

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Escalate to human review

Escalate — gated path requires human/CODEOWNERS review.

This PR modifies .github/workflows/deploy.yml, which matches the gated glob .github/workflows/** (gated_paths: true). The required review status is REVIEW_REQUIRED, not APPROVED, so the hard approval gate (decision table row 2) is not satisfied and this cannot be auto-approved.

Escalation reasons

  • .github/workflows/deploy.yml is under a hard approval gate (.github/workflows/**); review_decision is REVIEW_REQUIRED. Human/CODEOWNERS approval is required before merge.

Medium findings

  • .github/workflows/deploy.yml:601 — Whole-rollout log capture (up to 20k lines) widens the best-effort redaction blast radius into a repo-readable 30-day artifact; worth a conscious bound.

No Critical/High findings. The reviewer verified an explicit timeout 2400 on the background stream, bounded/reaped watcher subshell, errexit/pipefail before set +e, atomic .tmp+mv writes, [ -f ]-guarded globs, and that every uploaded file passes the redaction filter (raw logs stay out of the artifact). The single Medium does not block on its own, but the gated-path gate forces escalation.

Medium findings

  • .github/workflows/deploy.yml:601 — Whole-rollout log capture (20k lines) widens best-effort redaction blast radius into repo-readable artifact

Why human review

  • .github/workflows/deploy.yml is under a hard approval gate (.github/workflows/**) and review_decision is REVIEW_REQUIRED — human/CODEOWNERS approval required before merge.
  • This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

@aao-secretariat aao-secretariat Bot added the ladon/needs-human-review Ladon has escalated this PR for human review. label Nov 18, 2026
Review on #9088: uploading the whole rollout's log stream (up to 20,000
lines, including the serving fleet's request logs) through a best-effort
redaction regex into a 30-day artifact widened the exposure surface far
beyond what the diagnosis needs.

The stream now stays on the runner. On failure the forensics step uploads
only each green machine's own lines (redacted, last 3,000 per machine),
which are boot output from machines that serve no traffic until they pass
/ready, plus their redacted status, checks and events as before.
fly-logs-deploy.txt is gone.

Re-checked with a fake flyctl: no serving-fleet line reaches the
artifact, green machines' lines and a deleted machine's last snapshot
survive, and Bearer lines are redacted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Escalate to human review

Escalate — hard approval gate unsatisfied.

This PR modifies .github/workflows/deploy.yml, which matches the hard, path-based approval gate (.github/workflows/**). gated_paths: true and review_decision: REVIEW_REQUIRED (not APPROVED), so row 2 of the decision table fires: Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

The diff itself is in good shape. The prior Medium finding (deploy.yml:601 — whole-rollout 20k-line log capture widening redaction blast radius into the repo-readable artifact) is resolved: raw whole-rollout output now lands in /tmp/fly-live/logs-raw.txt, kept on the ephemeral runner and never uploaded, while the uploaded forensics artifact gets only per-green-machine, redacted, tail-limited lines. New code adds a 2400s timeout, an EXIT trap, glob guards for zero green machines, and a hoisted redaction pattern. No new Critical/High/Medium findings. This is strictly better than the prior pass.

Escalation reasons:

  • .github/workflows/deploy.yml (modified) is under a hard approval gate (.github/workflows/**) and review_decision is REVIEW_REQUIRED, not APPROVED. Human/CODEOWNERS approval is required before merge.

Once a human/CODEOWNERS approval is recorded, this clean diff falls through to approve.

Why human review

  • .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge.
  • This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

…step

deploy-workflow.test.ts runs the Deploy step's flyctl calls by slicing
the script from the first `set +e` to `deploy_exit=$?`. The green-machine
watcher added a `set +e` of its own earlier in the script, so the slice
started inside the watcher loop and ran it in the foreground forever:
Server unit tests (2/4) hung until the 20-minute job timeout on both CI
runs of #9088 (the test runs in 64ms on main).

The watcher now uses `set +o errexit`, so the slice starts at the
deploy's own toggle again. deploy-workflow.test.ts passes (24 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread .github/workflows/deploy.yml Resolved

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Escalate to human review

Escalate — gated workflow path + unsatisfied review status, plus one medium finding.

This PR modifies .github/workflows/deploy.yml, which matches the hard approval gate .github/workflows/** (gated_paths: true), and review_decision is REVIEW_REQUIRED, not APPROVED. Per decision rule row 2, the outcome must be escalate until a human/CODEOWNERS approval is recorded — regardless of how clean the diff is.

Escalation reasons

  • .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED. Human/CODEOWNERS approval is required before merge.

Medium findings

  • .github/workflows/deploy.yml:275 — Green-machine narrowing fails open: a transient flyctl machines list failure leaves before-ids.txt empty, so every machine is classified green and serving-fleet request logs (redaction non-exhaustive) can reach the uploaded artifact. Prefer fail-closed: skip capture when the list fails.

Note: the sticky-escalation prior decision carries forward via row 2 regardless; the new medium finding is surfaced here for the human reviewer.

Medium findings

  • .github/workflows/deploy.yml:275 — Green-machine narrowing fails open when initial flyctl machines list fails

Why human review

  • .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge.
  • Medium: .github/workflows/deploy.yml:275 green-machine narrowing fails open on flyctl machines list failure, potentially leaking serving-fleet request logs (non-exhaustive redaction) into the uploaded forensics artifact.
  • This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

Review on #9088: green machines are "every machine not listed before the
deploy", and `flyctl machines list ... || true` left that list empty on
a transient Machines API failure, which would have classified the serving
fleet as green and uploaded its request logs on a failed deploy.

The list is now retried three times and must be non-empty. Without it,
the step warns and captures nothing: no log stream, no watcher, so no
machine's lines can reach the artifact. stop_capture tolerates either
background job being absent.

Simulated with a fake flyctl: with the list available only the green
machines are captured and no serving-fleet line reaches the artifact;
with the list failing, nothing is captured. deploy-workflow.test.ts
passes (24 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread .github/workflows/deploy.yml Resolved

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Escalate to human review

Escalate — gated path under hard approval gate, review not yet approved.

This PR modifies .github/workflows/deploy.yml, which matches the hard approval gate .github/workflows/** (gated_paths: true). The current review_decision is REVIEW_REQUIRED, not APPROVED, so the gate is NOT satisfied (decision table row 2). Human/CODEOWNERS approval must be recorded before this can merge.

Iteration credit: The prior Medium finding (green-machine narrowing fails open on initial flyctl machines list failure) is resolved — the pre-deploy list now retries 3× and skips capture fail-closed if it never succeeds, with before-ids.txt guaranteed non-empty. Net improvement.

Medium findings

  • .github/workflows/deploy.yml:317 — Watch-loop flyctl list/status calls (L298/L303) lack a timeout while the log stream got timeout 2400; stop_capture's wait "$watch_pid" can hang the Deploy step's EXIT trap if an flyctl call hangs during the exact deploy-failure the capture targets.

Only one current Medium (operational/infra path, not data-loss/schema), which on its own would not force escalate — but row 2 (gated path, review not APPROVED) fires first. Human/CODEOWNERS review required.

Escalation reasons

  • .github/workflows/deploy.yml is under the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED. Human/CODEOWNERS approval required before merge.

Medium findings

  • .github/workflows/deploy.yml:317 — Watch-loop flyctl calls lack timeout; EXIT-trap wait can hang the Deploy job

Why human review

  • .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge.
  • This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

Review on #9088: the EXIT trap waited for the watcher, whose flyctl calls
had no timeout, so a hung Machines API call during a failing deploy (the
case this capture exists for) would have held the Deploy step until the
job timeout, with no forensics uploaded.

Each watcher flyctl call now runs under `timeout 30`, and stop_capture
kills the watcher before reaping it. Snapshots are moved into place
whole, so the last good one survives the kill.

Simulated with a fake flyctl whose `machine status` hangs: the step
exits as soon as the deploy work does, and the snapshot taken before the
hang is kept. deploy-workflow.test.ts passes (24 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Escalate to human review

Escalate — gated path requires human/CODEOWNERS approval.

This PR modifies .github/workflows/deploy.yml, which falls under the hard, non-overridable approval gate (.github/workflows/**). gated_paths is true and review_decision is REVIEW_REQUIRED (not APPROVED), so Row 2 of the decision table fires: the outcome must be escalate until a real human/CODEOWNERS approval is recorded.

Review quality note (not blocking): The diff is clean. It responds to the prior Medium (deploy.yml:317 watch-loop/EXIT-trap hang) and the fix holds — every new watch-loop flyctl call is now bounded (timeout 30 on machines list/machine status, timeout 2400 on the log stream), stop_capture kills logs_pid without waiting and does kill before wait, so the EXIT trap can no longer hang the Deploy job. Green-capture is fail-closed (unlistable fleet skips capture rather than uploading serving-fleet request logs). The prior Medium is resolved and not re-surfaced; no new blocking or Medium findings. Were it not for the path gate, this would be an approve — but the gate is a hard, deterministic rule and only a recorded approval lifts it.

Escalation reason: .github/workflows/deploy.yml (modified) matches .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval is required before merge.

Why human review

  • .github/workflows/deploy.yml (modified) matches the hard approval gate .github/workflows/** and review_decision is REVIEW_REQUIRED, not APPROVED — human/CODEOWNERS approval required before merge.
  • This PR touches a path under a hard, non-overridable approval gate (.github/workflows/deploy.yml (modified) matches .github/workflows/**) and the current GitHub review decision is 'REVIEW_REQUIRED', not APPROVED. This is a hard gate enforced in code — Ladon cannot auto-approve until a human/CODEOWNERS approval is recorded, regardless of how clean the diff is.

@bokelley
bokelley merged commit 4a3d19c into main Nov 18, 2026
31 checks passed
@bokelley
bokelley deleted the ci/deploy-green-machine-forensics branch November 18, 2026 16:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ladon/needs-human-review Ladon has escalated this PR for human review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant