Skip to content

Capture expected CI aggregate errors in workflow tests - #1407

Merged
taras merged 2 commits into
mainfrom
codex/ci-aggregate-output-capture
Nov 18, 2026
Merged

taras merged 2 commits into
mainfrom
codex/ci-aggregate-output-capture

Conversation

@taras

@taras taras commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Why

Passing CI workflow tests emitted GitHub error annotations. Their deliberately rejected aggregate subprocesses forwarded raw workflow commands, so successful jobs appeared to contain CI failures.

What changes

The tests capture expected aggregate stdout/stderr in the complete child result. Unexpected assertion failures still report labeled exit status and escaped full output. Scoped Stdio suppression restores normal forwarding on exit; existing zero/nonzero expectations stay intact.

How to verify it

  • Original Planner and the same retained independent Codex Astra reviewer: PASS at signed 7a60d872eb0e2ef37963d21b25b272f01ca1ddec, based on main 1ce93ad5.
  • Focused Deno verification on this head passed 9 suites/51 steps and emitted zero raw workflow commands.
  • Earlier applicable Node/Bun, typecheck, formatting and suppression-removal evidence is explicitly inherited. The changed file, its imports and the entire annotation patch are unchanged; the newer base adds the unrelated History wait.
  • The regression checks both captured streams, escaped LF/CR diagnostics and restored outer forwarding. Removing suppression makes it fail on the leaked child output.
  • Hosted Bun 3/10 on the preceding annotation revision passed with zero ::error:: commands, confirming the actual output improvement.
  • Required CI passed on the current restacked head: run 44234784763, 62 successful jobs and one expected skip.

Scope

Only scripts/tests/ci-workflow.test.ts changes. This is the separate output follow-up to the completed main repair. Local Deno 2.9.7 differs from CI's recorded 2.9.5.

Scope confirmation

  • Every changed file supports the purpose described above.
  • Unrelated cleanup and formatting changes are excluded.
  • The description distinguishes inherited evidence from new-head verification.

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Independent review: PASS at 47ac54071541ee16af1ef8cbe5b5e330e0eb8e97.

Reviewer: Codex gpt-6-astra, using the documented replacement after the retained Fable conversation could not resume. Original Planner also passed this exact revision.

The reviewer traced every runGreen consumer and the installed process/context APIs. Scoped Stdio suppression preserves the complete ProcessResult and existing exit expectations; JSON-escaped failure diagnostics retain both streams without emitting executable workflow commands. The regression verifies capture, restored outer forwarding, LF/CR safety and a real suppression-removal failure.

No blocking findings. Applicable retained Deno/Node/Bun evidence covers 51 tests/steps per runtime; focused typecheck and formatting passed. The tested and published commits have identical trees. Local Deno 2.9.7 differs from CI's recorded 2.9.5; hosted checks remain required.

Review receipt SHA-256: da7b74f80af64203a4a0fde6e77bcc336f8d4d3a8349c0f16c2243c8c259ca8a. This output repair is separate from #1406 and does not resolve the current main failure.

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

All 60 test/build/check jobs passed in run 44227118849, with the normal one-job skip. The only failing prerequisite is main-green, which correctly saw old main b2bc6f1 fail its macOS bench; green consequently failed too. The original gate output is preserved.

#1406 has since merged as 1ce93ad5, and its authoritative main run is still running. Once that run is green, only the failed health/aggregate checks need refreshing. No source change or blanket test retry is indicated by this result.

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Hosted output verification: Bun 3/10 passed, ran scripts/tests/ci-workflow.test.ts, and its complete decoded log contains zero ::error:: commands. The original passing shard emitted the aggregate's expected errors as runner annotations. The replacement regression and the real hosted output now both confirm the capture boundary.

runGreen joins the aggregate under scoped Stdio suppression, so the
expected `::error::` lines of rejected aggregates stay in the captured
result instead of reaching the test process's streams, where a CI
runner raises them as annotations. Exit assertions report the captured
streams JSON-escaped when they fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

The failed health/aggregate checks were refreshed only after current main recovered. Attempt 2 is now successful: 62 jobs passed and one skipped, with unchanged source 47ac54071541ee16af1ef8cbe5b5e330e0eb8e97.

The repository ruleset additionally requires the branch to be current with main. A mechanical restack onto green main 1ce93ad5 is in progress through the retained role workflow. The annotation file and patch are unchanged; a focused regression passed on the new signed head. Original Planner and selected Astra exact-head reconciliation are closing before publication. This is base maintenance, with no added functionality or reset of correction limits.

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Mechanical restack published as signed 7a60d872eb0e2ef37963d21b25b272f01ca1ddec, sole parent current green main 1ce93ad5. The annotation file blob and complete branch patch are unchanged (patch SHA-256 a139e91915bb18c30943dedd5c0cbd03eb58f1d9f04ac0e743ccc6d8f60ba1cb). Old refs and records are preserved; publication used an exact old-SHA lease.

Original Planner: PASS on the new head. The same retained independent Astra reviewer 01a1ec76-3f63-7db3-8b48-3c080c5574f3: PASS on the new head, with no blocking findings and all four original acceptance clauses retained. Review receipt SHA-256: 1d50a5e4f5736176b936a88e4498d92d548df4b3e811f49e80aa0668acb6f698.

New-head focused Deno verification passed 9 suites/51 steps with zero raw workflow commands. Prior Node/Bun/type/format/control evidence is explicitly inherited, not described as rerun. The newer base changes only the unrelated History wait. Required CI on this exact new head is the remaining delivery gate.

@taras
taras force-pushed the codex/ci-aggregate-output-capture branch from 47ac540 to 7a60d87 Compare November 18, 2026 04:04
@taras
taras marked this pull request as ready for review November 18, 2026 04:29

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Final required CI passed on exact reviewed restack 7a60d872eb0e2ef37963d21b25b272f01ca1ddec: run 44234784763, 62 successful jobs and one expected skip. Original Planner and same retained Astra reviews are PASS. The branch is based on current green main 1ce93ad5; the PR is now ready for maintainer review/merge.

#1402 is also reviewed and green at its existing maintainer head. After the first of these PRs merges, any necessary strict-base refresh of the other will preserve its patch and exact-head review requirements. The final corpus remeasurement for #1409 will follow both merges.

@taras
taras merged commit 5095bb3 into main Nov 18, 2026
62 of 63 checks passed
@taras
taras deleted the codex/ci-aggregate-output-capture branch November 18, 2026 07:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant