You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(reporting): shorten and instrument installed production gates #1196
The installed reporting production gates run close to their hard deadlines and provide little or no progress output while the isolated child process is active. This makes a production deadlock, a slow phase, and a runner-capacity problem look alike until the outer timeout expires.
The rc.4 adoption incident in #1195 is the immediate example:
vcs-pg-installed-production produced no stdout/stderr for 1,800 seconds
the exact local equivalent completed in 1,416.56 seconds
CI learned only that the child exceeded the aggregate deadline
useful PostgreSQL evidence existed, but it was not correlated to a named inner phase or wait state
This issue is for test performance and observability. It must not replace or conceal the production concurrency investigation in #1195.
Desired changes
Emit bounded, structured progress for each production-contract phase.
Apply per-phase deadlines beneath the aggregate job deadline.
Capture process-tree and PostgreSQL wait-state metadata when progress stalls.
Identify which test/seed/replay/restart phase is active without printing fixtures, provider responses, credentials, or customer data.
Profile duplicated setup and expensive work across VCS/sdist and base/PG cells.
Safely reuse or shard immutable build inputs where isolation guarantees remain intact.
Retain original logs and cleanup outcomes as CI artifacts.
Acceptance criteria
A stalled lane reports its active phase and last progress timestamp before termination.
Phase timeouts are deterministic and leave no owned child/database process behind.
Normal installed production lanes finish with meaningful headroom under their deadlines on supported runners.
Runtime reductions do not remove VCS/sdist, base/PG, restart, historical-artifact, or installed-origin coverage.
Performance changes preserve package isolation and exact artifact identity checks.
At least one negative control proves a deliberately stalled phase fails with actionable diagnostics.
Classification: Enhancement — test infrastructure / CI observability Bucket(s): test-infrastructure (no repo label match; noted in run summary) Status: deferred
What the experts said:
dx-expert: well-scoped and actionable; "no credentials/fixtures" constraint is workable via phase-ID + timestamp + pg_stat_activity; acceptance criteria are concrete; implementation straightforward.
code-reviewer: problem statement is sound, but the referenced files (scripts/reporting_test_harness.py, tests/conformance/reporting/*_installed_pg.py, _hardening_installed.py, _hardening_packaging.py) do not exist on main; they live on the stacked reporting branches. On main this is greenfield, but the required infrastructure is already built on the branch.
My take: The targets of this issue exist only on the stacked reporting branch chain (currently in flight at #1191 and its parents). Implementation can happen now as a parallel PR on that branch stack, or be folded into #1191 before it merges — either way it is blocked on main until the chain lands.
Fresh evidence from PR #1193, workflow run 35510429312 attempt 1:
Test Python 3.12 job 106077250223 reached the workflow's 45-minute ceiling in Run tests with coverage.
Pytest was still making forward progress with passing tests at 93%; the last visible test passed immediately before GitHub emitted The action 'Run tests with coverage' has timed out after 45 minutes.
This is a test-runtime/gate-budget failure, not a product assertion failure. The rerun should not be treated as a waiver; a repeat remains blocking and strengthens the case for profiling/sharding the gate.
Won’t do additional installed-gate performance work in Wave 2: current main already has phase progress, bounded stall diagnostics, and cleanup checks; #1195 covers the remaining PostgreSQL correctness work.
Activity
bokelley commentedon Sep 20, 2026
Triage
Classification: Enhancement — test infrastructure / CI observability
Bucket(s): test-infrastructure (no repo label match; noted in run summary)
Status: deferred
What the experts said:
pg_stat_activity; acceptance criteria are concrete; implementation straightforward.scripts/reporting_test_harness.py,tests/conformance/reporting/*_installed_pg.py,_hardening_installed.py,_hardening_packaging.py) do not exist onmain; they live on the stacked reporting branches. On main this is greenfield, but the required infrastructure is already built on the branch.My take: The targets of this issue exist only on the stacked reporting branch chain (currently in flight at #1191 and its parents). Implementation can happen now as a parallel PR on that branch stack, or be folded into #1191 before it merges — either way it is blocked on main until the chain lands.
Blocked-on: #1191 — resurfaces on merge
Triaged by Claude Code. Session: https://claude.ai/code/session_01EJ4QPh5Lb1wavJ9j5zzDZn
Generated by Claude Code
bokelley commentedon Sep 20, 2026
Fresh evidence from PR #1193, workflow run 35510429312 attempt 1:
Test Python 3.12job 106077250223 reached the workflow's 45-minute ceiling inRun tests with coverage.The action 'Run tests with coverage' has timed out after 45 minutes.pg-vcstimeout remains tracked with the production forward-progress risk in fix(reporting): guarantee forward progress across production PostgreSQL lock paths #1195.bokelley commentedon Sep 29, 2026
Won’t do additional installed-gate performance work in Wave 2: current main already has phase progress, bounded stall diagnostics, and cleanup checks; #1195 covers the remaining PostgreSQL correctness work.