Skip to content

test(reporting): shorten and instrument installed production gates #1196

Closed as not planned

Description

@bokelley

Problem

The installed reporting production gates run close to their hard deadlines and provide little or no progress output while the isolated child process is active. This makes a production deadlock, a slow phase, and a runner-capacity problem look alike until the outer timeout expires.

The rc.4 adoption incident in #1195 is the immediate example:

  • vcs-pg-installed-production produced no stdout/stderr for 1,800 seconds
  • the exact local equivalent completed in 1,416.56 seconds
  • CI learned only that the child exceeded the aggregate deadline
  • useful PostgreSQL evidence existed, but it was not correlated to a named inner phase or wait state

This issue is for test performance and observability. It must not replace or conceal the production concurrency investigation in #1195.

Desired changes

  • Emit bounded, structured progress for each production-contract phase.
  • Apply per-phase deadlines beneath the aggregate job deadline.
  • Capture process-tree and PostgreSQL wait-state metadata when progress stalls.
  • Identify which test/seed/replay/restart phase is active without printing fixtures, provider responses, credentials, or customer data.
  • Profile duplicated setup and expensive work across VCS/sdist and base/PG cells.
  • Safely reuse or shard immutable build inputs where isolation guarantees remain intact.
  • Retain original logs and cleanup outcomes as CI artifacts.

Acceptance criteria

  • A stalled lane reports its active phase and last progress timestamp before termination.
  • Phase timeouts are deterministic and leave no owned child/database process behind.
  • Normal installed production lanes finish with meaningful headroom under their deadlines on supported runners.
  • Runtime reductions do not remove VCS/sdist, base/PG, restart, historical-artifact, or installed-origin coverage.
  • Performance changes preserve package isolation and exact artifact identity checks.
  • At least one negative control proves a deliberately stalled phase fails with actionable diagnostics.

Activity

  1. added
    enhancementNew feature or request
    pythonPull requests that update python code
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 20, 2026
  2. bokelley commented on Sep 20, 2026

    @bokelley
    ContributorAuthor

    Triage

    Classification: Enhancement — test infrastructure / CI observability
    Bucket(s): test-infrastructure (no repo label match; noted in run summary)
    Status: deferred

    What the experts said:

    • dx-expert: well-scoped and actionable; "no credentials/fixtures" constraint is workable via phase-ID + timestamp + pg_stat_activity; acceptance criteria are concrete; implementation straightforward.
    • code-reviewer: problem statement is sound, but the referenced files (scripts/reporting_test_harness.py, tests/conformance/reporting/*_installed_pg.py, _hardening_installed.py, _hardening_packaging.py) do not exist on main; they live on the stacked reporting branches. On main this is greenfield, but the required infrastructure is already built on the branch.

    My take: The targets of this issue exist only on the stacked reporting branch chain (currently in flight at #1191 and its parents). Implementation can happen now as a parallel PR on that branch stack, or be folded into #1191 before it merges — either way it is blocked on main until the chain lands.

    Blocked-on: #1191 — resurfaces on merge


    Triaged by Claude Code. Session: https://claude.ai/code/session_01EJ4QPh5Lb1wavJ9j5zzDZn


    Generated by Claude Code

  3. added and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 20, 2026
  4. bokelley commented on Sep 20, 2026

    @bokelley
    ContributorAuthor

    Fresh evidence from PR #1193, workflow run 35510429312 attempt 1:

    • Test Python 3.12 job 106077250223 reached the workflow's 45-minute ceiling in Run tests with coverage.
    • Pytest was still making forward progress with passing tests at 93%; the last visible test passed immediately before GitHub emitted The action 'Run tests with coverage' has timed out after 45 minutes.
    • This is a test-runtime/gate-budget failure, not a product assertion failure. The rerun should not be treated as a waiver; a repeat remains blocking and strengthens the case for profiling/sharding the gate.
    • The separate installed pg-vcs timeout remains tracked with the production forward-progress risk in fix(reporting): guarantee forward progress across production PostgreSQL lock paths #1195.
  5. bokelley commented on Sep 29, 2026

    @bokelley
    ContributorAuthor

    Won’t do additional installed-gate performance work in Wave 2: current main already has phase progress, bounded stall diagnostics, and cleanup checks; #1195 covers the remaining PostgreSQL correctness work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions