Skip to content

Make the Delivery Markdown suite finish reliably under Node #1408

Open

Description

@taras

Story

The Delivery Markdown suite should complete reliably under the repository's Node test runner while retaining its review, continuity and progress-completion checks.

Current gap

In PR #1402, Node shard 4/18, pnpm test:node 4/18 ran examples/adversarial-implementation/tests/Delivery.test.md at c4098d631d8a3bba7b01cfc2d7679a16fa472ec7. The document reached its 300000ms deadline and was stopped.

The document began at 02:27:44.593Z. Its final progress-completion scenario began at 02:32:06.130Z, about 262 seconds into that budget. Cancellation interrupted a Prompt after about 39 seconds of the case's 90-second allowance. The outer document and child journals were uploaded. The nested journal shows three completed variants and the fourth variant's opening Prompt pending.

This differs from the duplicate-success count failures tracked in #1232 and #1176.

Diagnosis and evidence

Native Fable's read-only diagnosis identifies cumulative document cost as the supported cause. Several phases took about 1.2–1.5 times as long as an earlier green run; the trace does not demonstrate a blocked final Prompt. The journal changes on #1402 do not execute on the pre-failure path.

Across the sampled earlier runs, the suite took roughly 175–293 seconds. The slowest passing comparison left about seven seconds of the five-minute budget. Runner speed is an inference from correlated timings, not a controlled hardware measurement.

The original failed log was saved before any retry: SHA-256 fd81d4c140f946187d28b34c2ef06356681137cac204c2af3342539764b93c08. Failure journal artifact

A read-only local probe of the identical commit passed in 219.76 seconds without source changes; its final scenario took about 27 seconds. It used the normal outer Node CLI with Deno child commands, on macOS with Node 26.5.1/Deno 2.9.7. CI used Linux with Node 22.23.4/Deno 2.9.5. This is observed timing variation at identical source, not an identical-host CI retry. The first local probe log preserves selected timestamped lines truncated to 400 characters each and the joined exit receipt; it is not full raw output.

Proposed repair

Move the two existing nested wrapper groups into two independently discovered Markdown suites. Retain their Deno child execution and each authored allowance: the summary-repair wrapper has 120 seconds and the progress-completion wrapper has 90 seconds. Preserve the nested documents' existing timeouts as well. Register each suite with the existing 120-second adversarial deadline class, preserving Delivery's 300-second deadline. Remeasure the changed corpus weights through the supported hosted measurement workflow.

Planner review must confirm this fits the existing testing contract before implementation. Keep this repair separate from #1402's journal implementation.

Acceptance

  • Distinguish normal subprocess/startup cost from an operation that stops progressing, using the preserved first failure.
  • Preserve every behavior assertion, refusal case, bounded failure cleanup and failure-journal capture.
  • Verify the affected suites under Deno, Node and Bun, plus discovery/deadline regression checks.
  • Record discriminating evidence for the correction and the required hosted weight measurement. An unchanged green retry does not close this issue.

Activity

  1. added
    flakeIntermittent or timing-sensitive failure that can pass without a code change
    on Nov 18, 2026
  2. taras commented on Nov 18, 2026

    @taras
    OwnerAuthor

    Identical-revision hosted recurrence evidence is now available. After preserving the first failure and completing Fable's diagnosis, the single failed-job retry attempt 2 passed on unchanged c4098d631d8a3bba7b01cfc2d7679a16fa472ec7. Node 4/18, job 134558966498 passed; the entire PR run is successful. No test, source or deadline changed between those attempts.

    This supplies the previously missing same-revision hosted pass and confirms this failure is intermittent. It does not close the issue. The causal suite-isolation correction is in #1409, with independent review, required CI and hosted corpus measurement still in progress.

  3. taras commented on Nov 18, 2026

    @taras
    OwnerAuthor

    The source correction in #1409 is merged, together with #1402's journal coverage and #1407's output capture. Current source is main 6207d248231be67c9093689abade6609769d05d4.

    The required final corpus measurement is now run 44263905322, attempt 1, pinned to that exact commit on codex/final-test-weights-1408. The corpus includes the journal regression added by #1402: 583 applicable Deno files, 468 Node files and 467 Bun files. The earlier cancelled run's partial records will not be reused.

    This issue remains open for complete matching runtime records, supported artifact reassembly/provenance verification and a reviewed weights-only follow-up. Current merged-main CI is 44263482456; its macOS bench has passed 42/0 and the remaining jobs are still underway.

  4. taras commented on Nov 18, 2026

    @taras
    OwnerAuthor

    The final measurement is blocked by the existing #1331 Bun flake. In run 44263905322, attempt 1, Bun job 134663622215 failed repl-headless-submit-process.test.ts: the held-cleanup Stop case returned exit status 1 instead of 130. Its record is failed after 107/467 files. The first full log and runtime artifact were preserved before any retry.

    The log confirms that the test saved its journals locally, but this measurement workflow uploaded only its runtime JSON. The original #1331 Planner has now produced an accepted diagnostic plan to expose the captured child output/final receipt and upload failure journals separately from weights records. The exit-1 cause is still unconfirmed; production exit behavior will not be changed without a witness.

    Deno and Node remain running. This failed attempt cannot produce the final weights set, and its records will not be combined with another attempt. #1408 remains open for a complete measurement and reviewed weights-only follow-up. No replacement measurement has been dispatched.

  5. taras commented on Nov 18, 2026

    @taras
    OwnerAuthor

    The diagnostic amendment is now PR #1414, signed abc64a127b51bbd286698a550fa2723c8b836f2b, with original Planner PASS and independent Codex Astra PASS. Deno, Node and the single Linux x64 Bun probe pass; the latter used emulation, so it does not diagnose the intermittent hosted exit-1 failure. #1331 remains open.

    One fresh complete measurement is now run 44279296345, attempt 1, pinned to that exact reviewed source on immutable ref codex/measure-1408-abc64a12. It enables separate failure-journal uploads while preserving measurement record layout and complete-only assembly. The first dispatch transport failed; GitHub confirmed no run existed before the successful transport retry, and exactly one matching run is now confirmed.

    The failed 44263905322 and cancelled 44232626922 datasets remain ineligible. Only three complete matching records from this new run/attempt can advance the supported reassembly and reviewed weights-only follow-up. A recurrence will preserve its first diagnostics and hold the weights work rather than trigger a blind retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    flakeIntermittent or timing-sensitive failure that can pass without a code change

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions