Repository navigation
test(breg): native timer fairness test fails intermittently in harness cleanup #2505
Description
native_bounded_fairness_interleaves_all_timer_lanes_across_types_and_replicas (crates/registry-breg/tests/postgres_record_timer_native.rs) failed once in a merge-queue run and passed on the next attempt with no code change.
- Run: https://github.com/registrystack/registry-stack/actions/runs/40929857218 (job "Base Registry Engine product contracts (postgres:2/3)", 2026-10-29)
- Output:
panicked at crates/registry-breg/tests/postgres_record_timer_native.rs:3126:35:
native timer result Ok(()); native timer cleanup found 1 failure(s); logs retained at /tmp/breg-record-timer-native-HQOq99
The journey itself returned Ok(()). The failure came from NativeTimerHarness::finish in crates/registry-breg/tests/support/record_timer_native.rs, which collects errors from replica.stop() and assert_logs_safe() but reports only their count. The retained logs live on the runner and are gone, so the cause can't be recovered from this run.
Suggested first step: include the collected failure messages in the finish error so the next occurrence says whether a replica failed to stop or a log check tripped. Then fix the actual cause.
Possibly related: three other PostgreSQL tests failed locally under heavy machine load and passed when rerun alone. They have not been seen failing on CI:
postgres_package::signed_schema_fingerprint_mismatch_is_refused_before_maintenance_and_never_readypostgres_proposal::automatic_lease_expiring_during_a_write_rolls_back_and_recovers_within_its_budgetpostgres_record_timer_native::native_reviewed_successor_refreshes_held_occurrence_once_after_restart
Activity
jeremi commentedon Oct 30, 2026
Second occurrence, different test in the same file and the same harness cleanup path:
native_database_session_loss_rolls_back_active_transaction_for_peer_recoveryfailed withnative timer result Ok(()); native timer cleanup found 1 failure(s)in merge-queue run https://github.com/registrystack/registry-stack/actions/runs/41016229506 (job "Base Registry Engine product contracts (postgres:2/3)", 2026-10-30). That PR does not touch the timer tests. So this looks like a harness cleanup problem rather than one test's logic.jeremi commentedon Nov 5, 2026
Two more merge-queue occurrences in the same job ("Base Registry Engine product contracts (postgres:2/3)"), neither from a PR that touches the timer tests:
native_database_session_loss_rolls_back_active_transaction_for_peer_recovery,native timer result Ok(()); native timer cleanup found 1 failure(s)at line 3126 (the same signature as the second occurrence).native_fixture_pause_resume_fences_pre_pause_claim_across_processes(line 3141):native timer result Err("native timer durable evidence query failed"); native timer cleanup found 1 failure(s): native timer process log warning/error codes differed: expected_ranges={}, actual={"<none>": 1}. The cleanup error now names its cause: one warning or error log line with no code.native_fault_stale_resumed_owner_cannot_settle_after_peer(line 3135):native timer completion was not one atomic exact-once effect, with the timer stillpending/scheduledat revision 1 and no outbox entry or action receipt.That makes four failures in eight days, across four different tests in
postgres_record_timer_native.rs, all in the 2/3 shard. The uncoded log line in the latest run looks like the next thing to chase: finding which process emitted it should say whether this is harness cleanup or a real ordering problem. The second failure in that run asserts on the product's exactly-once evidence rather than on cleanup, so it may not be only the harness.jeremi commentedon Nov 8, 2026
Two more occurrences on 2026-11-07, both with an empty expected range and one unexpected process log code, so the same harness cleanup path:
native_bounded_fairness_interleaves_all_timer_lanes_across_types_and_replicasin the fix(render): declare seal assurance once per runtime, matching Evidence #2594 merge-queue run: https://github.com/registrystack/registry-stack/actions/runs/42379627389 (job "Base Registry Engine product contracts (postgres:2/3)"). Panic atpostgres_record_timer_native.rs:3141; actual codes{"record_timer.worker.iteration_failed": 1}. Passed on re-enqueue with no code change.native_timer_metrics_are_current_bounded_and_value_free(crates/registry-breg/tests/postgres_record_timer_metrics.rs:494) on main: https://github.com/registrystack/registry-stack/actions/runs/42414340097 (job "Base Registry Engine product contracts (postgres:3/3)"). A replica exited before readiness with status 1, and cleanup found actual codes{"<none>": 1}.The second one is a different test on the same harness, and this time a replica failed before readiness rather than during the journey. The suggested first step still stands: the retained replica logs are on the runner and gone, so
finishand the readiness failure should print the replica log tail into the panic message.jeremi commentedon Nov 11, 2026
Another occurrence, merge-queue run https://github.com/registrystack/registry-stack/actions/runs/42977045944 (job "Base Registry Engine product contracts (postgres:2/3)", 2026-11-11), from #2644, which touches no hook or timer runtime code.
native_database_session_loss_rolls_back_active_transaction_for_peer_recoverypanicked atpostgres_record_timer_native.rs:3182. This time the message names the code (thefinishchange helped):{"hook.claim.identity_refused": 1, "hook.worker.iteration_failed": 1, "record_timer.worker.iteration_failed": 1}level="WARN" message="hook state transition failed" code="hook.claim.identity_refused"hook.claim.identity_refusedis not in this test's expected ranges, which allow(0, 2)for the otherhook.claim.*codes.So, at least for this test, the hook worker on the replica whose database session was killed can log one refused claim identity before it recovers. If that is the intended recovery path, the fix is to allow
hook.claim.identity_refused(0, 2)in the session-loss test's expected ranges, like the other hook claim codes. If it is not intended, the hook claim path after session loss needs a look.