Repository navigation
fix(peer): settle canceled requests without inferring recovery - #105
Conversation
Release canceled probe permits without clearing failure history. Preserve positive evidence only within a fully settled non-failing cohort, and distinguish caller abandonment from client-owned timeouts across block, roster and delegated-origin requests.
Coldwings
left a comment
There was a problem hiding this comment.
Self-review complete for the current change.
Checked all admitted request exits in block, roster and delegated-origin paths, including validation cancellation and body Close. Neutral completion releases only its own probe, preserves failure history, and cannot override a newer generation. A real success freezes half-open admission until the admitted cohort settles; a failing sibling vetoes recovery. Roster bypass retains discovery behavior without consuming data-probe slots. Completed HEAD/empty responses still supply positive evidence; unread body close does not.
The original real-HTTP reproduction fails on the base (zero requests but circuit closed) and passes on this change (zero requests and circuit half-open). Main/provider vet, race, coverage and build checks, independent goldens and all eight documentation checks pass. The source code adds no wire or routing changes. ADR-0013 is marked accepted in the merge candidate so the decision and implementation land atomically.
|
Independent review found two delegated-origin boundary issues and one documentation mismatch:
The reviewers found the neutral cohort, generation isolation and roster admission accounting sound. These findings will be fixed and the updated head independently reviewed again before merging. |
Recognize body deadlines imposed by the underlying HTTP client without relying on cancellation of the wrapper context. Use HTTP response semantics to settle HEAD and empty responses despite timeout body wrappers. Pin entry reuse and exactly-once close accounting and clarify validator context-error behavior.
There was a problem hiding this comment.
🟡 Changes recommended
Response handling misclassifies complete HEAD/zero-length responses and client body-read timeouts, potentially leaving half-open circuits unrecovered or incorrectly clearing failures.
2 open issues
What changed in this PR
Fixes #104 by separating caller cancellation from peer-health evidence, so canceled requests release admission without falsely recovering a circuit or erasing failure history.
Changes:
- Applies neutral cancellation accounting to block reads, validation, roster exchange, and delegated-origin requests.
- Adds regression coverage for probe cohorts, cancellation timing, stale completions, and origin-body lifecycle.
- Documents the recovery contract in ADR-0013, peer documentation, and the changelog.
| File | Description |
|---|---|
| internal/peer/roster.go | Applies neutral accounting to canceled roster requests. |
| internal/peer/peer.go | Distinguishes caller cancellation from failures during block requests. |
| internal/peer/origin.go | Updates delegated-origin completion and cancellation accounting. |
| internal/peer/origin_test.go | Checks breaker accounting for idle timeouts. |
| internal/peer/neutral_cancel_test.go | Adds cancellation and recovery regression tests. |
| internal/peer/breaker.go | Settles neutral outcomes while preserving cohort recovery rules. |
| docs/peer.md | Updates cancellation, recovery, and validation documentation. |
| docs/adr/README.md | Registers ADR-0013. |
| docs/adr/0013-settle-canceled-peer-requests-neutrally.md | Defines neutral settlement and probe-cohort recovery. |
| CHANGELOG.md | Records the cancellation-accounting fix. |
Review effort: Balanced
|
Addressed the independent review findings. Delegated bodies now recognize HTTP client/transport timeout errors even when the wrapper context remains live. HEAD, explicit empty responses and 204/304 complete according to HTTP semantics, and their body/timer wrappers are closed. Validator API documentation now describes neutral caller cancellation and soft client-owned context timeouts. Added real HTTP timeout and empty-response regressions, and strengthened tests so stale same-generation entry completion and repeated body Close would detect accidental release of another probe. Main and provider vet/race/coverage/build checks passed after the source fixes; focused peer vet/race/coverage and all documentation checks passed after final test strengthening. Updated-head independent reviews are in progress. |
Close the HTTP timeout wrapper before replacing a semantically empty response body with http.NoBody, preserving normal EOF reads for callers.
|
Independent review completed for the updated change and its final empty-response cleanup. Both reviewers report no blocking objections.
The second review found the earlier findings resolved; the final follow-up also confirmed recovery occurs at headers and body reads return normal EOF. Focused race tests, original independent reproductions, documentation checks and diff checks pass. One non-blocking coverage opportunity remains: explicitly expiring a caller deadline during an active body read; current tests separately cover active body cancellation, caller deadlines before dispatch, and client-owned body deadlines. |
Coldwings
left a comment
There was a problem hiding this comment.
Final self-review complete. The HTTP timeout and complete-empty-response findings are fixed, tested, replied to and resolved. Empty response wrappers are closed before returning http.NoBody; caller abandonment remains neutral, client-owned deadlines remain failures, and genuine complete responses remain positive evidence.
Verified the final source with main-module vet/race/coverage/build, and the affected peer regression after the final test assertion update. Provider vet/race/coverage/build passed after the substantive fixes. Independent goldens and all eight documentation checks pass. Both independent reviewers report no blockers for the final head. No wire or placement changes; merge remains gated on final-head CI and Copilot review.
Go 1.22 HTTP timeout errors implement net.Error but do not necessarily match context.DeadlineExceeded. Assert Timeout while retaining the live-caller and reopened-circuit checks.
|
The Go 1.22 CI failure was in the new timeout test's error-type assertion: the logged circuit state was correctly open and the caller was still live, but that runtime's HTTP timeout error did not match context.DeadlineExceeded. Updated the assertion to the portable net.Error.Timeout contract while retaining the live-caller and reopened-circuit checks. Production code already handles that timeout form and is unchanged. Peer vet, race and coverage checks pass; the new head is undergoing the full CI matrix and refreshed review. |
|
Both independent reviewers checked the final Go 1.22 assertion adjustment and extend their no-blocker verdicts to the current head. The change is test-only and retains the timeout, live-caller and open-circuit assertions; production behavior is unchanged. Focused tests and diff checks pass. The final Go 1.22 result is being verified by CI. |
Coldwings
left a comment
There was a problem hiding this comment.
Self-review of the final test-only update is complete: the timeout assertion now uses the Go 1.22-compatible net.Error contract and retains the health-state checks. Production code is unchanged from the fully verified revision; focused peer vet/race/coverage pass. Both independent reviewers confirm no blockers for this head.

Fixes #104
A request canceled before HTTP dispatch could close a half-open peer circuit despite receiving no response. Caller cancellation now settles admission neutrally: it releases its probe slot without clearing failure history. Recovery requires a genuine current-generation answer and completion of every admitted data probe without failure; all-canceled cohorts remain probeable.
Apply the same distinction to block reads, validation, roster exchange and delegated-origin requests. Closing an unfinished origin body is neutral; complete HEAD/empty responses remain positive evidence. Client-owned timeouts remain failures, while origin stream failures remain distinguishable from malformed cached blocks. Entry/generation isolation and discovery admission bypass are preserved. ADR-0013 records the contract; no wire, placement or configuration upgrade is needed.
Validation: the real-HTTP reproduction reports zero server requests in both versions, but the old version incorrectly closes the circuit and the fix keeps it half-open. Deterministic tests cover cancellation before dispatch, before headers and during body reads; failure history, probe cohorts, stale completions, caller deadlines, early close and internal timeout accounting. Main and
providers/k8smodules passedgo vet ./...,go test ./... -race -count=1,go test ./... -cover -count=1, andgo build ./.... Independent golden checks (DART_GOLDEN_REF=1) and all eight documentation checks passed.Review includes self-review, two independent reviews of concurrency and contract/test coverage, and GitHub Copilot review of the current head.
Checklist
go vet ./.../go test ./... -race -count=1/go test ./... -cover -count=1all pass (AGENTS.md)docs/peer.mdupdated; no new packagebash scripts/check-docs.shpasses