Skip to content

fix(peer): settle canceled requests without inferring recovery - #105

Merged
Coldwings merged 4 commits into
data-accelerator:mainfrom
Coldwings:fix/neutral-peer-cancellation
Nov 9, 2026
Merged

Coldwings merged 4 commits into
data-accelerator:mainfrom
Coldwings:fix/neutral-peer-cancellation

Conversation

@Coldwings

Copy link
Copy Markdown
Collaborator

Fixes #104

A request canceled before HTTP dispatch could close a half-open peer circuit despite receiving no response. Caller cancellation now settles admission neutrally: it releases its probe slot without clearing failure history. Recovery requires a genuine current-generation answer and completion of every admitted data probe without failure; all-canceled cohorts remain probeable.

Apply the same distinction to block reads, validation, roster exchange and delegated-origin requests. Closing an unfinished origin body is neutral; complete HEAD/empty responses remain positive evidence. Client-owned timeouts remain failures, while origin stream failures remain distinguishable from malformed cached blocks. Entry/generation isolation and discovery admission bypass are preserved. ADR-0013 records the contract; no wire, placement or configuration upgrade is needed.

Validation: the real-HTTP reproduction reports zero server requests in both versions, but the old version incorrectly closes the circuit and the fix keeps it half-open. Deterministic tests cover cancellation before dispatch, before headers and during body reads; failure history, probe cohorts, stale completions, caller deadlines, early close and internal timeout accounting. Main and providers/k8s modules passed go vet ./..., go test ./... -race -count=1, go test ./... -cover -count=1, and go build ./.... Independent golden checks (DART_GOLDEN_REF=1) and all eight documentation checks passed.

Review includes self-review, two independent reviews of concurrency and contract/test coverage, and GitHub Copilot review of the current head.

Checklist

  • go vet ./... / go test ./... -race -count=1 / go test ./... -cover -count=1 all pass (AGENTS.md)
  • docs/peer.md updated; no new package
  • bash scripts/check-docs.sh passes
  • ADR-0013 records neutral cancellation and probe-cohort recovery

Release canceled probe permits without clearing failure history. Preserve positive evidence only within a fully settled non-failing cohort, and distinguish caller abandonment from client-owned timeouts across block, roster and delegated-origin requests.

@Coldwings Coldwings left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review complete for the current change.

Checked all admitted request exits in block, roster and delegated-origin paths, including validation cancellation and body Close. Neutral completion releases only its own probe, preserves failure history, and cannot override a newer generation. A real success freezes half-open admission until the admitted cohort settles; a failing sibling vetoes recovery. Roster bypass retains discovery behavior without consuming data-probe slots. Completed HEAD/empty responses still supply positive evidence; unread body close does not.

The original real-HTTP reproduction fails on the base (zero requests but circuit closed) and passes on this change (zero requests and circuit half-open). Main/provider vet, race, coverage and build checks, independent goldens and all eight documentation checks pass. The source code adds no wire or routing changes. ADR-0013 is marked accepted in the merge candidate so the decision and implementation land atomically.

@Coldwings

Copy link
Copy Markdown
Collaborator Author

Independent review found two delegated-origin boundary issues and one documentation mismatch:

  • A configured http.Client.Timeout can expire during a body read without canceling the peer wrapper's own context. That path was still recorded as an answer, incorrectly recovering a half-open circuit. A real HTTP reproduction confirmed it.
  • The same HTTP timeout setting wraps http.NoBody, so pointer identity cannot establish completion of HEAD or explicitly empty responses. Closing those completed responses could leave a half-open circuit unrecovered. Real HTTP GET/HEAD reproductions confirmed it.
  • The GetValidated API comment and package document still described every validator error as a hard failure, omitting the new cancellation/deadline distinction.

The reviewers found the neutral cohort, generation isolation and roster admission accounting sound. These findings will be fixed and the updated head independently reviewed again before merging.

Recognize body deadlines imposed by the underlying HTTP client without relying on cancellation of the wrapper context. Use HTTP response semantics to settle HEAD and empty responses despite timeout body wrappers. Pin entry reuse and exactly-once close accounting and clarify validator context-error behavior.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Response handling misclassifies complete HEAD/zero-length responses and client body-read timeouts, potentially leaving half-open circuits unrecovered or incorrectly clearing failures.

2 open issues
What changed in this PR

Fixes #104 by separating caller cancellation from peer-health evidence, so canceled requests release admission without falsely recovering a circuit or erasing failure history.

Changes:

  • Applies neutral cancellation accounting to block reads, validation, roster exchange, and delegated-origin requests.
  • Adds regression coverage for probe cohorts, cancellation timing, stale completions, and origin-body lifecycle.
  • Documents the recovery contract in ADR-0013, peer documentation, and the changelog.
File Description
internal/​peer/​roster.go Applies neutral accounting to canceled roster requests.
internal/​peer/​peer.go Distinguishes caller cancellation from failures during block requests.
internal/​peer/​origin.go Updates delegated-origin completion and cancellation accounting.
internal/​peer/​origin_test.go Checks breaker accounting for idle timeouts.
internal/​peer/​neutral_cancel_test.go Adds cancellation and recovery regression tests.
internal/​peer/​breaker.go Settles neutral outcomes while preserving cohort recovery rules.
docs/​peer.md Updates cancellation, recovery, and validation documentation.
docs/​adr/​README.md Registers ADR-0013.
docs/​adr/​0013-settle-canceled-peer-requests-neutrally.md Defines neutral settlement and probe-cohort recovery.
CHANGELOG.md Records the cancellation-accounting fix.

Review effort: Balanced

Comment thread internal/peer/origin.go Resolved
Comment thread internal/peer/origin.go Resolved
@Coldwings

Copy link
Copy Markdown
Collaborator Author

Addressed the independent review findings. Delegated bodies now recognize HTTP client/transport timeout errors even when the wrapper context remains live. HEAD, explicit empty responses and 204/304 complete according to HTTP semantics, and their body/timer wrappers are closed. Validator API documentation now describes neutral caller cancellation and soft client-owned context timeouts.

Added real HTTP timeout and empty-response regressions, and strengthened tests so stale same-generation entry completion and repeated body Close would detect accidental release of another probe. Main and provider vet/race/coverage/build checks passed after the source fixes; focused peer vet/race/coverage and all documentation checks passed after final test strengthening. Updated-head independent reviews are in progress.

Close the HTTP timeout wrapper before replacing a semantically empty response body with http.NoBody, preserving normal EOF reads for callers.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the change separating caller cancellation from peer-health accounting.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@Coldwings

Copy link
Copy Markdown
Collaborator Author

Independent review completed for the updated change and its final empty-response cleanup. Both reviewers report no blocking objections.

  • Correctness/concurrency review verified neutral probe cohorts, failure veto, entry/generation isolation, discovery bypass and delegated-origin completion. Both original real-HTTP timeout/HEAD reproductions now pass.
  • Contract/tests review verified HTTP client timeout handling, complete HEAD/empty/204/304 responses, readable empty bodies, validator error documentation, and tests that detect stale-entry or repeated-close release of another probe.

The second review found the earlier findings resolved; the final follow-up also confirmed recovery occurs at headers and body reads return normal EOF. Focused race tests, original independent reproductions, documentation checks and diff checks pass. One non-blocking coverage opportunity remains: explicitly expiring a caller deadline during an active body read; current tests separately cover active body cancellation, caller deadlines before dispatch, and client-owned body deadlines.

@Coldwings Coldwings left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final self-review complete. The HTTP timeout and complete-empty-response findings are fixed, tested, replied to and resolved. Empty response wrappers are closed before returning http.NoBody; caller abandonment remains neutral, client-owned deadlines remain failures, and genuine complete responses remain positive evidence.

Verified the final source with main-module vet/race/coverage/build, and the affected peer regression after the final test assertion update. Provider vet/race/coverage/build passed after the substantive fixes. Independent goldens and all eight documentation checks pass. Both independent reviewers report no blockers for the final head. No wire or placement changes; merge remains gated on final-head CI and Copilot review.

Go 1.22 HTTP timeout errors implement net.Error but do not necessarily match context.DeadlineExceeded. Assert Timeout while retaining the live-caller and reopened-circuit checks.
@Coldwings

Copy link
Copy Markdown
Collaborator Author

The Go 1.22 CI failure was in the new timeout test's error-type assertion: the logged circuit state was correctly open and the caller was still live, but that runtime's HTTP timeout error did not match context.DeadlineExceeded. Updated the assertion to the portable net.Error.Timeout contract while retaining the live-caller and reopened-circuit checks. Production code already handles that timeout form and is unchanged. Peer vet, race and coverage checks pass; the new head is undergoing the full CI matrix and refreshed review.

@Coldwings

Copy link
Copy Markdown
Collaborator Author

Both independent reviewers checked the final Go 1.22 assertion adjustment and extend their no-blocker verdicts to the current head. The change is test-only and retains the timeout, live-caller and open-circuit assertions; production behavior is unchanged. Focused tests and diff checks pass. The final Go 1.22 result is being verified by CI.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the circuit-breaker cancellation-accounting changes.

No open issues 🎉

Review effort: Balanced

@Coldwings Coldwings left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the final test-only update is complete: the timeout assertion now uses the Go 1.22-compatible net.Error contract and retains the health-state checks. Production code is unchanged from the fully verified revision; focused peer vet/race/coverage pass. Both independent reviewers confirm no blockers for this head.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the change separating canceled-request cleanup from peer-health accounting.

No open issues 🎉

Review effort: Balanced

@Coldwings
Coldwings merged commit a57320a into data-accelerator:main Nov 9, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Canceled block requests can falsely recover a peer circuit

2 participants