[Coldwings]

Fixes #102

New readers joining during a tracker's first tick can all receive the first reader's singleton snapshot. Keeping that response for half a lease sends startup traffic directly to the root despite root-fanout=1. A real HTTP regression with a warm root and eight cold readers delivered 32 MiB from the root and no reader-to-reader payload on the base; immediate publication delivers 8 MiB from the root and 24 MiB through readers, with the same eight JOINs and correct data.

Publish growth during JOIN while retaining tick-batched departures, last-reader deletion and idle eviction. Client refreshes are demand-driven and shared per object/tracker view: start at 50 ms, double after successful refreshes, and cap at half the granted lease. Completed reads cause no background polls. Fresh cache hits reuse tracker selection for the identical immutable membership view; replacement views recheck address/liveness even if the ordering epoch is unchanged. Cancellation, failed refreshes and late responses preserve caller ownership and newer cached views. Owner selection, HRW order, cross-hop block coalescing and terminal supply remain unchanged.

Add dart_peer_fetch_total{target="root|parent|ancestor"} for dispatched block HTTP attempts and dart_reader_snapshot_size for successful JOIN response sizes. Admission/circuit rejections and cache hits are excluded; shared JOIN waiters do not duplicate histogram observations. These metrics have no per-object or per-peer labels and do not claim to measure per-target wire bytes.

The reproducible startup experiment compares frozen publication, ordinary refresh, one early refresh and bounded doubling. Doubling reduces root payload for gradual arrivals at additional JOIN cost; it does not uniformly improve p99. The 50 ms start is an engineering choice, not an optimal-cluster claim. Synthetic loopback results, request costs and limitations are documented in docs/tracker.md; python3 contrib/bench/tracker-startup.py --count 3 reproduces the policy comparison without changing sources.

No wire, hashing or configuration format changes. -tracker-tick now controls removal/expiry batching; new JOINs do not wait for it. Mixed versions retain read correctness but deliver partial startup improvement. A coordinated upgrade is recommended. ADR-0012 records the contract and rejected synchronization alternatives.

Validation

• Main module: go vet ./..., go test ./... -race -count=1, go test ./... -cover -count=1, go build ./....
• Separate providers/k8s module: the same vet/race/coverage/build checks.
• DART_GOLDEN_REF=1 go test ./internal/chunk ./internal/hashring ./internal/cluster -count=1.
• New refresh/cache-hit, dispatch-metric and hedge-admission regressions repeated 30 times with -race; peer dispatch and client-timeout tests repeated 100 times.
• Zero-allocation stable snapshot hits, same-epoch address/liveness changes, and lookup-only microbenchmarks at 32/256/4096 members.
• Real HTTP startup byte accounting, all four experimental policies, and bash scripts/check-docs.sh (all eight checks).

Checklist

• go vet ./... / go test ./... -race -count=1 / go test ./... -cover -count=1 all pass (AGENTS.md)
• Package documentation updated; no new package
• bash scripts/check-docs.sh passes
• ADR: 0012 (tracker publication and client refresh contract)

Self-review completed for this head.

• Traced immediate JOIN growth, renewal without epoch churn, departure/expiry batching, sorted response copies and last-reader deletion. New readers do not postpone the removal tick.
• Traced shared refresh ownership through independent waiter cancellation, leader replacement, failure sharing and late responses from another tracker view. Refresh is demand-driven; no background lifecycle or new cluster synchronization is introduced. Existing owner selection, earlier-rank forwarding and terminal supply separation are preserved.
• Checked target counters against actual HTTP requests, including breaker-promoted ancestors and root fallback after a rejected hedge. Snapshot-size observations count successful shared JOINs, excluding waiters and cache hits. Metrics are attempts and raw published sizes, not per-target traffic or a proof of global tree agreement.
• Confirmed the startup regression counts actual HTTP payload writes and compares all client bytes. The opt-in policy experiment records 1 MiB latency and JOIN costs; docs retain the slow-growth p99 tradeoff and synthetic-root limitation.

Main and separate Kubernetes module vet/race/coverage/build checks passed, as did independent golden checks, 30 race repetitions of refresh/metric/admission regressions, and all eight documentation checks. No local blocking objection remains; independent review and remote CI are tracked separately.

Two independent read-only reviews completed for the initial implementation.

• Correctness/concurrency: no blocking objections. Reviewed publication/expiry boundaries, cancellation ownership, canceled-leader replacement, response ordering, lease caps, and unchanged directed routing/supply dependencies. Residual limit: tracker HRW selection now precedes snapshot-cache lookup, and large-membership CPU cost has not been measured.
• API/docs/metrics/experiments: no blocking objections. Dispatch classes and snapshot observations match their documentation; experimental evidence explicitly retains extra JOIN cost, worse slow-growth p99, and synthetic-network limits. One non-blocking coverage gap: the backoff regression does not yet change its returned reader list despite documenting that list changes cannot reset the interval.

I will add that regression assertion and have both reviewers verify the resulting head. The reviewers inspected source and contracts; they did not independently reproduce the measured medians or rerun the full test suite.

Both independent reviewers have verified the final head and report no blocking objections or outstanding findings.

• The concurrency reviewer confirmed that the production implementation and its cancellation/lifetime assessment are unchanged by the follow-up.
• The contract/metrics reviewer confirmed that changing reader IDs and publication epoch on every simulated JOIN closes the backoff coverage finding.

Residual limits remain explicit: large-membership CPU cost of tracker selection has not been measured, and the synthetic experiment is not a cluster-scalability or NIC-throughput result. The reviewers remained read-only and did not independently rerun the benchmark medians. Final local main-module vet/race/coverage checks and 30 race repetitions of the refresh tests passed; all production changes retain the previously completed separate Kubernetes/build/golden/documentation validation.

Final-head self-review: the only follow-up changes the backoff regression's reader IDs and publication epoch on each JOIN. The expected 50/100/200/.../3000 ms sequence and idle-lease reset still pass, directly covering the independent review finding. Production behavior is unchanged. Main-module vet, full race and coverage checks passed again, together with 30 race repetitions of the refresh regressions. No local blocking objection remains; both independent reviewers also verified this head.

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

2 open issues

• This overlay patching depends on exact string matches including a literal tab prefix and a specific… · New
• ctx.Err() being non-nil here means the request was canceled before any HTTP dispatch occurred,… · New

What changed in this PR

This PR changes tracker publication semantics to publish reader growth immediately (while batching removals on ticks), adds demand-driven bounded backoff + coalescing for engine tracker JOIN refreshes, and introduces new observability metrics for peer attempt targeting and reader snapshot sizes.

Changes:

• Publish newly joined readers during Registry.Join (epoch advances on growth), while keeping departures/expiry tick-batched.
• Add engine-side shared refresh flights with cancelable waiters and bounded exponential backoff tied to lease grants.
• Add peer.Client.GetIntoObserved to observe pre-dispatch attempts and record new dart_peer_fetch_total + dart_reader_snapshot_size metrics; update docs and add ADR-0012.

File	Description
internal/​tracker/​tracker_test.go	Updates/adds tests for immediate growth, renewal epoch stability, and tick-batched departures.
internal/​tracker/​tracker.go	Implements immediate publication of JOIN growth and updates comments/semantics accordingly.
internal/​tracker/​client.go	Updates client comment to “published” set terminology.
internal/​peer/​peer.go	Adds GetIntoObserved and a pre-dispatch callback hook in the shared get path.
internal/​peer/​observed_test.go	Adds tests validating observation timing and non-notification on pre-dispatch rejections.
internal/​node/​node.go	Updates -tracker-tick flag description to reflect new batching semantics.
internal/​engine/​tracker_startup_test.go	Adds end-to-end startup traffic test proving growth-before-tick improves P2P usage.
internal/​engine/​tracker_experiment_test.go	Adds opt-in experiment for measuring startup policies and refresh tradeoffs.
internal/​engine/​routing_metrics_test.go	Adds tests validating new peer target metrics classification.
internal/​engine/​routing.go	Records per-attempt peer target metrics via GetIntoObserved callback.
internal/​engine/​roles_golden_test.go	Updates cached reader-set construction to include tracker basis (addr/epoch).
internal/​engine/​review_regression_test.go	Adds assertions ensuring dispatch metrics don’t count unadmitted hedges.
internal/​engine/​readers_test.go	Updates tests to reflect JOIN growth being published immediately.
internal/​engine/​readers.go	Reworks reader snapshot caching: shared refresh flights, bounded backoff, basis keyed by tracker view.
internal/​engine/​reader_refresh_test.go	Adds comprehensive tests for coalescing, cancellation, overwrite protection, and backoff schedule.
internal/​engine/​metrics.go	Adds dart_peer_fetch_total counters and dart_reader_snapshot_size histogram + recording helpers.
internal/​engine/​engine.go	Adds rsCalls map to support shared refresh flights.
docs/​tracker.md	Updates tracker documentation to “immediate growth / batched departures” + implementation details.
docs/​peer.md	Documents GetIntoObserved semantics and test coverage.
docs/​observability.md	Documents new metrics and PromQL examples for startup diagnosis.
docs/​engine.md	Updates engine documentation for demand-driven refresh, shared calls, and backoff behavior.
docs/​dart.md	Updates CLI flag docs for -tracker-tick.
docs/​adr/​README.md	Adds ADR-0012 to the ADR index.
docs/​adr/​0012-publish-reader-growth-immediately.md	Introduces ADR-0012 describing the decision and consequences.
docs/​README.md	Updates tracker documentation blurb to reflect new semantics.
contrib/​bench/​tracker-startup.py	Adds a benchmark script using Go overlays to compare refresh/publication policies.
CHANGELOG.md	Adds changelog entries for new metrics and the ADR-0012 behavior change.

_{Review effort: Lite}

Follow-up review identified one introduced timeout-accounting regression, superseding the prior no-blocker status until fixed.

Both independent reviewers distinguished the original caller context from the client-owned request timeout: the new pre-dispatch check saw the derived deadline and unconditionally recorded outcomeAnswered, whereas the existing transport-error path records the client's own deadline as a failure. A focused regression reproduced zero HTTP requests and zero dispatch notifications while the half-open breaker incorrectly closed.

The narrow correction preserves the existing caller-versus-client-timeout distinction in the early return. The broader false recovery on caller cancellation is independently reproduced on both main and this PR and is tracked separately in #104. It is not introduced by the tracker change. Full validation and final-head reviews will be repeated after the correction.

Both independent reviewers have re-reviewed the corrected final head and report no blocking objections. The introduced pre-dispatch client-timeout regression is fixed and covered by a real HTTP/circuit-state test. Both agree that the independently reproduced caller-cancellation false-recovery defect belongs in #104 and is not claimed as fixed by this PR.

The earlier publication, refresh-concurrency, metrics and experiment assessments remain valid. The exact-anchor experiment script is retained as an explicitly fail-fast maintenance tradeoff. Large-membership tracker selection cost and cluster scalability remain unmeasured.

Validation was repeated for the correction: main and separate Kubernetes module vet/race/coverage/build, independent goldens, all eight documentation checks, and 100 race repetitions of the peer dispatch/timeout tests passed. Reviewers remained read-only and did not independently rerun benchmarks or the full test suite.

Self-review of the corrected final head: traced both context sources through the early-return and HTTP-error paths. Client-owned timeouts now retain failure accounting; caller cancellation retains the pre-existing policy tracked in #104. The new regression confirms zero HTTP dispatch and no false breaker recovery, with 100 race repetitions. Full main/Kubernetes validation, goldens and docs checks passed again. Both independent reviewers verified this head with no blocking objections. Prior review comments have explicit dispositions and are resolved.

🔵 Needs a closer look

Tracker selection now runs before the fresh-cache check, adding O(N log N) member ranking and cluster-sized allocations to every reader-snapshot lookup, including cache hits.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Avoid reranking cluster members on reader-snapshot cache hits

internal/​engine/​readers.go:81

Calling trackerAddr before the fresh-cache check now hashes and sorts every Ready member and allocates three cluster-sized slices on each reader-snapshot lookup, even on cache hits. This adds O(N log N) cluster-wide work per routed block, including when only a few members are active readers. Cache tracker selection per object and immutable membership view so unchanged-view hits avoid ranking; do not key this optimization solely by View.Epoch(), which excludes address and liveness changes.

2 resolved since last review

• ctx.Err() being non-nil here means the request was canceled before any HTTP dispatch occurred,…
• This overlay patching depends on exact string matches including a literal tab prefix and a specific…

_{Review effort: Balanced}

Addressed the additional tracker-selection cost raised in the latest review summary. Fresh snapshot hits now reuse selection for the identical immutable membership view, avoiding cluster-wide ranking. A changed view always re-evaluates selection, including address or liveness changes with an unchanged ordering epoch. If selection remains the same, only an immutable view marker is updated; lease/refresh deadlines and response age are preserved.

Added stable-hit allocation, canceled-hit and same-epoch address/liveness regressions, plus a lookup-only microbenchmark. On the local machine, 256-member hits changed from about 40 microseconds / 10680 bytes / 5 allocations to about 64 nanoseconds / zero allocations. This measures only snapshot lookup, not application throughput. The same zero-allocation result holds at 32 and 4096 members.

Full main/Kubernetes vet, race, coverage and build checks, independent goldens and all eight docs checks passed again; reader refresh/cache tests passed 30 race repetitions. Current-head independent review and Copilot re-review are being completed.

Both independent reviewers verified the final snapshot-cache optimization and report no blocking objections on this head.

Identical-view reuse follows the immutable membership-view contract. Replacement views recheck addresses/liveness despite unchanged epochs. Same-basis updates copy the cache entry under its mutex without extending deadlines or publication age; cancellation and stale-flight protections remain intact. Retention is one shared view pointer per entry, without a historical chain, released through replacement or existing lazy eviction.

The previously noted stable-hit ranking cost is addressed. Allocation tests and benchmarks measure lookup cost only; broader scalability remains unproven. The timeout fix remains intact and pre-existing #104 remains separate. Reviews were read-only; the reviewers did not rerun builds or benchmark measurements.

Final-head self-review: verified that fresh same-view hits bypass ranking, while new immutable views re-evaluate address/liveness rather than relying solely on ordering epoch. Same-basis reuse preserves cache age and all deadlines; cancellation is checked before the fast return. Confirmed existing refresh ownership and the client-timeout fix remain unchanged. Full main/Kubernetes checks, goldens and docs checks pass, as do 30 race repetitions of refresh/cache regressions. Both independent reviewers verified this head with no blocking objections.

Note

Copilot was unable to run its full agentic suite in this review.

Copilot review overview

5 open issues

• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)… · New
• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)… · New
• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)… · New
• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)… · New
• The field name frozen now holds the published set that can grow immediately on JOIN, so it’s no… · New

_{Review effort: Lite}

🟡 Changes recommended

The seeded test entry lacks view: view, causing remote-tracker cases to bypass supplied snapshots and leaving the intended snapshot-churn and non-mutation checks ineffective.

1 open issue

• Seeded entry bypasses snapshot coverage for remote trackers · New

5 resolved since last review

• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)…
• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)…
• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)…
• expiresAt and leaseUntil are computed from now captured before performing the (possibly slow)…
• The field name frozen now holds the published set that can grow immediately on JOIN, so it’s no…

_{Review effort: Balanced}

Both independent reviewers assessed the five latest comments on the unchanged final head and retain their no-blocking-objection verdict.

Four comments repeat the same timestamp concern. Request-start accounting conservatively bounds a lease granted during server processing; moving it to response completion can overestimate the lease after a delayed response. Refresh spacing is start-to-start and demand-driven, so a slow JOIN can consume its interval without defeating doubling or creating a background retry loop. The comments have been replied to and resolved with this explanation.

The remaining private-field naming suggestion is optional cleanup; current comments and immediate-publication tests describe the semantics correctly. No source changes were needed for these dispositions. Reviews remained read-only.

Both independent reviewers verified the final test-only fixture correction and report no blocking objections. The supplied reader snapshots now carry their immutable view marker, and explicit consumption assertions prevent silently testing only owner-plus-self trees. Tracking-disabled golden cases remain unchanged, as do all independently generated expectations and production code.

The prior concurrency, timeout, timestamp-policy and metric assessments remain applicable. Reviewers stayed read-only and did not rerun tests. Main-module vet/race/coverage and 30 focused race repetitions passed for this follow-up; existing production/Kubernetes/build/golden/documentation validation remains applicable. Issue #104 is separate.

Final-head self-review: both seeded-snapshot fixtures now include the correct immutable view and assert that the intended raw snapshot is consumed before routing checks. The new assertion reproduced the original bypass, and the corrected tests passed 30 race repetitions plus full main-module vet/race/coverage checks. No production code or golden expected value changed. Both independent reviewers inspected this head with no blocking objections; the review thread has been replied to and resolved.

🟢 Approval recommended

No active review issues are reported for the reader publication and snapshot refresh changes.

No open issues 🎉

1 resolved since last review

• Seeded entry bypasses snapshot coverage for remote trackers

_{Review effort: Balanced}