Publish tracker readers immediately and bound startup refreshes

CHANGELOG.md

@@ -9,6 +9,9 @@ every release's upgrade notes.
 
 ### Added
 
+- Expose block HTTP attempts by topology target and published reader snapshot
+  sizes to diagnose startup distribution without per-connection traffic capture.
+
 - Configure the preferred owner with `-root-fanout` (default 1), independently
   of descendant `-fanout` (default 2), reducing normal first-hop replication
   (#100, ADR-0011). Hedging, root fallback and snapshot churn remain unrestricted
@@ -30,6 +33,10 @@ every release's upgrade notes.
 
 ### Fixed
 
+- Publish new tracker readers immediately and refresh active client snapshots
+  with bounded, demand-driven backoff (#102, ADR-0012). Concurrent JOINs share
+  one request; departures remain tick-batched and no background polling is added.
+
 - Coalesce normal block misses across relay hops (#97, ADR-0010). Include eligible
   self in local reader trees and preserve strictly earlier parent/ancestor edges.
   Terminal supply remains independent of waiting relays; cold requests crossing

contrib/bench/tracker-startup.py

@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+"""Compare startup publication/refresh policies using temporary Go overlays."""
+import argparse
+import json
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument('--count', type=int, default=3)
+parser.add_argument('--policy', choices=['all', 'frozen', 'normal', 'once', 'backoff'], default='all')
+parser.add_argument('--scenario', choices=['burst', 'staggered', 'reverse', 'slow-growth'])
+args = parser.parse_args()
+if args.count < 1:
+    parser.error('--count must be positive')
+repo = Path(__file__).resolve().parents[2]
+reader_path = repo / 'internal/engine/readers.go'
+tracker_path = repo / 'internal/tracker/tracker.go'
+readers = reader_path.read_text()
+start = readers.index('func readerRefreshDelay(')
+end = readers.index('\n// trackerAddr', start)
+normal = readers[:start] + '''func readerRefreshDelay(previous, ceiling time.Duration) time.Duration {
+    return ceiling
+}
+''' + readers[end:]
+once = readers[:start] + '''func readerRefreshDelay(previous, ceiling time.Duration) time.Duration {
+    if previous <= 0 { return min(initialReaderRefresh, ceiling) }
+    return ceiling
+}
+''' + readers[end:]
+tracker = tracker_path.read_text()
+start = tracker.index('\t// Growth must be visible in this JOIN response.')
+end = tracker.index('\n\treturn JoinResponse{', start)
+frozen = tracker[:start] + tracker[end:]
+policies = ['frozen', 'normal', 'once', 'backoff'] if args.policy == 'all' else [args.policy]
+with tempfile.TemporaryDirectory(prefix='dart-startup-overlay-') as tmp:
+    tmp = Path(tmp)
+    for policy in policies:
+        replacements = {}
+        if policy != 'backoff':
+            candidate = tmp / 'readers.go'
+            candidate.write_text(once if policy == 'once' else normal)
+            replacements[str(reader_path)] = str(candidate)
+        if policy == 'frozen':
+            candidate = tmp / 'tracker.go'
+            candidate.write_text(frozen)
+            replacements[str(tracker_path)] = str(candidate)
+        overlay = tmp / 'overlay.json'
+        overlay.write_text(json.dumps({'Replace': replacements}))
+        env = dict(os.environ, DART_TRACKER_STARTUP_EXPERIMENT='1', DART_REFRESH_POLICY=policy)
+        env.setdefault('GOMAXPROCS', '2')
+        pattern = '^TestTrackerStartupExperiment$'
+        if args.scenario:
+            pattern += '/^' + args.scenario + '$'
+        subprocess.run(['go', 'test', '-overlay', str(overlay), './internal/engine',
+                        '-run', pattern, '-count', str(args.count), '-timeout', '5m', '-v'],
+                       cwd=repo, env=env, check=True)

docs/README.md

@@ -15,7 +15,7 @@ This directory holds DART's **official documentation** (tracked in git). It targ
 | [fetch.md](./fetch.md) | `internal/fetch` — origin HTTP range fetch + singleflight coalescing | Implemented, tests passing |
 | [engine.md](./engine.md) | `internal/engine` — read-through orchestration (cache/peer/origin), whole-block relay, bounded prefetch, hedging | Implemented, tests passing |
 | [peer.md](./peer.md) | `internal/peer` — node-to-node block transport (server + pooled client) | Implemented, tests passing |
-| [tracker.md](./tracker.md) | `internal/tracker` — per-file active reader set (leases, tick freeze) | Implemented, tests passing |
+| [tracker.md](./tracker.md) | `internal/tracker` — per-file active reader set (leases, immediate growth, batched departures) | Implemented, tests passing |
 | [registry.md](./registry.md) | `internal/registry` — container registry pull-through mirror (OverlayBD/containerd path) | Implemented, tests passing |
 | [observability.md](./observability.md) | `internal/metrics` + `internal/admin` — Prometheus exporter, admin endpoints | Implemented, tests passing |
 | [node.md](./node.md) | `internal/node` — node assembly: flags, store/engine wiring, discovery-scheme dispatch | Implemented, tests passing |

docs/adr/0012-publish-reader-growth-immediately.md

@@ -0,0 +1,70 @@
+# ADR-0012: Publish reader growth immediately and refresh on bounded demand
+
+- Status: accepted
+- Date: 2026-11-09
+- Supersedes: none
+- Binds: docs/tracker.md publication, integration and Stability Contract; docs/engine.md reader refresh and Stability Contract; docs/observability.md topology metrics; docs/dart.md tracker-tick
+
+## Context
+
+Issue [#102](https://github.com/data-accelerator/dart/issues/102) identifies two
+startup delays: freezing the first JOIN's singleton reader set until a tracker
+tick, then retaining that response in client caches for half a reader lease.
+Local self completion keeps routing acyclic but cannot discover other readers.
+Even with root fanout one, many different singleton snapshots can send all
+readers directly to the root. Normal reader forwarding already follows strictly
+earlier HRW ranks under common member weights; correctness does not require
+synchronized reader snapshots or frozen topology.
+
+## Decision
+
+Publish newly joined readers before replying, batch departures on ticks, and
+refresh client snapshots only on routing demand with shared per-object/view
+JOINs and time-based doubling from 50 ms to half the granted lease.
+
+## Consequences
+
+Renewing an already published reader changes no publication epoch unless a due
+removal sweep changes the set. New JOINs do not postpone removal processing;
+last-reader deletion and idle eviction retain their immediate/lazy semantics.
+Snapshots remain sorted copies, and publication epochs remain local to one
+tracker file entry. JSON and peer wire framing do not change.
+
+An active client's successful refresh intervals double independently of reader
+list changes. Short grants cap even the initial interval; missing grants retain
+a two-second ceiling. Expired leases or a different tracker membership view
+restart the schedule. There are no background refresh workers. Waiters may
+cancel independently; a canceled leader can be replaced without publishing its
+late result. Failed JOINs are shared but not cached. A response started before
+a more recently installed view cannot overwrite that view.
+
+A synthetic warm-root experiment supports repeated early refresh for gradual
+arrivals, while a single early refresh is cheaper for completed bursts. Fifty
+milliseconds is a starting engineering choice, not an optimal cluster constant.
+Late arrivals after the schedule reaches its ceiling still wait for the next
+routing lookup. Refresh traffic scales with active node/object pairs; this
+change makes no thousand-reader control-plane scalability claim.
+
+Expose low-cardinality counters for dispatched root/parent/ancestor block
+attempts and a histogram of reader counts in successful JOIN replies. Exclude
+pre-dispatch rejections, local hits and coalesced JOIN waiters. Attempts are not
+wire-byte counters; raw reader counts are not filtered tree sizes.
+
+The preferred owner, local self completion, strict earlier-rank forwarding,
+normal cross-hop coalescing, terminal supply flights and ordering-epoch boundary
+remain unchanged. Mixed versions retain read correctness but obtain only
+partial startup performance improvement; coordinated upgrade is recommended.
+
+## Alternatives considered
+
+- Tick-frozen growth and normal half-lease refresh preserve the initial root
+  bottleneck despite a healthy, warm root and available reader peers.
+- A single early refresh has lower control-plane cost for bursts but can finish
+  before later readers join; simple doubling also covers gradual startup.
+- Repeated fixed fast polls continue spending requests after startup. Background
+  refresh also spends requests on completed reads.
+- Notifications, gossip convergence, reader-change-triggered resets and global
+  synchronization add state or RPC dependencies that ordered forwarding does
+  not need. This remains a trusted-cluster design.
+- Per-object or per-peer metric labels have unbounded operational cardinality;
+  aggregate topology classes and snapshot-size observations suffice here.

docs/adr/README.md

@@ -135,3 +135,4 @@ acceptable in current law — it is a pointer, not a narrative.
 | [0009](./0009-root-eligible-peer-roles.md) | Anchor distribution at role-eligible owners | accepted |
 | [0010](./0010-coalesce-directed-block-loads.md) | Coalesce directed normal block loads across hops | accepted |
 | [0011](./0011-separate-root-fanout.md) | Configure root fanout independently of reader branching | accepted |
+| [0012](./0012-publish-reader-growth-immediately.md) | Publish reader growth immediately and refresh on bounded demand | accepted |

docs/dart.md

@@ -55,7 +55,7 @@ dart [flags]
 | `-root-fanout` | `1` | preferred root normal branching factor (≥1); not a limit on hedge/fallback transfers |
 | `-admin` | `:9100` | admin/metrics listen address; empty disables |
 | `-reader-tree` | `true` | build the distribution tree over the active reader set (per-file tracker) |
-| `-tracker-tick` | `3s` | how often a tracker republishes a reader set |
+| `-tracker-tick` | `3s` | removal/expiry batching interval; new readers publish immediately |
 | `-replicas` | `1` | HRW candidates that authoritatively hold a chunk (the owned budget) |
 | `-owned-fraction` | `0.8` | owned share in [0,1]; 0 is borrowed-only, 1 owned-only; invalid/non-finite values fail startup |
 | `-mem-size` | `256 MiB` | Hybrid hot-set size; ignored in memory/disk modes; below one block disables the Hybrid front tier |

docs/engine.md

@@ -32,14 +32,23 @@ never become descendants; non-root peers never contact origin. Without P2P,
 ordinary nodes operate as single-node caches; node assembly requires P2P for
 restricted roles.
 
-Reader-set lookups are **cached per object for half the granted tracker lease**
-(and the cache doubles as the lease renewal: a hit re-serves the frozen set, a
-miss re-JOINs, which refreshes this node's own lease). Caching for half the
-lease guarantees renewal always lands before the lease lapses; a tracker that
-grants nothing falls back to a 2 s period. Expired entries are swept as the
-cache grows past a few hundred objects, so the map tracks objects currently
-being read and stays small. When the tracker is unreachable the engine falls
-back to the preferred owner for that lookup (the reader set is soft state).
+Reader-set lookups are cached per object and tracker membership view (ADR-0012).
+On continued routing demand, successful refreshes use 50 ms, 100 ms, 200 ms, ...,
+capped at half the granted tracker lease. A short lease caps even the first
+interval; a missing grant uses a 2 s ceiling. Each miss re-JOINs to renew the
+lease. The schedule restarts after lease expiry or tracker-view change, never
+because the reader list changed. There are no background polls or refresh RPCs
+on an ordinary cache hit. Concurrent JOINs share one inline caller; waiters can
+cancel independently and retry after leader cancellation. Failed refreshes are
+not cached, and late responses cannot replace a more recently installed view.
+Entries past their lease are swept as the map grows beyond 256 objects. When
+the tracker is unreachable that lookup routes to the preferred owner.
+A fresh cache hit reuses tracker selection for the identical immutable membership
+view, avoiding full-cluster ranking and allocations. A replacement view is
+re-evaluated even when its ordering epoch is unchanged, because addresses and
+liveness are not part of that epoch.
+See [tracker measurements](./tracker.md#startup-measurement-and-tradeoffs) for
+startup traffic and the additional control-plane request cost.
 
 ## 2. Concepts
 
@@ -579,6 +588,10 @@ block sources. A supply root records its proxied response bytes as `origin_in`.
 
 ## 6. Determinism / Stability Contract
 
+Reader JOIN growth is immediate; client refresh uses demand-driven bounded
+backoff and shared per-object/view calls (ADR-0012, [tracker](./tracker.md)).
+This does not alter HRW order, normal cross-hop flights or terminal supply.
+
 - ADR-0011 defaults the preferred root to one child and descendants to two.
   Explicit equal degrees preserve the uniform tree. Changing degrees preserves
   the strict HRW dependency order; no flight key or membership epoch change is
@@ -718,7 +731,8 @@ go test ./internal/engine/ -cover -count=1
 | `TestBlockDoesNotCacheRangeIgnored` | defense in depth: bytes sliced from a Range-ignored 200 are served, never cached |
 | `TestHedgeWinMetricsRequireFiredHedge` | no hedge fired → no hedge-win counters (the backup_won/fired comparison stays meaningful) |
 | `TestMissesDoNotFeedLatencyEstimator` | fast 404 misses never arm the hedge delay |
-| `TestReaderSetCacheFollowsGrantedLease` | the reader-set cache renews at half the granted lease, never lapsing mid-read |
+| `TestReaderSetCacheFollowsGrantedLease` | a short granted lease bounds the reader-set cache renewal period |
+| `TestPeerFetchTargetMetrics` | real root, parent and ancestor attempts match metrics, including breaker-promoted ancestors |
 | `TestRelayParentErrorFallsBackToOrigin` | an open-circuit/errored parent is skipped to origin, not propagated as a 500 |
 | `TestEmptyObjectGetServes200` | empty object: 200 empty on both paths; a Range request stays 416 |
 | `TestParseRangeSuffixEmptyObject` | a suffix range on an empty object is unsatisfiable (no inverted `[0,-1]` interval); non-empty suffix behavior unchanged |

docs/observability.md

@@ -77,6 +77,8 @@ and safe for concurrent use, since it runs on the scrape path.
 | `dart_relay_total` | counter | `result=served\|declined` | relay requests handled for peers |
 | `dart_block_fetch_seconds` | histogram | `source=peer\|origin` | block-path fetch latency: cache fills and peer reads only — Size probes and the uncached passthrough path are **not** observed here (`dart_passthrough_total` counts the latter). A coalesced fetch still observes its latency (the caller genuinely waited) even though its bytes are not double-counted in `dart_bytes_total` |
 | `dart_hedge_total` | counter | `event=fired\|primary_won\|backup_won` | tail-latency hedges: duplicates launched, and which contender served the block |
+| `dart_peer_fetch_total` | counter | `target=root,parent,ancestor` | dispatched block HTTP attempts; root includes fallback supply roots; parent means the direct tree parent, ancestor a higher non-root ancestor |
+| `dart_reader_snapshot_size` | histogram | — | reader count in successful JOIN replies; one observation per completed shared refresh, not per waiter or cache hit |
 | `dart_peer_failover_total` | counter | — | definite peer failures escalated to the next ancestor (not rate limited, unlike hedges) |
 | `dart_passthrough_total` | counter | `reason=range_unsupported` | requests proxied without block caching (direct for root-capable nodes, delegated for non-root nodes; see docs/engine.md §3.9) |
 | `dart_peer_circuits_open` | gauge | — | peers whose circuit is currently open |
@@ -109,6 +111,31 @@ just briefly slow.
 The store and breaker metrics are wired by `engine.RegisterStoreMetrics(reg, store)`
 and `engine.RegisterPeerMetrics(reg, breaker)`; `cmd/dart` calls both.
 
+### Inspecting reader-tree startup
+
+`dart_peer_fetch_total` classifies from the requesting node's tree snapshot.
+A root is counted as `root` even when it is the direct parent; a breaker-skipped
+parent does not turn the promoted ancestor into `parent`. Buffer admission
+rejections, circuit rejection and local hits do not count. Failed or canceled
+after-dispatch attempts and hedges do count; internal HTTP retries do not add
+another observation. These are attempts, not successful blocks or wire bytes.
+
+```promql
+# Root share of dispatched block attempts across the selected instances.
+sum(rate(dart_peer_fetch_total{target="root"}[1m]))
+  / sum(rate(dart_peer_fetch_total[1m]))
+# Fraction of observed JOIN responses with at most one published reader.
+sum(rate(dart_reader_snapshot_size_bucket{le="1"}[1m]))
+  / sum(rate(dart_reader_snapshot_size_count[1m]))
+```
+
+The histogram measures raw published IDs before membership/role filtering and
+local-self completion, not the size of every currently cached client view.
+It also observes local-registry JOINs. No file, URL or peer-ID label is emitted.
+Compare root share, singleton observations and existing byte/hedge/failover
+metrics over an active interval. These signals reveal likely startup collapse;
+they cannot prove a globally consistent tree or measure per-target bandwidth.
+
 ## 4. `internal/admin` endpoints
 
 | Method | Path | Response |

docs/peer.md

@@ -147,6 +147,8 @@ type LeaseSource func(context.Context, BlockRequest) (store.Lease, bool, error)
 // Server.Borrow LeaseSource takes precedence over Server.Src.
 func (c *Client) GetInto(ctx context.Context, addr string, req BlockRequest,
     dst []byte, validate func(context.Context, int64) error) ([]byte, bool, error)
+func (c *Client) GetIntoObserved(ctx context.Context, addr string, req BlockRequest,
+    dst []byte, validate func(context.Context, int64) error, onStart func()) ([]byte, bool, error)
 ```
 
 `LeaseSource` transfers immutable pinned bytes to the server, including any lease
@@ -163,6 +165,14 @@ It reads at most len(dst)+1 bytes, without a growing payload allocation. The
 known-length GetValidated compatibility API allocates one expected-size payload
 buffer. Read-only sharing and cache publication belong to the engine (ADR-0010).
 
+`GetIntoObserved` adds an inline callback immediately before HTTP dispatch. It
+must return promptly and runs outside breaker locks. Breaker rejection, invalid
+request construction and already-canceled contexts do not notify. Dispatch can
+still fail before bytes reach the network; net/http internal retries are not
+separate observations. The client's own deadline expiring before dispatch still
+records a breaker failure, as it does during transport; it is distinct from
+caller cancellation. This is attempt instrumentation, not a byte counter.
+
 ### 3.4 Validated whole-block reads: `Client.GetValidated`
 
 ```go
@@ -435,6 +445,11 @@ go test ./internal/peer/ -race -count=1
 
 ### Test list (property each guards)
 
+`TestGetIntoObservedDispatch` verifies pre-dispatch observation on real HTTP
+success/miss/error and exclusion of breaker/setup/cancellation rejections.
+`TestGetIntoObservedClientTimeoutBeforeDispatch` verifies that an undispatched
+client-owned timeout cannot falsely recover a half-open circuit.
+
 | Test | Property guarded |
 |---|---|
 | `TestServerClientRoundtrip` | store → server → client returns exact block bytes |