Skip to content

fix(rpc): harden backup failover blacklisting and probes - #128

Merged
anhthii merged 1 commit into
feat/rpc-backup-providersfrom
fix/rpc-backup-providers-review
Nov 12, 2026
Merged

anhthii merged 1 commit into
feat/rpc-backup-providersfrom
fix/rpc-backup-providers-review

Conversation

@vietddude

Copy link
Copy Markdown
Collaborator

Follow-up to #127, targeting its branch. Fixes from reviewing the backup provider / escalating blacklist / probe changes.

Changes

  • One strike per incident. BlacklistEscalating added a strike per failed request, so several in-flight requests failing in one blip jumped the cooldown from 3m to the 30m cap. Failures during an active blacklist no longer add a strike; they only extend the expiry when the new cooldown is longer (e.g. a 403).
  • Force-rotate reaches the backup. With a backup configured, a primary over ForceRotateThreshold was picked again by the primaries-first pass, so the backup was never used. It is now blacklisted for 1m (escalating, probe-gated), which sends traffic to the backup and brings the primary back on the timer. Configs without a backup keep the previous rotation behaviour.
  • Total outage does not escalate. A failure right after emergency recovery reuses the current strike, so a short network or DNS outage leaves providers at their base cooldown instead of the cap.
  • Stale probe results are dropped. A generation counter on the provider lets finishProbe ignore a result when the provider was recovered or re-blacklisted while the probe ran.
  • Less locking. switchToPrimary takes the failover write lock only when a primary is available. Probe gating is one locked method (settleExpiredBlacklist) instead of three round-trips; RequiresProbe, ReblacklistEscalating, tryStartProbe and endProbe are gone.
  • Cleanup. GetLatestBlockTimestamp uses utils.ParseHexUint64; comments trimmed to one line.

Not changed

A single Success() still resets strikes. Keeping strikes across successes would hold a rate-limited free primary out for longer and push more traffic to the paid backup.

Tests

  • New: concurrent failures count as one strike, force-rotate reaches the backup and returns, total outage does not escalate, stale probe results are dropped.
  • TestExecuteWithRetryProvider_FallsBackToBackup no longer sleeps the 5s retry interval.
  • go test -race ./internal/rpc/ and go test ./internal/rpc/evm/ ./internal/worker/ pass.

- count one blacklist strike per incident instead of per failed request
- blacklist a force-rotated provider when a backup exists so traffic reaches the backup and returns on a timer
- reuse the strike after emergency recovery so a total outage does not escalate cooldowns
- drop probe results when the provider changed state while the probe ran
- take the failover write lock only when a primary is available
- remove the 5s sleep from the backup fallback test
@anhthii
anhthii merged commit d35e4c5 into feat/rpc-backup-providers Nov 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants