Repository navigation
fix(rpc): harden backup failover blacklisting and probes - #128
Merged
anhthii merged 1 commit intoNov 12, 2026
Merged
Conversation
- count one blacklist strike per incident instead of per failed request - blacklist a force-rotated provider when a backup exists so traffic reaches the backup and returns on a timer - reuse the strike after emergency recovery so a total outage does not escalate cooldowns - drop probe results when the provider changed state while the probe ran - take the failover write lock only when a primary is available - remove the 5s sleep from the backup fallback test
3 of 4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #127, targeting its branch. Fixes from reviewing the backup provider / escalating blacklist / probe changes.
Changes
BlacklistEscalatingadded a strike per failed request, so several in-flight requests failing in one blip jumped the cooldown from 3m to the 30m cap. Failures during an active blacklist no longer add a strike; they only extend the expiry when the new cooldown is longer (e.g. a 403).ForceRotateThresholdwas picked again by the primaries-first pass, so the backup was never used. It is now blacklisted for 1m (escalating, probe-gated), which sends traffic to the backup and brings the primary back on the timer. Configs without a backup keep the previous rotation behaviour.finishProbeignore a result when the provider was recovered or re-blacklisted while the probe ran.switchToPrimarytakes the failover write lock only when a primary is available. Probe gating is one locked method (settleExpiredBlacklist) instead of three round-trips;RequiresProbe,ReblacklistEscalating,tryStartProbeandendProbeare gone.GetLatestBlockTimestampusesutils.ParseHexUint64; comments trimmed to one line.Not changed
A single
Success()still resets strikes. Keeping strikes across successes would hold a rate-limited free primary out for longer and push more traffic to the paid backup.Tests
TestExecuteWithRetryProvider_FallsBackToBackupno longer sleeps the 5s retry interval.go test -race ./internal/rpc/andgo test ./internal/rpc/evm/ ./internal/worker/pass.