[anhthii]

Summary

• Backup providers. New backup: true flag on node config. A backup provider is only used while every primary is unavailable. Traffic goes back to a primary as soon as one recovers. Wired through every chain builder in worker/factory.go.
• Escalating blacklist cooldowns. Each blacklist since the last successful request doubles the cooldown, capped at MaxBlacklistDuration (default 30m, or the base cooldown if that's larger). A successful request resets the count.
• Recovery probe (EVM only). A provider blacklisted for errors doesn't go back into rotation on a timer alone. When its cooldown expires, a background probe must pass first; see the diagrams below. Capability blacklists still recover on a timer, and other chains keep timer-based recovery.
• Fix: emergency recovery always set currentIndex = 0. It now points at the provider it actually recovered.

How it works

1. Failover to the backup

Normal:            indexer ──requests──▶ Primary ✅

Primary breaks:    indexer ──requests──▶ Primary ❌  (timeouts / errors)
                                         └─▶ blacklisted for base cooldown (strike 1)

                   indexer ──requests──▶ Backup ✅   (only used while every primary is down)

2. Recovery probe: test the primary before giving it traffic again

Before this PR, a blacklisted provider got real traffic again as soon as its cooldown expired. If it was still broken, real requests failed again. Now a small test runs in the background first, while real traffic stays on the backup:

 Primary's cooldown expires
            │
            ▼
 ┌──────────────────────────────────────────────────────────────┐
 │ PROBE  (background goroutine, 5s timeout, one at a time)     │
 │                                                              │
 │  Step 1 ─ "Are you alive?"                                   │
 │     Primary: eth_getBlockByNumber("latest", false)           │
 │     error / timeout ───────────────────────────────▶ FAIL    │
 │                                                              │
 │  Step 2 ─ "Are you up to date?"   (run in parallel w/ step 1)│
 │     Active provider (backup): same call                      │
 │     compare latest-block timestamps                          │
 │     primary more than 60s behind ──────────────────▶ FAIL    │
 │                                                              │
 │  otherwise ────────────────────────────────────────▶ PASS    │
 └──────────────────────────────────────────────────────────────┘
            │                                  │
          PASS                               FAIL
            │                                  │
            ▼                                  ▼
   un-blacklist Primary              blacklist again, cooldown x2
   traffic moves back to Primary     (capped at 30m), probe again
                                     when it expires

Step 2 matters because a node can be alive but stuck: it answers instantly but returns an old block. Step 1 alone would let it back in, and the indexer would stop moving forward.

Primary latest block: 10:00:00   Backup: 10:00:12   → 12s behind → PASS
Primary latest block: 09:50:00   Backup: 10:00:12   → 10m behind → FAIL (stuck / syncing)

When step 2 is skipped. If there's nothing reliable to compare against, a broken reference shouldn't keep a working primary out, so the probe passes on step 1 alone:

Situation	Step 2	Result if step 1 passes
Backup active and healthy	compare	pass only if ≤ 60s behind
Only one provider configured	skip	pass
Active provider itself unavailable	skip	pass
Active provider's probe call errors	skip	pass

3. Why timestamps instead of block numbers

A fixed block count means very different amounts of time on each EVM chain. A time limit adapts by itself:

60s limit  ≈   5 blocks on Ethereum   (12s blocks)
           ≈  30 blocks on Base/OP    (2s blocks)
           ≈  80 blocks on BSC        (0.75s blocks)
           ≈ 240 blocks on Arbitrum   (0.25s blocks)

Both heads are fetched concurrently, so the time between the two calls isn't counted as lag on fast chains.

4. Cooldown escalation (primary stays broken)

fail ─▶ base ─probe ❌─▶ 2×base ─probe ❌─▶ 4×base ─probe ❌─▶ ... ─▶ 30m ─▶ 30m ─▶ ...
                                                                       (cap)
e.g. a 3m base:  3m → 6m → 12m → 24m → 30m → 30m ...
base depends on error type (1–5m); bases above the cap (e.g. 24h) are kept as-is
any successful request ─▶ strikes reset to 0

A dead primary costs at most about 2 lightweight calls every 30 minutes. With every provider healthy, the probe makes zero calls. Probe calls go through the same per-endpoint rate limiter as normal traffic.

5. Single provider

There's nowhere to fail over to, so emergency recovery puts the only provider back immediately. Cooldowns and the probe never take effect, and the indexer keeps retrying, paced by the rate limiter and the worker's retry logic. That's the same behavior as before this PR.

Config

nodes:
  - url: "https://primary-rpc.example.com"
  - url: "https://backup-rpc.example.com"
    backup: true   # only used while every non-backup node is unavailable

Notes

• EthereumAPI gains GetLatestBlockTimestamp. It decodes only timestamp from eth_getBlockByNumber("latest", false), because with false the node returns transactions as hash strings, which can't be decoded into Block.Transactions []Txn.

Test plan

• go test -race ./internal/rpc/ ./internal/worker/: primary preference, return to primary, falling back to the backup on retry, cooldown escalation and reset, probe fail/pass/lag, timer recovery for capability blacklists
• GetLatestBlockTimestamp unit tests covering a hash-only transaction list and a null block
• Live eth_getBlockByNumber("latest", false) against an Ethereum public RPC: transactions came back as hash strings and the latest block was 4s old
• Run the EVM indexer with a primary and a backup node and confirm failover and return to primary in the logs

[vietddude]

Reviewed this and put the suggested fixes in a follow-up PR against this branch: #128

Main points:

• BlacklistEscalating adds a strike per failed request, so concurrent failures from one blip jump straight to the 30m cap.
• With 1 primary + 1 backup, force-rotate on unclassified errors re-picks the same primary and never reaches the backup.
• During a total outage, emergency recovery plus escalation leaves every provider at the 30m cap after the network returns.
• runProbe applies its result even if the provider was recovered or re-blacklisted while the probe ran.
• switchToPrimary takes the write lock on every request while on a backup.

Left as is: a single Success() resetting strikes, since keeping them would hold the free primary out longer and send more traffic to the paid backup.

[vietddude]

Ran the last unchecked test-plan item ("Run the EVM indexer with a primary and a backup node and confirm failover and return to primary in the logs").

Setup. Built from the #128 branch (a166808, this PR plus the follow-up fixes), so this does not cover this PR's head on its own. ethereum_mainnet, regular worker only. Primary: a local proxy in front of a public Ethereum RPC that can be switched to return errors. Backup (backup: true): a second public RPC.

Scenario 1: primary returns HTTP 429

14:32:24 WRN Switching provider due to error provider=ethereum_mainnet-1 error_type=rate_limit blacklist_strikes=1 blacklist_duration=5m0s
14:32:29 INF Switching to provider provider=ethereum_mainnet-2 backup=true state=healthy
14:37:25 WRN Recovery probe failed, keeping provider blacklisted provider=ethereum_mainnet-1 blacklist_duration=10m0s
14:41:40      (primary restored)
14:47:26 INF Recovery probe passed, recovering provider provider=ethereum_mainnet-1
14:47:26 INF Returning to primary provider from_provider=ethereum_mainnet-2 to_provider=ethereum_mainnet-1

From 14:32 to 14:47 the indexer processed 78 blocks (26397384 to 26397461) with no gaps and no error-level log lines; /status reported healthy, pending_blocks: 0 afterwards.

Scenario 2: primary returns unclassified errors (HTTP 500)

This is the force-rotate path, which only reaches the backup with the #128 change.

14:24:37–47 WRN Unknown provider error type provider=ethereum_mainnet-1 (x3)
14:24:47 WRN Blacklisting force-rotated provider provider=ethereum_mainnet-1 blacklist_duration=1m0s
14:24:47 INF Switching to provider provider=ethereum_mainnet-2 backup=true state=healthy
14:25:50 WRN Recovery probe failed, keeping provider blacklisted blacklist_duration=2m0s
14:27:50 WRN Recovery probe failed, keeping provider blacklisted blacklist_duration=4m0s
14:31:33      (primary restored)
14:31:51 INF Recovery probe passed, recovering provider provider=ethereum_mainnet-1
14:31:53 INF Returning to primary provider from_provider=ethereum_mainnet-2 to_provider=ethereum_mainnet-1

Failover, cooldown escalation on failed probes, and return to primary all show up as expected in both scenarios.

[vietddude]

Approving. Suggested fixes are in #128 and the live failover run is in the comment above.