feat: pos ci wider coverage by jhkimqd · Pull Request #254 · agglayer/e2e

jhkimqd · 2026-03-31T08:10:53Z

Description

Increase testing coverage to cover consensus correctness in heimdall-v2
Update bor-erigon exclusion pairs which are not compatible (bor-2.6.5 vs erigon-3.5.0)
Expanded PoS E2E test coverage — new test cases covering Heimdall consensus correctness (checkpoints, milestones, spans, liveness, node health), validator set safety, state sync ordering, Bor span/sprint boundary safety, clerk bridge sync, fork transition state consistency, cross-client receipt/state-root comparisons, resilience (reorg resistance, chain liveness under stress, parallel execution safety, RPC stability), gas metering across forks, precompile fork-transition safety, basefee boundary validation, and system contract fork safety.
Cross-client (Bor + Erigon) CI support — Added Erigon as a first-class CI participant with auto-detection of the latest Erigon release from GitHub, a new erigon-image workflow input, and cross-client gatekeeping to gracefully skip tests on unsupported version combinations.
Refactored version matrix strategy — Reworked the compatibility matrix from tracking N major.minor lines to a focused 3-entry Bor strategy (latest, second-latest, previous minor line) that targets the most production-relevant upgrade paths. Updated update-compat-versions.py accordingly.
Exclusion pairs with metadata — Excluded pairs in compat-versions.yml now carry reason and link fields; the CI summary table surfaces these so reviewers can see why a combination was skipped and trace it to the failing run. - Graceful handling of mixed-version/unactivated-fork networks — Tests detect version incompatibilities early and skip with clear annotations instead of failing, avoiding wasted CI time on known-broken combinations.
CI workflow improvements
Shared test helpers clean up

Testing

CI seems to be passing:

- cross-client state root oracle tests - golden state regression tests - sprint-boundary fork tests Signed-off-by: Ji Hwan <jkim@polygon.technology>

Signed-off-by: Ji Hwan <jkim@polygon.technology>

…devnet node-health.bats: - Use CometBFT RPC (L2_CL_RPC_URL) instead of REST API (L2_CL_API_URL) for /status endpoint, which returns the nested {result: {node_info, sync_info}} format the jq selectors expect - Fix jq catching_up extraction: `false // empty` evaluates to empty in jq because false is falsy; use `has()` + `tostring` instead - Remove double http:// prefix when kurtosis port print already returns a URL span-in-turn.bats: - Fix producer validator ID field name: .val_id not .id in selected_producers - Allow Bor block to be before latest span start (Heimdall prepares spans in advance), only fail when Bor block exceeds span end Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…r heimdall-v2 The /stake/validators endpoint returns "Not Implemented" on heimdall-v2. Replace it with a 3-tier fallback: /stake/validators-set (heimdall-v2), /stake/validators (heimdall-v1), and /bor/spans/latest (span producers). The existing base64 root hash handling and string-to-int field parsing were already correct and validated against a live kurtosis-pos devnet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add 7 Heimdall test steps (node-health, checkpoint-safety, span-milestone-safety, span-in-turn, bridge-sync, validator-set-safety, validator-state) to the pos-fork-transition job, and 4 resilience test files (blockstm, chain-liveness, consensus-finality, rpc-stability) to the pos-execution-specs job. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The /stake/validators REST endpoint returns "Not Implemented" on heimdall-v2. Switch _get_validators() to read from /bor/spans/latest -> span.validator_set.validators which provides val_id, signer, jailed, voting_power, and proposer_priority fields. Also fix CometBFT /validators call (height=0 is rejected), normalise double http:// in RPC URL from kurtosis, and update jq selectors to prefer voting_power over power and val_id over id. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Fix Heimdall span endpoint from /bor/span/latest (501) to /bor/spans/latest - Parse Heimdall v2 response under .span key instead of .result - Fix double http:// prefix in second Bor node discovery (kurtosis port print already returns full URL) - Fix teardown failure in rpc-node-stability: bare && short-circuit returned exit 1 when _test_tmpdir was unset Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Replace broken counter/burner contract bytecode with solc 0.8.24 output - Add --gas-limit to contract deployment cast calls (gas estimation fails) - Fix double http:// in cross-node RPC URL discovery (kurtosis port print already returns full URL) - Fund wallet B with 1 ETH in PIP-16 dependency test so B->C transfer succeeds regardless of A->B async timing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Remove double http:// prefix in L2_CL_RPC_URL (kurtosis port print already returns a full URL) - Fix REST /status jq path to use flat .latest_block_height instead of .result.sync_info.latest_block_height (Heimdall REST is not CometBFT) - Fix event record ID max comparison to use tonumber (IDs are strings, so lexicographic max gave wrong results e.g. "2" > "10") Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Signed-off-by: Ji Hwan <jkim@polygon.technology>

In bor-2.7.0-vs-bor-2.6.5 mixed-version networks, the chain stalls at block 1023 near a sprint/span boundary. This causes all transaction-sending tests to fail with 'replacement transaction underpriced' because cast send submissions pile up with the same nonce when no blocks are mined. Changes: - Add _fund_ephemeral helper to pos-setup.bash: creates ephemeral wallet and funds it, calling skip on chain stall errors - Add _send_or_skip helper for inline cast send with stall detection - Update 12 test files using cast send in setup() to use _fund_ephemeral - Update 14 fuzz tests creating wallets inline to use _fund_ephemeral - Update RPC conformance tests (7 cast sends) with inline stall handling - Update RPC concurrent tests (2 multi-wallet tests) with stall handling - Update setup_file() in gas-metering and parallel-fork-tests with stall detection flag (_CHAIN_STALLED) checked in per-test setup() - Fix basefee-fork-boundary-validation to filter out disabled forks (FORK_GIUGLIANO=999999999) when computing max fork block - Add stall handling to kzg, pip79, and contract-creation test bodies Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Signed-off-by: Ji Hwan <jkim@polygon.technology>

leovct · 2026-04-08T07:57:33Z

@claude review

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd added 8 commits March 31, 2026 10:33

feat: wider pos ci tests coverage

573f374

- cross-client state root oracle tests - golden state regression tests - sprint-boundary fork tests Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: tests + helper scripts for ci coverage

7086130

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: kurtosis url parsing

5fb8082

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: coinbase address != producer address

84b8332

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: heimdall-v2 correctness tests

15d9ffd

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: heimdall milestone check logic

2b32115

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: exclusion pairs should include bor-erigon

91bd126

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: erigon manual input cases handling

d49ca35

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd marked this pull request as draft March 31, 2026 08:10

fix: shellcheck + cleanup

d5fa65e

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd marked this pull request as ready for review March 31, 2026 08:17

jhkimqd and others added 19 commits March 31, 2026 17:33

feat: fast detection for version incompatibilities

088f14d

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: erigon gates

82b5dd5

Signed-off-by: Ji Hwan <jkim@polygon.technology>

chore: bump tests inven

2ccf7a7

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: add skips to expected failures

695f698

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: cross client gatekeeping

a24d1d3

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: heimdall correctness tests

a2098e1

Signed-off-by: Ji Hwan <jkim@polygon.technology>

chore: bump tests inven

bf1763e

Signed-off-by: Ji Hwan <jkim@polygon.technology>

chore: fix shellcheck

5553ca5

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: kurtosis topology cleanup in ci

5f73eaf

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: serialize job in ci for stability

60a2e25

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: bump foundry-toolchain version

6218e88

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: add skip to features not supported across versions

c157d97

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd added 8 commits April 1, 2026 14:13

feat: add skipped tests summary + compact kurtosis enclave

d5656a9

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: manual trigger behaviors fix

ca7ff4e

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: additional test cases coverage for bor + heimdall

77ba7d7

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: remove dupes + cleanup

4f3952e

Signed-off-by: Ji Hwan <jkim@polygon.technology>

chore: bump testsinven

3c7f594

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: add exclusion pairs in ci summary table

0f571df

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: reduce parallel jobs

79d4812

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: bump parallel jobs to 5

b4ad15d

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd requested a review from a team April 3, 2026 00:33

jhkimqd and others added 5 commits April 3, 2026 15:31

fix: pos e2e failing tests due to tests setup

77e1335

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: bats handling

0bcc3b6

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: skip span overlap tests in mixed version networks

e4d3dfa

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: ci handling for unactivated forks

0247281

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd marked this pull request as draft April 3, 2026 13:51

feat: archive node topology + tests fixes

1d3408d

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd marked this pull request as ready for review April 4, 2026 04:39

jhkimqd added 8 commits April 6, 2026 08:55

fix: ci checks

98e528c

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: update-tests-inventory script pipe

1b30621

Signed-off-by: Ji Hwan <jkim@polygon.technology>

chore: cleanup

d7e83cd

Signed-off-by: Ji Hwan <jkim@polygon.technology>

feat: update pos matrix

55e12af

Signed-off-by: Ji Hwan <jkim@polygon.technology>

chore: cleanup

32daa7c

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: use latest erigon for latest bor

ede627f

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: ci annotations fix + tests logic cleanup

6d30bd8

Signed-off-by: Ji Hwan <jkim@polygon.technology>

fix: skipping tests should be tested

08280ac

Signed-off-by: Ji Hwan <jkim@polygon.technology>

leovct approved these changes Apr 8, 2026

View reviewed changes

feat: exclude fork trans 2.7.1-beta+2.6.5

d7145c0

Signed-off-by: Ji Hwan <jkim@polygon.technology>

jhkimqd merged commit 39157bf into main Apr 8, 2026
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: pos ci wider coverage#254

feat: pos ci wider coverage#254
jhkimqd merged 51 commits intomainfrom
feat/pos-ci-wider-coverage

jhkimqd commented Mar 31, 2026 •

edited

Loading

Uh oh!

leovct commented Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jhkimqd commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

leovct commented Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jhkimqd commented Mar 31, 2026 •

edited

Loading