Skip to content

fix: clear stale apt lists before apt-get update to fix ARM64 build#54

Merged
Marketen merged 2 commits into
mainfrom
copilot/fix-docker-arm64-build-issues
Apr 8, 2026
Merged

fix: clear stale apt lists before apt-get update to fix ARM64 build#54
Marketen merged 2 commits into
mainfrom
copilot/fix-docker-arm64-build-issues

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 8, 2026

ARM64 (linux/arm64) builds were failing during apt-get update with a GPG signature error against ports.ubuntu.com. Root cause: stale/corrupted APT metadata cached inside the docker buildx builder instance.

Changes

  • beacon-chain/Dockerfile and validator/Dockerfile: prepend rm -rf /var/lib/apt/lists/* to the RUN step that runs apt-get update, ensuring package lists are always fetched fresh regardless of builder cache state.
RUN rm -rf /var/lib/apt/lists/* && \
    apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh
Original prompt

Problem

The ARM64 (linux/arm64) Docker build is failing for both the beacon-chain and validator targets during the apt-get update step. The error is:

GPG error: http://ports.ubuntu.com/ubuntu-ports jammy InRelease: At least one invalid signature was encountered.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy InRelease' is not signed.

This occurs at line 19-20 in beacon-chain/Dockerfile and line 18-19 in validator/Dockerfile. The root cause is stale or corrupted APT repository metadata cached in the buildx builder instance, causing GPG signature verification to fail during cross-architecture builds.

Failing job: https://github.com/dappnode/DAppNodePackage-lighthouse-generic/actions/runs/24122984921/job/70406505104

Solution

Add rm -rf /var/lib/apt/lists/* before apt-get update in both Dockerfiles to clear any stale cached package metadata, making the build resilient to corrupted builder caches.

beacon-chain/Dockerfile — change the RUN command (line 19-20) from:

RUN apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh

to:

RUN rm -rf /var/lib/apt/lists/* && \
    apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh

validator/Dockerfile — change the RUN command (line 18-19) from:

RUN apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh

to:

RUN rm -rf /var/lib/apt/lists/* && \
    apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh

The following is the prior conversation context from the user's chat exploration (may be truncated):

User: Please find a solution for this failing job. Use the logs, job definition, and any referenced files where the failure occurred. Keep your response focused on the solution and include code suggestions when appropriate.

Assistant:

I'll start by fetching the job logs and the workflow definition to understand the failure.

Assistant: Let me look at the Dockerfiles involved in this build.

Assistant: ## Failure Analysis

The build is failing on the ARM64 (linux/arm64) Docker build for both the beacon-chain and validator targets. The error occurs at line 18-19 in both Dockerfiles during apt-get update:

GPG error: http://ports.ubuntu.com/ubuntu-ports jammy InRelease: At least one invalid signature was encountered.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy InRelease' is not signed.

This is a well-known issue with cross-architecture Docker builds using docker buildx with the docker-container driver. The GPG signature verification fails because of stale or corrupted apt cache/lists inside the buildx builder instance. The ports.ubuntu.com URLs confirm this is the ARM64 build specifically (x86 uses archive.ubuntu.com).

Solution

There are two approaches to fix this:

Option 1 (Recommended): Clean apt lists before updating

Add rm -rf /var/lib/apt/lists/* before apt-get update in both Dockerfiles to clear any stale cached package metadata:

beacon-chain/Dockerfile — change line 19:

RUN rm -rf /var/lib/apt/lists/* && \
    apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh

validator/Dockerfile — change line 18:

RUN rm -rf /var/lib/apt/lists/* && \
    apt-get update && apt-get --yes install curl && apt-get clean && \
    chmod +rx /usr/local/bin/entrypoint.sh /etc/profile.d/consensus_tools.sh

Option 2: Prune the buildx builder on the runner

If the issue is a corrupted buildx builder cache on the staking-test-hoodi runner, you can reset it by adding a step before the build in .github/workflows/release.yml:

- name: Prune buildx builder cache
  run: docker buildx prune --all --force
- name: Build and upload
  run: npx @dappnode/dappnodesdk build ...

Why this happens

The sigp/lighthouse base image uses Ubuntu Jammy (22.04). When building for linux/arm64 via buildx with a docker-container driver, the builder instance can cache stale or corrupted APT repository metadata. The GPG signatures then fail to verify because the cac...

This pull request was created from Copilot chat.

Copilot AI changed the title [WIP] Fix ARM64 Docker build issues for beacon-chain and validator fix: clear stale apt lists before apt-get update to fix ARM64 build Apr 8, 2026
Copilot AI requested a review from Marketen April 8, 2026 11:00
@Marketen Marketen marked this pull request as ready for review April 8, 2026 11:00
@Marketen Marketen merged commit 8642aee into main Apr 8, 2026
1 check passed
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Apr 8, 2026

✅ SYNC TEST REPORT - PASSED

This CI runs on a real DAppNode using the same RPC calls a user would make to configure a staker setup: execution client, consensus client, web3signer, MEV-boost, and relays. The self-hosted runner is pre-loaded with synced (or near-synced) execution client volumes, enabling fast sync and attestation tests with any client combination. Consensus clients use checkpoint sync, so no pre-synced volume is required.

Sync Test: Verifies that both execution and consensus clients reach a synced state. This test ensures the staker configuration is valid and clients can synchronize with the network.

📦 Clients Used

Component DNP Name DNP Version
Execution hoodi-nethermind.dnp.dappnode.eth 0.1.12
Consensus lighthouse-hoodi.dnp.dappnode.eth 0.1.6
Web3Signer web3signer-hoodi.dnp.dappnode.eth 0.1.3
MEV Boost mev-boost-hoodi.dnp.dappnode.eth 0.1.1
Network hoodi

🔖 Version Tracking

Execution Client Versions

Stage Version
Before Install Nethermind/v1.36.0+31cb81b7/linux-x64/dotnet10.0.1
After Install Nethermind/v1.36.0+31cb81b7/linux-x64/dotnet10.0.1

Consensus Client Versions

Stage Version
Before Install Lighthouse/v8.1.3-176cce5/x86_64-linux
After Install Lighthouse/v8.1.3-176cce5/x86_64-linux

⏱️ Timing Measurements

Environment Setup

Operation Duration Status
SetStakerConfig 46.933s
PackageInstall 19.22s

Test Execution

Operation Duration Status
WaitForBeaconchainSync 24.015s
WaitForExecutionSync 1ms

Total Duration: 2m5s

🔴 Container Error Logs

⚠️ Showing up to 3 error lines per container. See CI logs for complete details.

DAppNodePackage-validator.lighthouse-hoodi.dnp.dappnode.eth:

2026-04-08T11:12:30.001098839Z Apr 08 11:12:30.001 ERROR No synced beacon nodes                        total: 1, available: 1, synced: 0
2026-04-08T11:12:42.000635195Z Apr 08 11:12:42.000 ERROR No synced beacon nodes                        total: 1, available: 1, synced: 0

DAppNodePackage-beacon-chain.lighthouse-hoodi.dnp.dappnode.eth:

2026-04-08T11:12:30.896922390Z Apr 08 11:12:30.896 INFO  Could not UPnP map Discovery port             error: Gateway does not support UPnP
2026-04-08T11:12:44.607538098Z Apr 08 11:12:44.607 WARN  Error processing HTTP API request             elapsed_ms: 0.164788, status: 400 Bad Request, path: /eth/v1/validator/liveness/87064, method: POST

📋 View full CI logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants