[matthewgrossman]

Summary

Stalled Nix store downloads held CI jobs until the job timeout, because Nix waits 300 s with no data before it retries a file. This PR makes Nix give up on a stalled download after 60 s. It also limits the first nix develop try in setup-nix to 15 minutes, so the existing retry runs in a fresh process.

Related Issue

Closes #5421

Changes

• setup-nix passes extra-conf to the Determinate installer:
  • stalled-download-timeout = 60 (default 300).
  • connect-timeout = 15. This is the current Determinate default. Setting it here keeps it from changing under us.
• The first nix develop try runs under a 900 s watchdog. If the watchdog fires, it stops Nix with SIGTERM and the existing single retry runs. The retry has no limit of its own; the job timeout still applies. A small bash watchdog is used because macOS runners have no timeout/gtimeout, and their bash is 3.2.
• The retry warning now includes the exit status, so you can tell a timeout (143) from a Nix failure.

Evidence and chosen values

What a stall looks like. In job 122735958171, NAR downloads from openshell.cachix.org retried exactly every 300 s ("Less than 1 bytes/sec transferred the last 300 seconds"). Two files stalled on every attempt until the job was cancelled at 45 minutes. The flake census for 2026-10-21 to 2026-11-04 found 4 such stalls: 2 on macOS, held for 45 min, and 2 on tmachine, held for 90 min. That is about 5.7 runner hours. It also found 5 fast Nix download failures, which the existing retry already handles.

Why 60 s only cuts true stalls. Nix implements stalled-download-timeout with curl's low-speed limit of 1 byte/s. A transfer is aborted only if it receives less than 1 byte/s for the whole window. Slow but working downloads are not affected. With 5 attempts, a stuck file now fails in about 5 minutes instead of 25.

Why 900 s for the first try. These are times for the whole Run ./.github/actions/setup-nix step (install, Cachix and shell). They come from about 3,270 successful jobs in the last 25 successful runs each of Branch Checks, Branch E2E Checks and Release Dev:

Platform	Jobs	Median	p90	p99	Max
Linux	2797	8 s	52 s	56 s	373 s
macOS	470	112 s	131 s	229 s	322 s

900 s is about 2.4 times the slowest successful run. If a slow but healthy cold build hits the limit, the paths it already fetched or built stay in the store, and the retry continues from there. So a false trigger costs only the in-progress work. On macOS (45-minute job timeout), a stall now leaves at least 30 minutes for the retry.

Settings must be in nix.conf. The Nix daemon performs substitutions, and it does not read a client's NIX_CONFIG. extra-conf writes the settings into /etc/nix/nix.custom.conf before the daemon starts.

Other Nix callers

Every other Nix call in CI runs after setup-nix on the same runner and uses the same daemon, so all of them get the new download settings:

• nix build X || nix build X in integration-runner.yml (tmachine), e2e-suites.yml, prepare-integration-inputs.yml, build-vm-driver.yml and check-protobuf-compatibility.
• shell: nix develop ... steps across the workflows and in build-rust-binary.

Both tmachine stalls in the census were in nix build .#tmachine || .... With the 60 s stall timeout, the first nix build now fails within minutes, so its || retry can run. Those call sites do not get the 15-minute wall-clock limit. That limit is only in setup-nix.

Testing

• Ran the step script under /bin/bash 3.2 (macOS) and bash 5.3 with -eo pipefail and a fake nix, with a 3 s limit. A passing first try exits at once. A failing first try retries. A hung first try is stopped at the limit and the retry succeeds. No sleep processes are left behind.
• shellcheck on the step script, and zizmor on the action: no findings.
• Linux and macOS lanes pass in CI. The macOS setup-nix log shows extra-conf applied.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)

[matthewgrossman]

/ok to test 39552a8

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.