Skip to content

Stop integration go test before the CI runner's 6h job limit - #7749

Merged
denik merged 1 commit into
mainfrom
denik/integration-timeout-under-job-limit
Nov 18, 2026
Merged

denik merged 1 commit into
mainfrom
denik/integration-timeout-under-job-limit

Conversation

@denik

@denik denik commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Changes

Lower the go test timeout of the integration and integration-short tasks from 12h to 5h20m.

Why

The integration jobs run on GitHub-hosted runners, which cancel a job after 6h. Since 2026-11-16, most main runs during working hours need more than 6h: every test is about 1.65x slower (median) under the current CI load, and some, such as bundle/resources/job_runs/failed_run, take about 45 minutes instead of 2. Off-peak runs still finish in about 3h10m to 3h50m.

The runner cancels the job about 350 minutes into the test step. With -timeout=12h the Go timeout never fires, so the cancelled job leaves:

  • a [package] failure with no output;
  • every unfinished test reported as FAIL ... (-1.00s) (about 350 tests per job);
  • no known-failures masking, because the test runner never gets to analyze the results.

At 5h20m, go test panics first. Its output lists the tests that were still running, and the reporting steps still have about 25 minutes before the job limit. This makes slow runs readable; it does not make them faster. CI load is outside this repo.

The 12h timeout came from #5658, when the goal was to stop go test from cutting runs short at 2h. Nothing on these runners can use more than 6h. The slowest off-peak run I found took 4h47m of test-step time, which fits under 5h20m.

Dashboard:

error key lines
error/1272 ERROR failed to interrupt 'go test': not supported by windows / FAIL Package acceptance (5h46m59.789s) / DONE 2985 tests, 1348 skipped, 353 failures in 20937.308s / ##[error]The operation was canceled.

Tests

No behavior change outside CI. I checked that task --list-all still parses the Taskfile. The next long main run shows the effect.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: da0d0f0

Run: 44245242686

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 3:33
💚​ aws-windows-latest-4core-16gb 1 272 13 4:04
💚​ azure linux-2core-8gb 1 269 15 3:43
💚​ azure-windows-latest-4core-16gb 1 271 13 4:25
💚​ gcp linux-2core-8gb 1 270 15 3:52
💚​ gcp-windows-latest-4core-16gb 1 272 13 3:13
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:20 azure-windows-latest-4core-16gb TestAccept
3:58 aws-windows-latest-4core-16gb TestAccept
3:08 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 18, 2026 06:02
@denik
denik requested a review from a team as a code owner November 18, 2026 06:02
@denik
denik requested review from Sankalp-Mittal, andrewnester and rugpanov and removed request for rugpanov November 18, 2026 06:02

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@denik
denik added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit f1ca859 Nov 18, 2026
42 checks passed
@denik
denik deleted the denik/integration-timeout-under-job-limit branch November 18, 2026 11:26
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants