Required Go tests exhaust hosted storage after successful cleanup

🤖 Generated by the Agentic Engineer

Description

Required Go validation failed for KSail at 42d3bba2f3d66c9e4d852d9bbea810b62433391d after its normal cleanup succeeded. The linker and test fixtures report disk quota exhaustion and no space left. This blocks protected delivery even though the separate Home Isolation Guard passed at the same head.

Reproduction evidence

Required Test job 125662151650, run attempt 1, executed the shared workflow at 15e2fb1af4b5525ff96202661ca4b87d6affade0 with Go 1.27.3 and go test ./....
Cleanup was enabled and raised measured root availability from 18.65 to 38.61 GiB. The exact module-cache restore was 2,191,583,898 compressed bytes; compilation cache missed.
The first linker/write failures appear around 6m36s into testing. Disk exhaustion also prevented fixture copies and compilation. The log does not establish the responsible filesystem, quota/inode limit, expanded cache size or peak footprint.
The completed Reduce repeated disk-cleanup work in Go workflow fixtures #527 evaluation used an older KSail revision, Go 1.26.8 and no restored caches. Its outcomes do not establish capacity for this workload. Existing measurement is disabled in this required run and samples only the workspace filesystem.

Expected behavior

The unchanged complete test scope finishes successfully within a measured storage envelope. Unreadable storage observations remain unknown and cannot justify a cleanup bypass or success claim.

Acceptance criteria

Capture one bounded native observation of the current failing source/toolchain/cache envelope, including post-setup and post-restore component sizes, filesystem identity and available bytes/inodes for workspace, temporary storage and both Go caches. Record readable quota evidence or explicitly retain that limitation. Preserve failure-time and sampled observations without publishing machine paths or raw private output.
Use the observations to implement a root-cause capacity correction. Preserve the existing test command's coverage, security checks, cleanup protection and fail-closed behavior; do not rely on retrying until a different runner happens to pass.
Exercise refusal/error controls and verify actual required Go tests at the corrected workflow/source, retaining the original failure and complete run, attempt, cache and toolchain identities. Compare execution and feedback times without treating summed parallel intervals as elapsed time.
Document the resulting capacity requirement, limitations and native post-merge consumer readback. Keep KSail #8129's separate original home-isolation acceptance intact.

Priority rationale: confirmed required-CI failure blocks started delivery. Initial effort: Medium. Part of #287.

🔗 Related PRs

#789 - perf(ci): compare Go runner capacity policies [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

🤖 Generated by the Agentic Engineer

An existing green run supplies a useful comparison:

Observation	Failed Test 125662151650	Green Test 125665864734
Runner image	ubuntu-26.04 / 20261018.50.1	ubuntu-26.04 / 20261025.179.1
Root available before cleanup	18.650 GiB	87.350 GiB
Root available after cleanup	38.609 GiB	110.826 GiB
Compilation and schema cache restores	Misses	Misses
Go and command	1.27.3 / complete unit suite	Same

Both restored the same exact module-cache key and report 2,191,583,898 compressed bytes. Current cache metadata identifies one matching main-branch entry, ID 9907466533, with that size. No archive content digest was recorded in the original jobs. Both used the same shared cleanup at 15e2fb1af4b5525ff96202661ca4b87d6affade0.

The actual tested sources were synthetic merges bab0963e3c16b91e8a236e9679a29ffcde276b8f and 12e0e340b1c74760d3f8dfe4e031d3a1fdf7dc37, both based on 620cdc63eae95e0145eaebe38d4ea6cd75c3cf55. Their go.sum bytes match; go.mod only moves two unchanged selected module versions from indirect to direct requirements. The probe in devantler-tech/ksail#8153 is corrected to use the actual failed synthetic merge.

This establishes a 72.217 GiB post-cleanup availability difference, not its cause. It does not support claiming an OS-major migration or changing to Ubuntu 24.04. The native probe will retain filesystem/peak-use limitations instead of treating the passing run as a universal capacity guarantee.

Failed run, green run.

🤖 Generated by the Agentic Engineer

A further native Build failure now preserves the disk-exhaustion symptom: job 125779701708, associated with KSail PR #8147 candidate eba4b382f013558146c80d77c8a39d1ea57536d9, executes shared validation at 628106a325f4acbe642702dfb302d23624351724 with Go 1.27.3. At 07:21:20 UTC, the external linker reports final link failed: No space left on device; the build remains failed.

This is a build observation, not a new full-suite or home-isolation result. The retained log does not establish the exhausted filesystem, quota or peak component footprint. It supports continuing this existing capacity investigation; it does not justify changing the runner image, retrying until a larger runner appears, or weakening the command. The bounded measurement in devantler-tech/ksail#8154 is still pending delivery. The earlier adverse receipts remain valid.

🤖 Generated by the Agentic Engineer

The historical replay in KSail#8153 cannot continue with its original input: the exact module archive is no longer available, and no original archive-content digest was retained. Its first native observation refused preparation before executing the test workload. KSail#8161 retires that unusable probe; it does not establish a capacity requirement or resolve this issue.

The next implementation will extend the existing default-off collector on the actual required Go Test path. It will observe ordinary current-source execution, retain source/run/attempt/toolchain and cache identities, distinguish filesystem bytes and inodes from component sizes, and preserve explicitly unknown quota limits. It will not reconstruct the old archive, change the tested command, move caches, weaken cleanup or substitute a passing retry for the original failure.

The source owner is the shared Go validation workflow, currently observed natively at aa06994a86a2dea6c4c90e1f482a9d8816359c94. A separately called KSail wrapper is not the required job that failed. Both disabled and enabled behavior, failure/cancellation preservation, incomplete observations and generated-workflow parity must pass before publication. Consumer adoption and a real required-run observation remain subsequent gates. The original adverse evidence and acceptance criteria stay intact.

chore: agent-claim #838 nonce=508b785dabc98e2da6f1ebab9490df11

chore: agent-claim #838 nonce=ca76d0ba2c576ea7fdab6a47685eaf38

🤖 Generated by the Agentic Engineer

Another current-source required Test failure: KSail #7827, job 126228613082, source ec59b8c8f9b7d36be0ffd4ed06e4b97a50b9f8dc, Go 1.27.3, shared workflow c496acfebce54b27513c01d22be1c19d5c97ec32. Normal cleanup succeeded and reported 40,469,072 KiB available afterward. The restored module archive reports 2,191,199,546 compressed bytes; this is not represented as the historical archive. The first linker quota failure was at 22:16:28 UTC, about six minutes into go test ./..., followed by more linker and temporary-file write failures. Both observation switches were false, so this does not identify the exhausted filesystem/quota or peak footprint.

Two harness assertions also failed in that run. Both pass in a fresh local focused run, but that does not establish their hosted cause or erase the adverse receipt.

KSail #7827 now includes signed source 16f6c85e0b3f025138a411d594b7e270a179fced, enabling the existing bounded ordinary-Test observer with its closed-schema consumer declaration. Its real immutable reader was checked disabled/absent and enabled; the workload, cache handling, cleanup, deadlines and required gates remain intact. The next native observation and an evidence-based capacity correction are still outstanding. No replay of the unavailable historical archive, larger-runner substitution or recovery claim.

🤖 Generated by the Agentic Engineer

A fresh passing comparison is now available from KSail #8163 Test job 126237968828, alongside the failed #7827 Test job 126228613082.

Observation	Failed Test	Passing Test
Ubuntu 26.04 image	`20261018.50.1`	`20261025.179.1`
Available after normal cleanup	40,469,072 KiB	116,199,228 KiB
Restored module archive	2,191,199,546 compressed bytes	same reported size and matched key
Compilation cache	miss	miss
Go / command / shared workflow	1.27.3 / `go test ./...` / `c496acfe`	same

I read the actual tested synthetic merges from both checkout logs: c06b7349aabf732f53597f147c367dbe34a32819 and e5f3e79757f1f2a29f0f94b67e757e90e36a439b. Their native contents metadata confirms identical go.mod blob 8a0c631d5c12dbe3fbcc1f2dba7374476457ab74 and go.sum blob 1a9d7d5860e52a2aa5ab5ac7e1a06f35996423a0. Both logs name the same restored module-cache key ending 20bd79af987dd1c66891b844d9d6f6eaa5d2ee9153adb23189d0de96d7fdc0ee; no archive-content digest was recorded. The passing run executes 201 tested packages, with 22 packages reporting no tests.

These remain different PR sources and uncontrolled runner observations. The larger measured free-space envelope correlates with success; it does not establish why the runner layouts differ, which filesystem/quota was exhausted, peak component use or a capacity correction. Do not change OS versions, rerun until a larger runner appears, or erase the failing harness assertions from this comparison. The instrumented #7827 head 16f6c85e0b3f025138a411d594b7e270a179fced remains the next ordinary required-Test observation, not a claimed repair.

🤖 Generated by the Agentic Engineer

One more uncontrolled observation, this time on KSail main rather than a pull request: Validate Main Test job 126538243329 at 423fc542aae21d12d41eb3e20acae6874513dc2b.

Observation	This run
Ubuntu 26.04 image	`20261018.50.1` (same as the failed row above)
Available before / after normal cleanup	19,545,632 / 40,473,804 KiB
Shared workflow	`522038c41420f0c9508a586cb944eda995ad0068`
Go	1.27.3
First quota / no-space error	about 6m15s into testing (linker on `pkg/cli/clusterapi.test`)

It failed the same way as before: linker disk quota exceeded / No space left on device, then the fixture-write failures in internal/ciharness (20 --- FAIL lines, all of them writes that hit the quota). The earlier runs on main today (08:58Z, 07:48Z, 01:19Z) passed. I did not read which image they used.

So the old image with about 40 GB free now fails on both pull-request and main sources. That is still a correlation, not a cause. In line with the guidance above, I have not re-run it to fish for a larger runner.

fix(go): refuse a multi-line RUNNER_TEMP before writing GITHUB_ENV

🤖 Generated by the Agentic Engineer (interactive Claude Code session)

Root cause, from the failing main log (job 126538243329): every write that failed was under the system temporary directory. That covers the linker output, vet.cfg and importcfg in Go's work directory, and then the test fixtures. None was on the runner disk, which still had about 40 GiB free after cleanup. The error is disk quota exceeded, which points to a small, quota-limited tmpfs on image 20261018.50.1. An instrumented passing run on 20261026.180.1 found the temporary directory on the runner disk itself.

#871 routes GOTMPDIR to RUNNER_TEMP in the build, test and coverage jobs. It prints a before/after GO_TEMP_ROUTE capacity receipt and documents the capacity requirement. This issue stays open after #871 merges. Its last criteria need KSail to bump to the released workflow and run its required Test on both images, which is the native readback.

🤖 Generated with Claude Code

fix(go): route the go command's work directory to the runner disk (#871)

🤖 Generated by the Agentic Engineer

Post-merge verification, 8 November 2026. The fix works in KSail's required Go tests, so this issue can close.

What KSail runs now. KSail's ci.yaml calls validate-go-project.yaml at 15372ea4 (v20.1.10). That release came out after #871 merged (#871 at 11:09Z, v20.1.10 at 11:13Z on 7 November).

Native readback. The required 🧪 Test job passed on KSail main in the last three CI runs. Each one printed the GO_TEMP_ROUTE receipt and showed Go's work directory on the runner disk (ext4, about 111 GiB free):

Run	Head	Test job	Runner image
42421106430	`66a5697`	128549660028	ubuntu-26.04 `20261101.186.1`
42467829548	`c97b896`	128691525411	ubuntu-26.04 `20261101.186.1`
42510034346	`6ef410b`	128821063161	ubuntu-26.04 `20261101.186.1`

No repeat of the failure. I checked all 19 failed KSail ci.yaml runs since v20.1.10 was released, across all branches. None of them failed in the required 🧪 Test job.

Limitation. The quota-limited image 20261018.50.1 is no longer used. On the current image the system temporary directory is also on the runner disk, so these runs show that the route works, but they cannot reproduce the original quota case. If an image with a small temporary filesystem comes back, the GO_TEMP_ROUTE receipt will show it: system_temp_fstype will differ from go_temp_fstype.

KSail #8129's separate home-isolation acceptance is unchanged and still tracked there.

Navigation Menu

Repository navigation

Required Go tests exhaust hosted storage after successful cleanup #838

Description

Reproduction evidence

Expected behavior

Acceptance criteria

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Required Go tests exhaust hosted storage after successful cleanup #838

Description

Description

Reproduction evidence

Expected behavior

Acceptance criteria

Activity

coderabbitai commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 3, 2026

devantler commented on Nov 4, 2026

devantler commented on Nov 4, 2026

devantler commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions