Skip to content

test(testkit): fail stalled steps before the runner stops the test - #5455

Open
elezar wants to merge 1 commit into
test/5416-program-row-timeout/elezarfrom
test/5333-surface-test-stalls/elezar
Open

elezar wants to merge 1 commit into
test/5416-program-row-timeout/elezarfrom
test/5333-surface-test-stalls/elezar

Conversation

@elezar

@elezar elezar commented Nov 10, 2026

Copy link
Copy Markdown
Member

Stacked on #5450. Review that PR first; this diff is against its branch.

Summary

A hung CLI call in a conformance lane showed only nextest's generic TIMEOUT, with no command name or output, and the gateway log tail began after the stalled test.

  • Time budget. The runner exports the per-test limit as OPENSHELL_TEST_TIMEOUT_SECS, from the same constant as the nextest profile and the program's --timeout. testkit caps its CLI, sandbox exec, sandbox wait and poll timeouts to the time left minus 5 s, so a stuck step fails with its own error. Cleanup keeps its own timeouts. Without the variable, nothing is capped.
  • CLI output. A command that times out reports its command name and the output it printed so far. A command that takes over 10 s, exits non-zero, or is still running (every 30 s) is reported on stderr. Fast successful commands print nothing.
  • Gateway logs. The k3s diagnostics collect gateway logs from the run's start time (--since-time, written by the runner in the guest), with a 30 minute fallback, keeping the first 64 KiB and last 512 KiB.

Related Issue

Part of #5333. Not covered here, so the issue stays open:

Testing

  • cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test pass for tests/testkit and tests/runner.
  • New unit tests cover the budget cap and CLI timeout reporting.
  • collect-k3s.sh was only syntax-checked (bash -n), and no e2e lane was run.

A hung CLI call only showed nextest's generic timeout, with no command name or output, and the gateway log tail started after the stalled test.

- The runner exports the per-test limit as OPENSHELL_TEST_TIMEOUT_SECS. testkit caps its CLI, exec, wait and poll timeouts to the time left, so a stuck step fails with its own error. Cleanup keeps its own timeouts.
- A CLI command that times out reports its output so far. A command that is slow, exits non-zero, or is still running is reported while the test runs.
- The k3s diagnostics collect gateway logs from the run's start time, with a 30 minute fallback, instead of the last 300 lines.

Part of #5333.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant