Repository navigation
Open
Labels
Description
User Story
I maintain OpenShell's end-to-end tests. Runner rows now run a prebuilt testkit program (--program, #5332), and more rows and provisioners will follow. The program path should give the same timeout, filter and diagnostics guarantees as the nextest path, and should be easy to reuse from provisioners other than tmachine.
Description
Follow-ups found while reviewing #5332. All of them are in tests/runner unless noted.
- Per-test timeout.
openshell-conformance rundefaults to 900 s per test. The nextest profile stopped a test after about 120 s (slow-timeout = { period = "30s", terminate-after = 4 }). The runner passes neither--timeoutnor--grace(execute.rs,Kind::Program), so one hung test can use much of the 45-minuteRUN_TIMEOUT. Pass an explicit timeout and grace for program rows. --filterhas two meanings. It is a nextest filterset without--programand a test-name substring with it (main.rs). Reject filterset syntax when--programis set, or use a separate flag.- Move invocation out of
run_tests. TheKindbranch inexecute.rshardcodes the program'sreadyandrun --junitarguments and its exit codes, inside the tmachine-specific code. Host provisioners (VM driver, macOS, Windows) would copy it. Build the argv and map the exit code in one place that provisioners share. Share the exit-code constants with testkit instead of repeating them (PROGRAM_TEST_FAILEDvsFAILED_EXIT). - Check the program before booting a VM.
bundle::programonly checks that the path is a file. A wrong-architecture or dynamically linked binary fails later as "wait for the gateway" with an exec-format error. Check the ELF architecture and that it is static. - Name the log for both kinds. The program's output is saved as
nextest.log. Use a neutral name such astests.log. - Use the prebuilt program in the other callers. Release Dev and release qualification still build the conformance suites in every row (
release-dev.yml,release-qualification.yml). Features and drivers rows still build per row (about 3 minutes each); that needs those families to ship atestkit::main()program.
Context
Measured on #5332: the five conformance rows dropped from 542-904 s to 278-700 s per job. The slowest row went from 904 s to 700 s.
Related: #5295 (per-family settings; these should reach program rows too), #5333 (diagnostics; the program writes junit.xml only after the last test, so a run-level timeout loses it).
Part of #5281.
Definition of Done
- Program rows pass an explicit per-test timeout and grace.
-
--filtercannot be misread across the two paths. - A provisioner other than tmachine can run a program without copying its argv or exit-code handling.
- A wrong-architecture or dynamic program fails before the VM boots.
- Release Dev and release qualification use the prebuilt program.
Activity