Skip to content

test(runner): follow-ups to running a prebuilt testkit program on runner rows #5416

Open

Description

@matthewgrossman

User Story

I maintain OpenShell's end-to-end tests. Runner rows now run a prebuilt testkit program (--program, #5332), and more rows and provisioners will follow. The program path should give the same timeout, filter and diagnostics guarantees as the nextest path, and should be easy to reuse from provisioners other than tmachine.

Description

Follow-ups found while reviewing #5332. All of them are in tests/runner unless noted.

  • Per-test timeout. openshell-conformance run defaults to 900 s per test. The nextest profile stopped a test after about 120 s (slow-timeout = { period = "30s", terminate-after = 4 }). The runner passes neither --timeout nor --grace (execute.rs, Kind::Program), so one hung test can use much of the 45-minute RUN_TIMEOUT. Pass an explicit timeout and grace for program rows.
  • --filter has two meanings. It is a nextest filterset without --program and a test-name substring with it (main.rs). Reject filterset syntax when --program is set, or use a separate flag.
  • Move invocation out of run_tests. The Kind branch in execute.rs hardcodes the program's ready and run --junit arguments and its exit codes, inside the tmachine-specific code. Host provisioners (VM driver, macOS, Windows) would copy it. Build the argv and map the exit code in one place that provisioners share. Share the exit-code constants with testkit instead of repeating them (PROGRAM_TEST_FAILED vs FAILED_EXIT).
  • Check the program before booting a VM. bundle::program only checks that the path is a file. A wrong-architecture or dynamically linked binary fails later as "wait for the gateway" with an exec-format error. Check the ELF architecture and that it is static.
  • Name the log for both kinds. The program's output is saved as nextest.log. Use a neutral name such as tests.log.
  • Use the prebuilt program in the other callers. Release Dev and release qualification still build the conformance suites in every row (release-dev.yml, release-qualification.yml). Features and drivers rows still build per row (about 3 minutes each); that needs those families to ship a testkit::main() program.

Context

Measured on #5332: the five conformance rows dropped from 542-904 s to 278-700 s per job. The slowest row went from 904 s to 700 s.

Related: #5295 (per-family settings; these should reach program rows too), #5333 (diagnostics; the program writes junit.xml only after the last test, so a run-level timeout loses it).

Part of #5281.

Definition of Done

  • Program rows pass an explicit per-test timeout and grace.
  • --filter cannot be misread across the two paths.
  • A provisioner other than tmachine can run a program without copying its argv or exit-code handling.
  • A wrong-architecture or dynamic program fails before the VM boots.
  • Release Dev and release qualification use the prebuilt program.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions