Repository navigation
Wait for and print the results of databricks pipelines test - #7298
Conversation
2c84257 to
8542365
Compare
|
Commit: Regression test reportSelected 8 changed tests (limit=20, 0 not selected) 8 tests: regression=7 unreleased=0 coverage=1 failing=0
TestAccept/pipelines/test/arguments: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): TestAccept/pipelines/test/immutable: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): TestAccept/pipelines/test/pipeline-id: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): TestAccept/pipelines/test/results/failed: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): same output as main (865ee2a) TestAccept/pipelines/test/results/incomplete: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): same output as main (865ee2a) TestAccept/pipelines/test/results/passed: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): same output as main (865ee2a) TestAccept/pipelines/test/selection: branch ✅ | main (865ee2a) ❌ | latest (v1.26.0) ❌main (865ee2a): latest (v1.26.0): |
✅ Integration test reportCommit: b45969f
Top 3 slowest tests (at least 2 minutes):
|
8542365 to
cb0a526
Compare
025649f to
23e0bb0
Compare
| if incompleteReason == "" { | ||
| summary := run.summaries[0] | ||
| testsFailed := summary.Total == 0 || summary.Failed > 0 || summary.Errors > 0 | ||
| if !testsFailed && state == pipelines.UpdateInfoStateCompleted { |
There was a problem hiding this comment.
A COMPLETED update whose tests all passed returns nil here without checking run.fatalError, so an update that also reported a fatal error exits 0. Could we add run.fatalError == "" to this condition, with a TestTestRunError case for it?
Related: a run where every test is SKIPPED also returns nil here. That matches the design (an all-skipped run completes), but CI and agents will read it as green, so the help should say so explicitly.
There was a problem hiding this comment.
Done. testRunError also checks run.fatalError == "", with a test case. The help says an all-skipped run exits 0.
| UpdateId: tw.updateID, | ||
| }) | ||
| if err != nil { | ||
| return "", err |
There was a problem hiding this comment.
A GetUpdate error (and the same for the events read just below) ends the wait with the raw error and no --resume hint, even though the update keeps running. The SDK client already retries transient HTTP errors, so no extra retry logic is needed here. Could the error just include the same resume command as the timeout path?
There was a problem hiding this comment.
I'll do this in a follow-up.
| updateID := resumeUpdateID | ||
| if !resume { | ||
| req := newStartTestUpdateRequest(target.selections, pytestArgs) | ||
| updateID, err = startTestUpdate(ctx, target.w, target.pipelineID, req) |
There was a problem hiding this comment.
This POST uses the signal context, so a Ctrl+C during Start Update loses the update ID, and the update runs with no way to find it from this output. Could the POST use cmd.Context()?
Also on this line: when the service rejects the start because an update is already active, a short static hint after its message would give agents the next step, for example "Run 'databricks pipelines stop KEY' to cancel it, or wait for it to finish." There's no need to parse the ID out of the message.
There was a problem hiding this comment.
Done in #7426: Start Update uses cmd.Context(), tested by TestInterruptDuringStartUpdateKeepsTheUpdateID. The already-active hint: I'll do that in a follow-up.
| fmt.Fprintln(out) | ||
|
|
||
| if incompleteReason != "" { | ||
| fmt.Fprintf(out, "RUN INCOMPLETE: %s.\n", incompleteReason) |
There was a problem hiding this comment.
The most likely cause of INCOMPLETE is events still arriving after the terminal state, and the update can still be read later. Could we print the --resume command here too?
There was a problem hiding this comment.
I'll do this in a follow-up.
|
|
||
| // testVerdictErrorPrefix starts the message of the error by which the service reports that tests | ||
| // failed, such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed (1 passed, 0 skipped).". | ||
| const testVerdictErrorPrefix = "[PIPELINE_TEST_FAILED]" |
There was a problem hiding this comment.
Recognizing the service verdict by message prefix is fragile. The exit code doesn't depend on it, only the output does. The SDK's ErrorDetail has no error class yet, so a comment saying this is temporary until a class is exposed would help.
There was a problem hiding this comment.
Done in #7432. A comment says the message is matched because ErrorDetail has no error class yet.
| // update ended in another way or reports an error of its own. The service's own verdict, | ||
| // such as "[PIPELINE_TEST_FAILED] 1 of the pipeline's tests failed", is not. | ||
| ended := state == pipelines.UpdateInfoStateFailed || state == pipelines.UpdateInfoStateCompleted | ||
| if testsFailed && ended && run.fatalError == "" { |
There was a problem hiding this comment.
Every failure path exits 1. Test failures and zero-collected runs print nothing to stderr, while infra failures print Error:, so an agent tells "fix the code" from "retry" only from the output. pipelines run behaves the same, so distinct exit codes would be a CLI-wide decision. For this stack, could the help just say what exit 0 and 1 mean? Small related nit: with --junit-xml, the JUnit report: line prints after the summary, so tail -1 misses the summary; printing it before the summary would keep the summary last.
There was a problem hiding this comment.
Done. The help says what exit 0 and 1 mean. The JUnit report: line prints before the summary from #7434.
| summary := run.summaries[0] | ||
| switch summary.Total { | ||
| case 0: | ||
| fmt.Fprintln(out, "No tests were collected.") |
There was a problem hiding this comment.
Agents hit this often, and the message gives no hint. Naming what was run would make it self-correcting, e.g. No tests were collected from <testfile>. Check the test selection and any --pytest-arg filters. (I'd avoid restating pytest's naming rules here, since projects can change them in their pytest configuration.) But if we can give a bit better message that would be nice.
There was a problem hiding this comment.
I'll do this in a follow-up.
There was a problem hiding this comment.
Fixed in 86122f3. The message names the selection, or the pipeline's test sources, and points to the --pytest-arg filters when they were given. It does not restate pytest's naming rules.
| testCasePassed testCaseStatus = "PASSED" | ||
| testCaseFailed testCaseStatus = "FAILED" | ||
| testCaseSkipped testCaseStatus = "SKIPPED" | ||
| testCaseError testCaseStatus = "ERROR" |
There was a problem hiding this comment.
should this be errored? I see we do error and errored at some places would be good if we can do consistent.
There was a problem hiding this comment.
These follow the proto: the case status is ERROR and the summary field is errored. The summary line uses the status names (ERROR=).
| @@ -0,0 +1,799 @@ | |||
| # 200 cases emit 601 events, more than two pages of List Pipeline Events. | |||
There was a problem hiding this comment.
Non-blocking: instead of committing 800 lines of identical tests, could the script generate this file before pipelines deploy? The fake collects tests with a regex over the deployed files (libs/testserver/pipelines_testing.go:106), so this works the same and runs on bash 3.2 and Git Bash:
for i in $(seq 0 199); do printf 'def test_case_%03d():\n assert True\n\n' "$i"; done > tests/test_many.pyAlso worth checking how many cases you actually need. The CLI doesn't send max_results, so the fake pages at its default of 25 events (pipelines_testing.go:332), and the output shows 8 pages read. Fewer cases may cover the same paths. The page math reads better in the script than as a comment in a generated file.
There was a problem hiding this comment.
Done. The unit test TestReadTestEvents200Cases in #7432 replaces it. The reader sends max_results=250.
| assert.Equal(t, 400, response.StatusCode) | ||
| } | ||
|
|
||
| func TestPipelineStartUpdateRejectsInvalidTestOptions(t *testing.T) { |
There was a problem hiding this comment.
Non-blocking: this file tests the fake more deeply than the CLI contract needs, and a few could go:
TestPipelineStartUpdateRejectsInvalidTestOptionsandTestPipelineStartUpdateRejectsTestOptionsWithoutTestOnly, together with the fake'spipelineTestOptionsError. The CLI validates--pytest-argand makes selections absolute before any request, and it always sendstest_only: true, sopipelines testnever reaches this validation.TestPipelineFiniteUpdateIsActiveUntilItEnds,TestPipelineUpdateThatIsNeverPolledEndsWithItsFirstStateandTestPipelineStopHidesEventsOfStatesNotReached. They pin the fake's state timing, and what the CLI sees (409, CANCELED) is golden inrun-controlandsubmit-error.TestPipelineEventsPages. Paging is covered by the CLI's own reader test andmany-cases.
TestCollectPipelineTests, TestPipelineTestUpdate and TestPipelineTestUpdateConflictAndStop cover what the acceptance tests rely on. Also, line 242 repeats the assertion on line 241.
There was a problem hiding this comment.
Done. The stack no longer changes libs/testserver.
66e7f7e to
62049a1
Compare
23e0bb0 to
b45969f
Compare
42a9409 to
8ff858f
Compare
d0d7138 to
cdc73e0
Compare
8ff858f to
40a204e
Compare
cdc73e0 to
86122f3
Compare
40a204e to
f2268c2
Compare
86122f3 to
ae6fcdc
Compare
f2268c2 to
6c5c865
Compare
The command waits for the test-only update, prints each test case as it finishes, then the failure details and a summary line. It exits with code 0 only for a complete run where at least one test ran and none failed or errored, and a completed update that reports a fatal error does not pass. Test IDs are shown relative to the project folder. With the no-wait flag, the command returns after the update starts. The wait takes the steps of the bundle pipeline runner's wait loop. The acceptance tests stub the service per case, as the other pipelines tests do. Co-authored-by: Isaac <no-reply@databricks.com>
The message names the test selection, or the pipeline's test sources, and points to the --pytest-arg filters when they were given. It does not restate pytest's naming rules, because a project can change them. The changelog entry for the results output moves to this PR. Co-authored-by: Isaac <no-reply@databricks.com>
ae6fcdc to
cfe688b
Compare
The entry is added with the other pipeline testing entries in #7439. Co-authored-by: Isaac <no-reply@databricks.com>
❌ Integration test reportCommit: 4e503ea
132 interesting tests: 119 FAIL, 7 RECOVERED, 6 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [**stack/pipelines-test-read-events**](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] ← _this PR_ - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - Read every page of an update's test events with the SDK's `ListPipelineEventsRequest` and `buildPipelineEventFilter`. Existing code is not changed. - Collect one result per test case and check that the results are complete. - Not used by the command yet. - Start test-only updates with `w.Pipelines.StartUpdate`: the Go SDK now has `test_only` and `test_options`. The body has the same fields and values; only the key order inside `test_options` follows the SDK type (one golden file). - The List Pipeline Events request stays a direct call: the SDK's `PipelineEvent` has no `details` or `origin.test_case_id`, which the test results come from. - The short help says *Experimental*, like the long help and the gate. A selection outside the synced files, and a relative selection with `workspace.sync_mode: immutable`, now say which /Workspace path to pass. Tests: unit tests, including 200 cases over 3 pages; acceptance `pipelines/test/*`. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [**stack/pipelines-test-print-results**](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] ← _this PR_ - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - Print test results: `<i> of <n> <STATUS> <test ID> ....... [<STATUS> in <s>s]`, failure details, and a last line `Done. PASSED=… TOTAL=…`. - Not used by the command yet. Tests: unit tests. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
) ## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [**stack/pipelines-test-a4-run-control**](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] ← _this PR_ - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - `--timeout` and Ctrl+C stop the local wait and print the `--resume` command; the update keeps running. - `--resume <update-id>` reads an existing test update. - A Ctrl+C during Start Update still prints the update ID; a second Ctrl+C ends the process. - `--resume` reads `test_only` with `w.Pipelines.GetUpdate`: the Go SDK now has `UpdateInfo.test_only`. As before, only an explicit `test_only: false` is rejected; an update without the field is accepted. - A failed read of the update or its events ends with the `--resume` command, as a timeout does. An incomplete result prints the `--resume` command before the summary. - When Start Update says an update is already active, the error says to wait, to stop it (bundle pipelines), or to read it with `--resume`. - The changelog entry for `--resume` and `--timeout` is in this PR. Tests: unit tests; acceptance in `acceptance/pipelines/test/run-control/*`. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
) ## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [**stack/pipelines-test-pr3-junit**](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] ← _this PR_ - [stack/pipelines-test-junit-flag](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] --------- - Build a JUnit XML report from a test run: one entry per case, plus a failing entry for an incomplete run. - The file is replaced atomically. - Not used by the command yet. - Every failing entry that the command adds is named `pipelines test`; the update ID is in the entry's text. Tests: unit tests. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
) ## 🥞 Stacked PR - [stack/pipelines-test-pr1-command](databricks#7297) [[Files changed](https://github.com/databricks/cli/pull/7297/files)] - [stack/pipelines-test-bundle-key](databricks#7431) [[Files changed](https://github.com/databricks/cli/pull/7431/files)] - [stack/pipelines-test-read-events](databricks#7432) [[Files changed](https://github.com/databricks/cli/pull/7432/files)] - [stack/pipelines-test-print-results](databricks#7433) [[Files changed](https://github.com/databricks/cli/pull/7433/files)] - [stack/pipelines-test-pr2-output](databricks#7298) [[Files changed](https://github.com/databricks/cli/pull/7298/files)] - [stack/pipelines-test-a4-run-control](databricks#7426) [[Files changed](https://github.com/databricks/cli/pull/7426/files)] - [stack/pipelines-test-pr3-junit](databricks#7299) [[Files changed](https://github.com/databricks/cli/pull/7299/files)] - [**stack/pipelines-test-junit-flag**](databricks#7434) [[Files changed](https://github.com/databricks/cli/pull/7434/files)] ← _this PR_ --------- - Add `--junit-xml <path>`, which writes the report on every exit after the flags parse. - The `JUnit report:` line prints before the summary line. - The help has examples: all tests, one file, one test by ID, CI with `--junit-xml`, `--timeout` then `--resume`, and `--pipeline-id`. - The changelog entry for `--junit-xml` is in this PR. Tests: acceptance reports (`out.*.xml`) in `acceptance/pipelines/test/*`. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
🥞 Stacked PR
--no-waitreturns after the update starts.--pytest-argfilters when they were given.Tests: unit tests; acceptance in
acceptance/pipelines/test/results/*, with the service stubbed per case.This pull request and its description were written by Isaac.