Skip to content

acc: print the invariant tests' LOG.*.err files on failure - #7779

Merged
denik merged 1 commit into
mainfrom
acc-invariant-print-err
Nov 18, 2026
Merged

denik merged 1 commit into
mainfrom
acc-invariant-print-err

Conversation

@denik

@denik denik commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Changes

  • acceptance/bundle/invariant/test.toml no longer ignores *.err. The harness checks the ignore patterns before the LOG prefix, so the LOG.*.err files these scripts write (bundle plan -o json > ... 2>LOG.plan_initial.err, LOG.planjson.err, ...) were never printed. A failing plan then left only Exit code: 1 in the test log. Every .err file under acceptance/bundle/invariant is a LOG.* file, so no test gains an unexpected output file.

Why

Split out of #7770 at review. This makes the silent invariant failures behind error/1045 diagnosable. It does not fix them: the recent ones have different causes (a vector search endpoint plan, an app plan).

Tests

TestAccept/bundle/invariant passes locally.

Dashboard

error verbatim snippet
error/1045 expected: "INPUT_CONFIG_OK\n", actual : "\nExit code: 1\n"

This pull request and its description were written by Isaac.

The "*.err" ignore pattern is matched before the LOG prefix, so the stderr
of a failing `bundle plan` was never printed and the test log showed only
"Exit code: 1". Every .err file these tests write is a LOG.* file.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 18, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 7c53dd9

Run: 44351118740

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 10:18
💚​ aws-windows-latest-4core-16gb 1 274 17 9:37
💚​ azure linux-2core-8gb 1 271 19 12:07
💚​ azure-windows-latest-4core-16gb 1 273 17 7:44
💚​ gcp linux-2core-8gb 1 272 19 10:00
💚​ gcp-windows-latest-4core-16gb 1 274 17 7:36
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 13 slowest tests (at least 2 minutes):
duration env testname
3:14 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:09 gcp-windows-latest-4core-16gb TestSyncFullFileSync
2:59 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
2:50 gcp-windows-latest-4core-16gb TestLock
2:49 gcp-windows-latest-4core-16gb TestAccept
2:38 azure-windows-latest-4core-16gb TestAccept
2:37 azure linux-2core-8gb TestSyncIncrementalFileSync
2:36 aws-windows-latest-4core-16gb TestAccept
2:14 aws-windows-latest-4core-16gb TestImportDir
2:10 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:08 azure-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
2:05 azure linux-2core-8gb TestImportDirWithOverwriteFlag
2:05 gcp-windows-latest-4core-16gb TestImportDirWithOverwriteFlag

@denik
denik marked this pull request as ready for review November 18, 2026 17:53
@denik
denik requested review from a team as code owners November 18, 2026 17:53
@denik
denik requested review from Sankalp-Mittal, andrewnester, janniklasrose and rclarey and removed request for rclarey November 18, 2026 17:53

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

No issues found.

@denik
denik added this pull request to the merge queue Nov 18, 2026
Merged via the queue into main with commit e822544 Nov 18, 2026
32 checks passed
@denik
denik deleted the acc-invariant-print-err branch November 18, 2026 18:28
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

❌ Integration test report

Commit: e822544

Run: 44370163437

Env ❌​FAIL 🟨​KNOWN 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
❌​ aws linux-2core-8gb 2 1 14 1629 1386 311:40
❌​ aws-windows-latest-4core-16gb 2 1 14 1545 1414 313:22
❌​ azure linux-2core-8gb 8 1 14 1603 1394 316:49
❌​ azure-windows-latest-4core-16gb 6 1 14 1521 1422 310:21
🔄​ gcp linux-2core-8gb 3 14 1616 1392 307:13
💚​ gcp-windows-latest-4core-16gb 15 1534 1420 306:28
23 interesting tests: 14 RECOVERED, 8 FAIL, 1 KNOWN
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🔄​f 💚​R
❌​ TestAccept/bundle/deployment/bind/alert ✅​p ✅​p ❌​F ❌​F ✅​p ✅​p
❌​ TestAccept/bundle/deployment/bind/alert/DMS= ✅​p ✅​p ❌​F ❌​F ✅​p ✅​p
❌​ TestAccept/bundle/deployment/bind/alert/DMS=true ✅​p ✅​p ❌​F ❌​F ✅​p ✅​p
❌​ TestAccept/bundle/generate/alert ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/generate/alert/DMS=true ✅​p ✅​p ❌​F ✅​p ✅​p ✅​p
❌​ TestAccept/bundle/resources/apps/inline_config ❌​F ❌​F ❌​F ❌​F 🔄​f ✅​p
❌​ TestAccept/bundle/resources/apps/inline_config/DMS= ✅​p ❌​F ❌​F ❌​F 🔄​f ✅​p
❌​ TestAccept/bundle/resources/apps/inline_config/DMS=true ❌​F ✅​p ❌​F ❌​F ✅​p ✅​p
💚​ TestAccept/bundle/resources/apps/lifecycle-started 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/apps/lifecycle-started/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/basic/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS= 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/postgres_synced_tables/recreate/DMS=true 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 50 slowest tests (at least 2 minutes):
duration env testname
11:48 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
11:09 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:54 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:33 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
10:21 azure-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
10:18 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:37 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:35 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
9:31 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:14 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
9:13 aws linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
9:11 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
8:51 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:44 gcp linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=
8:36 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started/DMS=true
8:12 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=
8:06 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
8:05 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:55 azure linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/data_security_mode/DMS=true
7:49 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:40 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:39 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
7:39 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
7:36 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=
7:35 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
7:34 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=true
7:33 aws linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
7:30 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=true
7:28 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
7:24 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:05 gcp linux-2core-8gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=1
7:05 gcp-windows-latest-4core-16gb TestAccept/bundle/quiet-levels/DMS=
7:03 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=
7:01 aws-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-toggle/DMS=
6:59 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
6:51 aws linux-2core-8gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:46 gcp linux-2core-8gb TestAccept/bundle/resources/clusters/deploy/local_ssd_count/DMS=true
6:45 azure-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started/DMS=true
6:43 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=
6:40 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_libraries.yml.tmpl/READPLAN=
6:22 aws linux-2core-8gb TestAccept/bundle/invariant/destroy_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:20 gcp linux-2core-8gb TestAccept/bundle/invariant/delete_idempotent/DMS=/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=
6:09 azure linux-2core-8gb TestAccept/pipelines/run/refresh-flags
6:09 azure linux-2core-8gb TestAccept/bundle/resources/apps/lifecycle-started-omitted/DMS=
6:07 gcp-windows-latest-4core-16gb TestAccept/bundle/quiet-levels/DMS=true
6:07 aws-windows-latest-4core-16gb TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=cluster_apply_policy_default_values.yml.tmpl/READPLAN=1
6:01 gcp-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true
5:59 gcp linux-2core-8gb TestAccept/bundle/templates/default-python/combinations/classic/DLT=yes/DMS=/NBOOK=no/PY=yes/READPLAN=1
5:58 gcp linux-2core-8gb TestAccept/bundle/config-remote-sync/multiple_resources/DMS=true
5:57 aws-windows-latest-4core-16gb TestAccept/bundle/resources/clusters/lifecycle-started-toggle/DMS=true

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants