Skip to content

acc: don't record quality monitor status and dashboard_id - #7495

Merged
denik merged 2 commits into
mainfrom
fix-qm-status-flaky
Nov 12, 2026
Merged

denik merged 2 commits into
mainfrom
fix-qm-status-flaky

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Project quality monitor GET responses and plan remote_state through a monitor_fields() allow-list in acceptance/bundle/resources/quality_monitors/script.prepare. This follows .agents/rules/testing.md and matches postgres_*/script.prepare. The allow-list leaves out status and dashboard_id. It applies to create/out.plan_noop.json, change_*/out.plan.json and change_table_name/out.get.json. The dashboard_id repl in test.toml had nothing left to match, so I removed it.

Why

These tests run bundle plan / quality-monitors get right after the monitor is created or recreated. When that happens, the backend sometimes still reports MONITOR_STATUS_PENDING and hasn't set dashboard_id yet. The golden then doesn't match, and the tests are flaky on AWS, Azure and GCP.

Decision: record less instead of polling until the monitor is ACTIVE. Both fields are output-only and change on the server's schedule. None of these tests is about them: they cover plan actions, requests and state, and those are all still recorded. Polling would make the tests slower and would depend on how long provisioning takes on the backend.

Dashboard:

error verbatim snippet
error/1405 "status": "MONITOR_STATUS_PENDING", (expected "status": "MONITOR_STATUS_ACTIVE",, "dashboard_id" missing) in create/out.plan_noop.json and change_table_name/out.get.json
error/2893 --- .../out.plan.json - "dashboard_id": "(redacted)", - "status": "MONITOR_STATUS_ACTIVE", + "status": "MONITOR_STATUS_PENDING",

Tests: TestAccept/bundle/resources/quality_monitors/create/DMS=, TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=, TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=true (also change_assets_dir and change_output_schema_name).

Tests

  • Local: go test ./acceptance -run TestAccept/bundle/resources/quality_monitors passes.
  • Cloud: all quality_monitors subtests pass on AWS (-count=1), both before and after the switch to the allow-list. Azure and GCP credentials were not available locally, so those clouds are left to CI.

This pull request and its description were written by Isaac.

Right after a monitor is created, the backend can still report
MONITOR_STATUS_PENDING without a dashboard_id, which made the
quality_monitors acceptance tests flaky on cloud.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 11, 2026
@github-actions github-actions Bot added DABs DABs related issues test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Commit: a9e4d5073 | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

8 tests: regression=0 unreleased=0 coverage=8 failing=0
test branch main (4668f26) latest (v1.26.0)
TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_table_name/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=true ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/create/DMS= ✅ ✅ ✅
TestAccept/bundle/resources/quality_monitors/create/DMS=true ✅ ✅ ✅

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: a9e4d50

Run: 43215830816

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 282 15 14:16
🔄​ aws-windows-latest-4core-16gb 1 1 283 13 9:18
🔄​ azure linux-2core-8gb 2 1 279 15 14:29
💚​ azure-windows-latest-4core-16gb 1 283 13 9:54
💚​ gcp linux-2core-8gb 1 282 15 11:15
💚​ gcp-windows-latest-4core-16gb 1 284 13 7:04
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalFileSync ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p
🔄​ TestSyncIncrementalSyncFileToPythonNotebook ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
🔄​ TestSyncNestedFolderSync ✅​p ✅​p 🔄​f ✅​p ✅​p ✅​p
Top 17 slowest tests (at least 2 minutes):
duration env testname
4:38 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:59 azure linux-2core-8gb TestAccept
3:53 aws linux-2core-8gb TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=
3:41 aws-windows-latest-4core-16gb TestAccept
3:39 azure-windows-latest-4core-16gb TestAccept
3:24 aws linux-2core-8gb TestAccept
3:18 gcp-windows-latest-4core-16gb TestAccept
3:12 azure-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=
3:11 gcp linux-2core-8gb TestAccept
2:42 aws-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true
2:30 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:25 aws linux-2core-8gb TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true
2:14 aws-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=
2:11 aws-windows-latest-4core-16gb TestAccept/bundle/resources/quality_monitors/change_output_schema_name/DMS=true
2:05 gcp-windows-latest-4core-16gb TestImportDir
2:02 azure linux-2core-8gb TestAccept/bundle/resources/quality_monitors/change_assets_dir/DMS=true
2:01 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions

@denik
denik marked this pull request as ready for review November 11, 2026 18:16
@denik
denik requested review from a team as code owners November 11, 2026 18:16
@denik
denik enabled auto-merge November 11, 2026 18:16
@andrewnester
andrewnester self-requested a review November 12, 2026 08:58

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integration report for this commit still marks change_table_name flaky on aws-windows (both variants) and azure-windows DMS=true. Was that attempt the same status/dashboard_id diff, or a different cause?

@denik
denik disabled auto-merge November 12, 2026 12:05
…-list

Replace the jq del() deny-list with an allow-list helper in
quality_monitors/script.prepare, as .agents/rules/testing.md asks, and reuse
it for remote_state in the plan goldens.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

AI agent (harness_auto, opus5.5, isaac_claude): Re the review question ("Was that attempt the same status/dashboard_id diff, or a different cause?"): it was a different cause, unrelated to this change. In all three flaky attempts on that run (aws-windows DMS= and DMS=true, azure-windows DMS=true), the first bundle deploy failed while uploading deployment state:

 Created quality_monitors.monitor1
+Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/Workspace%2FUsers%2F[USERNAME]%2F.bundle%2Fquality-monitor-update-[UNIQUE_NAME]%2Fdefault%2Fstate%2Fmetadata.json?overwrite=true": request timed out after 1m30s of inactivity

On azure-windows the timed-out file was state/resources.json. That attempt never reached the plan/get steps, so it never compared status or dashboard_id. The retry passed in each case. It looks like a transient workspace-files timeout on Windows runners, and this test only hits it because it sits in that environment. I'm not addressing it in this PR.

@denik
denik requested a review from andrewnester November 12, 2026 12:12
@denik
denik enabled auto-merge November 12, 2026 12:26
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 44261c3 Nov 12, 2026
27 checks passed
@denik
denik deleted the fix-qm-status-flaky branch November 12, 2026 12:56
@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…n goldens (databricks#7764)

## Changes

New acceptance helper `acceptance/bin/plan_filter.py`: reads `bundle
plan -o json` on stdin, drops `remote_state` from every plan entry and
removes the deployment stamp (reusing `nostamp.render`). With a resource
key it prints just that entry, and fails if the key isn't in the plan.
So

```
$CLI bundle plan -o json | nostamp | jq '(.plan."resources.X.y" // .) | del(.remote_state)'
```

becomes

```
$CLI bundle plan -o json | plan_filter.py resources.X.y
```

It's covered by a selftest (`acceptance/selftest/plan_filter`) and
documented in `.agents/rules/testing.md`, along with a rule to use it
unless a test checks `remote_state` itself.

Applied to the cloud postgres plan tests:
`postgres_branches/update_protected`, `postgres_databases/update`,
`postgres_roles/update`,
`postgres_projects/add_default_endpoint_settings`, plus
`postgres_projects/update_display_name` and
`postgres_endpoints/update_autoscaling`, which already dropped
`remote_state` with jq. The golden updates only remove `remote_state`
blocks (3 insertions, all from trailing-comma changes, 193 deletions).

## Why

`remote_state` is the backend's GET response copied into the plan as is:
status, timestamps, sizes, UIDs, and any field the backend starts
returning. If a golden records it, the test fails whenever the backend
changes, even when the CLI's decision (`action`, `new_state`, `changes`)
is the same, and the failure shows up on every cloud at once.
`changes[*].remote` still records the remote value of each field the
plan compares.

Measured on main before this PR:
- 115 goldens record `remote_state`, 42 of them in `Cloud = true` tests.
- Over 30 days of main runs, these "Plan differs" errors had every diff
line inside `remote_state`, so this filter would have prevented them:
[error/2961](http://go/cli/dashboard/error/2961?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d)
(13 failures),
[error/2893](http://go/cli/dashboard/error/2893?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d)
(10),
[error/2964](http://go/cli/dashboard/error/2964?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3587](http://go/cli/dashboard/error/3587?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3526](http://go/cli/dashboard/error/3526?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d).
databricks#7745 and databricks#7495 fixed some of these test by test.
- Not prevented: the postgres GET change from last week
([error/3534](http://go/cli/dashboard/error/3534?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3535](http://go/cli/dashboard/error/3535?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3537](http://go/cli/dashboard/error/3537?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d)
and siblings, fixed in databricks#7596). Once the backend echoed spec fields back,
`changes` changed too (`skip`/`spec:input_only` entries disappeared and
`remote` values appeared), so those tests would have failed with or
without `remote_state`. That's a real change in the plan, and the
goldens should catch it.

| error | snippet (inside `remote_state`) |
|---|---|
|
[error/2961](http://go/cli/dashboard/error/2961?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d)
| `+ "compute_last_active_time": "[TIMESTAMP]",` |
|
[error/2893](http://go/cli/dashboard/error/2893?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d)
| `Unexpected output "status": "MONITOR_STATUS_PENDING",` |

## Decisions

- Tests opt in with a helper; the runner doesn't strip `remote_state` by
default. Some tests check `remote_state` on purpose (dashboards'
`serialized_dashboard`, grants' `__embed__` principals,
`apps/config-env-no-drift`, `quality_monitors`' allow-list), and a
filter hidden in the runner would make those goldens misleading.
- `changes[*].remote` is kept. It's the input to the CLI's drift
decision, and out-of-band tests depend on it.
- When the resource key is missing, `plan_filter.py` fails. The old `jq
'.plan."KEY" // .'` silently wrote the whole plan instead. None of the
converted goldens relied on that fallback, and a missing entry should
fail loudly.
- For `skip` entries (the `no_change` goldens), only `action` is left,
because a skip has no `changes`. That's intended: the plan's decision is
the assertion. If a broken read changed what the CLI sees, the action
would turn into update or recreate, and that still shows up.
- It's a separate script rather than a flag on `nostamp.py`. `nostamp`
is also used for states and request bodies, where `remote_state` doesn't
apply. `plan_filter.py` reuses `nostamp.render`, so stamp handling stays
in one place.
- Scope is the postgres family, which has churned the most. The other
cloud goldens that record `remote_state` (dashboards, grants,
quality_monitors) check it on purpose or already filter it. Local-only
tests can adopt the helper when they're next touched.

## Tests

- `selftest/plan_filter` (local): whole plan, one entry, missing entry,
non-plan and empty input.
- All 12 variants (DMS= and DMS=true) of the six postgres tests pass on
AWS (`testme-aws`). The last commit message requests the same AWS run in
CI.

## Dashboard


[error/3534](http://go/cli/dashboard/error/3534?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3535](http://go/cli/dashboard/error/3535?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3537](http://go/cli/dashboard/error/3537?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d):
these motivated the change. They were already fixed on main by databricks#7596,
and this PR wouldn't have prevented them (see above). It does prevent
the class above:
[error/2961](http://go/cli/dashboard/error/2961?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3587](http://go/cli/dashboard/error/3587?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/2964](http://go/cli/dashboard/error/2964?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d),
[error/3526](http://go/cli/dashboard/error/3526?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d).

```
Plan differs (out.plan.update.json): Unexpected output "is_protected": false, "no_expiry": true,
Plan differs (out.plan.restore.json): Unexpected output "compute_last_active_time": "[TIMESTAMP]",
```

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants