Repository navigation
acc: don't record quality monitor status and dashboard_id - #7495
Conversation
Right after a monitor is created, the backend can still report MONITOR_STATUS_PENDING without a dashboard_id, which made the quality_monitors acceptance tests flaky on cloud. Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 4 changed tests (limit=20, 0 not selected) 8 tests: regression=0 unreleased=0 coverage=8 failing=0
|
✅ Integration test reportCommit: a9e4d50
Top 17 slowest tests (at least 2 minutes):
|
andrewnester
left a comment
There was a problem hiding this comment.
The integration report for this commit still marks change_table_name flaky on aws-windows (both variants) and azure-windows DMS=true. Was that attempt the same status/dashboard_id diff, or a different cause?
…-list Replace the jq del() deny-list with an allow-list helper in quality_monitors/script.prepare, as .agents/rules/testing.md asks, and reuse it for remote_state in the plan goldens. Co-authored-by: Isaac <no-reply@databricks.com>
|
AI agent (harness_auto, opus5.5, isaac_claude): Re the review question ("Was that attempt the same status/dashboard_id diff, or a different cause?"): it was a different cause, unrelated to this change. In all three flaky attempts on that run (aws-windows On azure-windows the timed-out file was |
…n goldens (databricks#7764) ## Changes New acceptance helper `acceptance/bin/plan_filter.py`: reads `bundle plan -o json` on stdin, drops `remote_state` from every plan entry and removes the deployment stamp (reusing `nostamp.render`). With a resource key it prints just that entry, and fails if the key isn't in the plan. So ``` $CLI bundle plan -o json | nostamp | jq '(.plan."resources.X.y" // .) | del(.remote_state)' ``` becomes ``` $CLI bundle plan -o json | plan_filter.py resources.X.y ``` It's covered by a selftest (`acceptance/selftest/plan_filter`) and documented in `.agents/rules/testing.md`, along with a rule to use it unless a test checks `remote_state` itself. Applied to the cloud postgres plan tests: `postgres_branches/update_protected`, `postgres_databases/update`, `postgres_roles/update`, `postgres_projects/add_default_endpoint_settings`, plus `postgres_projects/update_display_name` and `postgres_endpoints/update_autoscaling`, which already dropped `remote_state` with jq. The golden updates only remove `remote_state` blocks (3 insertions, all from trailing-comma changes, 193 deletions). ## Why `remote_state` is the backend's GET response copied into the plan as is: status, timestamps, sizes, UIDs, and any field the backend starts returning. If a golden records it, the test fails whenever the backend changes, even when the CLI's decision (`action`, `new_state`, `changes`) is the same, and the failure shows up on every cloud at once. `changes[*].remote` still records the remote value of each field the plan compares. Measured on main before this PR: - 115 goldens record `remote_state`, 42 of them in `Cloud = true` tests. - Over 30 days of main runs, these "Plan differs" errors had every diff line inside `remote_state`, so this filter would have prevented them: [error/2961](http://go/cli/dashboard/error/2961?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d) (13 failures), [error/2893](http://go/cli/dashboard/error/2893?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d) (10), [error/2964](http://go/cli/dashboard/error/2964?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3587](http://go/cli/dashboard/error/3587?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3526](http://go/cli/dashboard/error/3526?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d). databricks#7745 and databricks#7495 fixed some of these test by test. - Not prevented: the postgres GET change from last week ([error/3534](http://go/cli/dashboard/error/3534?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3535](http://go/cli/dashboard/error/3535?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3537](http://go/cli/dashboard/error/3537?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d) and siblings, fixed in databricks#7596). Once the backend echoed spec fields back, `changes` changed too (`skip`/`spec:input_only` entries disappeared and `remote` values appeared), so those tests would have failed with or without `remote_state`. That's a real change in the plan, and the goldens should catch it. | error | snippet (inside `remote_state`) | |---|---| | [error/2961](http://go/cli/dashboard/error/2961?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d) | `+ "compute_last_active_time": "[TIMESTAMP]",` | | [error/2893](http://go/cli/dashboard/error/2893?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d) | `Unexpected output "status": "MONITOR_STATUS_PENDING",` | ## Decisions - Tests opt in with a helper; the runner doesn't strip `remote_state` by default. Some tests check `remote_state` on purpose (dashboards' `serialized_dashboard`, grants' `__embed__` principals, `apps/config-env-no-drift`, `quality_monitors`' allow-list), and a filter hidden in the runner would make those goldens misleading. - `changes[*].remote` is kept. It's the input to the CLI's drift decision, and out-of-band tests depend on it. - When the resource key is missing, `plan_filter.py` fails. The old `jq '.plan."KEY" // .'` silently wrote the whole plan instead. None of the converted goldens relied on that fallback, and a missing entry should fail loudly. - For `skip` entries (the `no_change` goldens), only `action` is left, because a skip has no `changes`. That's intended: the plan's decision is the assertion. If a broken read changed what the CLI sees, the action would turn into update or recreate, and that still shows up. - It's a separate script rather than a flag on `nostamp.py`. `nostamp` is also used for states and request bodies, where `remote_state` doesn't apply. `plan_filter.py` reuses `nostamp.render`, so stamp handling stays in one place. - Scope is the postgres family, which has churned the most. The other cloud goldens that record `remote_state` (dashboards, grants, quality_monitors) check it on purpose or already filter it. Local-only tests can adopt the helper when they're next touched. ## Tests - `selftest/plan_filter` (local): whole plan, one entry, missing entry, non-plan and empty input. - All 12 variants (DMS= and DMS=true) of the six postgres tests pass on AWS (`testme-aws`). The last commit message requests the same AWS run in CI. ## Dashboard [error/3534](http://go/cli/dashboard/error/3534?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3535](http://go/cli/dashboard/error/3535?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3537](http://go/cli/dashboard/error/3537?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d): these motivated the change. They were already fixed on main by databricks#7596, and this PR wouldn't have prevented them (see above). It does prevent the class above: [error/2961](http://go/cli/dashboard/error/2961?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3587](http://go/cli/dashboard/error/3587?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/2964](http://go/cli/dashboard/error/2964?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d), [error/3526](http://go/cli/dashboard/error/3526?main=1&pr=1&end=2026-11-18T11:00:00Z&last=14d). ``` Plan differs (out.plan.update.json): Unexpected output "is_protected": false, "no_expiry": true, Plan differs (out.plan.restore.json): Unexpected output "compute_last_active_time": "[TIMESTAMP]", ``` This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Changes
Project quality monitor GET responses and plan
remote_statethrough amonitor_fields()allow-list inacceptance/bundle/resources/quality_monitors/script.prepare. This follows.agents/rules/testing.mdand matchespostgres_*/script.prepare. The allow-list leaves outstatusanddashboard_id. It applies tocreate/out.plan_noop.json,change_*/out.plan.jsonandchange_table_name/out.get.json. Thedashboard_idrepl intest.tomlhad nothing left to match, so I removed it.Why
These tests run
bundle plan/quality-monitors getright after the monitor is created or recreated. When that happens, the backend sometimes still reportsMONITOR_STATUS_PENDINGand hasn't setdashboard_idyet. The golden then doesn't match, and the tests are flaky on AWS, Azure and GCP.Decision: record less instead of polling until the monitor is
ACTIVE. Both fields are output-only and change on the server's schedule. None of these tests is about them: they cover plan actions, requests and state, and those are all still recorded. Polling would make the tests slower and would depend on how long provisioning takes on the backend.Dashboard:
"status": "MONITOR_STATUS_PENDING",(expected"status": "MONITOR_STATUS_ACTIVE",,"dashboard_id"missing) increate/out.plan_noop.jsonandchange_table_name/out.get.json--- .../out.plan.json- "dashboard_id": "(redacted)",- "status": "MONITOR_STATUS_ACTIVE",+ "status": "MONITOR_STATUS_PENDING",Tests: TestAccept/bundle/resources/quality_monitors/create/DMS=, TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=, TestAccept/bundle/resources/quality_monitors/change_table_name/DMS=true (also
change_assets_dirandchange_output_schema_name).Tests
go test ./acceptance -run TestAccept/bundle/resources/quality_monitorspasses.quality_monitorssubtests pass on AWS (-count=1), both before and after the switch to the allow-list. Azure and GCP credentials were not available locally, so those clouds are left to CI.This pull request and its description were written by Isaac.