Repository navigation
acc: add plan_filter.py to keep backend-owned remote_state out of plan goldens - #7764
Merged
Merged
Conversation
…n goldens remote_state in a plan entry is the backend's GET payload as is, so goldens that record it fail whenever the backend adds a field or a computed status value changes, even when the plan's action and changes stay the same. plan_filter.py drops remote_state from every entry and the deployment stamp (as nostamp does), optionally printing one entry. Use it in the postgres update tests, replacing their per-test jq filters. aws: TestAccept/bundle/resources/postgres_branches/update_protected TestAccept/bundle/resources/postgres_databases/update TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_endpoints/update_autoscaling TestAccept/bundle/resources/postgres_projects/update_display_name TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 7 changed tests (limit=20, 0 not selected) 13 tests: regression=0 unreleased=0 coverage=13 failing=0
|
Collaborator
✅ Integration test reportCommit: 1a7af48
Top 50 slowest tests (at least 2 minutes):
|
…ceback aws: TestAccept/bundle/resources/postgres_branches/update_protected TestAccept/bundle/resources/postgres_databases/update TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_endpoints/update_autoscaling TestAccept/bundle/resources/postgres_projects/update_display_name TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings Co-authored-by: Isaac <no-reply@databricks.com>
denik
marked this pull request as ready for review
November 18, 2026 13:03
denik
requested review from
Sankalp-Mittal,
andrewnester,
anton-107 and
renaudhartert-db
and removed request for
anton-107 and
renaudhartert-db
November 18, 2026 13:03
andrewnester
approved these changes
Nov 18, 2026
Sankalp-Mittal
approved these changes
Nov 18, 2026
Sankalp-Mittal
left a comment
Contributor
There was a problem hiding this comment.
[AI Generated] Review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.
No blocking or major issues found.
… shape aws: TestAccept/bundle/resources/postgres_branches/update_protected TestAccept/bundle/resources/postgres_databases/update TestAccept/bundle/resources/postgres_roles/update TestAccept/bundle/resources/postgres_endpoints/update_autoscaling TestAccept/bundle/resources/postgres_projects/update_display_name TestAccept/bundle/resources/postgres_projects/add_default_endpoint_settings Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
❌ Integration test reportCommit: 4c87e6a
1348 interesting tests: 1335 MISS, 6 PANIC, 4 RECOVERED, 3 FAIL
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
New acceptance helper
acceptance/bin/plan_filter.py: readsbundle plan -o jsonon stdin, dropsremote_statefrom every plan entry and removes the deployment stamp (reusingnostamp.render). With a resource key it prints just that entry, and fails if the key isn't in the plan. Sobecomes
It's covered by a selftest (
acceptance/selftest/plan_filter) and documented in.agents/rules/testing.md, along with a rule to use it unless a test checksremote_stateitself.Applied to the cloud postgres plan tests:
postgres_branches/update_protected,postgres_databases/update,postgres_roles/update,postgres_projects/add_default_endpoint_settings, pluspostgres_projects/update_display_nameandpostgres_endpoints/update_autoscaling, which already droppedremote_statewith jq. The golden updates only removeremote_stateblocks (3 insertions, all from trailing-comma changes, 193 deletions).Why
remote_stateis the backend's GET response copied into the plan as is: status, timestamps, sizes, UIDs, and any field the backend starts returning. If a golden records it, the test fails whenever the backend changes, even when the CLI's decision (action,new_state,changes) is the same, and the failure shows up on every cloud at once.changes[*].remotestill records the remote value of each field the plan compares.Measured on main before this PR:
remote_state, 42 of them inCloud = truetests.remote_state, so this filter would have prevented them: error/2961 (13 failures), error/2893 (10), error/2964, error/3587, error/3526. acc: ignore postgres project compute_last_active_time in update_display_name plans #7745 and acc: don't record quality monitor status and dashboard_id #7495 fixed some of these test by test.changeschanged too (skip/spec:input_onlyentries disappeared andremotevalues appeared), so those tests would have failed with or withoutremote_state. That's a real change in the plan, and the goldens should catch it.remote_state)+ "compute_last_active_time": "[TIMESTAMP]",Unexpected output "status": "MONITOR_STATUS_PENDING",Decisions
remote_stateby default. Some tests checkremote_stateon purpose (dashboards'serialized_dashboard, grants'__embed__principals,apps/config-env-no-drift,quality_monitors' allow-list), and a filter hidden in the runner would make those goldens misleading.changes[*].remoteis kept. It's the input to the CLI's drift decision, and out-of-band tests depend on it.plan_filter.pyfails. The oldjq '.plan."KEY" // .'silently wrote the whole plan instead. None of the converted goldens relied on that fallback, and a missing entry should fail loudly.skipentries (theno_changegoldens), onlyactionis left, because a skip has nochanges. That's intended: the plan's decision is the assertion. If a broken read changed what the CLI sees, the action would turn into update or recreate, and that still shows up.nostamp.py.nostampis also used for states and request bodies, whereremote_statedoesn't apply.plan_filter.pyreusesnostamp.render, so stamp handling stays in one place.remote_state(dashboards, grants, quality_monitors) check it on purpose or already filter it. Local-only tests can adopt the helper when they're next touched.Tests
selftest/plan_filter(local): whole plan, one entry, missing entry, non-plan and empty input.testme-aws). The last commit message requests the same AWS run in CI.Dashboard
error/3534, error/3535, error/3537: these motivated the change. They were already fixed on main by #7596, and this PR wouldn't have prevented them (see above). It does prevent the class above: error/2961, error/3587, error/2964, error/3526.
This pull request and its description were written by Isaac.