Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .agents/rules/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,12 +180,15 @@ Available on `PATH` during test execution (from `acceptance/bin/`):
- `find.py REGEX [--expect N]`: find files matching regex in current directory. `--expect N` asserts an exact count.
- `diff.py DIR1 DIR2` or `diff.py FILE1 FILE2`: recursive diff with test replacements applied.
- `print_state.py [-t TARGET] [--backup]`: print deployment state (`resources.json`, or a terraform state left by an older CLI).
- `plan_filter.py [RESOURCE_KEY]`: filter `bundle plan -o json` on stdin for a golden: drops every entry's `remote_state` and the deployment stamp (as `nostamp` does); with RESOURCE_KEY, prints only that entry. Errors if the key is not in the plan.
- `edit_resource.py TYPE ID < script.py`: fetch resource by ID, execute Python on it (resource in `r`), then update it. TYPE is `jobs` or `pipelines`.
- `gron.py`: flatten JSON into greppable discrete assignments (simpler than `jq` for searching JSON).
- `jq` is also available for JSON processing.

**RULE: Prefer `gron.py | grep <field>` over inline `jq` paths for single-value lookups.** The gron output prints the JSON path inline, so the test log explains itself.

**RULE: Record whole plan entries with `plan_filter.py` unless the test asserts on `remote_state` itself.** A narrower projection such as `nostamp | jq '.plan[KEY] | {action, changes}'` is fine as is. `remote_state` is the backend's GET payload as is (status, timestamps, sizes, fields the backend adds), so a golden that records it breaks on backend changes that leave the plan's action and changes untouched, on every cloud at once. `changes[*].remote` still records the remote value of each field with a change entry.

**RULE: Don't pass `--keep` to `print_requests.py` if a later `print_requests.py` call follows.** The buffer accumulates, so the second call double-prints the earlier requests.

**RULE: Filter recorded requests with `print_requests.py`, never with a hand-written `jq 'select(...)' out.requests.txt` pipeline** — inline, or hidden inside a local `print_requests()` shell function. The helper already excludes GET, normalizes query strings, optionally sorts, and deletes `out.requests.txt` afterwards. A copy-pasted `jq` wrapper reimplements all of that, drifts from the canonical output format, and is the single most common acceptance-test anti-pattern in this repo. Wrapping `print_requests.py` *itself* in a local function is fine — e.g. to send each variant's output to its own `out.requests.<name>.json`. Reach for `jq` on `out.requests.txt` only for what `print_requests.py` genuinely can't express: filtering on request *body* content, or deleting noisy body fields (prefer a `Repls` entry in `test.toml` even then).
Expand Down
56 changes: 56 additions & 0 deletions acceptance/bin/plan_filter.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
#!/usr/bin/env python3
"""Read `bundle plan -o json` on stdin and write the part of it a golden should record.

A plan entry's remote_state is the backend's GET payload, copied as is: it carries
backend-computed fields (status, timestamps, sizes, UIDs) and every field the backend
starts or stops returning. Recording it makes a golden fail on backend changes that
leave the CLI's decision (action, new_state, changes) untouched, often on every cloud
and test at once. This drops remote_state from every entry; changes[*].remote still
Comment thread
denik marked this conversation as resolved.
Resolved
records the remote value of each field with a change entry.

Also removes the deployment stamp, as nostamp.py does, so it replaces
`nostamp | jq '.plan["KEY"] | del(.remote_state)'`.

Usage:
$CLI bundle plan -o json | plan_filter.py # whole plan
$CLI bundle plan -o json | plan_filter.py resources.jobs.foo # one entry

A test that asserts on remote_state itself should not use this.
"""

import argparse
import json
import sys

import nostamp


def filter_plan(data, key=None):
entries = data.get("plan") if isinstance(data, dict) else None
if not isinstance(entries, dict):
raise SystemExit("plan_filter.py: input is not a plan: no 'plan' object")
for entry in entries.values():
if isinstance(entry, dict):
entry.pop("remote_state", None)
if key is None:
return data
if key not in entries:
raise SystemExit(f"plan_filter.py: no plan entry {key!r}; entries: {sorted(entries)}")
return entries[key]


def main():
parser = argparse.ArgumentParser()
parser.add_argument("key", nargs="?", help="resource key of the one entry to print, e.g. resources.jobs.foo")
args = parser.parse_args()
text = sys.stdin.read()
try:
data = json.loads(text)
except json.JSONDecodeError as ex:
raise SystemExit(f"plan_filter.py: input is not JSON ({ex}): {text[:200]!r}") from None
data = filter_plan(data, args.key)
sys.stdout.write(nostamp.render(data, 2) + "\n")


if __name__ == "__main__":
main()
Original file line number Diff line number Diff line change
Expand Up @@ -5,25 +5,5 @@
"label": "${resources.postgres_projects.my_project.id}"
}
],
"action": "skip",
"remote_state": {
"branch_id": "dev-branch",
"create_time": "[TIMESTAMP]",
"is_protected": false,
"name": "[DEV_BRANCH_ID]",
"no_expiry": true,
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
"status": {
"branch_id": "dev-branch",
"current_state": "READY",
"default": false,
"is_protected": false,
"source_branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"source_branch_lsn": "[LSN]",
"source_branch_time": "[TIMESTAMP]",
"state_change_time": "[TIMESTAMP]"
},
"uid": "[BRANCH_UID]",
"update_time": "[TIMESTAMP]"
}
"action": "skip"
}
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,6 @@
"parent": "projects/test-pg-proj-[UNIQUE_NAME]"
}
},
"remote_state": {
"branch_id": "dev-branch",
"create_time": "[TIMESTAMP]",
"is_protected": true,
"name": "[DEV_BRANCH_ID]",
"no_expiry": true,
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
"status": {
"branch_id": "dev-branch",
"current_state": "READY",
"default": false,
"is_protected": true,
"source_branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"source_branch_lsn": "[LSN]",
"source_branch_time": "[TIMESTAMP]",
"state_change_time": "[TIMESTAMP]"
},
"uid": "[BRANCH_UID]",
"update_time": "[TIMESTAMP]"
},
"changes": {
"is_protected": {
"action": "update",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,26 +14,6 @@
"parent": "projects/test-pg-proj-[UNIQUE_NAME]"
}
},
"remote_state": {
"branch_id": "dev-branch",
"create_time": "[TIMESTAMP]",
"is_protected": false,
"name": "[DEV_BRANCH_ID]",
"no_expiry": true,
"parent": "projects/test-pg-proj-[UNIQUE_NAME]",
"status": {
"branch_id": "dev-branch",
"current_state": "READY",
"default": false,
"is_protected": false,
"source_branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"source_branch_lsn": "[LSN]",
"source_branch_time": "[TIMESTAMP]",
"state_change_time": "[TIMESTAMP]"
},
"uid": "[BRANCH_UID]",
"update_time": "[TIMESTAMP]"
},
"changes": {
"is_protected": {
"action": "update",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ print_requests() {
title "Initial deployment"
trace $CLI bundle validate
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_branches.dev_branch" // .' > out.plan.create.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_branches.dev_branch > out.plan.create.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -28,7 +28,7 @@ trace $CLI postgres get-branch "${branch_name}" | branch_fields

title "Verify no_change (no changes)"
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '(.plan."resources.postgres_branches.dev_branch" // .) | del(.remote_state.status.logical_size_bytes)' > out.plan.no_change.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_branches.dev_branch > out.plan.no_change.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -39,7 +39,7 @@ trace update_file.py databricks.yml "is_protected: false" "is_protected: true"
trace cat databricks.yml

trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '(.plan."resources.postgres_branches.dev_branch" // .) | del(.remote_state.status.logical_size_bytes)' > out.plan.update.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_branches.dev_branch > out.plan.update.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -51,7 +51,7 @@ trace $CLI postgres get-branch "${branch_name}" | branch_fields
title "Restore is_protected to false"
trace update_file.py databricks.yml "is_protected: true" "is_protected: false"
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '(.plan."resources.postgres_branches.dev_branch" // .) | del(.remote_state.status.logical_size_bytes)' > out.plan.restore.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_branches.dev_branch > out.plan.restore.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,5 @@
"label": "${resources.postgres_roles.owner.id}"
}
],
"action": "skip",
"remote_state": {
"create_time": "[TIMESTAMP]",
"database_id": "my-database",
"name": "[MY_DATABASE_ID]",
"parent": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main",
"postgres_database": "initial_db_name",
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner",
"status": {
"database_id": "my-database",
"postgres_database": "initial_db_name",
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner"
},
"update_time": "[TIMESTAMP]"
}
"action": "skip"
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,6 @@
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner"
}
},
"remote_state": {
"create_time": "[TIMESTAMP]",
"database_id": "my-database",
"name": "[MY_DATABASE_ID]",
"parent": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main",
"postgres_database": "renamed_db_name",
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner",
"status": {
"database_id": "my-database",
"postgres_database": "renamed_db_name",
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner"
},
"update_time": "[TIMESTAMP]"
},
"changes": {
"postgres_database": {
"action": "update",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,20 +18,6 @@
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner"
}
},
"remote_state": {
"create_time": "[TIMESTAMP]",
"database_id": "my-database",
"name": "[MY_DATABASE_ID]",
"parent": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main",
"postgres_database": "initial_db_name",
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner",
"status": {
"database_id": "my-database",
"postgres_database": "initial_db_name",
"role": "projects/test-pg-proj-[UNIQUE_NAME]/branches/main/roles/app-owner"
},
"update_time": "[TIMESTAMP]"
},
"changes": {
"postgres_database": {
"action": "update",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ print_requests() {
title "Initial deployment"
trace $CLI bundle validate
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_databases.my_database" // .' > out.plan.create.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_databases.my_database > out.plan.create.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -29,7 +29,7 @@ trace $CLI postgres get-database "${database_name}" | database_fields

title "Verify no changes"
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_databases.my_database" // .' > out.plan.no_change.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_databases.my_database > out.plan.no_change.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -40,7 +40,7 @@ trace update_file.py databricks.yml "postgres_database: initial_db_name" "postgr
trace cat databricks.yml

trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_databases.my_database" // .' > out.plan.update.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_databases.my_database > out.plan.update.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -52,7 +52,7 @@ trace $CLI postgres get-database "${database_name}" | database_fields
title "Restore postgres_database to original value"
trace update_file.py databricks.yml "postgres_database: renamed_db_name" "postgres_database: initial_db_name"
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_databases.my_database" // .' > out.plan.restore.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_databases.my_database > out.plan.restore.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ print_requests() {
title "Initial deployment"
trace $CLI bundle validate
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_endpoints.my_endpoint" // . | del(.remote_state)' > out.plan.create.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_endpoints.my_endpoint > out.plan.create.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -29,7 +29,7 @@ trace $CLI postgres get-endpoint "${endpoint_name}" | endpoint_fields

title "Verify no changes"
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_endpoints.my_endpoint" // . | del(.remote_state)' > out.plan.no_change.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_endpoints.my_endpoint > out.plan.no_change.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -40,7 +40,7 @@ trace update_file.py databricks.yml "autoscaling_limit_max_cu: 8" "autoscaling_l
trace cat databricks.yml

trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_endpoints.my_endpoint" // . | del(.remote_state)' > out.plan.update.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_endpoints.my_endpoint > out.plan.update.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand All @@ -52,7 +52,7 @@ trace $CLI postgres get-endpoint "${endpoint_name}" | endpoint_fields
title "Restore endpoint autoscaling to original value"
trace update_file.py databricks.yml "autoscaling_limit_max_cu: 4" "autoscaling_limit_max_cu: 8"
trace $CLI bundle plan
trace $CLI bundle plan -o json | nostamp | jq '.plan."resources.postgres_endpoints.my_endpoint" // . | del(.remote_state)' > out.plan.restore.json
trace $CLI bundle plan -o json | plan_filter.py resources.postgres_endpoints.my_endpoint > out.plan.restore.json
rm -f "$OUT_REQUESTS"
trace $CLI bundle deploy

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"plan_version": [PLAN_VERSION],
"cli_version": "[CLI_VERSION]",
"lineage": "[UUID][0]",
"lineage": "[UUID]",
"serial": 1,
"plan": {
"resources.postgres_projects.my_project": {
Expand All @@ -18,31 +18,6 @@
"project_id": "test-pg-proj-[UNIQUE_NAME]"
}
},
"remote_state": {
"create_time": "[TIMESTAMP]",
"display_name": "Test Project for Adding Default Endpoint Settings",
"history_retention_duration": "604800s",
"name": "projects/test-pg-proj-[UNIQUE_NAME]",
"project_id": "test-pg-proj-[UNIQUE_NAME]",
"status": {
"branch_logical_size_limit_bytes": [BRANCH_SIZE_LIMIT],
"default_branch": "projects/test-pg-proj-[UNIQUE_NAME]/branches/production",
"default_endpoint_settings": {
"autoscaling_limit_max_cu": 1,
"autoscaling_limit_min_cu": 1,
"suspend_timeout_duration": "86400s"
},
"display_name": "Test Project for Adding Default Endpoint Settings",
"enable_pg_native_login": false,
"history_retention_duration": "604800s",
"owner": "[USERNAME]",
"pg_version": 16,
"project_id": "test-pg-proj-[UNIQUE_NAME]",
"synthetic_storage_size_bytes": 0
},
"uid": "[UUID][1]",
"update_time": "[TIMESTAMP]"
},
"changes": {
"default_endpoint_settings": {
"action": "update",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ trace update_file.py databricks.yml "#TO_ADD: " ""
# The direct plan's "changes" map is the structdiff artifact: it records one
# change at path default_endpoint_settings (the whole block) instead of descending
# to its leaves. (The top-level plan verdict is 'update' on both engines.)
$CLI bundle plan -o json | nostamp > out.plan.json
$CLI bundle plan -o json | plan_filter.py > out.plan.json

# The consequence shows up at deploy time, which diverges by engine, so route it to
# per-engine files. Masking the whole message means the API requires the suspension
Expand Down
Loading
Loading