Repository navigation
Add acceptance tests for the engine user-agent tag on a terraform state - #7775
Conversation
|
Commit: Regression test reportSelected 3 changed tests (limit=20, 0 not selected) 3 tests: regression=0 unreleased=0 coverage=3 failing=0
|
| @@ -0,0 +1,110 @@ | |||
| { | |||
There was a problem hiding this comment.
terraform.tfstate (110 lines) and terraform-requests.json (19 lines) are byte-for-byte identical in all three subdirectories. They could live once in the user-agent/ parent and each script would reference ../terraform-requests.json ../terraform.tfstate. That keeps them in sync and drops ~260 lines; replay_tfstate.py takes plain relative paths so it just works. If a test later needs a different fixture it can fork at that point.
There was a problem hiding this comment.
Done in 2f9d1c9: the fixtures now live once in acceptance/bundle/migrate/user-agent/. One correction: tests run in a temp copy of their directory, so a bare ../ doesn't resolve. The scripts read them via $TESTDIR/../, like bundle/validate/empty_resources.
This reply was written by Isaac.
✅ Integration test reportCommit: 7f47413
Top 3 slowest tests (at least 2 minutes):
|
Record which engine tag each request carries when a bundle still has terraform state: - race: the local terraform state disappears between the state pull and the auto-migration (as with two concurrent deploys). The deploy is tagged engine/terraform and then panics. - bind-unbind: pull-only commands tag engine/terraform before rejecting the terraform state. - engine-pinned: a terraform engine pinned via DATABRICKS_BUNDLE_ENGINE or bundle.engine. Co-authored-by: Isaac <no-reply@databricks.com>
db32498 to
2495c84
Compare
BSD uniq pads the count differently from GNU uniq, so the migrate/user-agent goldens failed on the macOS CI leg. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
left a comment
There was a problem hiding this comment.
[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.
No new issues found.
The three tests replayed byte-identical terraform-requests.json and terraform.tfstate copies; keep one in the parent directory and read it via $TESTDIR/.. like other acceptance tests with shared fixtures. Co-authored-by: Isaac <no-reply@databricks.com>
- race: require the failure with musterr and assert the panic with contains.py on LOG.deploy instead of a bare errcode piped into grep. - bind-unbind: read the job id with read_id.py. - engine-pinned: send summary output to LOG.summary, not /dev/null. - Move print_engines to bundle/migrate/script.prepare so other migrate tests can use it, and keep LOG.* files out of the bundle upload. Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
left a comment
There was a problem hiding this comment.
[AI Generated] Follow-up review by review-factory (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.
No new issues found.
❌ Integration test reportCommit: f696aca
967 interesting tests: 943 MISS, 15 RECOVERED, 5 FAIL, 4 PANIC
Top 50 slowest tests (at least 2 minutes):
|
…migration (databricks#7778) ## Changes `statemgmt.Migrate` now returns an error when the local terraform state is missing, instead of reporting that there is nothing to migrate. Its `bool` return was only false on that path, so it is dropped. The failure is recorded as a new deploy telemetry key, `direct_migrate_state_missing`, separate from `direct_migrate_error` (a conversion failure). ## Why Migration only runs when the pulled state is terraform, and the pull always leaves that state on disk, so a missing file means it changed after the pull, e.g. a concurrent deploy migrated it. Before, the deploy then continued on the terraform engine with no direct state open: it took the deploy lock, uploaded files, sent `engine/terraform` in the user agent, and panicked in `CalculatePlan` with `DeploymentState must be opened first`. This is what one production deploy on v1.24.0 looked like. ## Tests The `migrate/user-agent/race` golden from databricks#7775 (this PR is stacked on it) changes from the panic to the error, and the deploy now stops after the state pull. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…abricks#7690) ## Changes Commands that read and need the state (deploy, plan, destroy, run, summary) now tag the user agent with `engine/direct` before the bundle is loaded, instead of after the state is pulled. Pull-only commands (bind, unbind, `debug states`, `deployment migrate`) keep tagging the pulled state's engine. ## Why Requests made while loading the bundle, initializing, pulling the state and resolving `lookup` variables (`scim/v2/Me`, `workspace/export`, `workspace/get-status`, `jobs/list`, `pipelines`, ...) went out without an engine tag, as did the telemetry upload of runs that failed early. On CLI >= 1.20 that's 2.3% of `bundle_deploy` requests and 0.78% of deploy runs over the last 30 days, and dashboards that classify by engine count them as terraform. These commands always run on the direct engine since v1.20.0: a terraform state is migrated first, and a failed migration (including a state removed after the pull, databricks#7778) is fatal. ## Tests In `acceptance/bundle/user_agent`, the opening requests flip from MISS to OK; the only untagged ones left are `/.well-known/databricks-config`, which is sent before there is a command context, and `bundle validate`, which doesn't read state. `state_present` and `migrate/auto/run-no-commit` only change the tag order. `migrate/auto/plan-failure` asserts that every request of a deploy whose migration fails is tagged `engine/direct`; its deploy telemetry still reports `state_engine: terraform`, the state it started from. Stacked on databricks#7778, which is stacked on databricks#7775. In databricks#7775's tests, the `race` deploy is now tagged `engine/direct`, and deploys rejected for a terraform engine pinned via `DATABRICKS_BUNDLE_ENGINE` or `bundle.engine` now tag their telemetry upload `engine/direct`. `bind-unbind` is unchanged: those commands still tag `engine/terraform`. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Changes
Acceptance tests under
acceptance/bundle/migrate/user-agent/that record theengine/user-agent tag of every request when a bundle still has a terraform state:race: the local terraform state disappears between the state pull and the auto-migration, as when two deploys run at once. The deploy tagsengine/terraformand then panics withDeploymentState must be opened first.bind-unbind:deployment bindandunbindonly pull the state, so they tagengine/terraformbefore rejecting it.engine-pinned: a terraform engine pinned throughDATABRICKS_BUNDLE_ENGINEorbundle.engine.The shared
acceptance/bundle/migrate/script.preparegains theprint_engineshelper and keepsLOG.*files out of the bundle upload, for all migrate tests.Why
Telemetry classifies bundle requests by this tag, and
engine/terraformstill shows up from CLI >= 1.20. These tests pin down current behavior so changes to the tagging (#7690) and a fix for the panic show up as golden diffs.Tests
Test-only change; goldens record what main does today.
This pull request and its description were written by Isaac.