Repository navigation
Recover the deployment when its create response is lost - #7770
Conversation
When CreateDeployment fails after the server applied it (an HTTP/2 stream reset while the body is read), the deploy wrote its jobs without deployment_id, and the next plan, which finds the deployment at the state path, reported them as changed. On any create error other than a 4xx, look the deployment up at the state path and use it if it exists. Also print the invariant tests' LOG.*.err files on failure: the "*.err" ignore pattern hid them, so a failing `bundle plan` left only "Exit code: 1". aws: TestAccept/bundle/invariant/no_drift/DMS=true Co-authored-by: Isaac <no-reply@databricks.com>
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 127 tests: regression=2 unreleased=0 coverage=125 failing=0
TestAccept/bundle/dms/create-response-lost/DMS=true: branch ✅ | main (61f1fc5) ❌ | latest (v1.26.0) ❌main (61f1fc5): latest (v1.26.0): same output as main (61f1fc5) TestAccept/bundle/dms_fault/get_deployment/DMS=true: branch ✅ | main (61f1fc5) ❌ | latest (v1.26.0) ❌main (61f1fc5): latest (v1.26.0): |
✅ Integration test reportCommit: b376767
Top 50 slowest tests (at least 2 minutes):
|
…rsions A create the SDK retries after a 504 or a timeout can come back as 409 once the first attempt was applied. A deployment that already has versions existed before this run, so it is not taken over. aws: TestAccept/bundle/invariant/no_drift/DMS=true Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
left a comment
There was a problem hiding this comment.
[AI Generated] Review by review-factory (Claude Opus 5.5 + Claude Fable 5.2, cross-verified). Only points the reviewers agreed on are posted.
Design, could be simpler: mayHaveCreated picks by status code which create errors get a lookup, and the second commit already had to add 409 to it. Adoption is made safe by the LastVersionId == "" check, and a rejected create just finds no node. Looking up after every create error costs one get-status and drops mayHaveCreated, its test and the status list.
…d create Per review: a rejected create leaves no node, so the lookup can run after every error, and the status classification goes. The testserver's CreateDeployment now matches the service: a second create in the same folder fails with ALREADY_EXISTS (it used to overwrite the node), and a node with no record is finalized in place. create-response-lost covers the recovered 409 after an SDK retry and the refused deployment with versions. The LOG.*.err change moves to its own PR. aws: TestAccept/bundle/invariant/no_drift/DMS=true Co-authored-by: Isaac <no-reply@databricks.com>
|
@Sankalp-Mittal re the design point: agreed and done in b376767. The lookup runs after every create error, and |
❌ Integration test reportCommit: 73b204d
83 interesting tests: 57 MISS, 12 RECOVERED, 9 FAIL, 2 flaky, 2 PANIC, 1 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
…s#7779) ## Changes - `acceptance/bundle/invariant/test.toml` no longer ignores `*.err`. The harness checks the ignore patterns before the `LOG` prefix, so the `LOG.*.err` files these scripts write (`bundle plan -o json > ... 2>LOG.plan_initial.err`, `LOG.planjson.err`, ...) were never printed. A failing plan then left only `Exit code: 1` in the test log. Every `.err` file under `acceptance/bundle/invariant` is a `LOG.*` file, so no test gains an unexpected output file. ## Why Split out of databricks#7770 at review. This makes the silent invariant failures behind error/1045 diagnosable. It does not fix them: the recent ones have different causes (a vector search endpoint plan, an app plan). ## Tests `TestAccept/bundle/invariant` passes locally. ## Dashboard | error | verbatim snippet | |---|---| | [error/1045](http://go/cli/dashboard/error/1045?main=1&pr=1&end=2026-11-18T12:00:00Z&last=7d) | `expected: "INPUT_CONFIG_OK\n"`, `actual : "\nExit code: 1\n"` | This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
Changes
CreateDeploymentcan fail after the server has applied it: in CI the response body read fails with an HTTP/2 stream reset. In shadow mode the deploy then only warned and wrote its jobs withoutdeployment.deployment_id. The next plan finds the deployment at${workspace.state_path}/resources.deployment.jsonand stamps its id, so it reportsupdatefor every job. That is theno_driftfailure.cmd/bundle/utilstodms.FetchDeployment, sobundle/phasescan share it.CreateDeploymentnow matches the service: a second create in the same folder fails withALREADY_EXISTS(it used to overwrite the node with a new deployment), and a node with no record is finalized in place. As a result,bundle/dms_fault/get_deploymentnow shows the warning the cloud gives ("failed to create deployment: deployment [DEPLOYMENT_ID] already exists") where the fake used to register a replacement deployment silently.Why
Read back rather than retry, following the deploy lock create in #7594: the service rejects a second create in the same folder.
Tests
New
acceptance/bundle/dms/create-response-lost(local,fault.py --after-handler) has three cases:ALREADY_EXISTSis recovered;Without the fix, the recovered cases show the warning, then
update jobs.foo/Plan: 0 to add, 1 to change.bundle/acceptance passes locally.no_drift/DMS=trueran on AWS in this PR's integration run: 87 passed, 0 failed (linux and windows).The last commit asks CI to run
aws: TestAccept/bundle/invariant/no_drift/DMS=true.Dashboard
Warn: Could not enable deployment history for this deploy: failed to create deployment: response body: stream error: stream ID 99; INTERNAL_ERROR; received from peer...Unexpected action='update' for resources.jobs.foo,"deployment.deployment_id": {"action": "update", ...}Tests: TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=job_table_update_trigger.yml.tmpl/READPLAN=1, TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=job_escaped_refs.yml.tmpl/READPLAN=, TestAccept/bundle/invariant/no_drift/DMS=true/INPUT_CONFIG=job_permission_ref.yml.tmpl/READPLAN=1
This pull request and its description were written by Isaac.