Repository navigation
Recover the deployment when its create response is lost #7770
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
6 changes: 6 additions & 0 deletions
6
acceptance/bundle/dms/create-response-lost/databricks.yml.tmpl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| bundle: | ||
| name: dms-create-response-lost-$UNIQUE_NAME | ||
| resources: | ||
| jobs: | ||
| foo: | ||
| name: foo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["true"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
|
|
||
| === The first deploy's create is applied, but its response is lost | ||
|
|
||
| >>> fault.py --after-handler POST /api/2.0/bundle/deployments 500 0 1 | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-create-response-lost-[UNIQUE_NAME]/default/files... | ||
| Created jobs.foo | ||
| Files: 2 uploaded, 0 deleted | ||
| Resources: 1 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| === The deploy recovered the deployment: the job carries its id | ||
|
|
||
| >>> [CLI] jobs get [JOB_ID] | ||
| { | ||
| "deployment_id": "[DEPLOYMENT_ID]", | ||
| "kind": "BUNDLE", | ||
| "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-create-response-lost-[UNIQUE_NAME]/default/state/metadata.json", | ||
| "version_id": "1" | ||
| } | ||
|
|
||
| === No drift after the deploy | ||
|
|
||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged | ||
|
|
||
| === A create that finds an earlier deployment with versions does not adopt it | ||
|
|
||
| >>> fault.py GET /api/2.0/bundle/deployments/* 500 0 3 | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Warn: Could not read deployment history: Fault injected by test. | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-create-response-lost-[UNIQUE_NAME]/default/files... | ||
| Warn: Could not enable deployment history for this deploy: failed to create deployment: deployment [DEPLOYMENT_ID] already exists | ||
| Updated jobs.foo | ||
| Files: 0 uploaded, 0 deleted | ||
| Resources: 0 created, 1 changed, 0 deleted, 0 unchanged | ||
|
|
||
| === Destroy, so the next deploy creates a new deployment | ||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.jobs.foo | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/dms-create-response-lost-[UNIQUE_NAME]/default | ||
|
|
||
| Destroy: 1 deleted | ||
|
|
||
| === A create retried into ALREADY_EXISTS is recovered | ||
|
|
||
| >>> fault.py --after-handler POST /api/2.0/bundle/deployments 503 0 1 | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/dms-create-response-lost-[UNIQUE_NAME]/default/files... | ||
| Created jobs.foo | ||
| Files: 2 uploaded, 0 deleted | ||
| Resources: 1 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| >>> [CLI] jobs get [JOB_ID_2] | ||
| { | ||
| "deployment_id": "[DEPLOYMENT_ID_2]", | ||
| "kind": "BUNDLE", | ||
| "metadata_file_path": "/Workspace/Users/[USERNAME]/.bundle/dms-create-response-lost-[UNIQUE_NAME]/default/state/metadata.json", | ||
| "version_id": "1" | ||
| } | ||
|
|
||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
| state_node="/Workspace/Users/${CURRENT_USER_NAME}/.bundle/dms-create-response-lost-${UNIQUE_NAME}/default/state/resources.deployment.json" | ||
|
|
||
| show_job_deployment() { | ||
| job_id=$(jq -r '.state["resources.jobs.foo"].__id__' .databricks/bundle/default/resources.json) | ||
| add_repl "$job_id" JOB_ID | ||
| trace $CLI jobs get "$job_id" | jq '.settings.deployment' | ||
| } | ||
|
|
||
| # The service creates the deployment, but the client never sees the answer (in CI: an HTTP/2 | ||
| # stream reset while the body is read). | ||
| title "The first deploy's create is applied, but its response is lost" | ||
| trace fault.py --after-handler "POST /api/2.0/bundle/deployments" 500 0 1 | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "The deploy recovered the deployment: the job carries its id" | ||
| deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "$state_node" | jq -r .object_id) | ||
| add_repl "$deployment_id" DEPLOYMENT_ID | ||
| show_job_deployment | ||
|
|
||
| title "No drift after the deploy" | ||
| trace $CLI bundle plan | ||
|
|
||
| # The startup lookup fails, so the deploy tries to create the deployment again. The service | ||
| # rejects it as ALREADY_EXISTS, and the deployment it finds has versions: it existed before this | ||
| # run, so the deploy does not take it over. | ||
| title "A create that finds an earlier deployment with versions does not adopt it" | ||
| trace fault.py "GET /api/2.0/bundle/deployments/*" 500 0 3 | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Destroy, so the next deploy creates a new deployment" | ||
| trace $CLI bundle destroy --auto-approve | ||
|
|
||
| # The service answers 503 after the create was applied; the SDK retries it, and the retry is | ||
| # rejected as ALREADY_EXISTS. | ||
| title "A create retried into ALREADY_EXISTS is recovered" | ||
| trace fault.py --after-handler "POST /api/2.0/bundle/deployments" 503 0 1 | ||
| trace $CLI bundle deploy | ||
| deployment_id=$(MSYS_NO_PATHCONV=1 $CLI workspace get-status "$state_node" | jq -r .object_id) | ||
| add_repl "$deployment_id" DEPLOYMENT_ID_2 | ||
| show_job_deployment | ||
| trace $CLI bundle plan |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| RecordRequests = false | ||
| # Local only: the lost responses are injected by fault.py, which a cloud run does not have. | ||
| Cloud = false | ||
|
|
||
| # The startup lookup is retried after the injected 500s; do not wait between attempts. | ||
| Env.DATABRICKS_BUNDLE_RETRY_INTERVAL_MS = "1" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| package dms | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
| "path" | ||
| "strconv" | ||
|
|
||
| "github.com/databricks/cli/libs/retry" | ||
| "github.com/databricks/databricks-sdk-go" | ||
| "github.com/databricks/databricks-sdk-go/apierr" | ||
| "github.com/databricks/databricks-sdk-go/service/bundledeployments" | ||
| "github.com/databricks/databricks-sdk-go/service/workspace" | ||
| ) | ||
|
|
||
| // FetchDeployment returns the deployment recorded under statePath and its id, the object id of | ||
| // the workspace node the service creates there. It returns an empty id and a nil deployment when | ||
| // there is no node yet. | ||
| // | ||
| // TODO: a deployment is only usable when both the node and the service's record exist, and a | ||
| // half-created one - node present, record missing - blocks the bundle at startup even though | ||
| // CreateDeployment already recovers from it. Return an empty id and version unless both halves | ||
| // are there, leaving the caller to call CreateDeployment to create or finalize it. | ||
| // | ||
| // TODO: ask the service for a lookup by state path, so this is one round trip rather than two - a | ||
| // workspace lookup to turn the node into an id, then a get by that id. | ||
| func FetchDeployment(ctx context.Context, w *databricks.WorkspaceClient, statePath string) (string, *bundledeployments.Deployment, error) { | ||
| nodePath := path.Join(statePath, DeploymentNodeName) | ||
| obj, err := retry.With(ctx, retry.IsTransientRead, func() (*workspace.ObjectInfo, error) { | ||
| return w.Workspace.GetStatusByPath(ctx, nodePath) | ||
| }) | ||
| if errors.Is(err, apierr.ErrNotFound) || errors.Is(err, apierr.ErrResourceDoesNotExist) { | ||
| return "", nil, nil | ||
| } | ||
| if err != nil { | ||
| return "", nil, fmt.Errorf("looking up deployment at %s: %w", nodePath, err) | ||
| } | ||
| deploymentID := strconv.FormatInt(obj.ObjectId, 10) | ||
| deployment, err := retry.With(ctx, retry.IsTransientRead, func() (*bundledeployments.Deployment, error) { | ||
| return w.BundleDeployments.GetDeployment(ctx, bundledeployments.GetDeploymentRequest{ | ||
| Name: DeploymentName(deploymentID), | ||
| }) | ||
| }) | ||
| if err != nil { | ||
| return "", nil, err | ||
| } | ||
| return deploymentID, deployment, nil | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.