Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .nextchanges/bundles/destroy-empty-id-state.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* `bundle destroy` and `bundle deploy` no longer fail with "internal error, missing in state" when the state holds an empty-ID entry left by an interrupted recreate in an older CLI. ([#7676](https://github.com/databricks/cli/pull/7676))
11 changes: 11 additions & 0 deletions acceptance/bundle/destroy/empty-id-in-state/databricks.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
bundle:
name: empty-id-in-state

resources:
jobs:
foo:
name: foo
bar:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of two yaml files, use one with comments like this:

bar: # TO_REMOVE

then run grep -v when you need to get rid of bar

name: bar
baz:
name: baz
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
bundle:
name: empty-id-in-state

resources:
jobs:
foo:
name: foo
baz:
name: baz
2 changes: 2 additions & 0 deletions acceptance/bundle/destroy/empty-id-in-state/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = ["", "true"]
37 changes: 37 additions & 0 deletions acceptance/bundle/destroy/empty-id-in-state/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@

>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/empty-id-in-state/default/files...
Created jobs.bar
Created jobs.baz
Created jobs.foo
Files: 2 uploaded, 0 deleted
Resources: 3 created, 0 changed, 0 deleted, 0 unchanged

=== An empty-ID entry for a resource in config is planned as a create

>>> [CLI] bundle plan
create jobs.foo

Plan: 1 to add, 0 to change, 0 to delete, 2 unchanged

=== An empty-ID entry for a resource removed from config

>>> cp databricks_no_bar.yml databricks.yml

>>> [CLI] bundle deploy
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/empty-id-in-state/default/files...
Created jobs.foo
Files: 1 uploaded, 0 deleted
Resources: 1 created, 0 changed, 0 deleted, 1 unchanged
["resources.jobs.baz","resources.jobs.foo"]

=== An empty-ID entry on destroy

>>> [CLI] bundle destroy --auto-approve
The following resources will be deleted:
delete resources.jobs.foo

All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/empty-id-in-state/default

Destroy: 1 deleted
no state file
27 changes: 27 additions & 0 deletions acceptance/bundle/destroy/empty-id-in-state/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
state_file=".databricks/bundle/default/resources.json"

# CLIs before #5173 recorded an empty-ID entry when the create step of a recreate failed.
blank_id() {
jq ".state[\"resources.jobs.$1\"].__id__ = \"\"" "$state_file" > state.tmp && mv state.tmp "$state_file"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you use update_file.py helper? (it fails loudly if it cannot do replacement)

}

show_state() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can print_state.py be used?

if [ -f "$state_file" ]; then jq -c '.state | keys' "$state_file"; else echo "no state file"; fi
}

trace $CLI bundle deploy

title "An empty-ID entry for a resource in config is planned as a create"
blank_id foo
trace $CLI bundle plan

title "An empty-ID entry for a resource removed from config"
blank_id bar
trace cp databricks_no_bar.yml databricks.yml
trace $CLI bundle deploy
show_state

title "An empty-ID entry on destroy"
blank_id baz
trace $CLI bundle destroy --auto-approve
show_state
6 changes: 6 additions & 0 deletions acceptance/bundle/destroy/empty-id-in-state/test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Cloud = false

Ignore = [
"databricks_no_bar.yml",
".databricks/"
]
2 changes: 1 addition & 1 deletion bundle/deployplan/action.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ type Action struct {
ResourceKey string
ActionType ActionType
// Gone mirrors PlanEntry.Gone: the delete is a state-only cleanup because the
// resource no longer exists remotely.
// resource no longer exists remotely (confirmed by a read, or an empty-ID state entry).
Gone bool
// StateOnly mirrors PlanEntry.StateOnly: applying the action only changes the state
// (a delete of a resource without DoDelete, or a re-key).
Expand Down
8 changes: 5 additions & 3 deletions bundle/deployplan/plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,9 +130,11 @@ type PlanEntry struct {
ID string `json:"id,omitempty"`
DependsOn []DependsOnEntry `json:"depends_on,omitempty"`
Action ActionType `json:"action,omitempty"`
// Gone is set on Delete entries when planning confirmed the resource no longer
// exists remotely. Applying such an entry only removes it from the state, without
// calling the delete API, and approval prompts do not list it as a deletion.
// Gone is set on Delete entries when the resource no longer exists remotely: planning
// confirmed it with a read, or the state entry has an empty ID (left by a recreate in
// an older CLI whose create step failed after the delete). Applying such an entry only
// removes it from the state, without calling the delete API, and approval prompts do
// not list it as a deletion.
Gone bool `json:"gone,omitempty"`
// StateOnly marks an entry whose apply has no backend effect and only changes the state:
// - a Delete of a resource that implements no DoDelete. Like Gone, applying it only
Expand Down
7 changes: 5 additions & 2 deletions bundle/direct/bundle_plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -225,8 +225,11 @@ func (b *DeploymentBundle) CalculatePlan(ctx context.Context, client *databricks
if entry.Action == deployplan.Delete {
id := b.StateDB.GetResourceID(resourceKey)
if id == "" {
logdiag.LogError(ctx, fmt.Errorf("%s: internal error, missing in state", errorPrefix))
return false
// An empty-ID entry from an older partial-recreate failure (see the
// Create path below): its resource was deleted and never re-created,
// so there is nothing to delete remotely; applying only drops the entry.
entry.Gone = true
Comment thread
Sankalp-Mittal marked this conversation as resolved.
Resolved
return true
}

if !adapter.HasDoDelete() {
Expand Down
Loading