Repository navigation
acc: add bind test for a resource that references an undeployed one #7846
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
denik
wants to merge
1
commit into
main
Choose a base branch
from
denik/bind-test-undeployed-dependency
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+136
−0
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
7 changes: 7 additions & 0 deletions
7
acceptance/bundle/deployment/bind/job/undeployed-dependency/databricks.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| bundle: | ||
| name: bind-undeployed-dependency | ||
|
|
||
| resources: | ||
| jobs: | ||
| deployed: | ||
| name: deployed upstream |
14 changes: 14 additions & 0 deletions
14
acceptance/bundle/deployment/bind/job/undeployed-dependency/databricks_v2.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| bundle: | ||
| name: bind-undeployed-dependency | ||
|
|
||
| resources: | ||
| jobs: | ||
| deployed: | ||
| name: deployed upstream | ||
|
|
||
| new_upstream: | ||
| name: new upstream | ||
|
|
||
| job_1: | ||
| name: bound job | ||
| description: "runs after ${resources.jobs.deployed.id} and ${resources.jobs.new_upstream.id}" |
2 changes: 2 additions & 0 deletions
2
acceptance/bundle/deployment/bind/job/undeployed-dependency/out.test.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Cloud = false | ||
| EnvMatrix.DMS = ["", "true"] |
88 changes: 88 additions & 0 deletions
88
acceptance/bundle/deployment/bind/job/undeployed-dependency/output.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
|
|
||
| === Deploy the bundle with only the upstream job | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/bind-undeployed-dependency/default/files... | ||
| Created jobs.deployed | ||
| Files: 2 uploaded, 0 deleted | ||
| Resources: 1 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| === Add a job that references both the deployed job and a job that is not deployed yet | ||
|
|
||
| === Bind without auto-approve shows the changes and does not touch the state | ||
|
|
||
| >>> errcode [CLI] bundle deployment bind job_1 [EXTERNAL_JOB_ID] | ||
| Plan: update resources.jobs.job_1 | ||
|
|
||
| Changes detected: | ||
| ~ deployment: null -> {"kind":"BUNDLE","metadata_file_path":"/Workspace/Users/[USERNAME]/.bundle/bind-undeployed-dependency/default/state/metadata.json"} | ||
| ~ description: null -> "runs after [DEPLOYED_ID] and ${resources.jobs.new_upstream.id}" | ||
| ~ edit_mode: null -> "UI_LOCKED" | ||
| ~ format: null -> "MULTI_TASK" | ||
| ~ max_concurrent_runs: null -> 1 | ||
| ~ name: "external job" -> "bound job" | ||
| ~ queue: null -> {"enabled":true} | ||
|
|
||
| Error: this bind operation requires user confirmation, but the current console does not support prompting. | ||
| To proceed, use --auto-approve after reviewing the plan above. | ||
|
|
||
|
|
||
| Exit code: 1 | ||
|
|
||
| >>> jq .state | keys .databricks/bundle/default/resources.json | ||
| [ | ||
| "resources.jobs.deployed" | ||
| ] | ||
|
|
||
| === Bind with auto-approve records the job and its dependencies | ||
|
|
||
| >>> [CLI] bundle deployment bind job_1 [EXTERNAL_JOB_ID] --auto-approve | ||
| Successfully bound job with an id '[EXTERNAL_JOB_ID]' | ||
| Run 'bundle deploy' to deploy changes to your workspace | ||
|
|
||
| >>> jq {serial, job_1: .state["resources.jobs.job_1"] | {__id__, depends_on, description: .state.description}} .databricks/bundle/default/resources.json | ||
| { | ||
| "serial": 3, | ||
| "job_1": { | ||
| "__id__": "[EXTERNAL_JOB_ID]", | ||
| "depends_on": [ | ||
| { | ||
| "node": "resources.jobs.deployed", | ||
| "label": "${resources.jobs.deployed.id}" | ||
| }, | ||
| { | ||
| "node": "resources.jobs.new_upstream", | ||
| "label": "${resources.jobs.new_upstream.id}" | ||
| } | ||
| ], | ||
| "description": "runs after [DEPLOYED_ID] and ${resources.jobs.new_upstream.id}" | ||
| } | ||
| } | ||
|
|
||
| >>> ls .databricks/bundle/default | ||
| deployment.json | ||
| resources.json | ||
| sync-snapshots | ||
|
|
||
| === Deploy creates the new upstream and updates the bound job | ||
|
|
||
| >>> [CLI] bundle plan | ||
| update jobs.job_1 | ||
| create jobs.new_upstream | ||
|
|
||
| Plan: 1 to add, 1 to change, 0 to delete, 1 unchanged | ||
|
|
||
| >>> [CLI] bundle deploy | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/bind-undeployed-dependency/default/files... | ||
| Created jobs.new_upstream | ||
| Updated jobs.job_1 | ||
| Files: 1 uploaded, 0 deleted | ||
| Resources: 1 created, 1 changed, 0 deleted, 1 unchanged | ||
|
|
||
| >>> [CLI] jobs get [EXTERNAL_JOB_ID] | ||
| runs after [DEPLOYED_ID] and [NEW_UPSTREAM_ID] | ||
|
|
||
| === Nothing is left to deploy | ||
|
|
||
| >>> [CLI] bundle plan | ||
| Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged |
24 changes: 24 additions & 0 deletions
24
acceptance/bundle/deployment/bind/job/undeployed-dependency/script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| title "Deploy the bundle with only the upstream job" | ||
| trace $CLI bundle deploy | ||
|
|
||
| title "Add a job that references both the deployed job and a job that is not deployed yet" | ||
| cp databricks_v2.yml databricks.yml | ||
| job_id=$($CLI jobs create --json '{"name": "external job"}' | jq -r '.job_id') | ||
| add_repl "$job_id" EXTERNAL_JOB_ID | ||
|
|
||
| title "Bind without auto-approve shows the changes and does not touch the state" | ||
| trace errcode $CLI bundle deployment bind job_1 $job_id | ||
| trace jq '.state | keys' .databricks/bundle/default/resources.json | ||
|
|
||
| title "Bind with auto-approve records the job and its dependencies" | ||
| trace $CLI bundle deployment bind job_1 $job_id --auto-approve | ||
| trace jq '{serial, job_1: .state["resources.jobs.job_1"] | {__id__, depends_on, description: .state.description}}' .databricks/bundle/default/resources.json | ||
| trace ls .databricks/bundle/default | ||
|
|
||
| title "Deploy creates the new upstream and updates the bound job" | ||
| trace $CLI bundle plan | ||
| trace $CLI bundle deploy | ||
| trace $CLI jobs get $job_id | jq -r .settings.description | sed "s/$(read_id.py deployed)/[DEPLOYED_ID]/; s/$(read_id.py new_upstream)/[NEW_UPSTREAM_ID]/" | ||
|
|
||
| title "Nothing is left to deploy" | ||
| trace $CLI bundle plan |
1 change: 1 addition & 0 deletions
1
acceptance/bundle/deployment/bind/job/undeployed-dependency/test.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| Cloud = false | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[blocking] The script creates
.databricks/bundle/default/viabundle deployandbundle deployment bind, but no ancestortest.tomlignores.databricks(the parent chain only hasIgnore = ["databricks.yml"]and the framework auto-adds.cache). After the script finishes the harness lists the temp dir (acceptance_test.go:1157) and flags every file not in inputs, outputs, or the compiled ignore set as unexpected (t.Errorat line 1189). AddIgnore = [".databricks"]here.