Repository navigation
Detect remote permission changes on unmanaged resources in config-remote-sync #7536
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
0725cb9
Detect remote permission changes on unmanaged resources in config-rem…
ilyakuz-db 93201ef
Read unmanaged permissions in configsync, sync declared permissions, …
ilyakuz-db f958e79
Resolve permission IDs of unchanged resources, import ACLs of removed…
ilyakuz-db b9f3db2
Handle an empty remote ACL and resource keys with spaces
ilyakuz-db c6612a8
Merge main into config-remote-sync-unmanaged-permissions
ilyakuz-db 0e8de2a
Report unreadable ACLs as skipped changes and update a stale test doc
ilyakuz-db 55397be
Merge main into config-remote-sync-unmanaged-permissions
ilyakuz-db File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
25 changes: 25 additions & 0 deletions
25
acceptance/bundle/config-remote-sync/unmanaged_permissions/databricks.yml.tmpl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| bundle: | ||
| name: test-bundle-$UNIQUE_NAME | ||
|
|
||
| resources: | ||
| jobs: | ||
| my_job: | ||
| tasks: | ||
| - task_key: main | ||
| notebook_task: | ||
| notebook_path: /Users/{{workspace_user_name}}/main | ||
| new_cluster: | ||
| spark_version: $DEFAULT_SPARK_VERSION | ||
| node_type_id: $NODE_TYPE_ID | ||
| num_workers: 1 | ||
|
|
||
| pipelines: | ||
| my_pipeline: | ||
| name: test-pipeline-$UNIQUE_NAME | ||
| libraries: | ||
| - notebook: | ||
| path: /Users/{{workspace_user_name}}/notebook | ||
|
|
||
| targets: | ||
| default: | ||
| mode: development |
3 changes: 3 additions & 0 deletions
3
acceptance/bundle/config-remote-sync/unmanaged_permissions/out.test.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Cloud = false | ||
| GOOS.windows = false | ||
| EnvMatrix.DMS = ["", "true"] |
131 changes: 131 additions & 0 deletions
131
acceptance/bundle/config-remote-sync/unmanaged_permissions/output.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files... | ||
| Created jobs.my_job | ||
| Created pipelines.my_pipeline | ||
| Files: 2 uploaded, 0 deleted | ||
| Resources: 2 created, 0 changed, 0 deleted, 0 unchanged | ||
|
|
||
| === Add permissions remotely to resources that declare none | ||
|
|
||
| === An ACL that cannot be read is reported as skipped | ||
|
|
||
| >>> [CLI] bundle config-remote-sync --select-ids jobs:[MY_JOB_ID] -o json | ||
| { | ||
| "status": "success", | ||
| "skippedChanges": null | ||
| } | ||
|
|
||
| >>> [CLI] bundle config-remote-sync -o json | ||
| { | ||
| "status": "partial", | ||
| "changes": [ | ||
| "resources.jobs.my_job" | ||
| ], | ||
| "skippedChanges": [ | ||
| { | ||
| "resourceKey": "resources.pipelines.my_pipeline", | ||
| "reason": "permissions_permission_denied", | ||
| "error": "Fault injected by test." | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| === Detect and save changes | ||
| Detected changes in 2 resource(s): | ||
|
|
||
| Resource: resources.jobs.my_job | ||
| permissions: add | ||
|
|
||
| Resource: resources.pipelines.my_pipeline | ||
| permissions: add | ||
|
|
||
|
|
||
|
|
||
| === Configuration changes | ||
|
|
||
| >>> diff.py databricks.yml.backup databricks.yml | ||
| --- databricks.yml.backup | ||
| +++ databricks.yml | ||
| @@ -14,4 +14,9 @@ | ||
| num_workers: 1 | ||
|
|
||
| + permissions: | ||
| + - level: CAN_MANAGE | ||
| + user_name: viewer@example.com | ||
| + - level: CAN_MANAGE_RUN | ||
| + user_name: admin@example.com | ||
| pipelines: | ||
| my_pipeline: | ||
| @@ -20,4 +25,9 @@ | ||
| - notebook: | ||
| path: /Users/{{workspace_user_name}}/notebook | ||
| + permissions: | ||
| + - level: IS_OWNER | ||
| + user_name: owner@example.com | ||
| + - group_name: users | ||
| + level: CAN_VIEW | ||
|
|
||
| targets: | ||
|
|
||
| === Sync again: the permissions are now in the config | ||
| No changes detected. | ||
|
|
||
|
|
||
| === Deploy keeps the remote owner | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files... | ||
| Created jobs.my_job.permissions | ||
| Created pipelines.my_pipeline.permissions | ||
| Files: 1 uploaded, 0 deleted | ||
| Resources: 2 created, 0 changed, 0 deleted, 2 unchanged | ||
|
|
||
| >>> [CLI] permissions get pipelines [MY_PIPELINE_ID] | ||
| [{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"}] | ||
|
|
||
| === Change a level, add and remove principals remotely | ||
|
|
||
| === Detect and save changes | ||
| Detected changes in 1 resource(s): | ||
|
|
||
| Resource: resources.jobs.my_job | ||
| permissions[group_name='data-team']: add | ||
| permissions[user_name='admin@example.com']: remove | ||
| permissions[user_name='viewer@example.com'].level: replace | ||
|
|
||
|
|
||
|
|
||
| === Configuration changes | ||
|
|
||
| >>> diff.py databricks.yml.backup databricks.yml | ||
| --- databricks.yml.backup | ||
| +++ databricks.yml | ||
| @@ -15,8 +15,8 @@ | ||
|
|
||
| permissions: | ||
| - - level: CAN_MANAGE | ||
| + - level: CAN_VIEW | ||
| user_name: viewer@example.com | ||
| - - level: CAN_MANAGE_RUN | ||
| - user_name: admin@example.com | ||
| + - group_name: data-team | ||
| + level: CAN_MANAGE_RUN | ||
| pipelines: | ||
| my_pipeline: | ||
|
|
||
| === Deploy and sync again | ||
| Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files... | ||
| Files: 1 uploaded, 0 deleted | ||
| Resources: 0 created, 0 changed, 0 deleted, 4 unchanged | ||
| No changes detected. | ||
|
|
||
|
|
||
| >>> [CLI] bundle destroy --auto-approve | ||
| The following resources will be deleted: | ||
| delete resources.jobs.my_job | ||
| delete resources.pipelines.my_pipeline | ||
|
|
||
| This action will result in the deletion of the following Lakeflow Spark Declarative Pipelines along with the | ||
| Streaming Tables (STs) and Materialized Views (MVs) managed by them. Set 'cascade_on_destroy: false' on a pipeline to retain datasets on pipeline deletion: | ||
| delete resources.pipelines.my_pipeline | ||
|
|
||
| All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default | ||
|
|
||
| Destroy: 2 deleted |
65 changes: 65 additions & 0 deletions
65
acceptance/bundle/config-remote-sync/unmanaged_permissions/script
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| #!/bin/bash | ||
|
|
||
| envsubst < databricks.yml.tmpl > databricks.yml | ||
|
|
||
| cleanup() { | ||
| trace $CLI bundle destroy --auto-approve | ||
| } | ||
| trap cleanup EXIT | ||
|
|
||
| $CLI bundle deploy | ||
| job_id="$(read_id.py my_job)" | ||
| pipeline_id="$(read_id.py my_pipeline)" | ||
|
|
||
| title "Add permissions remotely to resources that declare none" | ||
| # The current user's owner entry is managed by deploy, so it must not be written to the config. | ||
| $CLI permissions set jobs $job_id --json '{"access_control_list":[ | ||
| {"permission_level":"IS_OWNER","user_name":"'"$CURRENT_USER_NAME"'"}, | ||
| {"permission_level":"CAN_MANAGE","user_name":"viewer@example.com"}, | ||
| {"permission_level":"CAN_MANAGE_RUN","user_name":"admin@example.com"} | ||
| ]}' > /dev/null | ||
| # An owner transferred to another user is kept. | ||
| $CLI permissions set pipelines $pipeline_id --json '{"access_control_list":[ | ||
| {"permission_level":"IS_OWNER","user_name":"owner@example.com"}, | ||
| {"permission_level":"CAN_MANAGE","user_name":"'"$CURRENT_USER_NAME"'"}, | ||
| {"permission_level":"CAN_VIEW","group_name":"users"} | ||
| ]}' > /dev/null | ||
|
|
||
| title "An ACL that cannot be read is reported as skipped" | ||
| fault.py "GET /api/2.0/permissions/pipelines/$pipeline_id" 403 0 2 PERMISSION_DENIED | ||
| trace $CLI bundle config-remote-sync --select-ids "jobs:$job_id" -o json | jq '{status, skippedChanges}' | ||
| trace $CLI bundle config-remote-sync -o json | jq '{status, changes: (.changes | keys), skippedChanges}' | ||
|
|
||
| title "Detect and save changes" | ||
| cp databricks.yml databricks.yml.backup | ||
| $CLI bundle config-remote-sync --save | ||
|
|
||
| title "Configuration changes" | ||
| trace diff.py databricks.yml.backup databricks.yml | ||
| rm databricks.yml.backup | ||
|
|
||
| title "Sync again: the permissions are now in the config" | ||
| $CLI bundle config-remote-sync | ||
|
|
||
| title "Deploy keeps the remote owner" | ||
| $CLI bundle deploy | ||
| trace $CLI permissions get pipelines $pipeline_id | jq -c '[.access_control_list[] | {user_name, group_name, level: .all_permissions[0].permission_level}]' | ||
|
|
||
| title "Change a level, add and remove principals remotely" | ||
| $CLI permissions set jobs $job_id --json '{"access_control_list":[ | ||
| {"permission_level":"IS_OWNER","user_name":"'"$CURRENT_USER_NAME"'"}, | ||
| {"permission_level":"CAN_VIEW","user_name":"viewer@example.com"}, | ||
| {"permission_level":"CAN_MANAGE_RUN","group_name":"data-team"} | ||
| ]}' > /dev/null | ||
|
|
||
| title "Detect and save changes" | ||
| cp databricks.yml databricks.yml.backup | ||
| $CLI bundle config-remote-sync --save | ||
|
|
||
| title "Configuration changes" | ||
| trace diff.py databricks.yml.backup databricks.yml | ||
| rm databricks.yml.backup | ||
|
|
||
| title "Deploy and sync again" | ||
| $CLI bundle deploy | ||
| $CLI bundle config-remote-sync |
4 changes: 4 additions & 0 deletions
4
acceptance/bundle/config-remote-sync/unmanaged_permissions/test.toml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| RecordRequests = false | ||
| Ignore = [".databricks", "databricks.yml", "databricks.yml.backup"] | ||
|
|
||
| Env.DATABRICKS_BUNDLE_ENABLE_EXPERIMENTAL_YAML_SYNC = "true" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unlike
skip_permissions, this test has noCloud = trueand uses principals that exist only in the fake server, so the import and redeploy never ran against a real ACL. Can it use built-in groups?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
factory, if this is doable, let's verify with a local cloud run
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[AI Generated] Still open at head:
unmanaged_permissions/test.tomlhas noCloud = true(out.test.tomlshowsCloud = false), and the script usesviewer@,admin@,owner@example.comanddata-team, which won't exist in a real workspace. Built-inuserscovers group grants.IS_OWNERneeds a real user or service principal, so create a service principal in the script. I can't push or run cloud tests from this checkout;deco env runon your side would settle it.review-factory reply (Claude Opus 5.5)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[AI] Not done yet: a local cloud run is not possible from the factory's machine, which has no test-workspace environment. Making the test cloud-ready also takes more than built-in groups: the transferred pipeline owner needs a real second principal (e.g. the test service principal, since a group cannot be owner), and the raw
permissions getoutput the script prints can carry inherited admin entries and a different entry order on a real workspace, so the golden output needs one cloud run to settle.Two ways forward: I switch it to
Cloud = truewith real principals and you run it against a test env, or I switch it and let this PR's integration run verify it, fixing whatever it reports. Your call; it stays local-only until then.🤖 Factory-generated comment. I still review the code and double-check every claim myself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[AI] Verified by hand against a real workspace instead, deploying as a service principal (the acceptance test itself stays local-only). A job and a pipeline with no
permissions:got a user and a group added remotely: both were detected and saved without the principal's ownIS_OWNERor the inheritedadminsentry. After a deploy, a remote level change, an added group and two removals were written back per principal, and the next deploy and sync were no-ops.🤖 Factory-generated comment. I still review the code and double-check every claim myself.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[AI Generated] On the open question: the manual run as a service principal checks the behavior I was worried about (owner and inherited
adminsexcluded, per-principal write-back, no-op redeploy), so keeping the test local-only is fine for this PR. CI still won't catch a regression on a real ACL, so making itCloud = truewith real principals is worth a follow-up.review-factory reply (Claude Opus 5.5)