Skip to content

Detect remote permission changes on unmanaged resources in config-remote-sync - #7536

Merged
ilyakuz-db merged 7 commits into
mainfrom
config-remote-sync-unmanaged-permissions
Nov 16, 2026
Merged

ilyakuz-db merged 7 commits into
mainfrom
config-remote-sync-unmanaged-permissions

Conversation

@ilyakuz-db

@ilyakuz-db ilyakuz-db commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Changes

bundle config-remote-sync now writes remote permission changes back to the config:

  • Resources that declare no permissions: the current ACL of each deployed resource is read and added as a permissions: block. The current user's IS_OWNER/CAN_MANAGE is left out because deploy adds it (FixPermissions); an owner transferred to someone else is kept, so the next deploy does not take ownership back. This is an import: the ACL cannot tell grants that existed before the deployment from ones added since.
  • Declared permissions: remote additions, level changes and removals are written back per principal (e.g. permissions[user_name='a'].level) from the plan's existing .permissions comparison. Only entries written in the resource's own config are changed; the current user's entry and bundle-level permissions are left alone, and grants are still skipped.

An ACL that cannot be read leaves that resource's permissions unchanged and is listed in skippedChanges (permissions_permission_denied, or permissions_read_failed with the error), so -o json returns status: partial while the rest still syncs.

The ACL read lives in configsync and uses the existing adapters, so CalculatePlan and deploy are unchanged. FixPermissions now keeps the entry that sets a principal's level as is when it does not change it. Before, it rebuilt every entry without a source location, so config-remote-sync could not find declared entries in the YAML.

Why

Permission changes made in the UI were ignored, so the synced config drifted and the next deploy reverted them.

Tests

Extended config-remote-sync/unmanaged_permissions: imports a job ACL and a pipeline ACL (the pipeline's transferred owner is kept), deploys, then remotely changes a level, adds a group and removes a user, and checks the second sync writes all three and the following deploy is a no-op. A fault-injected 403 on the pipeline ACL checks the partial result and that --select-ids drops skips of unselected resources. Unit tests for declared-permission changes (including an empty remote ACL), the owner filter, resource selection, ID lookup and FixPermissions locations. Ran bundle/... unit tests and the bundle/ acceptance tests locally (the ruff-based pydabs formatting checks could not run: no ruff on the machine); golangci-lint clean.

Also ran it by hand against a real workspace, deploying as a service principal: a job and a pipeline that declare no permissions got a user and a group added remotely, and the sync saved them without the principal's own IS_OWNER or the inherited admins entry. After a deploy, a remote level change, an added group and two removals were written back, and the next deploy and sync were no-ops.

This pull request and its description were written by Isaac.

@ilyakuz-db
ilyakuz-db requested review from a team as code owners November 12, 2026 10:29
@ilyakuz-db
ilyakuz-db requested a review from rugpanov November 12, 2026 10:29
@ilyakuz-db ilyakuz-db added the ai-auto Deprecated: use factory-generated instead. label Nov 12, 2026
@ilyakuz-db ilyakuz-db self-assigned this Nov 12, 2026
@github-actions github-actions Bot added the DABs DABs related issues label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 55397be1a | Workflow run

Regression test report

Selected 3 changed tests (limit=20, 0 not selected)

6 tests: regression=2 unreleased=0 coverage=4 failing=0
test branch main (e1316e5) latest (v1.26.0)
TestAccept/bundle/config-remote-sync/job_fields/DMS= ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/job_fields/DMS=true ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/pipeline_fields/DMS= ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/pipeline_fields/DMS=true ✅ ✅ ✅
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS= ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=true ✅ ❌ ❌
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=: branch ✅ | main (e1316e5) ❌ | latest (v1.26.0) ❌

main (e1316e5):

acceptance_test.go:1374: Diff:
        --- bundle/config-remote-sync/unmanaged_permissions/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncunmanaged_permissionsDMS=790766340/002/output.txt
        @@ -16,107 +16,37 @@
         
         >>> [CLI] bundle config-remote-sync -o json
         {
        -  "status": "partial",
        -  "changes": [
        -    "resources.jobs.my_job"
        -  ],
        -  "skippedChanges": [
        -    {
        -      "resourceKey": "resources.pipelines.my_pipeline",
        -      "reason": "permissions_permission_denied",
        -      "error": "Fault injected by test."
        -    }
        -  ]
        +  "status": "success",
        +  "changes": [],
        +  "skippedChanges": null
         }
         
         === Detect and save changes
        -Detected changes in 2 resource(s):
        +No changes detected.
         
        -Resource: resources.jobs.my_job
        -  permissions: add
         
        -Resource: resources.pipelines.my_pipeline
        -  permissions: add
        -
        -
        -
         === Configuration changes
         
         >>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -14,4 +14,9 @@
        -             num_workers: 1
         
        -+      permissions:
        -+        - level: CAN_MANAGE
        -+          user_name: viewer@example.com
        -+        - level: CAN_MANAGE_RUN
        -+          user_name: admin@example.com
        -   pipelines:
        -     my_pipeline:
        -@@ -20,4 +25,9 @@
        -         - notebook:
        -             path: /Users/{{workspace_user_name}}/notebook
        -+      permissions:
        -+        - level: IS_OWNER
        -+          user_name: owner@example.com
        -+        - group_name: users
        -+          level: CAN_VIEW
        -
        - targets:
        -
         === Sync again: the permissions are now in the config
         No changes detected.
         
         
         === Deploy keeps the remote owner
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Created jobs.my_job.permissions
        -Created pipelines.my_pipeline.permissions
        -Files: 1 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 2 unchanged
        +Files: 0 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
         
         >>> [CLI] permissions get pipelines [MY_PIPELINE_ID]
        -[{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"}]
        +Error: Fault injected by test.
         
        -=== Change a level, add and remove principals remotely
         
        -=== Detect and save changes
        -Detected changes in 1 resource(s):
        +Next steps:
        +  - Verify you have the required permissions for this operation
        +  - Check your identity: databricks auth describe
        +  - Consider setting up a profile: databricks auth login --profile <name>
         
        -Resource: resources.jobs.my_job
        -  permissions[group_name='data-team']: add
        -  permissions[user_name='admin@example.com']: remove
        -  permissions[user_name='viewer@example.com'].level: replace
        -
        -
        -
        -=== Configuration changes
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -15,8 +15,8 @@
        -
        -       permissions:
        --        - level: CAN_MANAGE
        -+        - level: CAN_VIEW
        -           user_name: viewer@example.com
        --        - level: CAN_MANAGE_RUN
        --          user_name: admin@example.com
        -+        - group_name: data-team
        -+          level: CAN_MANAGE_RUN
        -   pipelines:
        -     my_pipeline:
        -
        -=== Deploy and sync again
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 4 unchanged
        -No changes detected.
        -
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -129,3 +39,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 2 deleted
        +
        +Exit code: 1

latest (v1.26.0): same output as main (e1316e5)

TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=true: branch ✅ | main (e1316e5) ❌ | latest (v1.26.0) ❌

main (e1316e5):

acceptance_test.go:1374: Diff:
        --- bundle/config-remote-sync/unmanaged_permissions/output.txt
        +++ /tmp/TestAcceptbundleconfig-remote-syncunmanaged_permissionsDMS=t3643247119/002/output.txt
        @@ -16,107 +16,37 @@
         
         >>> [CLI] bundle config-remote-sync -o json
         {
        -  "status": "partial",
        -  "changes": [
        -    "resources.jobs.my_job"
        -  ],
        -  "skippedChanges": [
        -    {
        -      "resourceKey": "resources.pipelines.my_pipeline",
        -      "reason": "permissions_permission_denied",
        -      "error": "Fault injected by test."
        -    }
        -  ]
        +  "status": "success",
        +  "changes": [],
        +  "skippedChanges": null
         }
         
         === Detect and save changes
        -Detected changes in 2 resource(s):
        +No changes detected.
         
        -Resource: resources.jobs.my_job
        -  permissions: add
         
        -Resource: resources.pipelines.my_pipeline
        -  permissions: add
        -
        -
        -
         === Configuration changes
         
         >>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -14,4 +14,9 @@
        -             num_workers: 1
         
        -+      permissions:
        -+        - level: CAN_MANAGE
        -+          user_name: viewer@example.com
        -+        - level: CAN_MANAGE_RUN
        -+          user_name: admin@example.com
        -   pipelines:
        -     my_pipeline:
        -@@ -20,4 +25,9 @@
        -         - notebook:
        -             path: /Users/{{workspace_user_name}}/notebook
        -+      permissions:
        -+        - level: IS_OWNER
        -+          user_name: owner@example.com
        -+        - group_name: users
        -+          level: CAN_VIEW
        -
        - targets:
        -
         === Sync again: the permissions are now in the config
         No changes detected.
         
         
         === Deploy keeps the remote owner
         Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Created jobs.my_job.permissions
        -Created pipelines.my_pipeline.permissions
        -Files: 1 uploaded, 0 deleted
        -Resources: 2 created, 0 changed, 0 deleted, 2 unchanged
        +Files: 0 uploaded, 0 deleted
        +Resources: 0 created, 0 changed, 0 deleted, 2 unchanged
         
         >>> [CLI] permissions get pipelines [MY_PIPELINE_ID]
        -[{"user_name":"owner@example.com","group_name":null,"level":"IS_OWNER"},{"user_name":null,"group_name":"users","level":"CAN_VIEW"},{"user_name":"[USERNAME]","group_name":null,"level":"CAN_MANAGE"}]
        +Error: Fault injected by test.
         
        -=== Change a level, add and remove principals remotely
         
        -=== Detect and save changes
        -Detected changes in 1 resource(s):
        +Next steps:
        +  - Verify you have the required permissions for this operation
        +  - Check your identity: databricks auth describe
        +  - Consider setting up a profile: databricks auth login --profile <name>
         
        -Resource: resources.jobs.my_job
        -  permissions[group_name='data-team']: add
        -  permissions[user_name='admin@example.com']: remove
        -  permissions[user_name='viewer@example.com'].level: replace
        -
        -
        -
        -=== Configuration changes
        -
        ->>> diff.py databricks.yml.backup databricks.yml
        ---- databricks.yml.backup
        -+++ databricks.yml
        -@@ -15,8 +15,8 @@
        -
        -       permissions:
        --        - level: CAN_MANAGE
        -+        - level: CAN_VIEW
        -           user_name: viewer@example.com
        --        - level: CAN_MANAGE_RUN
        --          user_name: admin@example.com
        -+        - group_name: data-team
        -+          level: CAN_MANAGE_RUN
        -   pipelines:
        -     my_pipeline:
        -
        -=== Deploy and sync again
        -Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
        -Files: 1 uploaded, 0 deleted
        -Resources: 0 created, 0 changed, 0 deleted, 4 unchanged
        -No changes detected.
        -
        -
         >>> [CLI] bundle destroy --auto-approve
         The following resources will be deleted:
           delete resources.jobs.my_job
        @@ -129,3 +39,5 @@
         All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
         
         Destroy: 2 deleted
        +
        +Exit code: 1
        
FAIL

latest (v1.26.0): same output as main (e1316e5)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 55397be

Run: 43896722669

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 276 16 9:52
💚​ aws-windows-latest-4core-16gb 1 272 16 6:13
💚​ azure linux-2core-8gb 1 275 16 11:21
💚​ azure-windows-latest-4core-16gb 1 271 16 4:52
💚​ gcp linux-2core-8gb 1 276 16 11:09
💚​ gcp-windows-latest-4core-16gb 1 272 16 7:59
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:11 gcp-windows-latest-4core-16gb TestAccept
3:09 azure-windows-latest-4core-16gb TestAccept
2:49 aws-windows-latest-4core-16gb TestAccept

@rugpanov
rugpanov removed their request for review November 12, 2026 13:35
@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

The live check in e2-dogfood supports reading unmanaged permissions: three DAB jobs had backend ACLs but no .permissions entry in deployment state. Each had only a direct IS_OWNER grant and inherited admins: CAN_MANAGE, which the current filters exclude.

A few things to address before merging:

  • Existing permissions: we need to sync remote additions and level changes even when the bundle already declares permissions. Currently those nodes are skipped, so subsequent UI edits after the first import are ignored. Can we reuse the existing plan/state comparison for these?
  • Ownership: filtering every IS_OWNER entry can drop a different remote owner. Once we import a permissions block, FixPermissions adds the deploying identity as owner. We should preserve existing ownership.
  • Tests: extend unmanaged_permissions to change a permission level and add another principal remotely after the first import, then verify a second sync captures both. The existing output_no_changes cloud test already covers unchanged backend defaults.

For unmanaged permissions, this imports the current explicit ACL; it cannot distinguish pre-existing grants from changes made since the last deployment. That seems reasonable if documented as import behavior.

Comment thread bundle/configsync/diff.go Outdated
var result []dresources.StatePermission
for _, p := range remote.EmbeddedSlice {
isCurrentUser := p.UserName == currentUser || p.ServicePrincipalName == currentUser
if p.Level == iam.PermissionLevelIsOwner || (p.Level == iam.PermissionLevelCanManage && isCurrentUser) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops IS_OWNER for every principal, so an owner transferred in the UI is lost and the next deploy's FixPermissions reassigns it to the deployer. Drop only the current user's?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] Fixed in b9f3db2: only the current user's IS_OWNER/CAN_MANAGE is dropped now (withoutCurrentUserManage in bundle/configsync/permissions.go), so a transferred owner is imported and the next deploy keeps it, adding CAN_MANAGE for the deployer. The pipeline in unmanaged_permissions covers it.

ilyakuz-db and others added 3 commits November 13, 2026 14:26
…keep other owners

Co-authored-by: Isaac <no-reply@databricks.com>
… permissions, keep the max-level entry location

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db

Copy link
Copy Markdown
Contributor Author

[AI] Fixed in b9f3db2:

  • Existing permissions: remote additions, level changes and removals are now written back per principal from the existing plan/state comparison. Only entries written in the resource's own config are changed; bundle-level ones and the current user's are left alone.
  • Ownership: only the current user's IS_OWNER/CAN_MANAGE is dropped on import, so a transferred owner is kept and FixPermissions adds CAN_MANAGE for the deployer instead.
  • Tests: unmanaged_permissions now deploys after the first import, then changes a level, adds a group and removes a user remotely; the second sync writes all three and the following deploy is a no-op.
  • Import behavior for undeclared permissions is documented in the code and the PR description.

Also moved the ACL read into configsync, so CalculatePlan keeps its signature. FixPermissions now keeps unchanged entries as is (with their source locations), which config-remote-sync needs to find declared entries in the YAML.

Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db ilyakuz-db added the software-factory Deprecated. Use factory-generated instead. label Nov 15, 2026

@Sankalp-Mittal Sankalp-Mittal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI review (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.

continue
}
permissionsKey := resourceKey + ".permissions"
if permissions := plan.Plan[permissionsKey]; permissions != nil && permissions.Action != deployplan.Delete {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: Between a saved import and the next deploy, .permissions plans as Create with no remote state, so a later UI grant shows no changes and that deploy removes it. Can this case read the ACL?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Factory, can you elaborate on this issue? Is it still valid? More likely a niche case and safe to omit for now?

Current PR already improves permission handling significantly so this could be a follow up I guess

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] Yes, it still holds at 55397be, but only in a narrow window: after a sync has imported an unmanaged ACL and the user saved it as a permissions: block, and before the next deploy.

  • In that window the state has no resources.jobs.<key>.permissions entry, so makePlan plans the node as Create and never calls DoRead (the !hasEntry branch in bundle/direct/bundle_plan.go). The entry has no remote state.
  • addDeclaredPermissionsChanges only handles entries with a *PermissionsState remote state, so it skips it. unmanagedPermissionsAdapters skips it too, because the node exists and is not a Delete.
  • So a grant added in the UI during that window shows "No changes". The next deploy's ResourcePermissions.DoCreate calls Permissions.Set with the declared list, which drops that grant.

After that first deploy the node is in state, and the declared path picks up later UI edits again. On main the same thing happens to any hand-written permissions: block before its first deploy, since main skips permissions entirely. So this isn't a regression. The PR only makes the window more likely to come up, because sync now writes the block for you.

I agree it's niche and safe to leave out of this PR. Fix for a follow-up: when a .permissions node is Create but its parent KeepsID(), read the ACL with the existing readPermissions and diff it per principal against the declared entries, the same way the declared path does. Then add an acceptance step: import, save, remote grant before deploy, sync detects it, deploy keeps it. I've requested the follow-up internally.

🤖 Factory-generated comment. I still review the code and double-check every claim myself.

Comment thread bundle/configsync/permissions.go Resolved
@@ -0,0 +1,4 @@
RecordRequests = false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike skip_permissions, this test has no Cloud = true and uses principals that exist only in the fake server, so the import and redeploy never ran against a real ACL. Can it use built-in groups?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

factory, if this is doable, let's verify with a local cloud run

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] Still open at head: unmanaged_permissions/test.toml has no Cloud = true (out.test.toml shows Cloud = false), and the script uses viewer@, admin@, owner@example.com and data-team, which won't exist in a real workspace. Built-in users covers group grants. IS_OWNER needs a real user or service principal, so create a service principal in the script. I can't push or run cloud tests from this checkout; deco env run on your side would settle it.

review-factory reply (Claude Opus 5.5)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] Not done yet: a local cloud run is not possible from the factory's machine, which has no test-workspace environment. Making the test cloud-ready also takes more than built-in groups: the transferred pipeline owner needs a real second principal (e.g. the test service principal, since a group cannot be owner), and the raw permissions get output the script prints can carry inherited admin entries and a different entry order on a real workspace, so the golden output needs one cloud run to settle.

Two ways forward: I switch it to Cloud = true with real principals and you run it against a test env, or I switch it and let this PR's integration run verify it, fixing whatever it reports. Your call; it stays local-only until then.

🤖 Factory-generated comment. I still review the code and double-check every claim myself.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI] Verified by hand against a real workspace instead, deploying as a service principal (the acceptance test itself stays local-only). A job and a pipeline with no permissions: got a user and a group added remotely: both were detected and saved without the principal's own IS_OWNER or the inherited admins entry. After a deploy, a remote level change, an added group and two removals were written back per principal, and the next deploy and sync were no-ops.

🤖 Factory-generated comment. I still review the code and double-check every claim myself.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Generated] On the open question: the manual run as a service principal checks the behavior I was worried about (owner and inherited admins excluded, per-principal write-back, no-op redeploy), so keeping the test local-only is fine for this PR. CI still won't catch a regression on a real ACL, so making it Cloud = true with real principals is worth a follow-up.

review-factory reply (Claude Opus 5.5)

Comment thread bundle/configsync/diff_test.go Resolved
ilyakuz-db and others added 2 commits November 16, 2026 11:28
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@ilyakuz-db
ilyakuz-db added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 630891d Nov 16, 2026
27 checks passed
@ilyakuz-db
ilyakuz-db deleted the config-remote-sync-unmanaged-permissions branch November 16, 2026 16:29
petrapostelnicu-db added a commit that referenced this pull request Nov 17, 2026
Resolve conflicts with #7645 (lifecycle metadata) and #7536 (permissions):
- convertPlanChange applies skipJobRunAs before applyLifecycle.
- applyLifecycle no longer drops a job's whole run_as as a backend default;
  skipJobRunAs already limits it to jobs passed in --sync-run-as-ids.

Also validate --sync-run-as-ids inside PostStateFunc so an invalid value
is reported through WriteResult (status: failed with -o json) and telemetry,
like --select-ids.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik

denik commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

After this PR, the three config-remote-sync cluster policy tests fail on cloud (all clouds). The ACL that gets imported includes the CAN_USE the backend gives the creator of a new cluster policy, so a permissions block is written into databricks.yml. Fix: #7717. (AI agent, harness_auto)

@denik denik added factory-generated and removed software-factory Deprecated. Use factory-generated instead. ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…tabricks#7717)

## Changes

`bundle config-remote-sync` no longer imports the current user's
`CAN_USE` on a cluster policy that declares no permissions. The backend
grants the creator `CAN_USE` on a new policy, so this entry is there
without any remote edit. It is dropped like the deployer's own
`IS_OWNER`/`CAN_MANAGE` on other resources. `withoutCurrentUserManage`
is renamed to `withoutCurrentUserDefaults` and now takes the resource
type.

The testserver now gives the creator `CAN_USE` when a cluster policy is
created, matching the cloud, so local runs cover this case.

## Why

Since databricks#7536, config-remote-sync reads the ACL of deployed resources that
declare no permissions. On cloud, the three `config-remote-sync` cluster
policy tests began reporting `Resource:
resources.cluster_policies.policy permissions: add` and writing this
into databricks.yml:

```
+      permissions:
+        - level: CAN_USE
+          service_principal_name: [USERNAME]
```

After that, the next deploy created
`cluster_policies.policy.permissions`. They fail on AWS, Azure and GCP,
for both DMS variants. The testserver gave a new policy no ACL, which is
why databricks#7536 passed locally.

Decisions:
- The current user's `CAN_USE` on a cluster policy is always dropped,
also when other principals are imported with it. Only workspace admins
create or edit cluster policies, and admins can use every policy
regardless of its ACL. So the grant isn't needed in the config, and
writing it would apply one deployer's grant to other targets and
deployers. As a result, a deploy after such a sync removes the
deployer's direct `CAN_USE` entry, and they keep access as an admin.
- Only cluster policies are affected. Other resource types keep the
existing filter as is.
- The current user is taken to be the creator, so the policy's
`creator_user_name` is not read.

## Tests

- Unit tests for the cluster policy filter, alone and next to other
grants. The existing owner-filter test still passes.
- New acceptance test `config-remote-sync/cluster_policy_permissions`:
the first sync reports no changes. Then a group is granted `CAN_USE`
remotely, and the sync imports only the group. The deploy that follows
applies that ACL, and the plan is a no-op.
- Before the fix, the testserver change makes the three tests fail
locally the same way they fail on cloud. With the fix, the whole local
acceptance suite passes.
- Ran the three tests and the new one on AWS with the fix, both DMS
variants: all pass. The commit message asks CI to run them on every
cloud.

## Dashboard

Failing since the 2026-11-17 nightly (the dashboard has not ingested
that run yet; I will add the error links once it does):
-
[TestAccept/bundle/config-remote-sync/cluster_policy_cross_target](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=true?main=1&pr=1&end=2026-11-18T00:00:00Z&last=2d)
-
[TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=true?main=1&pr=1&end=2026-11-18T00:00:00Z&last=2d)
-
[TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=true?main=1&pr=1&end=2026-11-18T00:00:00Z&last=2d)

```
+Created cluster_policies.policy.permissions
-Resources: 0 created, 1 changed, 0 deleted, 1 unchanged
+Resources: 1 created, 1 changed, 0 deleted, 1 unchanged
```

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 20, 2026
…set (databricks#7494)

## Changes
- `config-remote-sync` now writes a remote job `run_as` into a config
that doesn't set one, unless it names the job's owner (`skipJobRunAs` in
`bundle/configsync/runas.go`). It replaces the blanket
`resources.jobs.*.run_as` skip.
- The owner is the principal a job without `run_as` runs as, per the
Jobs API (`run_as_user_name`): the `IS_OWNER` entry of the job's ACL, or
`creator_user_name` when job access control is off. The ACL comes from:
  - the plan, for declared permissions;
- the unmanaged-permissions read from databricks#7536, which now returns the raw
ACL (the current user's `IS_OWNER`/`CAN_MANAGE` is still dropped before
permissions are written);
  - a direct read, for declared permissions that were never deployed.
- The whole-object `run_as` change is still skipped when:
  - the config sets `run_as` (the plan reports the edit key by key);
  - the remote has no `run_as`;
  - the owner is unknown because the ACL can't be read.
- `applyLifecycle` no longer drops a job's whole `run_as` as a backend
default; `skipJobRunAs` decides it.

## Why
- A run-as change in the Jobs UI never reached a YAML without `run_as`,
which is most bundles.
- The Jobs UI always shows a principal (the owner when `run_as` is
unset) and only saves a different one, so the owner check doesn't drop
UI edits.
- Jobs deployed with terraform carry `run_as` set to the owner
([provider
code](https://github.com/databricks/terraform-provider-databricks/blob/76f95e4/jobs/resource_job.go#L844-L857)),
so they are skipped.
- A `run_as` removed from the config and deployed is written back:
`jobs/reset` without `run_as` keeps the previous value, so the job still
runs as that principal.
- This replaces the `--sync-run-as-ids` flag from an earlier revision,
so the caller needs no new flag.

## Tests
- `config-remote-sync/job_run_as`:
  - remote `run_as` on a job without one → synced;
- remote `run_as` naming the owner → not synced, both with the creator
as owner and after an ownership transfer;
  - `run_as` removed from config and deployed → written back;
- `run_as` added locally, not deployed, then changed remotely → one
key-level change;
  - the kind-switch cases from databricks#7493 are unchanged;
  - a second sync is a no-op.
- `config-remote-sync/terraform_state`: a terraform-pinned `run_as`
naming the owner isn't reported.
- Unit tests for `skipJobRunAs` and `jobOwners`.

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants