Repository navigation
Detect remote permission changes on unmanaged resources in config-remote-sync - #7536
Conversation
|
Commit: Regression test reportSelected 3 changed tests (limit=20, 0 not selected) 6 tests: regression=2 unreleased=0 coverage=4 failing=0
TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=: branch ✅ | main (e1316e5) ❌ | latest (v1.26.0) ❌main (e1316e5): latest (v1.26.0): same output as main (e1316e5) TestAccept/bundle/config-remote-sync/unmanaged_permissions/DMS=true: branch ✅ | main (e1316e5) ❌ | latest (v1.26.0) ❌main (e1316e5): latest (v1.26.0): same output as main (e1316e5) |
✅ Integration test reportCommit: 55397be
Top 3 slowest tests (at least 2 minutes):
|
|
The live check in e2-dogfood supports reading unmanaged permissions: three DAB jobs had backend ACLs but no A few things to address before merging:
For unmanaged permissions, this imports the current explicit ACL; it cannot distinguish pre-existing grants from changes made since the last deployment. That seems reasonable if documented as import behavior. |
| var result []dresources.StatePermission | ||
| for _, p := range remote.EmbeddedSlice { | ||
| isCurrentUser := p.UserName == currentUser || p.ServicePrincipalName == currentUser | ||
| if p.Level == iam.PermissionLevelIsOwner || (p.Level == iam.PermissionLevelCanManage && isCurrentUser) { |
There was a problem hiding this comment.
This drops IS_OWNER for every principal, so an owner transferred in the UI is lost and the next deploy's FixPermissions reassigns it to the deployer. Drop only the current user's?
There was a problem hiding this comment.
[AI] Fixed in b9f3db2: only the current user's IS_OWNER/CAN_MANAGE is dropped now (withoutCurrentUserManage in bundle/configsync/permissions.go), so a transferred owner is imported and the next deploy keeps it, adding CAN_MANAGE for the deployer. The pipeline in unmanaged_permissions covers it.
…keep other owners Co-authored-by: Isaac <no-reply@databricks.com>
… permissions, keep the max-level entry location Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
[AI] Fixed in b9f3db2:
Also moved the ACL read into |
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
left a comment
There was a problem hiding this comment.
🤖 AI review (Claude Fable 5.2 + Claude Opus 5.5, cross-verified). Only points the reviewers agreed on are posted.
| continue | ||
| } | ||
| permissionsKey := resourceKey + ".permissions" | ||
| if permissions := plan.Plan[permissionsKey]; permissions != nil && permissions.Action != deployplan.Delete { |
There was a problem hiding this comment.
Major: Between a saved import and the next deploy, .permissions plans as Create with no remote state, so a later UI grant shows no changes and that deploy removes it. Can this case read the ACL?
There was a problem hiding this comment.
Factory, can you elaborate on this issue? Is it still valid? More likely a niche case and safe to omit for now?
Current PR already improves permission handling significantly so this could be a follow up I guess
There was a problem hiding this comment.
[AI] Yes, it still holds at 55397be, but only in a narrow window: after a sync has imported an unmanaged ACL and the user saved it as a permissions: block, and before the next deploy.
- In that window the state has no
resources.jobs.<key>.permissionsentry, somakePlanplans the node as Create and never callsDoRead(the!hasEntrybranch inbundle/direct/bundle_plan.go). The entry has no remote state. addDeclaredPermissionsChangesonly handles entries with a*PermissionsStateremote state, so it skips it.unmanagedPermissionsAdaptersskips it too, because the node exists and is not a Delete.- So a grant added in the UI during that window shows "No changes". The next deploy's
ResourcePermissions.DoCreatecallsPermissions.Setwith the declared list, which drops that grant.
After that first deploy the node is in state, and the declared path picks up later UI edits again. On main the same thing happens to any hand-written permissions: block before its first deploy, since main skips permissions entirely. So this isn't a regression. The PR only makes the window more likely to come up, because sync now writes the block for you.
I agree it's niche and safe to leave out of this PR. Fix for a follow-up: when a .permissions node is Create but its parent KeepsID(), read the ACL with the existing readPermissions and diff it per principal against the declared entries, the same way the declared path does. Then add an acceptance step: import, save, remote grant before deploy, sync detects it, deploy keeps it. I've requested the follow-up internally.
🤖 Factory-generated comment. I still review the code and double-check every claim myself.
| @@ -0,0 +1,4 @@ | |||
| RecordRequests = false | |||
There was a problem hiding this comment.
Unlike skip_permissions, this test has no Cloud = true and uses principals that exist only in the fake server, so the import and redeploy never ran against a real ACL. Can it use built-in groups?
There was a problem hiding this comment.
factory, if this is doable, let's verify with a local cloud run
There was a problem hiding this comment.
[AI Generated] Still open at head: unmanaged_permissions/test.toml has no Cloud = true (out.test.toml shows Cloud = false), and the script uses viewer@, admin@, owner@example.com and data-team, which won't exist in a real workspace. Built-in users covers group grants. IS_OWNER needs a real user or service principal, so create a service principal in the script. I can't push or run cloud tests from this checkout; deco env run on your side would settle it.
review-factory reply (Claude Opus 5.5)
There was a problem hiding this comment.
[AI] Not done yet: a local cloud run is not possible from the factory's machine, which has no test-workspace environment. Making the test cloud-ready also takes more than built-in groups: the transferred pipeline owner needs a real second principal (e.g. the test service principal, since a group cannot be owner), and the raw permissions get output the script prints can carry inherited admin entries and a different entry order on a real workspace, so the golden output needs one cloud run to settle.
Two ways forward: I switch it to Cloud = true with real principals and you run it against a test env, or I switch it and let this PR's integration run verify it, fixing whatever it reports. Your call; it stays local-only until then.
🤖 Factory-generated comment. I still review the code and double-check every claim myself.
There was a problem hiding this comment.
[AI] Verified by hand against a real workspace instead, deploying as a service principal (the acceptance test itself stays local-only). A job and a pipeline with no permissions: got a user and a group added remotely: both were detected and saved without the principal's own IS_OWNER or the inherited admins entry. After a deploy, a remote level change, an added group and two removals were written back per principal, and the next deploy and sync were no-ops.
🤖 Factory-generated comment. I still review the code and double-check every claim myself.
There was a problem hiding this comment.
[AI Generated] On the open question: the manual run as a service principal checks the behavior I was worried about (owner and inherited admins excluded, per-principal write-back, no-op redeploy), so keeping the test local-only is fine for this PR. CI still won't catch a regression on a real ACL, so making it Cloud = true with real principals is worth a follow-up.
review-factory reply (Claude Opus 5.5)
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Resolve conflicts with #7645 (lifecycle metadata) and #7536 (permissions): - convertPlanChange applies skipJobRunAs before applyLifecycle. - applyLifecycle no longer drops a job's whole run_as as a backend default; skipJobRunAs already limits it to jobs passed in --sync-run-as-ids. Also validate --sync-run-as-ids inside PostStateFunc so an invalid value is reported through WriteResult (status: failed with -o json) and telemetry, like --select-ids. Co-authored-by: Isaac <no-reply@databricks.com>
|
After this PR, the three |
…tabricks#7717) ## Changes `bundle config-remote-sync` no longer imports the current user's `CAN_USE` on a cluster policy that declares no permissions. The backend grants the creator `CAN_USE` on a new policy, so this entry is there without any remote edit. It is dropped like the deployer's own `IS_OWNER`/`CAN_MANAGE` on other resources. `withoutCurrentUserManage` is renamed to `withoutCurrentUserDefaults` and now takes the resource type. The testserver now gives the creator `CAN_USE` when a cluster policy is created, matching the cloud, so local runs cover this case. ## Why Since databricks#7536, config-remote-sync reads the ACL of deployed resources that declare no permissions. On cloud, the three `config-remote-sync` cluster policy tests began reporting `Resource: resources.cluster_policies.policy permissions: add` and writing this into databricks.yml: ``` + permissions: + - level: CAN_USE + service_principal_name: [USERNAME] ``` After that, the next deploy created `cluster_policies.policy.permissions`. They fail on AWS, Azure and GCP, for both DMS variants. The testserver gave a new policy no ACL, which is why databricks#7536 passed locally. Decisions: - The current user's `CAN_USE` on a cluster policy is always dropped, also when other principals are imported with it. Only workspace admins create or edit cluster policies, and admins can use every policy regardless of its ACL. So the grant isn't needed in the config, and writing it would apply one deployer's grant to other targets and deployers. As a result, a deploy after such a sync removes the deployer's direct `CAN_USE` entry, and they keep access as an admin. - Only cluster policies are affected. Other resource types keep the existing filter as is. - The current user is taken to be the creator, so the policy's `creator_user_name` is not read. ## Tests - Unit tests for the cluster policy filter, alone and next to other grants. The existing owner-filter test still passes. - New acceptance test `config-remote-sync/cluster_policy_permissions`: the first sync reports no changes. Then a group is granted `CAN_USE` remotely, and the sync imports only the group. The deploy that follows applies that ACL, and the plan is a no-op. - Before the fix, the testserver change makes the three tests fail locally the same way they fail on cloud. With the fix, the whole local acceptance suite passes. - Ran the three tests and the new one on AWS with the fix, both DMS variants: all pass. The commit message asks CI to run them on every cloud. ## Dashboard Failing since the 2026-11-17 nightly (the dashboard has not ingested that run yet; I will add the error links once it does): - [TestAccept/bundle/config-remote-sync/cluster_policy_cross_target](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/config-remote-sync/cluster_policy_cross_target/DMS=true?main=1&pr=1&end=2026-11-18T00:00:00Z&last=2d) - [TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/config-remote-sync/cluster_policy_remote_addition/DMS=true?main=1&pr=1&end=2026-11-18T00:00:00Z&last=2d) - [TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields](http://go/cli/dashboard/test/acceptance.TestAccept/bundle/config-remote-sync/policy_injected_cluster_fields/DMS=true?main=1&pr=1&end=2026-11-18T00:00:00Z&last=2d) ``` +Created cluster_policies.policy.permissions -Resources: 0 created, 1 changed, 0 deleted, 1 unchanged +Resources: 1 created, 1 changed, 0 deleted, 1 unchanged ``` This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
…set (databricks#7494) ## Changes - `config-remote-sync` now writes a remote job `run_as` into a config that doesn't set one, unless it names the job's owner (`skipJobRunAs` in `bundle/configsync/runas.go`). It replaces the blanket `resources.jobs.*.run_as` skip. - The owner is the principal a job without `run_as` runs as, per the Jobs API (`run_as_user_name`): the `IS_OWNER` entry of the job's ACL, or `creator_user_name` when job access control is off. The ACL comes from: - the plan, for declared permissions; - the unmanaged-permissions read from databricks#7536, which now returns the raw ACL (the current user's `IS_OWNER`/`CAN_MANAGE` is still dropped before permissions are written); - a direct read, for declared permissions that were never deployed. - The whole-object `run_as` change is still skipped when: - the config sets `run_as` (the plan reports the edit key by key); - the remote has no `run_as`; - the owner is unknown because the ACL can't be read. - `applyLifecycle` no longer drops a job's whole `run_as` as a backend default; `skipJobRunAs` decides it. ## Why - A run-as change in the Jobs UI never reached a YAML without `run_as`, which is most bundles. - The Jobs UI always shows a principal (the owner when `run_as` is unset) and only saves a different one, so the owner check doesn't drop UI edits. - Jobs deployed with terraform carry `run_as` set to the owner ([provider code](https://github.com/databricks/terraform-provider-databricks/blob/76f95e4/jobs/resource_job.go#L844-L857)), so they are skipped. - A `run_as` removed from the config and deployed is written back: `jobs/reset` without `run_as` keeps the previous value, so the job still runs as that principal. - This replaces the `--sync-run-as-ids` flag from an earlier revision, so the caller needs no new flag. ## Tests - `config-remote-sync/job_run_as`: - remote `run_as` on a job without one → synced; - remote `run_as` naming the owner → not synced, both with the creator as owner and after an ownership transfer; - `run_as` removed from config and deployed → written back; - `run_as` added locally, not deployed, then changed remotely → one key-level change; - the kind-switch cases from databricks#7493 are unchanged; - a second sync is a no-op. - `config-remote-sync/terraform_state`: a terraform-pinned `run_as` naming the owner isn't reported. - Unit tests for `skipJobRunAs` and `jobOwners`. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Changes
bundle config-remote-syncnow writes remote permission changes back to the config:permissions:block. The current user'sIS_OWNER/CAN_MANAGEis left out because deploy adds it (FixPermissions); an owner transferred to someone else is kept, so the next deploy does not take ownership back. This is an import: the ACL cannot tell grants that existed before the deployment from ones added since.permissions[user_name='a'].level) from the plan's existing.permissionscomparison. Only entries written in the resource's own config are changed; the current user's entry and bundle-level permissions are left alone, and grants are still skipped.An ACL that cannot be read leaves that resource's permissions unchanged and is listed in
skippedChanges(permissions_permission_denied, orpermissions_read_failedwith the error), so-o jsonreturnsstatus: partialwhile the rest still syncs.The ACL read lives in
configsyncand uses the existing adapters, soCalculatePlanand deploy are unchanged.FixPermissionsnow keeps the entry that sets a principal's level as is when it does not change it. Before, it rebuilt every entry without a source location, so config-remote-sync could not find declared entries in the YAML.Why
Permission changes made in the UI were ignored, so the synced config drifted and the next deploy reverted them.
Tests
Extended
config-remote-sync/unmanaged_permissions: imports a job ACL and a pipeline ACL (the pipeline's transferred owner is kept), deploys, then remotely changes a level, adds a group and removes a user, and checks the second sync writes all three and the following deploy is a no-op. A fault-injected 403 on the pipeline ACL checks thepartialresult and that--select-idsdrops skips of unselected resources. Unit tests for declared-permission changes (including an empty remote ACL), the owner filter, resource selection, ID lookup andFixPermissionslocations. Ranbundle/...unit tests and thebundle/acceptance tests locally (the ruff-based pydabs formatting checks could not run: no ruff on the machine); golangci-lint clean.Also ran it by hand against a real workspace, deploying as a service principal: a job and a pipeline that declare no permissions got a user and a group added remotely, and the sync saved them without the principal's own
IS_OWNERor the inheritedadminsentry. After a deploy, a remote level change, an added group and two removals were written back, and the next deploy and sync were no-ops.This pull request and its description were written by Isaac.