Repository navigation
Read the current user from identity/me instead of SCIM Me - #7714
andrewnester wants to merge 27 commits into
Conversation
Tests and acceptance runs need identity/me, the transitive parent groups of the current user, the account group list and group lookups by ID. The fake describes the caller as a service principal for SP tokens and as a user otherwise. Group membership is a set that tests join explicitly (Server.JoinGroup), the parent-groups endpoint lists only those groups, and the group list is paged on page_token, as the account endpoint is. Co-authored-by: Isaac <no-reply@databricks.com>
Callers used SCIM Me, which does not resolve an assumed group. GetCurrentUser now calls GET /api/2.0/identity/me and returns its error when it fails. It retries HTTP 500 responses for up to 30 seconds, as the SCIM call did before; GetCurrentUserOnce makes a single attempt for probes that should fail fast. The principal is mapped onto the iam.User fields that callers already read. identity/me returns no groups. CurrentUserGroups lists the caller's parent group IDs from identity/me/transitive-parent-groups, reads all account groups in one paged list, and maps the IDs to names. That is one list call per 1000 account groups. A group ID missing from the account list is skipped, and a failed call fails the lookup so no partial list is taken as complete. Co-authored-by: Isaac <no-reply@databricks.com>
Every non-test call site that needed the current user now calls the helper, so they all read identity/me and resolve an assumed group the same way. The account-config probe in auth describe uses GetCurrentUserOnce, so a 500 fails at once instead of waiting out the retry window. Co-authored-by: Isaac <no-reply@databricks.com>
identity/me returns no groups, so PopulateCurrentUser reads the current user's groups with iamutil.CurrentUserGroups and sets current_user.groups, once per command. A failed read is a Warning and leaves groups empty. current_user.roles and current_user.entitlements are no longer set, as no IAM-v2 source provides them. Permission checks match a configured CAN_MANAGE group by name against the groups already on current_user, so they make no group requests. Production code no longer reads SCIM for group membership. Co-authored-by: Isaac <no-reply@databricks.com>
Fixtures that stubbed scim/v2/Me now answer identity/me with the same user, and tests that mocked CurrentUser.Me use a testserver workspace for the call that now goes to identity/me. The air tests share withAirIdentityMe for servers that do not assert on identity/me. TestGetWorkspaceAuthStatus asserts a successful status and the test user name against a testserver with auth type pat. Its AuthTypeMismatch check for DATABRICKS_AUTH_TYPE=azure-cli stays in TestGetWorkspaceAuthStatusError, because the SDK keeps an environment auth type over the resolved one, so the pat call would fail and azure-cli cannot authenticate in a unit test. Co-authored-by: Isaac <no-reply@databricks.com>
Adds fake-server tests for a group returned by identity/me, and cloud-enabled tests for auth describe and for bundle validate reading the current user's groups. Request logs now show identity/me where they showed scim/v2/Me, plus GET /api/2.0/identity/me/transitive-parent-groups and GET /api/2.0/identity/groups for bundle commands: one account group list per command, with no per-group lookups. The liteswap check follows the bundle's first principal request to identity/me. Golden changes that are not a request-log substitution: - bundle summary and validate current_user gains active and emails, because the mapped iam.User carries them. Real SCIM Me already returned both, so production output is unchanged; only the testserver's SCIM fake lacked them. - bundle/python experimental-compatibility byte offsets shift by the length of those added fields. Co-authored-by: Isaac <no-reply@databricks.com>
Names the cloud-enabled acceptance tests so the PR's cloud run covers the auth describe path, bundle validate's group read, and two existing bundle tests that read the current user. cloud: TestAccept/cmd/auth/describe/identity-me-cloud TestAccept/bundle/validate/identity-me-cloud TestAccept/bundle/resources/models/basic TestAccept/bundle/local_state_staleness Co-authored-by: Isaac <no-reply@databricks.com>
25eb0f0 to
c476b41
Compare
|
Commit: Regression test reportSelected 20 changed tests (limit=20, 7 not selected) 39 tests: regression=32 unreleased=1 coverage=6 failing=0
TestAccept/air/list: branch ✅ | main (8b7e509) ✅ | latest (v1.27.0) ❌latest (v1.27.0): TestAccept/auth/bundle_default_profile: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/deployment/bind/pipelines/update/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/deployment/bind/pipelines/update/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/python/experimental-compatibility-both-equal/DMS=/PYDAB_VERSION=0.266.0: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/python/experimental-compatibility-both-equal/DMS=/PYDAB_VERSION=current: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): TestAccept/bundle/python/experimental-compatibility-both-equal/DMS=true/PYDAB_VERSION=0.266.0: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/python/experimental-compatibility-both-equal/DMS=true/PYDAB_VERSION=current: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/python/experimental-compatibility/DMS=/PYDAB_VERSION=0.266.0: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/python/experimental-compatibility/DMS=/PYDAB_VERSION=current: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): TestAccept/bundle/python/experimental-compatibility/DMS=true/PYDAB_VERSION=0.266.0: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/python/experimental-compatibility/DMS=true/PYDAB_VERSION=current: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/resources/jobs/request-summary: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/inline-script/databricks-cli/target-is-passed/default/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/inline-script/databricks-cli/target-is-passed/default/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/inline-script/databricks-cli/target-is-passed/from_flag/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/inline-script/databricks-cli/target-is-passed/from_flag/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/scripts/databricks-cli/profile-is-passed/from_flag/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/scripts/databricks-cli/profile-is-passed/from_flag/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/scripts/databricks-cli/target-is-passed/default/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/run/scripts/databricks-cli/target-is-passed/default/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/templates-machinery/helpers-error/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/templates-machinery/helpers-error/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/validate/group-principal/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/validate/group-principal/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): same output as main (8b7e509) TestAccept/bundle/validate/identity-me-cloud/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): TestAccept/bundle/validate/identity-me-cloud/DMS=true: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): TestAccept/bundle/validate/workspace-local-groups/DMS=: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌main (8b7e509): latest (v1.27.0): 5 more failure details omitted, see the full report in the run. |
✅ Integration test reportCommit: 7300c24
Top 8 slowest tests (at least 2 minutes):
|
shreyas-goenka
left a comment
There was a problem hiding this comment.
[AI] Two issues to address before merging: the new endpoint is still disabled in the cloud test workspaces, and resolving memberships through account groups loses workspace-local groups.
Focused tests passed locally for libs/iamutil, bundle/permissions, bundle/config/mutator, and cmd/auth.
|
|
||
| func getIdentityMe(ctx context.Context, w *databricks.WorkspaceClient) (*iam.User, error) { | ||
| var me identityMe | ||
| if err := identityGet(ctx, w, identityMePath, nil, &me); err != nil { |
There was a problem hiding this comment.
[AI] [P1] Complete the endpoint rollout before making this switch.
The cloud test logs show Me API is not enabled. Existing bundle initialization and sync tests now fail, and the integration report shows failures across AWS, Azure, and GCP.
Because this call has no SCIM fallback, upgrading the CLI breaks these commands in workspaces where the endpoint is disabled. Please complete the rollout before shipping this change, or preserve compatibility until the endpoint is available.
There was a problem hiding this comment.
Agreed. This PR stays on hold until the endpoint is enabled everywhere, and it has no SCIM fallback by design. The cloud Integration Tests will stay red until then, and I will re-run them once the rollout completes.
| var groups []iam.ComplexValue | ||
| for _, id := range ids { | ||
| name, ok := names[id] | ||
| if !ok { |
There was a problem hiding this comment.
[AI] [P2] Preserve workspace-local group memberships.
ListGroupsProxyAll lists account groups, so it cannot resolve workspace-local groups such as admins and users. Skipping IDs absent from that list drops memberships that SCIM previously returned.
For example, a bundle granting CAN_MANAGE to the workspace users group will no longer recognize the caller as a member and will emit an incorrect permission recommendation. workspace.current_user.groups also becomes incomplete.
Please retain a way to resolve workspace-local memberships and add a test covering one of these groups.
There was a problem hiding this comment.
Fixed in 2d0673c. The parent-groups response already carries group names for workspace-local groups such as admins and users (checked against a live workspace), so current_user.groups is now built from that single response and the account group list is no longer read. Tests: unit tests in libs/iamutil and bundle/permissions, and the new acceptance test bundle/validate/workspace-local-groups with CAN_MANAGE on admins and users.
asnare
left a comment
There was a problem hiding this comment.
Review limited to labs/project/, lgtm: dead code removal.
rugpanov
left a comment
There was a problem hiding this comment.
I reviewed this at c476b41.
I agree with the open threads on the hard cutover and on workspace-local groups:
#7714 (comment)
#7714 (comment)
I also read the cloud log for job 133964047518. It says "Me API is not enabled." So Integration Tests still fails on this head. It fails because this PR switches to identity/me before the endpoint is enabled in the test workspaces.
I added four inline comments: the assumed-group user name, the missing current_user.name, the 500-retry probes, and a question about the JSON key.
Smaller points:
- Groups are fetched on every bundle command, even on a cache hit.
CurrentUserGroupsshould return beforeListGroupsProxyAllwhen there are no parent groups. - Two comments were removed: the 500-retry note in
GetCurrentUserand the "Only userName is needed" note incmd/air/runlaunch.go. The rule in.agents/rules/comments.mdsays to keep comments in code the author did not write. ctxinanalyzeBundlePermissionsis unused.
| g := me.Group | ||
| return &iam.User{ | ||
| Id: g.GroupId, | ||
| UserName: g.GroupName, |
There was a problem hiding this comment.
For an assumed group, UserName is the group name, and the bundle code treats it as a user. expand_workspace_root.go then uses /Workspace/Users/<group name> as the default root. fix_permissions.go also adds the group as a user_name permission on the bundle resources.
I expect bundle deploy to fail for these callers. I have not run it. Could you reject group principals in bundle commands with a clear error? Or write a group_name permission and a valid root? A bundle test with a group principal would catch this.
There was a problem hiding this comment.
Fixed in 38651b1. Bundle commands now fail early with a clear error when the caller is an assumed group, instead of using the group name as a user. The check is shared with air run. auth describe and other non-bundle commands still work for group principals. Tests: a unit test on the populate mutator and the acceptance test bundle/validate/group-principal.
| return &iam.User{ | ||
| Id: u.UserId, | ||
| UserName: u.Username, | ||
| DisplayName: strings.TrimSpace(u.FullName.GivenName + " " + u.FullName.FamilyName), |
There was a problem hiding this comment.
Name is not set here. SCIM set name.givenName and name.familyName. acceptance/bundle/refschema/output/root.txt still lists those paths as valid references, so bundles that use them lose the value.
Could you set Name from FullName? A unit test on meToUser would keep this from coming back.
There was a problem hiding this comment.
Fixed in 38651b1: Name is set from the given and family names that identity/me returns (nil when both are empty), with a unit test on meToUser.
| return | ||
| } | ||
| _, err = w.CurrentUser.Me(callCtx, iam.MeRequest{}) | ||
| _, err = iamutil.GetCurrentUser(callCtx, w) |
There was a problem hiding this comment.
This probe used to make one call. GetCurrentUser now retries a 500 for up to 30 seconds, so a broken profile waits that long. The same change affects the credential probe in cmd/air/get.go:154 and shell completion in cmd/sync/completion.go:48. GetCurrentUserOnce keeps the old behavior.
Also, acceptance/bundle/templates-machinery/helpers-error/test.toml now returns 400 instead of 500, to avoid the wait. That test no longer covers the retry path. I would keep the 500 stub and accept the wait, or cover the retry in a unit test.
There was a problem hiding this comment.
Fixed in 4135519. The profiles probe, the air credential probe and shell completion now use the single-attempt call, so a broken profile fails at once. The retry path has its own unit test. helpers-error keeps its 400 stub, and its comment says it does not cover the 500 retry.
| var page struct { | ||
| Groups []struct { | ||
| GroupId string `json:"group_id"` | ||
| } `json:"me_transitive_parent_groups"` |
There was a problem hiding this comment.
Question: is the key me_transitive_parent_groups right? The SDK type for the sibling endpoint decodes transitive_parent_groups (service/iamv2/model.go:1159 in v0.200.0). If identity/me returns that key instead, groups is always empty. Which key does the API return?
There was a problem hiding this comment.
Yes. identity/me/transitive-parent-groups has its own response type and returns me_transitive_parent_groups. The SDK type for the sibling endpoint decodes a different key, so the code decodes the Me key directly. The unit tests and the dev-workspace check use that key.
pardis-beikzadeh-db
left a comment
There was a problem hiding this comment.
Review limited to cmd/air/, lgtm: the call sites now use iamutil.GetCurrentUser. The open threads on the rest of the PR still apply.
| func currentUserEmail(ctx context.Context, w *databricks.WorkspaceClient) (string, error) { | ||
| // Only userName is needed; the full response also resolves groups and entitlements. | ||
| me, err := w.CurrentUser.Me(ctx, iam.MeRequest{Attributes: "userName"}) | ||
| me, err := iamutil.GetCurrentUser(ctx, w) |
There was a problem hiding this comment.
Non-blocking: for an assumed group, UserName is the group name, so air run would upload to /Workspace/Users/<group name>. If group principals get rejected for bundles (as suggested on libs/iamutil/current_user.go), the same check would cover this.
There was a problem hiding this comment.
Covered by the same check as the bundle commands (38651b1): air run now rejects a group principal with a clear error.
The parent-groups response already names each group, including workspace-local groups such as admins and users. Reading names from the account group list missed workspace-local groups, so a CAN_MANAGE grant to them did not match. Current user groups now come from that one paged response, with no account group list read. Co-authored-by: Isaac <no-reply@databricks.com>
An assumed group has no user identity, so bundle commands fail early with a clear error instead of using the group name as a user name. air run uses the same check. Other commands, including auth describe, still identify the group. identity/me returns the given and family names, so current_user.user.name is set from them. Co-authored-by: Isaac <no-reply@databricks.com>
These probes only decide a fast answer, so a transient 500 should fail them at once instead of waiting out the 30 second retry used for command startup. Co-authored-by: Isaac <no-reply@databricks.com>
…ions Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Review items addressed in 2d0673c..0b79be3 (merged with current main):
Still open on purpose: the rollout gate. The cloud Integration Tests need the endpoint enabled in the test workspaces, so this stays on hold until then. |
Co-authored-by: Isaac <no-reply@databricks.com>
rugpanov
left a comment
There was a problem hiding this comment.
I re-reviewed this at 411ef59. The items from my earlier review are fixed. Bundle commands and air run reject an assumed group early, current_user.name is set, the probes make a single attempt, and the removed comments are back. Workspace-local groups now come from the parent-groups response, and Integration Tests pass on this head.
Risk: this PR changes the identity source of every bundle command and of about 20 other call sites, and it has no SCIM fallback. A workspace that does not serve identity/me fails every bundle command after the upgrade, so the open rollout thread must close before merge. The comments below are not blocking.
Outside the diff:
acceptance/air/cancel/test.toml:25andacceptance/air/list/test.toml:13: [nit] The comments say that the user comes from the built-inscim/v2/Mehandler. The user now comes from the identity/me handler. Update the two comments.
| // GetWorkspaceHomeDir returns the current user's workspace home folder. | ||
| func GetWorkspaceHomeDir(ctx context.Context, client *databricks.WorkspaceClient) (string, error) { | ||
| me, err := client.CurrentUser.Me(ctx, iam.MeRequest{}) | ||
| me, err := iamutil.GetCurrentUser(ctx, client) |
There was a problem hiding this comment.
[nice to have] GetCurrentUser accepts an assumed group and returns the group name as UserName. The doc of GetUserPrincipal names paths under /Workspace/Users as its use case.
For an assumed group, this function returns /Workspace/Users/<group name>. That folder does not exist, so the SSH setup fails later with an unclear error. The PR description says that SCIM /Me does not resolve an assumed group, so before this PR the command failed at the lookup.
The same pattern is in experimental/ssh/internal/keys/secrets.go:17, libs/sync/path.go:45 and the user_name and short_name helpers in libs/template/helpers.go. Use GetUserPrincipal at these call sites. Also narrow .nextchanges/cli/7714.md: it says that the current-user lookups now work for an assumed group.
There was a problem hiding this comment.
Fixed in e00aa91. The home-folder lookup, the SSH key secret scope, the fuse user metadata, the IDE launch, the Postgres target and the sync path check, and the template user_name, short_name and is_service_principal helpers now use GetUserPrincipal. An assumed group fails with the same error as the bundle commands. Tests: TestGetWorkspaceHomeDirRejectsAssumedGroup (new file workspace_test.go) and TestGetUserPrincipalRejectsAssumedGroup. The .nextchanges/cli entry now says only that databricks auth describe works for an assumed group (2492aaf).
| // Groups are not part of the cached principal: membership can change while the | ||
| // fingerprint stays the same, so they are read on every command. | ||
| var diags diag.Diagnostics | ||
| groups, err := iamutil.CurrentUserGroups(ctx, w) |
There was a problem hiding this comment.
[nice to have] CurrentUserGroups does not retry an HTTP 500, but the identity/me call does. When this call fails, Groups stays empty and the command continues with a warning.
Then PermissionDiagnostics does not see a CAN_MANAGE grant that the user has through a group. When the only CAN_MANAGE grant of the user is through a group, the user gets a second, incorrect recommendation to add their own identity to permissions.
Retry the 500 here, as pollIdentityMe does. Also, do not show the group-based recommendation when the groups could not be read.
There was a problem hiding this comment.
Retry: 5fc2207 retries a 500 on each parent-groups page, with the same helper and timeout as identity/me. Test: TestCurrentUserGroupsRetriesOn500.
Diagnostics: when the groups cannot be read, the permission recommendation to add the identity is not shown (f8acf5e). The flag is on the bundle, not the config, because config is converted between mutators. The warning about the groups stays the only message. Tests: TestPermissionDiagnosticsSkipsRecommendationWhenGroupsUnknown, TestPopulateCurrentUserWarnsWhenGroupsFail.
| // Groups are not part of the cached principal: membership can change while the | ||
| // fingerprint stays the same, so they are read on every command. |
There was a problem hiding this comment.
[Question] Before this PR, the groups came in the cached SCIM response, with the same staleness. Now each bundle command makes one more uncached request, also on a cache hit and when no permission uses group_name. Can the groups be cached with the principal, or read only when a group_name grant exists? The answer sets the request cost of every bundle command.
There was a problem hiding this comment.
Groups are now cached together with the principal, under the same key and invalidation (f8acf5e, final form 86c4ff5). A cache hit makes no identity/me and no parent-groups request. A failed groups read is not cached, so the next command reads them again. current_user.groups is still in bundle validate output. Tests: TestPopulateCurrentUserCachesGroupsWithPrincipal and the acceptance test bundle/validate/workspace-local-groups, whose request log shows the parent-groups call once and none on the second command.
| @@ -0,0 +1 @@ | |||
| * `bundle validate` and `bundle summary` no longer include `current_user.roles` or `current_user.entitlements` in their output. ([#7714](https://github.com/databricks/cli/pull/7714)) | |||
There was a problem hiding this comment.
[nit] The entry names only the removed roles and entitlements. The current_user output of bundle validate and bundle summary also gets the new fields active and emails (see acceptance/bundle/validate/job-references/output.txt). Add them to the entry.
There was a problem hiding this comment.
2492aaf: the bundles entry now also names current_user.active and current_user.emails.
| @@ -0,0 +1,5 @@ | |||
| # The current user belongs to the workspace-local groups admins and users. The | |||
| # parent-groups response names them, so their CAN_MANAGE grant counts as manageable. | |||
| [[Server]] | |||
There was a problem hiding this comment.
[nit] .agents/rules/testing.md says to model API behavior in libs/testserver/, not in per-test [[Server]] stubs. This PR adds group membership to the fake (groupMembers, AddGroups), but only Go tests can set it. Thus the acceptance tests stub the route.
Let acceptance tests set the membership in the fake, for example from the members of groups that GroupsCreate makes. Then remove this stub and the local stub in identity-me-cloud/test.toml.
There was a problem hiding this comment.
4033581: GroupsCreate makes the caller a member of a group whose members list names the caller, so acceptance tests set membership in the fake. The [[Server]] stubs in bundle/validate/workspace-local-groups/test.toml and bundle/validate/identity-me-cloud/test.toml are removed. The workspace-local-groups test creates admins and users through the groups API.
| err=$(mktemp) | ||
| $CLI bundle validate -o json 2>"$err" | jq '.workspace.current_user.groups | type' | ||
| grep -q "Could not read the groups" "$err" && { echo "unexpected groups warning"; exit 1; } || true | ||
| rm -f "$err" |
There was a problem hiding this comment.
[nit] This script does not follow .agents/rules/testing.md. The command has no trace, the negative check uses grep -q ... || true, and stderr goes to a mktemp file. If bundle validate fails, output.txt shows no command line and no exit code.
Wrap the command in trace. Write stderr to LOG.validate, and check it with contains.py '!Could not read the groups'.
There was a problem hiding this comment.
4033581: the command runs under trace, stderr goes to LOG.validate, and contains.py '!Could not read the groups' checks it. No mktemp, no || true.
| require.ErrorContains(t, diags.Error(), "may include: nosuchgroup.") | ||
| } | ||
|
|
||
| // A group the current user belongs to, matched by ID, grants CAN_MANAGE. The report |
There was a problem hiding this comment.
[nit] The comment says that the group is matched by ID. isMemberOfGroup matches the group by its Display name. Change "matched by ID" to "matched by name".
There was a problem hiding this comment.
f8acf5e: the test comment now says "matched by name".
| // identityGet sends a GET to path with the workspace's auth headers and decodes the | ||
| // response into out. | ||
| func identityGet(ctx context.Context, w *databricks.WorkspaceClient, path string, query map[string]any, out any) error { | ||
| c, err := databricksclient.New(w.Config) |
There was a problem hiding this comment.
[nit] identityGet makes a new client with databricksclient.New for each request. Thus each 500 retry and each page of parent groups builds a new client. Make one client in the caller, and use it for all retries and pages.
There was a problem hiding this comment.
5fc2207: one client is built per call and reused for all retries and pages.
…ity client The parent-groups call now retries a 500 the same way identity/me does. Each command builds one HTTP client and reuses it for all retries and pages. Co-authored-by: Isaac <no-reply@databricks.com>
…lder is derived Paths under /Workspace/Users and user names used as Postgres or VS Code identities need a user principal. These callers now use GetUserPrincipal, so an assumed group fails with the same error as the bundle commands instead of building a folder named after the group. Co-authored-by: Isaac <no-reply@databricks.com>
A cache hit now makes no identity/me or parent-groups request. Groups are cached under the same fingerprint as the principal, and a failed groups read is not cached. When the groups could not be read, the permission recommendation to add the identity is withheld, since a grant through a group may already apply. The warning about the groups is the only message. Co-authored-by: Isaac <no-reply@databricks.com>
GroupsCreate makes the caller a member of a new group that lists the caller, so the acceptance tests set workspace-local group membership without a per-test route stub. The identity-me-cloud script now checks the warning with contains.py and traces its command. Co-authored-by: Isaac <no-reply@databricks.com>
…ibe note Co-authored-by: Isaac <no-reply@databricks.com>
cache.Get bypassed the local cache telemetry (attempt, hit, miss), which changed the telemetry of every bundle command. The groups are now read inside GetOrCompute, which keeps the telemetry, and a failed groups read returns an error from the compute function so nothing is stored. The cache key carries a version, so entries written before the groups were cached are not read. Acceptance goldens drop the parent-groups request of a cached command. url_ref has no request left to record for destroy, so print_requests reports the missing log. Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts: # acceptance/bundle/user_agent/output.txt # acceptance/bundle/user_agent/simple/out.requests.summary.json
…quest Co-authored-by: Isaac <no-reply@databricks.com>
|
/review |
There was a problem hiding this comment.
Automated review at 8f71438. Each finding was verified against the code.
Not ready yet: 1 blocking issue(s), tagged [blocking].
This PR replaces SCIM /Me with the IAM v2 identity/me endpoint for resolving the current user, adds a second call to transitive-parent-groups for group membership, and rejects assumed-group principals in commands that need a user home directory. The implementation is thorough and well-tested, with one inconsistency in the psql commands where an assumed group would slip through.
| } | ||
|
|
||
| user, err := w.CurrentUser.Me(ctx, iam.MeRequest{}) | ||
| user, err := iamutil.GetCurrentUser(ctx, w) |
There was a problem hiding this comment.
[blocking] GetCurrentUser accepts an assumed group and passes its group name as the Postgres Username (line 86), which would fail with a confusing database auth error. The sibling code in experimental/postgres/cmd/targeting.go (resolveProvisioned line 117 and resolveAutoscaling line 178) already uses GetUserPrincipal, which rejects a group with a clear message. The same issue applies to psql_provisioned.go:30. Should these two call sites use GetUserPrincipal for consistency?
There was a problem hiding this comment.
Fixed in 4dd8246. psql autoscaling uses GetUserPrincipal as well, so both psql paths reject an assumed group with the same error.
rugpanov
left a comment
There was a problem hiding this comment.
Re-review at 8f71438. Thanks for working through the earlier rounds: the group-principal rejection, Name from identity/me and the single client per call all look good, and CI is green.
[Question] Rollout gate. This is still the one thing that blocks the merge for me. The PR has no fallback to SCIM /Me. I could not confirm that identity/me and its transitive-parent-groups endpoint are served in every workspace. Green Integration Tests show that our test workspaces serve identity/me. They do not show that every customer workspace does. Can you confirm with the identity team that both endpoints are enabled in all workspaces (all clouds, gov and private regions)? If that cannot be confirmed before the release, please add a fallback to SCIM /Me on 404 / ENDPOINT_NOT_FOUND / FEATURE_DISABLED, or put the switch behind an opt-in until the rollout is done. Both shreyas-goenka and I raised this before, and it is still open.
Risk: HIGH. PopulateCurrentUser runs at the start of every bundle command. auth describe, auth profiles, psql, apps import, sync and the template helpers also go through iamutil. There is no flag, so a workspace without identity/me breaks all of them, and rollback needs a new CLI release.
Other points (inline):
- Error on a missing endpoint is not actionable.
- A failed groups call is a warning, so
bundle validate --strictnow fails in CI. - The assumed-group guard is not applied at every call site (
apps import,psql, sync completion). current_useroutput changes more than the changelog says.- One test check was weakened, the paging loop has no bound, plus some nits.
Outside the diff:
- My earlier nit is still open.
acceptance/air/cancel/test.toml:25andacceptance/air/list/test.toml:14still say the user comes from "the built-in scim/v2/Me handler". Both commands now go to identity/me. - FYI:
libs/auth/workspace_id.gostill calls SCIM/Me(throughw.CurrentWorkspaceID) when no workspace ID is set. So the CLI now depends on both endpoints. Your PR body already says this; I mention it only because it matters for the rollout question.
|
|
||
| func getIdentityMe(ctx context.Context, c *identityClient) (*currentPrincipal, error) { | ||
| var me identityMe | ||
| if err := c.get(ctx, identityMePath, nil, &me); err != nil { |
There was a problem hiding this comment.
This is now the only source of the current user for every caller, with no fallback. On a workspace that does not serve identity/me yet, all bundle commands, auth describe, psql, apps import and the template helpers fail. auth profiles also marks valid profiles as invalid. See the rollout question in the review body.
Also, the error is not actionable today. populate_current_user.go returns diag.FromErr(err), and the other callers only add "failed to get current user: %w", so the user sees only the raw server text. If the fallback does not land, please at least detect 404 / ENDPOINT_NOT_FOUND / FEATURE_DISABLED here. Then return an error that says identity/me is not enabled in this workspace and that the user should use an older CLI version for now.
There was a problem hiding this comment.
Done in 4dd8246. A 404, 501, ENDPOINT_NOT_FOUND, FEATURE_DISABLED or NOT_IMPLEMENTED from identity/me now fails with "the identity/me API is not enabled in this workspace, so the current user cannot be read; use an older Databricks CLI version for now", followed by the server message. The bundle path reports that text, and so do auth describe and the other callers, which wrap the error. No SCIM /Me fallback was added. See the issue comment for the rollout.
| user.Groups = cached.Groups | ||
| case groupsErr != nil && errors.Is(err, groupsErr): | ||
| user = principal | ||
| diags = diag.Diagnostics{{ |
There was a problem hiding this comment.
Two things here:
- On
mainthis warning could not happen, because the groups came inline from SCIM/Me.bundle validate --strict(cmd/bundle/validate.go) turns any warning into an error. So iftransitive-parent-groupsreturns 403/404 (endpoint not enabled, or the SP cannot read it), CI that runsvalidate --strictnow fails with "Warnings are treated as errors in strict mode". The failure is also not cached, so every command repeats both calls and prints the warning again. Could this be a log line at debug or info level?CurrentUserGroupsUnknownalready suppresses the permission recommendation. Or skip the warning in strict mode. - Nit: AGENTS.md says to use
diag.Warningffor diagnostics with severity:diag.Warningf("Could not read the groups of the current user: %v", groupsErr). This also removes thefmtimport.
There was a problem hiding this comment.
Changed in 4dd8246. A failed groups read is now logged at info level, not reported as a warning, so bundle validate --strict passes. The permission recommendation is still withheld while the groups are unknown. Failed reads are not cached on purpose: a cached failure would keep the groups missing after the endpoint or the permission is fixed. A command repeats the two identity calls only while the groups read keeps failing.
|
|
||
| // Get current user to filter apps | ||
| me, err := w.CurrentUser.Me(ctx, iam.MeRequest{}) | ||
| me, err := iamutil.GetCurrentUser(ctx, w) |
There was a problem hiding this comment.
This uses GetCurrentUser, which accepts an assumed group. For a group caller the command runs MkdirAll(outputDir) and downloads the app source into it. It writes databricks.yml, and only then does PopulateCurrentUser fail with ErrGroupPrincipal. The directory stays, so a retry fails with "directory ... already exists". GetUserPrincipal here would fail before anything is written.
The same split exists elsewhere. cmd/sync/completion.go:48 (GetCurrentUserOnce) offers /Users/<group>/ and /Repos/<group>/ completions. That is harmless, but the PR body says sync fails early, and only libs/sync/path.go does.
There was a problem hiding this comment.
Fixed in 4dd8246. apps import now uses GetUserPrincipal. The check runs before the output directory is created, so an assumed group fails before anything is written.
| } | ||
|
|
||
| user, err := w.CurrentUser.Me(ctx, iam.MeRequest{}) | ||
| user, err := iamutil.GetCurrentUser(ctx, w) |
There was a problem hiding this comment.
GetCurrentUser accepts an assumed group, so user.UserName here is the group name, and it is passed as the Postgres Username. psql_autoscaling.go:41 has the same pattern. experimental/postgres/cmd/targeting.go uses GetUserPrincipal for the same purpose and rejects the group early. Should psql do the same, so that the two paths behave the same way?
There was a problem hiding this comment.
Fixed in 4dd8246. psql provisioned uses GetUserPrincipal, so an assumed group is rejected before connecting, the same as experimental/postgres targeting.
| @@ -0,0 +1 @@ | |||
| * `bundle validate` and `bundle summary` no longer include `current_user.roles` or `current_user.entitlements` in their output. `current_user` now includes `active` and `emails`. ([#7714](https://github.com/databricks/cli/pull/7714)) | |||
There was a problem hiding this comment.
The current_user shape in bundle validate -o json and in ${workspace.current_user.*} changes more than this entry says:
displayNameis nowgiven + " " + familyfrom identity/me, not the SCIMdisplayName. It is""when both name parts are empty.emailsis built fromuserName.activecomes from the account status.groupsentries have onlyvalueanddisplay, with no$refortype, and they are now transitive.
Also, "now includes active and emails" reads as new fields, but real SCIM /Me already returned them. Only the old fake left them out. Please list the real differences, so that users who reference these fields know what changed.
There was a problem hiding this comment.
Updated in 82dcb5a. The entry now lists the real differences: name holds the given and family names, displayName joins them with a space, emails holds the user name, active comes from the account status, and groups have only value and display and are transitive.
| query["page_token"] = pageToken | ||
| } | ||
| var page struct { | ||
| Groups []parentGroup `json:"me_transitive_parent_groups"` |
There was a problem hiding this comment.
No Cloud = true test checks that the group list is non-empty. identity-me-cloud checks only userName and that the warning is absent, and workspace-local-groups runs only against the fake from this PR. A server-side rename would give an empty group list with no warning. Can a cloud test assert at least one group (for example users)?
There was a problem hiding this comment.
Done in 4dd8246. identity-me-cloud now also asserts that the caller is in the users group. The fake lists the built-in users group for every identity, so the same assertion runs locally.
| // call runs at the start of every bundle command, so an unretried 500 fails the | ||
| // whole command; we close that gap here. | ||
| // identity/me is a read-only endpoint, so retrying any 500 is safe. The Databricks SCIM | ||
| // API intermittently returns "500 TEMPORARILY_UNAVAILABLE" (a condition it otherwise |
There was a problem hiding this comment.
Nit: this comment still uses SCIM's intermittent "500 TEMPORARILY_UNAVAILABLE" as the reason to retry. The call now goes to identity/me, so please update the reason or point to the identity/me behaviour.
There was a problem hiding this comment.
Updated in 4dd8246. The comment says identity/me is read-only and explains why the SDK does not retry a 500 here. The SCIM reference is gone.
| // Only userName is needed; the full response also resolves groups and entitlements. | ||
| me, err := w.CurrentUser.Me(ctx, iam.MeRequest{Attributes: "userName"}) | ||
| // Only userName is needed; the groups are read separately. | ||
| me, err := iamutil.GetUserPrincipal(ctx, w) |
There was a problem hiding this comment.
air run (and air list, through currentUserEmail) now goes to identity/me and fails early for an assumed group. pr-checklist/SKILL.md asks for a .nextchanges/air/ entry for a user-visible change to a databricks air command. Nit: the comment on line 24 ("the groups are read separately") is left over from the old Attributes: "userName" filter. air reads no groups.
|
|
||
| w := cmdctx.WorkspaceClient(ctx) | ||
| me, err := w.CurrentUser.Me(ctx, iam.MeRequest{}) | ||
| me, err := iamutil.GetCurrentUser(ctx, w) |
There was a problem hiding this comment.
On main this branch made a single CurrentUser.Me attempt. Now GetCurrentUser retries 500s for up to 30 s, so if identity/me returns 500, auth describe waits about 30 s before it tries the account fallback or reports the error. The PR body describes only the single-attempt account probe. Was the extra wait intended here?
There was a problem hiding this comment.
Good catch. In 4dd8246 auth describe makes a single attempt again through GetCurrentUserOnce, as on main. The PR body says so.
| @@ -0,0 +1,5 @@ | |||
| # The caller is an assumed group. Bundle commands act as a user, so validate fails | |||
| # early instead of using the group name as a user name. | |||
| [[Server]] | |||
There was a problem hiding this comment.
Nit: .agents/rules/testing.md says "Model API behavior in libs/testserver/, not in per-test [[Server]] response stubs." This stub and acceptance/cmd/auth/describe/identity-me-group/test.toml stub identity/me with two different group bodies. FakeWorkspace.IdentityMe already picks a user or an SP by token prefix, so it could model the assumed-group case in the same way.
There was a problem hiding this comment.
Done in 4dd8246. The assumed-group identity is modelled in libs/testserver through a token prefix, enabled by the IsAssumedGroup test flag. Both tests use the flag instead of [[Server]] stubs.
- Report a workspace without identity/me with an actionable error that says to use an older CLI version for now, in the bundle path and the other callers. - A failed parent-groups read only logs; it no longer warns, so `bundle validate --strict` does not fail on it. The permission recommendation stays withheld. - Reject a repeated next_page_token in the parent-groups paging loop. - Reject assumed-group principals in psql, apps import, air cancel --all, apps run locally and shell completion for remote paths. - auth describe makes a single attempt for the identity/me check. - The fake lists the built-in users group for every identity and can act as an assumed group, so acceptance tests set group membership through the fake instead of per-test stubs. - Strengthen the auth describe fallback test and the cloud group check. cloud: TestAccept/cmd/auth/describe/identity-me-cloud TestAccept/bundle/validate/identity-me-cloud TestAccept/bundle/resources/models/basic TestAccept/bundle/local_state_staleness Co-authored-by: Isaac <no-reply@databricks.com>
… error Co-authored-by: Isaac <no-reply@databricks.com>
|
Replying to the review body of 5991106017 and to the outside-the-diff notes. Rollout: merge stays on hold until the server-side identity/me rollout completes. There is still no SCIM /Me fallback and no opt-in. I have not confirmed with the identity team that identity/me and its transitive-parent-groups endpoint are served in every workspace, on every cloud, including gov and private regions. This comment makes no such claim. I will confirm with the identity team before merging. On the CLI side, a workspace that does not serve the API now gets an actionable error (see the thread on it). Outside the diff: acceptance/air/cancel and acceptance/air/list now say the user comes from the built-in identity/me handler (4dd8246). libs/auth/workspace_id.go still calls SCIM /Me when no workspace ID is set. That is pre-existing and unchanged here. Also: auth profiles only shows a valid or invalid flag, so the new error text is not shown there. I left the table unchanged rather than add a column. |
The merge of main became the head commit, so the tests named for the cloud run were no longer selected. cloud: TestAccept/cmd/auth/describe/identity-me-cloud TestAccept/bundle/validate/identity-me-cloud TestAccept/bundle/resources/models/basic TestAccept/bundle/local_state_staleness Co-authored-by: Isaac <no-reply@databricks.com>
| return false | ||
| // isMemberOfGroup reports whether the current user belongs to the group named name, | ||
| // by matching the name against the groups read in PopulateCurrentUser. | ||
| func isMemberOfGroup(b *bundle.Bundle, name string) bool { |
There was a problem hiding this comment.
Does it work with the nested groups? Not sure about behavior here
Changes
GET /api/2.0/identity/methroughiamutil.GetCurrentUser, replacing the SCIM/Mecall at every non-test call site. Theauth describeaccount probe uses a single-attempt variant.current_user.groupsin bundle config is populated from the caller's transitive parent groups (/api/2.0/identity/me/transitive-parent-groups), which returns group names for account and workspace-local groups such asadminsandusers. A failure of that call is logged, not reported as a warning, sobundle validate --strictstill passes; groups stay empty and the command continues.CAN_MANAGEgroup_nameagainstcurrent_user.groupsby name, with no extra API calls. Bundle commands,air run, and the places that derive a user folder (ssh, sync, templates, postgres targets) fail early with a clear error when the caller is an assumed group, since a group has no user home directory oruser_namepermission. This covers psql,apps import,air cancel --allandapps run-localtoo. When the groups cannot be read, the permission recommendation to add your own identity is withheld, because membership is unknown.current_user.rolesandcurrent_user.entitlementsare no longer set, since identity/me does not return them.bundle validateandbundle summaryoutput drops these fields.auth describeandbundle validate, and local tests cover workspace-local groups and group principals.Why
identity/me is the IAM v2 replacement for SCIM
/Me. It resolves principals SCIM does not, in particular when the caller is an assumed group, and it is the supported path going forward. Group membership moves to the IAM v2 endpoints for the same reason.Rollout notes:
/Me. Workspaces must have the identity/me API enabled before this release. A workspace that does not serve it fails with an error that says so and suggests an older CLI version.auth describe,auth profiles, the air credential check, shell completion) make a single attempt; other callers retry a 500 for up to 30 seconds.libs/authstill uses SCIM/Me, because it reads theX-Databricks-Org-Idresponse header. The generatedcurrent-user mecommand is unchanged.Tests
libs/iamutil(user, service principal and group principals, 500 retry, parent-groups paging, group name resolution),libs/testserver,cmd/auth,cmd/air,bundle/permissions,bundle/config/mutator.cmd/auth/describe/identity-me-*,bundle/validate/identity-me-cloud,bundle/validate/workspace-local-groups,bundle/validate/group-principal,bundle/permissions, and the request goldens. The fake lists the built-inusersgroup for every identity, so validate JSON output now includes it. The full acceptance suite passes.cloud:line runscmd/auth/describe/identity-me-cloud,bundle/validate/identity-me-cloud,bundle/resources/models/basicandbundle/local_state_stalenessagainst a real workspace.This pull request and its description were written by Isaac.