Skip to content

Split access tokens that don't fit in one OS keyring item - #7845

Open
simonfaltum wants to merge 6 commits into
mainfrom
keyring-large-tokens
Open

simonfaltum wants to merge 6 commits into
mainfrom
keyring-large-tokens

Conversation

@simonfaltum

@simonfaltum simonfaltum commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Changes

databricks auth login fails on macOS and Windows with store token: data passed to Set was too big when the OAuth access token doesn't fit in one OS keyring item, and later commands then fail because the new refresh token was never stored. Fixes #7707. A token that doesn't fit is now stored across several items; a token that fits is stored exactly as before.

  • The item under the cache key keeps the token without its access token, plus the part count and the access token's SHA-256. The access token goes into 2048-byte parts under <key>#access-token-<n> in the same databricks-cli service.
  • That item holds the refresh token and is written first, so a refresh whose part write fails still succeeds; a login fails instead and can be run again. If parts are missing or don't match the hash, Lookup reads again for up to about 375 ms, in case another process is writing or deleting them, and then returns the token without an access token, which PersistentAuth.Token() refreshes.
  • Delete, and so auth logout, removes every part, the last one first, so that an interrupted logout leaves parts the next one finds.
  • auth login rejects profile names that contain #access-token- in any letter case (Windows Credential Manager ignores case).
  • On macOS the store doesn't pass go-keyring a secret it is sure to reject, because go-keyring starts security -i before its size check and leaves the process behind.

Why

Windows Credential Manager caps a credential at 2560 bytes, and go-keyring's macOS backend base64-encodes the entry into one security -i line of at most 4096 bytes. One item therefore holds an access token of about 2.8 KB on macOS and 2.4 KB on Windows; the refresh token is only 36 bytes. The tokens in #7707 are about 3,250 bytes.

Decisions to review

  • Released versions (v1.0.0 to v1.27.0) read an empty top-level token from a split entry: not nil, because releases before v1.24 dereference it, and without the refresh token, because they would rotate it and then fail to store the result. They fail with token refresh: cached token has no refresh token, which asks for a new login.
  • Splitting over keeping only the refresh token in the keyring (a refresh on every command, each replacing the refresh token), Apple's Security framework (needs cgo or a stable code signature, no help on Windows), or a plaintext fallback (it would have to move every profile on the machine to plaintext, since reads use the keyring whenever it is reachable).
  • Costs: a large token costs one keyring read per part on every command, and each Delete makes one extra call. Parts stay behind when a token shrinks or an older version logs out, until the next Delete. A profile that an older version saved with #access-token- in its name can collide with another profile's parts.

Tests

  • On every OS, against a fake backend with both caps: the one-item boundary per cap and key length, round trips up to 5,000 bytes, write order on a failed part write, stale and miscounted parts, another process writing or logging out between two reads, an interrupted Delete, size changes on one key, and part errors and timeouts. Also a refresh without an access token and after a failed part write, the reserved name in any case at --profile (acceptance) and at the prompt, and TestDarwinCommandTooBig.
  • Breaking each rule in turn (no split, parts first, no hash check, no second read, Delete keeping parts or starting from the first, a nil or refresh-carrying top-level token, 4096-byte parts, and more) fails a named test.
  • In checkouts of v1.0.0, v1.23.0, v1.26.0 and v1.27.0, a split entry fails as described without a panic or a change to it, and a token that fits is written byte for byte as here.
  • On CI against the real macOS Keychain and Windows Credential Manager (the tests skip unless CI is set): round trips, a token that fits staying one item, size changes, the one-item boundary, a refresh through PersistentAuth, and auth logout (TestKeyringStore_RealBackend_*, TestToken_RefreshesLargeTokenInOSKeyring, TestLogoutRemovesSplitTokenFromOSKeyring). On macOS, TestZalandoBackend_Set_RejectsLargeSecretWithoutSecurityProcess checks that rejecting a large secret leaves no security process behind.
  • Not tested: a workspace that issues tokens over the limit; the tests use made-up tokens.
  • ./task test and lint for linux, darwin and windows pass.

This pull request and its description were written by Isaac.

databricks auth login failed on macOS and Windows with "store token: data
passed to Set was too big" when the OAuth access token did not fit in one
OS keyring item (#7707). The keyring store now splits such a token: the
item under the cache key keeps the token without its access token, the part
count and the access token's SHA-256, and the access token goes into
2048-byte parts under <key>#access-token-<n>. The item with the refresh
token is written first. A missing or mismatched part returns the token
without an access token, so that it is refreshed. Delete removes every
part, and auth login rejects profile names that contain #access-token-.
A token that fits is stored exactly as before.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the auth label Nov 20, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Commit: a83d3a769 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (36fdeb1) latest (v1.27.0)
TestAccept/cmd/auth/login/device-code/non-interactive/errors ✅ ❌ ❌
TestAccept/cmd/auth/login/device-code/non-interactive/errors: branch ✅ | main (36fdeb1) ❌ | latest (v1.27.0) ❌

main (36fdeb1):

acceptance_test.go:1402: Diff:
        --- cmd/auth/login/device-code/non-interactive/errors/output.txt
        +++ /tmp/TestAcceptcmdauthlogindevice-codenon-interactiveerrors643215903/002/output.txt
        @@ -39,83 +39,16 @@
         === Profile name that marks an access token part
         
         >>> musterr [CLI] auth login --host [DATABRICKS_URL] --profile test#access-token-0 --use-device-code
        -Error: profile name "test#access-token-0" must not contain #access-token-, which marks the parts of large access tokens
        +Using a browser on another device, visit:
        +[DATABRICKS_URL]/device/verification
         
        -=== Account host without an account ID
        +And enter the code:
        +WDJB-0001
         
        ->>> musterr [CLI] auth login --host https://accounts.cloud.databricks.test --use-device-code --user-code WDJB-0001 --confirmation-code FTFA-QQKY
        -Warn: [hostmetadata] failed to fetch host metadata for https://accounts.cloud.databricks.test, will skip for 1m0s
        -Error: the command is being run in a non-interactive environment, please specify an account ID using --account-id
        +Once finished, complete the login by running the same command with
        +--user-code WDJB-0001 and --confirmation-code set to the confirmation code
        +provided in your browser.
         
        -=== Profile scoped to resources
        +Unexpected success
         
        ->>> musterr [CLI] auth login --profile resources --use-device-code --user-code WDJB-0001 --confirmation-code FTFA-QQKY
        -Error: --use-device-code cannot be used with profile resources because its login is scoped to resources. Log in without --use-device-code
        -
        -=== No pending login
        -
        ->>> musterr [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0001 --confirmation-code FTFA-QQKY
        -Error: no pending device code login for profile test. Run the same command without --user-code and --confirmation-code to start a new login
        -
        ->>> musterr [CLI] auth login --host [DATABRICKS_URL] --use-device-code --user-code WDJB-0001 --confirmation-code FTFA-QQKY
        -Error: no pending device code login for host [DATABRICKS_URL]. Run the same command without --user-code and --confirmation-code to start a new login
        -
        ->>> [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code
        -
        -=== Wrong confirmation code
        -
        ->>> musterr [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0001 --confirmation-code AAAA-AAAA
        -Error: token exchange: Invalid confirmation code (error code: invalid_request). Check the confirmation code and approve the login if you haven't, then rerun this command. Run the same command without --user-code and --confirmation-code to start a new login if it still fails
        -
        ->>> AGENT=test-agent musterr [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0001 --confirmation-code AAAA-AAAA
        -Error: token exchange: Invalid confirmation code (error code: invalid_request). Check the confirmation code and approve the login if you haven't, then rerun this command. Run the same command without --user-code and --confirmation-code to start a new login if it still fails
        -
        -Note for AI agents (test-agent): ask the user to check the confirmation code
        -and to approve the login if they haven't. Do not approve it yourself
        -
        -=== Rerun with the right confirmation code
        -
        ->>> [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code wdjb-0001 --confirmation-code ftfa-qqky
        -Profile test was successfully saved
        -
        -=== Rerun after the login was completed
        -
        ->>> musterr [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0001 --confirmation-code FTFA-QQKY
        -Error: no pending device code login for profile test. Run the same command without --user-code and --confirmation-code to start a new login
        -
        -=== Device code unknown to the server
        -
        ->>> [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code
        -
        ->>> cp home/.databricks/token-cache.json home/token-cache.pending.json
        -
        ->>> [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0002 --confirmation-code FTFA-QQKY
        -Profile test was successfully saved
        -
        ->>> cp home/token-cache.pending.json home/.databricks/token-cache.json
        -
        ->>> musterr [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0002 --confirmation-code FTFA-QQKY
        -Error: token exchange: Invalid device code (error code: invalid_request). Run the same command without --user-code and --confirmation-code to start a new login
        -
        ->>> cache_keys
        -[
        -  "[DATABRICKS_URL]",
        -  "test"
        -]
        -
        -=== Scopes changed between the steps
        -
        ->>> [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --scopes sql
        -
        ->>> musterr [CLI] auth login --host [DATABRICKS_URL] --profile test --use-device-code --user-code WDJB-0003 --confirmation-code FTFA-QQKY
        -Error: the login for profile test was started with different scopes. Complete it with the same command that started it. Run the same command without --user-code and --confirmation-code to start a new login
        -
        ->>> print_requests.py --oneline //oidc/v1/device/auth //oidc/v1/token
        -{"method": "POST", "path": "/oidc/v1/device/auth", "raw_body": "client_id=databricks-cli&scope=offline_access+all-apis"}
        -{"method": "POST", "path": "/oidc/v1/token", "raw_body": "client_id=databricks-cli&confirmation_code=AAAAAAAA&device_code=device-code-1&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code"}
        -{"method": "POST", "path": "/oidc/v1/token", "raw_body": "client_id=databricks-cli&confirmation_code=AAAAAAAA&device_code=device-code-1&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code"}
        -{"method": "POST", "path": "/oidc/v1/token", "raw_body": "client_id=databricks-cli&confirmation_code=FTFAQQKY&device_code=device-code-1&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code"}
        -{"method": "POST", "path": "/oidc/v1/device/auth", "raw_body": "client_id=databricks-cli&scope=offline_access+all-apis"}
        -{"method": "POST", "path": "/oidc/v1/token", "raw_body": "client_id=databricks-cli&confirmation_code=FTFAQQKY&device_code=device-code-2&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code"}
        -{"method": "POST", "path": "/oidc/v1/token", "raw_body": "client_id=databricks-cli&confirmation_code=FTFAQQKY&device_code=device-code-2&grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code"}
        -{"method": "POST", "path": "/oidc/v1/device/auth", "raw_body": "client_id=databricks-cli&scope=offline_access+sql"}
        +Exit code: 1
        
FAIL

latest (v1.27.0): same output as main (36fdeb1)

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: a83d3a7

Run: 44750166166

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 5:08
💚​ aws-windows-latest-4core-16gb 1 274 18 5:23
💚​ azure linux-2core-8gb 1 271 20 4:54
💚​ azure-windows-latest-4core-16gb 1 273 18 3:55
💚​ gcp linux-2core-8gb 1 272 20 5:44
💚​ gcp-windows-latest-4core-16gb 1 274 18 4:34
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:29 gcp-windows-latest-4core-16gb TestAccept
4:14 aws-windows-latest-4core-16gb TestAccept
3:54 azure-windows-latest-4core-16gb TestAccept

@simonfaltum

Copy link
Copy Markdown
Member Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 6bc858e. Each finding was verified against the code.

No blocking issues found.

This PR adds multi-item keyring storage for OAuth access tokens that exceed OS keyring item size limits on macOS and Windows, fixing databricks auth login failures with store token: data passed to Set was too big. The implementation is careful: the refresh token is written first, a SHA-256 hash guards against torn reads, and tokens that fit in one item stay in the format older CLI versions understand. The code is well tested across both fake and real OS keyring backends, with coverage for size transitions, partial failures, and backward compatibility. No concrete defects were found.

simonfaltum and others added 3 commits November 20, 2026 12:20
Lookup reads a split entry again (25 ms doubling, 5 attempts) when its
access token parts are missing or do not match, so a reader that catches
another process writing or deleting the entry neither refreshes with a
refresh token that process already used up nor restores a deleted login.

Delete counts the parts and deletes them from the last, so a Delete that
fails part way leaves parts that the next one still finds.

A refresh whose part write fails still succeeds when the store kept the new
refresh token: recoverStoreUpdate accepts that partial update, so losing the
access token only costs the next command a refresh.

auth login rejects #access-token- in any letter case, because Windows
Credential Manager ignores case in credential names.

On macOS, zalandoBackend rejects a secret that go-keyring is sure to reject
without calling it, because go-keyring starts `security -i` before its size
check and leaves the process unreaped.

keyringEntry embeds a struct that Entry converts to and from again, so a
field added to Entry cannot be dropped silently. Also: string slicing in
putSplit, doc comments, the service name constant in the new tests, and the
original wording of the comments from #7727 except where they became wrong.

Co-authored-by: Isaac <no-reply@databricks.com>
The keyring store's Put now wraps a failed access token part write in
storage.ErrAccessTokenNotStored, after the entry with the refresh token was
stored. PersistentAuth.refresh treats that error as stored and logs it at
debug level, instead of recoverStoreUpdate guessing the state from the
store's contents, which also matched a Put that wrote nothing when the
server does not rotate refresh tokens. recoverStoreUpdate is back to its
original form.

A split entry now keeps every Entry field, not only the token, so the
compile-time Entry mirror holds for both layouts. The Store and Entry docs
say what the keyring store does differently.

Also: the macOS fd test runs first in its file and requires an equal count,
with a named helper instead of a closure; doc comment and wording fixes.

Co-authored-by: Isaac <no-reply@databricks.com>
Lookup now waits splitLookupDelay when the store sets no retry delay, so a
keyring store built without one still gives a concurrent writer time to
finish, and a test pins that schedule. auth login keeps failing on
storage.ErrAccessTokenNotStored, unlike a refresh: a comment says why. The
new debug line uses libs/log, validateLoginProfileName keeps its original
sentence, and TestDarwinCommandTooBig uses require.

Co-authored-by: Isaac <no-reply@databricks.com>
@simonfaltum
simonfaltum marked this pull request as ready for review November 20, 2026 13:39
@simonfaltum
simonfaltum requested review from a team as code owners November 20, 2026 13:39

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 8ba8b84. Each finding was verified against the code.

No blocking issues found. 2 non-blocking comment(s) below.

This PR splits oversized OAuth access tokens across multiple OS keyring items so that databricks auth login works on macOS and Windows when a token exceeds the single-item limit. The implementation is thorough: it stores the refresh token first for safety, verifies reassembled parts with SHA-256, retries on concurrent writes, and cleans up parts on delete. The test coverage is extensive across both fake and real backends. I found only minor style issues; no correctness bugs survived verification.

// openFileCount returns the number of files this process has open.
func openFileCount(t *testing.T) int {
t.Helper()
entries, err := os.ReadDir("/dev/fd")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] openFileCount reads /dev/fd, which doesn't exist on Windows, but this file's build constraint is darwin || windows. The sole caller skips on non-darwin, so nothing breaks today, but a future Windows test in this file that calls openFileCount would fail at runtime with no compile-time hint. Moving the function (and TestZalandoBackend_Set_RejectsLargeSecretWithoutSecurityProcess) into a //go:build darwin-only file would prevent that.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the test and openFileCount to keyring_darwin_test.go in a83d3a7, so the /dev/fd read only builds on macOS. That file still sorts before the one with TestKeyringSet_RejectsLargeSecret, so the test keeps running first.

RefreshToken: "first",
Expiry: time.Now().Add(-time.Minute),
}})
if err != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This new file uses t.Fatalf/t.Errorf throughout (lines 52, 59, 80, 86, 90, 98, …) instead of require/assert, which the testing rule prescribes. The same applies to the new tests in persistent_auth_test.go (TestToken_RefreshesTokenWithoutAccessToken, TestToken_RefreshesWithRefreshTokenStoredByFailedStoreUpdate). The pre-existing code in persistent_auth_test.go uses the same bare-testing style, so this is locally consistent, but other new test files in this PR (logout_keyring_os_test.go, keyring_size_limit_os_test.go) already use testify. Worth aligning the new file at least, or not — your call.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched the new file to require/assert in a83d3a7. I kept persistent_auth_test.go, including its two new tests, in the t.Fatalf style the rest of that file uses.

…2m keychain test

openFileCount reads /dev/fd, which Windows lacks, so the test and its helper
now live in keyring_darwin_test.go instead of a darwin || windows file. The
new u2m real-keychain test uses require and assert like the other new tests.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

auth login fails on macOS with "store token: data passed to Set was too big" when the access token is over ~3 KB

2 participants