Repository navigation
Conversation
When a lock write fails without a definite answer (timeout, 5xx, cut connection), the server may apply it after the command released the lock. Record such lock IDs locally and, when the next Lock finds a lock with one of them, delete it and create the lock again. aws: TestAccept/bundle/resources/secrets/basic TestAccept/bundle/select/basic Co-authored-by: Isaac <no-reply@databricks.com>
aws: TestAccept/bundle/resources/secrets/basic TestAccept/bundle/select/basic Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
✅ Integration test reportCommit: adaa1fe
Top 3 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 18, 2026 08:48
denik
requested review from
Sankalp-Mittal,
andrewnester,
rclarey and
shreyas-goenka
and removed request for
rclarey
November 18, 2026 08:48
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the deploy, locker changes before approving.
| @@ -0,0 +1 @@ | |||
| * Replace a stale deploy lock that a timed-out lock write left behind after this checkout released the lock, instead of failing the next deploy with "deploy lock acquired by ...". ([#7753](https://github.com/databricks/cli/pull/7753)) | |||
Contributor
There was a problem hiding this comment.
Does it need to go in changelog? Seems to be pretty small internal detail?
| return struct{}{}, locker.filer.Write(ctx, LockFileName, bytes.NewReader(buf), filer.CreateParentDirectories) | ||
| mode = append(mode, filer.OverwriteIfExists) | ||
| } | ||
| write := func() error { |
Contributor
There was a problem hiding this comment.
can it be extracted in a named function instead?
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Lockernotes when a lock write fails without a definite answer from the server: a timeout, a 5xx, a cut connection, or a cancelled request. In other words, any error other than "already exists", permission denied or not found. The server may still apply such a write later.Unlock(even a failed one), or aLockthat fails, adds the locker's ID to a local file,.databricks/bundle/<target>/released-deploy-locks.json, which keeps the last 20 IDs. The file is written to a temporary file and renamed into place.Lockreads the lock. If the lock's ID is on that list, it deletes the lock and creates it again. Only this checkout uses those IDs, and they belong to commands that already finished, so the lock is stale. Locks with any other ID are left alone and still fail with "deploy lock acquired by ...".IsTransientReadcheck and the same 3 attempts as before throughretry.Filer), so they get the same tracking. For that,Lockerkeeps the unretried filer for lock writes (lockFiler).libs/locker:Lockreplaces it;Lockfails, and one attempt lands later: the nextLockreplaces it;Why
In the stalls on 11-17, the workspace files API answered some lock creates (
import-filewithoverwrite=false) only after the CLI's 90s inactivity timeout. The server still applied those creates, after the command that sent them had released the lock or given up. The next command in the same bundle dir then found a lock that nobody would release:bundle deployfailed with the create timeout, and the followingbundle destroyfailed with "deploy lock acquired by" (quality_monitors/create, main, 11-17 15:28Z).retrying after Post .../state/deploy.lock?overwrite=false: request timed out after 1m30s of inactivitywarning earlier in the same test, followed by every later command in the test failing on the lock.#7594 already describes this case as a known edge case. These failures show it is common during a stall, so it needs handling.
Judgment calls:
--force-lock, as it does today.--force-lockbetween our read and our delete, our delete removes their lock. Forced locks already give up exclusive access, so I accepted this.Delaystalls every matching request, so it cannot stall the first create and let a later command through. The unit tests simulate the late write directly.Not changed: error/123 (the lock create still timed out after every retry) and error/1010 (the unlock's
get-statusread timed out after its retries). Those are stalls that outlast the retry budget, with 3 and 1 failures on main.Tests
libs/locker, listed above.bundle/deploy,bundle/destroy,bundle/telemetry,bundle/state,bundle/deploymentall pass, includingbundle/deploy/lock-retryandbundle/deploy/lock-timeout.Dashboard:
Error: Failed to acquire deployment lock: deploy lock acquired by [USERNAME] at [TIMESTAMP] +0000 UTC.afterWarn: retrying after Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/...%2Fstate%2Fdeploy.lock?overwrite=false": request timed out after 1m30s of inactivityExample tests: TestAccept/bundle/resources/grants/volumes/DMS=, TestAccept/bundle/select/basic/DMS=/READPLAN=1, TestAccept/bundle/resources/secrets/basic/DMS=true.
This pull request and its description were written by Isaac.