Skip to content

Replace a stale deploy lock left by a timed-out lock write - #7753

Open
denik wants to merge 2 commits into
mainfrom
denik/stale-deploy-lock
Open

denik wants to merge 2 commits into
mainfrom
denik/stale-deploy-lock

Conversation

@denik

@denik denik commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • Locker notes when a lock write fails without a definite answer from the server: a timeout, a 5xx, a cut connection, or a cancelled request. In other words, any error other than "already exists", permission denied or not found. The server may still apply such a write later.
  • When that happened, Unlock (even a failed one), or a Lock that fails, adds the locker's ID to a local file, .databricks/bundle/<target>/released-deploy-locks.json, which keeps the last 20 IDs. The file is written to a temporary file and renamed into place.
  • When a lock create hits "already exists", Lock reads the lock. If the lock's ID is on that list, it deletes the lock and creates it again. Only this checkout uses those IDs, and they belong to commands that already finished, so the lock is stale. Locks with any other ID are left alone and still fail with "deploy lock acquired by ...".
  • Forced lock writes go through the same retry loop as non-forced ones (same IsTransientRead check and the same 3 attempts as before through retry.Filer), so they get the same tracking. For that, Locker keeps the unretried filer for lock writes (lockFiler).
  • Unit tests in libs/locker:
    • a create times out, the command unlocks, and the late write then lands: the next Lock replaces it;
    • every attempt times out and Lock fails, and one attempt lands later: the next Lock replaces it;
    • a live lock held by someone else is kept, even when the list has entries.

Why

In the stalls on 11-17, the workspace files API answered some lock creates (import-file with overwrite=false) only after the CLI's 90s inactivity timeout. The server still applied those creates, after the command that sent them had released the lock or given up. The next command in the same bundle dir then found a lock that nobody would release:

  • Before #7594: bundle deploy failed with the create timeout, and the following bundle destroy failed with "deploy lock acquired by" (quality_monitors/create, main, 11-17 15:28Z).
  • After Retry the deploy lock create after transient errors #7594, the command that hit the timeout succeeds on retry and releases the lock. The late create lands after that, so the next command fails. All 12 such failures on main between 11-17 15:40Z and 18:10Z (AWS, Azure, GCP) show a retrying after Post .../state/deploy.lock?overwrite=false: request timed out after 1m30s of inactivity warning earlier in the same test, followed by every later command in the test failing on the lock.

#7594 already describes this case as a known edge case. These failures show it is common during a stall, so it needs handling.

Judgment calls:

  • The record is local. Only the process that wrote a lock knows its ID, and only an ID that we know was released can be removed safely. A remote "released" marker would make older CLI versions treat the lock as held. A local record covers the common case of commands running one after another in the same checkout, which is the case CI hits; a deploy from another checkout still needs --force-lock, as it does today.
  • If another client takes the lock with --force-lock between our read and our delete, our delete removes their lock. Forced locks already give up exclusive access, so I accepted this.
  • Stale-lock removal deletes the lock, then creates it again without overwrite, so a deployer that takes the lock in between still wins, and we fail with the usual holder error.
  • No acceptance test: the testserver's Delay stalls every matching request, so it cannot stall the first create and let a later command through. The unit tests simulate the late write directly.

Not changed: error/123 (the lock create still timed out after every retry) and error/1010 (the unlock's get-status read timed out after its retries). Those are stalls that outlast the retry budget, with 3 and 1 failures on main.

Tests

  • New unit tests in libs/locker, listed above.
  • Local acceptance tests bundle/deploy, bundle/destroy, bundle/telemetry, bundle/state, bundle/deployment all pass, including bundle/deploy/lock-retry and bundle/deploy/lock-timeout.

Dashboard:

error snippet
error/132 Error: Failed to acquire deployment lock: deploy lock acquired by [USERNAME] at [TIMESTAMP] +0000 UTC. after Warn: retrying after Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/...%2Fstate%2Fdeploy.lock?overwrite=false": request timed out after 1m30s of inactivity

Example tests: TestAccept/bundle/resources/grants/volumes/DMS=, TestAccept/bundle/select/basic/DMS=/READPLAN=1, TestAccept/bundle/resources/secrets/basic/DMS=true.

This pull request and its description were written by Isaac.

When a lock write fails without a definite answer (timeout, 5xx, cut
connection), the server may apply it after the command released the lock.
Record such lock IDs locally and, when the next Lock finds a lock with one
of them, delete it and create the lock again.

aws: TestAccept/bundle/resources/secrets/basic TestAccept/bundle/select/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto Deprecated: use factory-generated instead. label Nov 18, 2026
aws: TestAccept/bundle/resources/secrets/basic TestAccept/bundle/select/basic

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 18, 2026
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: adaa1fe

Run: 44266632702

Env 🤯​MISS 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 280 19 5:01
💚​ aws-windows-latest-4core-16gb 1 282 17 3:58
🤯​ azure linux-2core-8gb 2 1 271 19 4:03
🤯​ azure-windows-latest-4core-16gb 2 1 273 17 2:51
🤯​ gcp linux-2core-8gb 2 1 272 19 3:59
🤯​ gcp-windows-latest-4core-16gb 2 1 274 17 3:26
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🤯​ TestAccept/bundle/resources/secrets/basic ✅​p ✅​p 🤯​M 🤯​M 🤯​M 🤯​M
🤯​ TestAccept/bundle/select/basic ✅​p ✅​p 🤯​M 🤯​M 🤯​M 🤯​M
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:22 gcp-windows-latest-4core-16gb TestAccept
3:04 aws-windows-latest-4core-16gb TestAccept
2:47 azure-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 18, 2026 08:48
@denik
denik requested review from a team as code owners November 18, 2026 08:48
@denik
denik requested review from Sankalp-Mittal, andrewnester, rclarey and shreyas-goenka and removed request for rclarey November 18, 2026 08:48

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me — no issues found.

@andrewnester andrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking findings from a full pass; I'll take a manual look at the deploy, locker changes before approving.

@@ -0,0 +1 @@
* Replace a stale deploy lock that a timed-out lock write left behind after this checkout released the lock, instead of failing the next deploy with "deploy lock acquired by ...". ([#7753](https://github.com/databricks/cli/pull/7753))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it need to go in changelog? Seems to be pretty small internal detail?

Comment thread libs/locker/locker.go
return struct{}{}, locker.filer.Write(ctx, LockFileName, bytes.NewReader(buf), filer.CreateParentDirectories)
mode = append(mode, filer.OverwriteIfExists)
}
write := func() error {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be extracted in a named function instead?

@denik denik added factory-generated and removed ai-auto Deprecated: use factory-generated instead. labels Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues factory-generated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants