Skip to content

Add experimental LSP server for bundle files#4714

Draft
shreyas-goenka wants to merge 9 commits intomainfrom
cli-dabs-lsp
Draft

Add experimental LSP server for bundle files#4714
shreyas-goenka wants to merge 9 commits intomainfrom
cli-dabs-lsp

Conversation

@shreyas-goenka
Copy link
Copy Markdown
Contributor

@shreyas-goenka shreyas-goenka commented Mar 12, 2026

Summary

Adds an experimental LSP server for Databricks Asset Bundle (DABs) YAML files, invoked via databricks experimental bundle-lsp.

Features:

  • Autocomplete: ${...} interpolation paths including sequence indices (tasks[0]), var shorthand, and computed keys (bundle.target, workspace.current_user.*, bundle.git.*, workspace.root_path, etc.)
  • Go-to-definition: Navigate from ${...} references to their definitions in the merged config tree, and from resource keys to all interpolation references
  • Hover: Rich hover with resource IDs, URLs, and per-target deployment state
  • Document links: Clickable links to open resources in the Databricks workspace
  • Diagnostics: Warnings for unresolvable ${...} references (typos, missing keys)
  • File watching: Auto-reload when bundle configs or deployment state change
  • Multi-target: Hover shows deployment state across all configured targets

Technical details:

  • Uses creachadair/jrpc2 for JSON-RPC over stdio
  • Thread-safe state access with sync.RWMutex (file I/O performed outside the lock)
  • Reuses dyn.Value tree and dynvar.InterpolationRegexp from the existing bundle system
  • Notifications use jrpc2.Notify() (not Callback() which blocks)

This pull request was AI-assisted by Isaac.

shreyas-goenka added a commit to databricks/databricks-vscode that referenced this pull request Mar 12, 2026
Introduces a BundleLSPClient that spawns `databricks bundle lsp` via
stdio to provide deployment-aware features for bundle YAML files:

- Document links: Ctrl+click on resource keys to open in workspace
- Hover: Shows resource ID, name, and workspace link

The client connects to the CLI's new hidden LSP server and activates
for all bundle YAML files (databricks.yml, databricks.yaml, etc.).
Stacks with the existing Red Hat YAML extension.

Depends on: databricks/cli#4714

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@eng-dev-ecosystem-bot
Copy link
Copy Markdown
Collaborator

eng-dev-ecosystem-bot commented Mar 12, 2026

Commit: a27b67f

Run: 22982511783

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ azure windows 2 9 273 777 3:27
💚​ gcp linux 2 9 267 782 4:37
11 interesting tests: 9 SKIP, 2 RECOVERED
Test Name azure windows gcp linux
💚​ TestAccept 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S
💚​ TestAccept/ssh/connection 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
3:44 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:09 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:13 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:08 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

@eng-dev-ecosystem-bot
Copy link
Copy Markdown
Collaborator

eng-dev-ecosystem-bot commented Mar 12, 2026

Commit: 411026e

Run: 23663608224

Env 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
💚​ aws linux 7 10 270 810 5:19
💚​ aws windows 7 10 272 808 4:09
💚​ aws-ucws linux 7 10 366 726 6:25
💚​ aws-ucws windows 7 10 368 724 4:10
💚​ azure linux 1 12 273 808 5:53
💚​ azure windows 1 12 275 806 4:19
💚​ azure-ucws linux 1 12 371 722 6:23
💚​ azure-ucws windows 1 12 373 720 4:25
💚​ gcp linux 1 12 269 811 5:47
💚​ gcp windows 1 12 271 809 4:54
17 interesting tests: 10 SKIP, 7 RECOVERED
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 💚​R 💚​R 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 💚​R 💚​R 💚​R 💚​R
💚​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/postgres_branches/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/update_protected 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_branches/without_branch_id 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_endpoints/recreate 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/postgres_projects/update_display_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/synced_database_tables/basic 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
Top 20 slowest tests (at least 2 minutes):
duration env testname
5:22 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:45 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:36 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:20 gcp linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
3:12 azure linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:08 gcp windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
3:08 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:47 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:46 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:45 aws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:44 aws-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:40 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:39 aws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:39 aws-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:24 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:10 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=terraform
2:05 azure windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:04 azure-ucws linux TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct
2:03 azure-ucws windows TestAccept/bundle/resources/apps/inline_config/DATABRICKS_BUNDLE_ENGINE=direct

@varundeepsaini
Copy link
Copy Markdown
Contributor

this is so cool !!!

shreyas-goenka and others added 9 commits April 13, 2026 15:28
Two proposals:
1. Replace regex-based ${...} parsing with a proper two-mode character scanner
2. Add "did you mean?" suggestions for invalid variable references

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces a hidden `databricks bundle lsp` command that starts a
Language Server Protocol server for bundle YAML files. The server
provides:

- Document links: Ctrl+click on resource keys to open them in the
  Databricks workspace browser
- Hover: Shows resource ID, name, and a link to open in Databricks
  when hovering over resource keys

The server reads deployment state from local state files (both direct
engine and Terraform) to resolve resource IDs and construct workspace
URLs without requiring authentication.

Uses github.com/creachadair/jrpc2 for JSON-RPC 2.0 transport (same
library used by terraform-ls).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Ctrl+click on ${...} references navigates to where the path is defined
- Ctrl+click on resource keys shows all interpolation references (peek view)
- Hover on resource keys shows URLs for all targets (up to 10)
- Merged tree loading supports include file resolution
- var.X shorthand resolves to variables.X definitions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use PathToURI() instead of "file://" + path for cross-platform URI construction
- Replace platform-specific TestURIToPath with TestURIToPathRoundTrip
- Fix gofmt alignment in handler map declarations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The command is now at `databricks experimental bundle-lsp` instead of
`databricks bundle lsp`. This better reflects its experimental status.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add interpolation autocomplete with var shorthand, sequence expansion,
  and computed keys (bundle.target, workspace.current_user.*, etc.)
- Add diagnostic publishing for unresolvable ${...} references
- Fix race condition: add sync.RWMutex to Server for concurrent access
- Fix buildDisplayPath leading-dot bug for ${var} without trailing dot
- Extract duplicate URL-building into populateResourceURLs helper
- Change handleDefinition return type from any to []LSPLocation
- Remove terraform state support (direct engine only)
- Remove redundant file watcher patterns

Co-authored-by: Isaac
Use jrpc2.Notify() instead of jrpc2.Callback() for LSP notifications
like textDocument/publishDiagnostics. Callback sends a request and
blocks waiting for a response, but LSP notifications are fire-and-forget.
This caused the server to hang after sending the first diagnostics
notification. Also run registerFileWatchers in a goroutine since its
Callback blocks waiting for the client response.

Co-authored-by: Isaac
- Fix exhaustive switch for dyn.Kind in classifyValue
- Fix gofmt formatting in protocol.go and server.go
- Fix testifylint: use assert.Positive
- Fix isComputedPath false positive (bundle.targets matching bundle.target)
- Add missing computed keys (workspace.root_path, bundle.git.branch, etc.)
- Add exit notification handler per LSP spec
- Sync integration test handler map with server handler map
- Fix contradictory hover text when resource has ID but no URL
- Remove unused diagnostic severity constants

Co-authored-by: Isaac
- Remove os.Exit from handleExit; use jrpcServer.Stop() instead
- Move all file I/O out of the lock in loadBundleInfo; hold lock
  only to swap the new state
- Export InterpolationRegexp from dynvar and use it in LSP instead of
  duplicating the regex
- Unify computedPrefixes and computedKeys into a single source of truth
  (computedKeys in completion.go, used by diagnostics.go)
- Use bytes.NewReader instead of strings.NewReader(string(data))
- Convert loadMergedTree/loadAllTargetState to standalone functions
- Remove design docs from PR

Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants