Repository navigation
aitools: raise the plugin probe timeout from 5s to 20s - #7472
Conversation
2e276b6 to
848f252
Compare
cc3a109 to
3e0bf18
Compare
✅ Integration test reportCommit: b221ddf
Top 19 slowest tests (at least 2 minutes):
|
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 3e0bf18. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 open design question.
This PR raises the aitools plugin probe timeout from 5s to 20s so that a slow agent CLI cold start isn't reported as unavailable, and adds a changelog fragment. The change is correct and easy to revert, CI is green, and the description lays out the trade-offs. One question is open: whether a timed-out probe should get its own reason, since it is still reported as cli-not-on-path and telemetry can't confirm the fix. Beyond that, there are a few non-blocking cleanups: the stale 5s constant in agents/detect.go, the scope of the constant's comment, the changelog wording, and the PR description template.
Questions
- A probe that times out is still reported as
cli-not-on-path. Everyplugin --helperror maps toReasonCLINotOnPathinplugin.go:228-229, and the killed process returnssignal: killedrather thancontext.DeadlineExceeded, so telemetry records CLI_NOT_ON_PATH instead of the existing TIMEOUT category. Raising the limit moves the cutoff, but nothing will show whether 20s is enough, and a hung agent CLI now costs 20s per plugin agent instead of 5s. Do you want to keep the constant bump as is, add a distinct timeout reason (here or in a follow-up) so the funnel telemetry can confirm the fix, or treat a probe timeout as inconclusive and let the 60s install step decide?
Comments outside the changed lines
- [nit]
.github/PULL_REQUEST_TEMPLATE.md: The description uses## Summary/## Evidence/## Merge Danger..agents/skills/pr-checklist/SKILL.mdasks agent-authored PRs to follow.github/PULL_REQUEST_TEMPLATE.mdexactly (## Changes,## Why,## Tests) and not to add sections such as## Summary. Most recently merged PRs use the template. Could you move Summary into Changes/Why and Evidence into Tests?
| // Agent CLIs are large Node/Rust binaries whose cold start (first run, | ||
| // antivirus scan, slow container disk) can take several seconds, and a probe | ||
| // that times out reports the agent as unavailable, so leave headroom. | ||
| pluginProbeTimeout = 20 * time.Second |
There was a problem hiding this comment.
[nice to have] libs/aitools/agents/detect.go:20 still has its own pluginProbeTimeout = 5 * time.Second for the same <agent> plugin --help probe in (*Agent).ProbePlugin. After this PR, two constants with the same name disagree (5s vs 20s). Only tests call ProbePlugin today. Its doc, though, says it is "meant to be called once, for a selected agent, right before install", so wiring it in later would bring back the 5s false "unavailable" result this PR fixes. Please either raise that constant too, or delete ProbePlugin with its constant and tests so only one probe is left.
There was a problem hiding this comment.
Raised pluginProbeTimeout in libs/aitools/agents/detect.go to 20s as well, with a comment saying it matches the installer probe, so ProbePlugin no longer disagrees if it gets wired in. I kept ProbePlugin rather than deleting it, to keep this PR to the timeout change. (410f869)
| // pluginProbeTimeout bounds the `<agent> plugin --help` capability check. | ||
| pluginProbeTimeout = 5 * time.Second | ||
| // Agent CLIs are large Node/Rust binaries whose cold start (first run, | ||
| // antivirus scan, slow container disk) can take several seconds, and a probe | ||
| // that times out reports the agent as unavailable, so leave headroom. | ||
| pluginProbeTimeout = 20 * time.Second |
There was a problem hiding this comment.
[nit] This comment explains 20s by the cold start of plugin --help. The constant also bounds plugin marketplace list (line 124), where a timeout doesn't mark the agent unavailable. Instead marketplaceRegistered returns true, install doesn't record InstalledMarketplace, and a later uninstall leaves the marketplace registered. Anyone who tunes the probe timeout from this comment would change that behavior without knowing it. Please mention the second use here, or give marketplace list its own constant.
There was a problem hiding this comment.
The comment now also covers plugin marketplace list in marketplaceRegistered: on a timeout the marketplace is assumed already registered, so install does not record it and uninstall leaves it registered. (410f869)
| @@ -0,0 +1 @@ | |||
| * `databricks aitools install` gives a coding agent's CLI 20 seconds instead of 5 to answer the plugin capability check, so a slow first start no longer reports the agent as unavailable. ([#7472](https://github.com/databricks/cli/pull/7472)) | |||
There was a problem hiding this comment.
[nit] Three small things in this fragment. (1) The same probe runs in databricks aitools update when it migrates raw skills to the plugin (cmd/aitools/update.go:253 → InstallPluginForAgent), but the entry names only install. (2) "a slow first start no longer reports the agent as unavailable" is true only for starts under 20s. (3) .agents/skills/pr-checklist/SKILL.md asks for an imperative verb or "<command> now …". Suggested wording: "* databricks aitools install and databricks aitools update now wait up to 20 seconds instead of 5 for a coding agent's CLI to answer the plugin capability check, so a slow first start is less likely to report the agent as unavailable. (#7472)"
There was a problem hiding this comment.
Switched the fragment to your suggested wording. It names install and update, says "less likely", and keeps the PR link. (410f869)
3e0bf18 to
410f869
Compare
848f252 to
90c8726
Compare
410f869 to
88362b5
Compare
90c8726 to
125c8ca
Compare
88362b5 to
679dd3e
Compare
125c8ca to
0148734
Compare
679dd3e to
ca64d39
Compare
0148734 to
7d1dfc8
Compare
ca64d39 to
9d4b058
Compare
|
Replies to the two review-body items: Question 1 (timed-out probe reported as PR description template: I've kept the current format, because every PR in this stack uses the same Summary / Evidence / Merge Danger layout and that keeps them easy to compare. |
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 9d4b058. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 open design question.
This PR raises the aitools plugin probe timeout from 5s to 20s, so a slow agent CLI cold start is less likely to be reported as unavailable. It also updates the unused copy in agents/detect.go and adds a changelog fragment. The change is correct, CI is green, and every point from the earlier review has been addressed or answered. Nothing is blocking. One new non-blocking item: on Windows, npm wrapper CLIs aren't actually bounded by this timeout, because runAgentCmd doesn't use process.WithProcessGroup(). One question remains about the leftover ProbePlugin constant.
Questions
(*Agent).ProbePlugininlibs/aitools/agents/detect.gostill has only test callers, so itspluginProbeTimeoutis a second copy kept equal to the installer's value only by a comment. Do you expectProbePluginto get a production caller? If not, would you delete it with its constant in a follow-up, or haveinstallerandagentsshare one exported constant, so there's a single probe timeout to tune? Keeping both copies as they are is also fine if you'd rather.
Agreeing with existing threads
- [nice to have] A timed-out probe is still reported as cli-not-on-path: #7472 (review)
- [nit] The PR description doesn't use the repo's PR template: #7472 (review)
| // It also bounds `<agent> plugin marketplace list` in marketplaceRegistered, | ||
| // where a timeout means the marketplace is assumed already registered, so | ||
| // install does not record it and uninstall leaves it registered. | ||
| pluginProbeTimeout = 20 * time.Second |
There was a problem hiding this comment.
[nice to have] runAgentCmd (lines 66-70) calls process.Background without process.WithProcessGroup(). That leaves Cmd.WaitDelay at zero, and only the direct child is killed when this timeout fires. On Windows, an npm-installed Codex or Copilot runs as codex.cmd → cmd.exe → node → the native binary, with inherited stdio. If plugin --help hangs, the grandchild keeps the stdout/stderr pipes open and Wait blocks past 20s, possibly indefinitely, so "blocks for up to 20s" doesn't hold there. This predates the PR. Since #7475 already touches runAgentCmd, consider passing process.WithProcessGroup() there, as libs/localenv/uv.go does. It sets WaitDelay and signals the whole process group.
There was a problem hiding this comment.
Agreed it's worth doing, but I'd keep it out of this PR, which only changes the timeout value. Two reasons:
- aitools: typed fetch, state and agent CLI timeout errors #7475 rewrites
runAgentCmd, including this exactprocess.Backgroundcall (it wraps the result in a timeout error). The option belongs there, or in a follow-up on top of it, so the stack doesn't edit the same line twice. - It changes behavior on both platforms, so it deserves its own test. On Unix the timeout would send SIGTERM to the whole group and allow a 10s grace period before SIGKILL, so a hung probe could take up to 30s instead of 20s. On Windows,
WithProcessGroupinlibs/process/group_other.goonly setsWaitDelay. That boundsWaitat timeout + 10s, but grandchildren such as node undercodex.cmdare not killed, so the "blocks for up to 20s" bound still won't hold exactly there.
I'll call out the Windows caveat in the Merge Danger section of the description.
There was a problem hiding this comment.
There was a problem hiding this comment.
Right, it isn't in the stack yet. As I answered above, it stays a follow-up rather than part of this PR, which only changes the timeout value. Agreed that a tracking issue would keep it from getting lost.
7d1dfc8 to
b53927b
Compare
9d4b058 to
149db0e
Compare
b53927b to
14f43cd
Compare
149db0e to
3edb030
Compare
14f43cd to
8e754f4
Compare
9dd6d21 to
7e2cf0e
Compare
3e979d0 to
99db1f7
Compare
7e2cf0e to
2920789
Compare
99db1f7 to
07cf845
Compare
2920789 to
66114a4
Compare
07cf845 to
d82eb6b
Compare
66114a4 to
7cd9908
Compare
d82eb6b to
e9ddafa
Compare
7cd9908 to
fdd8c89
Compare
e9ddafa to
b221ddf
Compare
fdd8c89 to
e10129b
Compare
b221ddf to
78b9500
Compare
e10129b to
5576f2b
Compare
78b9500 to
55c0421
Compare
5576f2b to
151571c
Compare
55c0421 to
5cb043a
Compare
151571c to
64f749d
Compare
`databricks aitools install` runs `<agent> plugin --help` to check that the agent's CLI can install plugins. Agent CLIs are large Node/Rust binaries, and a cold start (first run, antivirus scan, slow container disk) can take several seconds. When the probe times out, the agent is reported as unavailable. ```diff // libs/aitools/installer/plugin.go -pluginProbeTimeout = 5 * time.Second +pluginProbeTimeout = 20 * time.Second ``` The same constant also bounds `<agent> plugin marketplace list`. Changelog fragment: `.nextchanges/cli/aitools-plugin-probe-timeout.md`. Co-authored-by: Isaac <no-reply@databricks.com>
5cb043a to
d27386e
Compare
…he install on it (databricks#7473) Stacked on databricks#7472. Stacked on: aitools: raise the plugin probe timeout from 5s to 20s ## Summary Claude Code's `plugin marketplace update` with no name refreshes every marketplace the user has added, so one unreachable third-party marketplace failed the Databricks plugin install. The refresh now names the marketplace the Databricks plugin is listed in (`claude-plugins-official` for Claude Code), and a failed refresh no longer stops the install or the update: both go on from the local copy of the marketplace and, once that succeeds, warn and record the plugin version as unknown. Codex's `plugin marketplace upgrade` in `aitools update` names the marketplace too. ```diff InstallPluginForAgent (libs/aitools/installer/plugin.go) marketplace add (if absent) - claude plugin marketplace update - on error: remove the marketplace we added, return install-failed + claude plugin marketplace update <marketplace> + on error: keep the error, continue claude plugin install databricks@<marketplace> on error: remove the marketplace we added - return install-failed with the install error + return install-failed with the install error + + "(refreshing the marketplace also failed: <refresh error>)" if the refresh failed - record Version = <ref> + if the refresh failed and the marketplace was not just added by us: + warn that the plugin came from the local copy (with the agent's stderr) + record Version = "" (version unknown) + else record Version = <ref> UpdatePluginForAgent (aitools update) - claude plugin marketplace update + claude plugin marketplace update <recorded marketplace, or the agent's current one> - on error: return install-failed + on error: keep the error, continue claude plugin update ... - codex plugin marketplace upgrade + codex plugin marketplace upgrade <recorded marketplace, or the agent's current one> + on error: same as above codex plugin add ... on update error: return install-failed + + "(refreshing the marketplace also failed: <refresh error>)" if the refresh failed - record Version = <ref> + if the refresh failed and the record does not already hold <ref>: + warn that the plugin came from the local copy, record Version = "" (version unknown) + else record Version = <ref> ``` Changelog fragment: `.nextchanges/cli/aitools-marketplace-refresh.md`. ## Evidence `libs/aitools/installer/plugin_test.go`, run at this head with `plugin.go` temporarily reverted to the parent's version and then restored: - **Before:** ```text TestInstallPluginContinuesWhenRefreshFails FAIL claude-code: install-failed: /usr/bin/claude plugin marketplace update: boom TestInstallPluginRollsBackMarketplaceWhenRefreshAndInstallFail FAIL detail "...marketplace update: refresh boom" does not contain "install boom (refreshing the marketplace also failed: " ``` **After:** ```text TestInstallPluginContinuesWhenRefreshFails PASS (install runs, marketplace kept) TestInstallPluginRollsBackMarketplaceWhenRefreshAndInstallFail PASS (both errors in the detail, marketplace removed) ok github.com/databricks/cli/libs/aitools/installer ``` The existing tests now also check that the refresh command names the marketplace (`claude plugin marketplace update databricks-agent-skills`, `codex plugin marketplace upgrade databricks-agent-skills`) and that a failed refresh leaves the recorded version empty unless we just added the marketplace. `TestInstallPluginContinuesWhenRefreshFails` is a table: a just-added marketplace (version recorded, no warning) and an already registered one (install goes on, no `marketplace remove`, `InstalledMarketplace == false`, empty version, warning with the agent's stderr). `TestUpdatePluginForAgentClaudeOneStep` also covers a record without a marketplace (falls back to the agent's marketplace). The failed install and failed update tests assert that the local-copy warning is not printed. For `aitools update`: `TestUpdatePluginForAgentContinuesWhenRefreshFails` (Claude Code and Codex: the refresh fails, the update still runs), `TestUpdatePluginForAgentReportsRefreshWhenUpdateFails` (both errors in the detail), and `TestUpdateInstalledPluginsRecordsUnknownVersionWhenRefreshFails` (the update succeeds, the state records an empty version). `TestUpdateInstalledPluginsRecordsUnknownVersionWhenRefreshFails` also covers a record that already holds the current release (kept). `TestInstalledPluginAgents` covers plugin records without a version, and `TestUpdateCheckPrintsPluginState` the three `--check` statuses. `TestAgentStatusLabelKeepsUnknownStaleScope` covers the `aitools list` label when one scope has an unknown version and the other is current. At this head: `go build ./...`, unit tests for `libs/aitools/...` and `cmd/aitools/...`, `task test-exp-aitools`, the `cmd/aitools` acceptance tests, lint and format on changed files, `task checks`, and `go vet` for windows and darwin on the aitools packages pass. Not run on Windows or macOS, and not run against a real `claude` binary. ## Merge Danger **Door:** two-way Behavior change: a failing refresh used to fail `aitools install` with an install-failed error and remove the marketplace we had just added. Now the install goes on using the local copy of the marketplace; if that copy is stale and lacks the plugin, the install fails and the error includes both messages. A refresh failure that is followed by a successful install prints a warning with the agent's stderr and a hint to run `databricks aitools update`, and the plugin is recorded as "version unknown" so `aitools update --check` and `aitools list` don't report it as up to date. The warning is printed only after the install succeeds, and not when we just added the marketplace (that copy is fresh, so the version is recorded as usual). `aitools list` now shows "version unknown · update available" when one scope has an unknown version and another is current, instead of the current scope's version. No exit code changes. A refresh that hangs until the 60s command timeout used to end the run; now the install or update runs next and can wait up to another 60s, so on a machine where GitHub hangs instead of failing fast, `aitools install` or `aitools update` can take about a minute longer than before. `aitools update` used to skip the plugin with a warning when the refresh failed; now, for both Claude Code and Codex, it still runs the plugin update and, unless the record already holds the current release, warns (with a hint to rerun `databricks aitools update` later) and records the version as unknown. When the record already holds the current release, it keeps that version and prints no warning. `aitools update --check` shows an unknown version as "version unknown (update may be available)", and `databricks version --check` still counts a plugin recorded without a version as installed, so it does not suggest installing Databricks tooling. If the update itself fails, the plugin is skipped as before and the error includes both messages. Both commands refresh only the one marketplace now. **Blast Radius:** small Only the Claude Code plugin install and update path and the Codex plugin update path in `databricks aitools`. This relies on `claude plugin marketplace update <name>` and `codex plugin marketplace upgrade <name>` accepting a marketplace name; that is checked only through the stubbed commands in the tests here. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
## Release v1.27.0 ### Notable Changes * `databricks aitools install`, `update` and `uninstall` now exit with code 1 when they do nothing they were asked to. `install` fails when it sets up nothing: no coding agent is detected, every agent is skipped or fails to install, or no skill in the release can be installed (also with `--path`). `update` fails when every update or check it attempts fails, and `uninstall` when every removal it attempts fails. Scripts that ran these commands as a best-effort step and relied on exit code 0 should handle the new exit code. ([#7481](#7481)) * Add `file:<path>` selectors to `pipelines run` and pipeline `bundle run`. `--refresh` and `--full-refresh` now use space-separated lists, such as `--refresh "orders payments"`, instead of comma-separated lists. Single names containing spaces now require backticks, e.g. ``--refresh 'table:`order details`'``. ([#7184](#7184)) * The Databricks CLI is now available on PyPI as `databricks`: run it with `uvx databricks`, or install it with `uv tool install databricks`, `pipx install databricks` or `pip install databricks`. The package contains only the prebuilt CLI binary. Uninstall the legacy `databricks-cli` package first, since both install a `databricks` command. ([#7704](#7704), [#7777](#7777)) ### CLI * Add `databricks sandbox fs` commands (`cp`, `ls`, `cat`, `mkdir`, `rm`) to transfer files between your local filesystem and sandboxes using `<sandbox-id>:/path`. ([#7372](#7372)) * `databricks fs` commands retry reads (stat, list, open) and deletes on DBFS and Volumes when the server returns a transient error or does not respond within the HTTP timeout. ([#7556](#7556)) * Retry UC Volumes file operations (`fs cp`, `fs ls`, `fs rm`, bundle uploads to Volumes) when the Files API returns HTTP 408, 429, 502, 503 or 504 (for example "AWS S3 is throttling requests"), or on a transient network error; this regressed in v1.10.0. ([#7559](#7559)) * Stop warning when the host metadata request (`/.well-known/databricks-config`) times out; the CLI already falls back to your config, and the message is still shown with `--debug`. ([#7752](#7752)) * Add `databricks aitools auto-install disable` and `enable` to turn off (or back on) the automatic installation of Databricks AI tools on CLI upgrades. A global-scope `databricks aitools uninstall` also turns it off. ([#7701](#7701)) * Stop printing the Databricks AI tooling install hint on `auth`, `configure`, `current-user me`, `version`, `help`, and shell-completion commands. ([#7452](#7452)) * When Codex, Cursor, GitHub Copilot CLI, OpenCode, Gemini CLI, Antigravity, Pi, Goose or Kiro run the CLI without Databricks skills installed for that agent, the CLI now prints a one-line hint to install them with `databricks aitools install --agents <agent> --scope global`, previously shown only to Claude Code. ([#7586](#7586)) * `databricks aitools install` now rejects an unknown `--agents` name before it shows the scope prompt, instead of after. ([#7469](#7469)) * `databricks aitools install` no longer asks "Proceed?" after the agent picker; the picker selection confirms the plan. The picker title now shows the install scope, its hint says Enter installs, `--skills-only` runs label plugin agents "skills" and pre-check every agent whose config dir exists, and the plan summary after it is reduced to the scope line. ([#7478](#7478)) * `databricks aitools install --output json` reports more specific `error_category` values: `SKILLS_FETCH_FAILED`, `RELEASE_LOOKUP_FAILED`, `RATE_LIMITED`, `MANIFEST_INVALID`, `STATE_INVALID` and `PLUGIN_CLI_TIMEOUT`. A coding agent CLI that does not answer in time is now reported as `timed out after <duration>`, followed by any stderr it wrote before it was stopped. ([#7475](#7475)) * `databricks aitools install` no longer prompts when a coding agent runs it, even in a pseudo-terminal; it uses the non-interactive defaults instead and prints a note saying so. ([#7477](#7477)) * `databricks aitools install` now prints the directory it installed skills into (for example `Installed 1 skill in ~/.databricks/aitools/skills.`), so agents can open the installed `SKILL.md` files without searching for them. ([#7668](#7668)) * `databricks aitools install` now ends a failed install with a step to recover, in the text output and in a new `recovery` field of `--output json`. ([#7480](#7480)) * `databricks aitools install` now installs plugins for the other coding agents even when installing skill files fails. ([#7479](#7479)) * `databricks aitools install` now installs skill files for a detected coding agent whose CLI is not on PATH (for example a Claude Code desktop or IDE install), unless it already has the plugin, says how to switch to the plugin later, and warns when the current project also has the plugin. `databricks aitools update` no longer tries to install the plugin for such an agent, and no longer writes skill files next to a plugin that such an agent already has. ([#7482](#7482)) * Fix `databricks aitools install` waiting a long time on an unreachable or slow workspace while it resolves authentication for telemetry; that wait is now limited to a few seconds. ([#7468](#7468)) * `databricks aitools install`, `update` and `list` now refuse a skills manifest that names a skill or file outside the skills directory, before writing any skill files. A skill whose name or file paths are not portable is skipped with a warning, and `update` keeps its installed copy. ([#7722](#7722)) * `databricks aitools install` and `databricks aitools update` refresh only the marketplace the Databricks plugin is listed in, so another unreachable marketplace no longer blocks them. When that refresh fails, both commands warn, install or update the plugin from the local copy of the marketplace, and record the plugin version as unknown. ([#7473](#7473)) * `databricks aitools install` and `databricks aitools update` now wait up to 20 seconds instead of 5 for a coding agent's CLI to answer the plugin capability check, so a slow first start is less likely to report the agent as unavailable. ([#7472](#7472)) * `databricks aitools install` no longer fails a plugin-only install when the lookup of the latest skills release on GitHub fails or is rate-limited. The plugin is installed and its version is recorded as unknown. ([#7474](#7474)) * `databricks aitools` commands now send usage telemetry, and the CLI records when it recommends installing Databricks AI tools. When the configured workspace is slow or unreachable, `aitools install`, `update` and `uninstall` can take up to about 6 seconds longer, and `list`, `version` and runs that stop early up to about 3 seconds. Set `DATABRICKS_CLI_DISABLE_TELEMETRY=1` to opt out. ([#7467](#7467), [#7469](#7469), [#7470](#7470)) * `databricks aitools install` and `databricks aitools update` now report a coding agent CLI shim (`.cmd` or `.bat`) on Windows that prints its plugin help and then exits with an error as `shim-exited-with-error`, with the command to run to see the error, instead of reporting the CLI as not on PATH. ([#7476](#7476)) * `apps init` no longer suggests installing skills when every detected coding agent already has the Databricks plugin. ([#7467](#7467)) * `databricks auth login` now gives up on the optional workspace lookup after about 5 seconds instead of retrying for about 5 minutes, and explains the fallback to the manual workspace ID prompt. ([#7522](#7522)) * Throttle Claude Code's native plugin install hint separately from the text hint: it prints at most once per 10 minutes per session, so the install dialog still comes up when a subagent, a hook, or a command outside a Claude Code subprocess runs the session's first `databricks` command. ([#7692](#7692)) * Print Claude Code's native plugin install hint alongside the text hint, so Claude Code offers a one-click install of the Databricks plugin. Claude Code now counts as having the tooling only when it loads it: skills only in `~/.databricks/aitools/skills` don't count, and a project- or local-scope plugin counts only when the CLI runs inside that project. `auth token`, `auth docker token` and CLI subprocesses no longer print install hints. ([#7585](#7585)) * Interactive prompts and pagers now work in Git for Windows' bash and other MSYS2 shells when they run under ConPTY (the default since Git for Windows 2.47) or directly in a Windows console. ([#7699](#7699)) * Add the `databricks feedback` command to send feedback about the CLI to Databricks. ([#7724](#7724)) * Fix shell completion in `databricks fs` commands for local paths typed with a leading `./` or repeated separators, which previously offered no suggestions. ([#7679](#7679)) * Fix a panic in `databricks labs` commands when a non-string SDK setting such as `DATABRICKS_DEBUG_HEADERS` or `DATABRICKS_RATE_LIMIT` is set in the environment. ([#7673](#7673)) * Fix a panic in `databricks labs install` for projects whose `labs.yml` has no `install` section. ([#7673](#7673)) * `NO_COLOR` no longer disables prompts, spinners or the list pager; they run without color. ([#7700](#7700)) * Add the Beta `databricks pipelines test` command, which runs the tests of a pipeline, and the `testing` property of pipelines in bundles. ([#7429](#7429)) * Add `databricks sandbox forward`, which serves an app running in a sandbox at a local URL. ([#7395](#7395)) * Bound the whole telemetry upload at exit, including config resolution and authentication, by its timeout, so a stalled auth or telemetry endpoint no longer keeps a command from exiting. ([#7467](#7467)) * `databricks version --check` now suggests `databricks aitools install` on stderr when a coding agent is detected without the Databricks plugin or skills. ([#7471](#7471)) ### AI Runtime * `air get -o json` now reports `PENDING` while the run waits for compute, matching `air list`, and includes the run's `status_message`. ([#7455](#7455)) * Wrap long failure reasons and log lines in the `air list` info and logs panes, keep the MLflow link (labeled with its retry) when a run's last retry failed before starting, and point `air logs` at earlier retries when the last one has no logs. ([#7456](#7456)) * `databricks air logs --node` now reports an out-of-range node index when it can determine the run's node count, instead of looking like a successful run with no logs. ([#7516](#7516)) * Add experiment, duration, accelerator, and dashboard metadata to `air list --output json`. For retried runs, `started_at` now reports the latest attempt, matching `air get`. ([#7568](#7568)) * `air list` shortens MLflow run names and unnamed-run IDs by keeping their end behind a leading `…`, and its columns no longer overflow on wide (for example CJK) text; `air get` shows an unnamed run's full MLflow run ID. ([#7582](#7582)) * Shrink the Experiment and User columns of the interactive `air list` table to fit the terminal width, keeping the end of a shortened experiment name. ([#7583](#7583)) * Show the experiment, accelerators and configuration in the `air list` info pane for runs where it showed `N/A`. ([#7587](#7587)) * Strip terminal control characters from the MLflow run, experiment and user names shown by `air list` and `air get`, so a run name cannot clear or retitle the terminal of whoever lists it. ([#7601](#7601)) * `air logs`, `air get`, and `air run --watch` no longer make an extra credential-check request before fetching the run, and `air get` reuses the configured workspace ID instead of looking it up. ([#7613](#7613)) * `air list` no longer waits on extra current-user requests before listing runs, which makes the default list about 150ms faster. ([#7615](#7615)) * `air list --all-status` requests larger index pages, so it makes far fewer round trips once the server supports `page_size`. ([#7616](#7616)) * Fit the interactive `air list` table to narrow terminals: when the Experiment and User columns are at their floors, shorten Started to `MM-DD HH:MM` and size the MLflow column to its labels, then hide Accelerators, User, MLflow and Progress in that order, naming the hidden columns in the hint line. ([#7629](#7629)) * `air get -o json` now includes `mlflow_run_id` and `mlflow_experiment_id`, so a run can be handed to MLflow-based steps without parsing `mlflow_url`. They identify the MLflow run `mlflow_url` links to, including one from an earlier retry that `mlflow_retry` names. ([#7636](#7636)) * `air get` and the `air list` info pane now show the environment of runs on the Databricks AI environment (`databricks_ai_v*`) instead of `N/A`. ([#7637](#7637)) * `air run` now rejects a `code_source.include_paths` path that selects no files instead of uploading a snapshot without it; glob patterns and pathspec magic are left to git. ([#7643](#7643)) * `air convert-to-dabs` now suggests the `--output-dir` that converts a working-tree workload whose `code_source.root_path` is outside the bundle directory, instead of suggesting `sync.paths`. ([#7644](#7644)) * `air run` now lists a `code_source.root_path` outside a git repository relative to `root_path`, so running from a subdirectory of `root_path` no longer uploads an empty or partial snapshot. ([#7646](#7646)) * `air list --all-status` now fills `--limit` past purged or inaccessible indexed runs. ([#7743](#7743)) * `air run` now warns about missing PyTorch when a pinned snapshot's `include_paths` selects a Python file. ([#7744](#7744)) * `air cancel JOB_RUN_ID` now rejects run IDs that are not AI Runtime runs instead of cancelling unrelated Jobs runs. In `-o json` output, the `failed[].error` text for a run that does not exist or that you cannot view now starts with `failed to get run <id>:` instead of the cancel API error. ([#7569](#7569)) * `air get -o json` now reports `mlflow_retry` when the MLflow link comes from an earlier retry because the run's last retry never reached MLflow. ([#7620](#7620)) * `databricks air run --help` now shows how to install the databricks-ai-runtime coding agent skill. ([#7652](#7652)) * `air run --watch -o json` now reports each lifecycle `STATUS` transition once and never reports a non-terminal state after the run has ended, including when logs are served from MLflow artifacts. ([#7621](#7621)) * `air convert-to-dabs` now explains the `--output-dir` workaround when `code_source.root_path` resolves to the bundle root, and notes when the generated job carries the default `max_retries`. ([#7579](#7579)) * `air run` now refuses to upload a `code_source` snapshot that selects no files (for example when `root_path` is git-ignored) and explains the cause, and `--dry-run` runs the snapshot file selection and reports the file count. ([#7570](#7570)) * Repeat the Jobs run link after live log streaming ends. ([#7571](#7571)) * `air get -o json` includes run details including flat `accelerator_type` and `num_accelerators`. Use `--include-config` to include stored config, which may contain plaintext `env_variables` and parameters; downloads over 1 MiB are omitted, and config-dependent `max_retries` is otherwise null. ([#7624](#7624)) * `air get` and the `air list` info pane no longer show `Max Retries 0` when the Jobs API leaves the task's retry limit out. For a run submitted with `air run`, the limit now comes from the run's uploaded config, shown as `3 (CLI default)` when the config doesn't set `max_retries`; it shows `N/A` when the limit can't be known (job runs, multi-container runs, legacy runs with no explicit value, or an unreadable config). ([#7623](#7623)) * Reorganize `databricks air` help: subcommands are grouped by task, `air run` has examples, and YAML fields are listed under headings, with pool fields marked optional and `command`, `compute.num_accelerators`, and `compute.accelerator_type` correctly marked required. ([#7512](#7512)) * `air run -h config` now lists the environment variables AI Runtime sets on every node (such as `MLFLOW_RUN_ID` and `CODE_SOURCE_PATH`), and `air logs --help` notes that `--download-to` cannot be combined with `--tail` or `--minutes`. ([#7655](#7655)) * Support full workspace accelerator types in `air list` filters while preserving family matching, and reject invalid filter values. ([#7515](#7515)) * `databricks air list --all-status` now warns when the AI Runtime run index is unavailable and it falls back to scanning Jobs runs. That fallback scan is capped at 250 runs, and a stuck index request fails after 15 seconds instead of retrying for minutes. `--all-status --filter user=<your email>` now uses the run index instead of scanning the workspace's full run history. ([#7619](#7619)) * Return structured JSON errors for invalid `air list --filter` values when `-o json` is used. ([#7612](#7612)) * `air list --all-status` no longer fails when a listed run has been purged. ([#7523](#7523)) * **Behavior change:** `databricks air logs` now prints the logs available now and exits, also for an active run, matching `kubectl logs`, `docker logs` and `databricks apps logs`. Pass `--follow` (`-f`) to stream new lines until the run completes; `--tail N --follow` prints the last N lines first. `databricks air run --watch` still submits and streams. ([#7658](#7658)) * `databricks air logs -o json` and `databricks air run --watch -o json` now emit a `WAITING` event, for example `{"type":"WAITING","message":"Waiting for accelerator compute capacity..."}`, when the reported waiting status changes. ([#7695](#7695)) * `databricks air logs` on a run that ended in the last two minutes now waits for the log index to settle before printing, instead of sometimes dropping lines that were indexed late. ([#7663](#7663)) * `air logs` and `air run --watch` now print the waiting status (for example `Waiting for accelerator compute capacity...`) as a line on stderr when stderr is not a terminal, instead of printing nothing until the run produces logs. ([#7657](#7657)) * Route AIR log links to the MLflow Logs page instead of the hidden artifact path. ([#7491](#7491)) * Invalid `compute.accelerator_type` errors and `air run -h config.compute` no longer list the CLI's built-in accelerator types, which could differ from the workspace's. ([#7520](#7520)) * `databricks air --help` now notes that GPU pools are not required to run workloads. ([#7581](#7581)) * `air run` now prints the job run URL with the `?o=<workspace-id>` suffix, matching `air get` and `air list`; `dashboard_url` in `air run -o json` and the `--watch` events gains it too. When `air list` cannot resolve the workspace ID, its links now omit `?o=` instead of ending in `?o=0`. ([#7573](#7573)) * `databricks air run` now reports an unknown workload YAML key by its field path, suggests the closest valid field, and points to `-h config` instead of naming an internal Go type. ([#7574](#7574)) * `air run` (including `--dry-run`) now warns before submitting when the `code_source` imports `torch` but PyTorch is not declared: the environment is the Standard environment (version 5 or later) with no `torch` dependency, so the run may fail after it waits for a GPU. ([#7656](#7656)) ### Bundles * Add `isolation_mode` to the `catalogs` resource, so a catalog can be deployed as `OPEN` or `ISOLATED`. ([#7463](#7463)) * `bundle validate` no longer warns when resource permissions and top-level or target permissions name the same principal. ([#7291](#7291)) * Retry bundle file uploads, reads and deletes when the server does not respond within the HTTP timeout ("request timed out after ... of inactivity"). ([#7521](#7521)) * Show how to recover when `bundle deploy` fails because an app's service principal does not exist or was deleted. ([#7533](#7533)) * Retry transient server errors when creating Lakebase (`postgres_*`) resources in the direct deployment engine. ([#7557](#7557)) * Fix `bundle plan` reporting the `spark_conf` and `custom_tags` the backend derives from `is_single_node: true` as drift on clusters. ([#7577](#7577)) * Stop planning a spurious update for Lakeflow Connect pipelines whose `ingestion_definition.data_staging_options` is filled in by the backend. ([#7603](#7603)) * Retry the root path check in `bundle destroy` on transient errors, so a dropped connection no longer fails the destroy. ([#7597](#7597)) * Use the Lakebase branch when its create request times out but the branch was created, instead of failing the deploy, in the direct deployment engine. ([#7602](#7602)) * Retry transient server errors when deleting resources in the direct deployment engine. ([#7708](#7708)) * Use the Lakebase project when its create request times out but the project was created, instead of failing the deploy with "already exists", in the direct deployment engine. ([#7747](#7747)) * Use the feature, model serving endpoint or dashboard that a create request created despite timing out, instead of failing the deploy, in the direct deployment engine. ([#7757](#7757)) * Use the Lakebase role when its create request fails with a transient error but the role was created, instead of failing the deploy, in the direct deployment engine. ([#7767](#7767)) * Use the MCP service, model service or model provider service that a create request created despite failing with a timeout, a 499 or a reset stream, instead of failing the deploy, in the direct deployment engine. Create recovery for features, serving endpoints and dashboards now also covers 499 and reset streams. ([#7782](#7782)) * Retry app updates and deletes in the direct deployment engine when they fail with 400 BAD_REQUEST because the app compute is `UPDATING`, after waiting for the compute to settle. ([#7606](#7606)) * Enable drift detection for `source_code_path` and `git_source` on apps. ([#6399](#6399)) * `bundle destroy` and `bundle deploy` no longer fail with "internal error, missing in state" when the state holds an empty-ID entry left by an interrupted recreate in an older CLI. ([#7676](#7676)) * Fix a feature whose `full_name` contains uppercase letters being recreated on every deploy. ([#7398](#7398)) * Fix `bundle deployment bind` (and `bundle generate --bind`) failing with `no such node "resources.internal_immutable_snapshots.immutable"` when `workspace.sync_mode` is `immutable`, as reported in [#7306](#7306). ([#7329](#7329)) * Retry creating the deployment lock (without `--force-lock`) when the request fails with a transient server error, times out ("request timed out after ... of inactivity") or loses its connection ("unexpected EOF"). ([#7594](#7594)) * Fix a materialization set member whose `feature_name` contains uppercase letters failing to deploy, or recreating the set on every deploy. ([#7398](#7398)) * Delete the old resource when its name follows a renamed schema. ([#7718](#7718)) * Fix `pipelines dry-run` starting a real run when given a job KEY, and a crash in its shell completion. ([#7672](#7672)) * Rename schemas in place; schemas and volumes follow a renamed catalog or schema. ([#7343](#7343)) * Fix a panic when validating or deploying jobs with local libraries and serverless job clusters. ([#7725](#7725)) * Fix `tgz` artifacts with `git` set producing an empty tarball or failing when the bundle is in a subdirectory of the git repository. ([#7674](#7674)) * Apply `presets.trigger_pause_status` and the development mode pause default to `triggers[]` entries on jobs. ([#7450](#7450)) ### Dependency Updates * Bump dependencies with known vulnerabilities. ([#7750](#7750)) * Update the Go SDK to [v0.198.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.198.0). ([#7530](#7530)) * Update the Go SDK to [v0.199.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.199.0). ([#7584](#7584)) * Update the Go SDK to [v0.200.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.200.0). ([#7653](#7653)) * Update the Go SDK to [v0.201.0](https://github.com/databricks/databricks-sdk-go/releases/tag/v0.201.0). ([#7769](#7769))
Stacked on #7471.
Stacked on: aitools: mention databricks aitools install in version --check
Summary
databricks aitools installruns<agent> plugin --helpto check that the agent's CLI can install plugins. Agent CLIs are large Node/Rust binaries, and a cold start (first run, antivirus scan, slow container disk) can take several seconds. When the probe times out, the agent is reported as unavailable.The same constant also bounds
<agent> plugin marketplace list. Changelog fragment:.nextchanges/cli/aitools-plugin-probe-timeout.md.Evidence
A temporary test (not committed) points
probePluginCLIat a fakeclaudescript that sleeps 7s and then prints usage. Linux, at this PR's head:go test ./libs/aitools/...passes. No committed test pins the timeout value, because such a test would have to wait out the timeout.Merge Danger
Door: two-way
Reverting is a one-line change.
Blast Radius: small
aitools installfor up to 20s at the probe instead of 5s. The same applies toplugin marketplace list, where a timeout makes install assume the marketplace is already registered..cmdshim, andrunAgentCmdkills only the direct child on timeout. A hung grandchild that keeps stdout/stderr open can make the probe wait past 20s. This predates the PR; bounding it withprocess.WithProcessGroup()is left to a follow-up.cli-not-on-pathreason as before.agents.ProbePlugininlibs/aitools/agents/detect.gohas its own copy of the constant, also raised to 20s. It has no non-test callers.This pull request and its description were written by Isaac.