Skip to content

aitools: Skip the apps init skills tip when Codex and Copilot have the plugin recorded - #7830

Merged
anton-107 merged 3 commits into
mainfrom
apps-init-tip-recorded-plugins
Nov 20, 2026
Merged

anton-107 merged 3 commits into
mainfrom
apps-init-tip-recorded-plugins

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

databricks apps init decides whether to recommend Databricks skills by calling agents.HasDatabricksToolingInstalled. With this change, that check also uses the plugins recorded in the CLI install state (installer.InstalledPluginAgents). It is used for agents that have no plugin manifest the CLI can read, which today means Codex and Copilot.

 for each detected agent:
   if agent has no plugin manifest reader (Codex, Copilot):
-    return false
+    if plugin not recorded in the CLI install state:
+      return false
+    continue
   if databricks plugin not active for the working directory:
     return false
 return true
  • HasDatabricksToolingInstalled and RecommendSkillsInstall now take an installedPlugins func(context.Context) []string argument. The agents package can't import the installer, so this follows the pattern MaybeVersionCheckTip already uses.
  • apps init passes installer.InstalledPluginAgents in both paths: the flags-mode tip and the interactive install prompt.
  • MaybeVersionCheckTip passes its existing installedPlugins through. Its behavior doesn't change, because it already returned early when any plugin was recorded.
  • Claude Code is unchanged. It still needs the plugin in its own manifest, active for the working directory. A recorded Claude Code plugin without a manifest entry doesn't count.
  • Added a changelog fragment under .nextchanges/cli/.

Why

When Codex or Copilot was detected, HasDatabricksToolingInstalled returned false unless skills were in the shared skills directory. The CLI can't read a plugin manifest for those agents, so it never saw their plugin. As a result, users who had already run databricks aitools install to get the Databricks plugin for Codex or Copilot were still told to install skills by apps init, and the tip was logged in telemetry.

The CLI install state is the only plugin source the CLI has for these agents. One gap remains: a plugin installed with the agent's own CLI, not with databricks aitools install, still isn't detected.

Tests

  • TestHasDatabricksToolingInstalledRecordedPlugins, a new table test with these cases:
    • Codex and Copilot both recorded: no recommendation.
    • Only Codex recorded while Copilot is also detected: recommendation shown.
    • Nothing recorded: recommendation shown.
    • Codex recorded alongside Cursor, which has no tooling: recommendation shown.
    • A Claude Code record without its manifest: recommendation shown.
  • TestRecommendSkillsInstallSkipsWhenPluginRecorded: no prompt, no stderr output and no hint telemetry when Codex and Copilot plugins are recorded.
  • Updated the existing HasDatabricksToolingInstalled and RecommendSkillsInstall tests for the new argument. They pass a stub that records no plugin, so their expectations don't change.
  • Acceptance test acceptance/cmd/aitools/telemetry/apps-init-tip: a new case writes an install state that records Codex and Copilot plugins. It then checks that apps init prints no tip and logs no telemetry event.

…e plugin recorded

## Changes

`databricks apps init` decides whether to recommend Databricks skills by calling `agents.HasDatabricksToolingInstalled`. With this change, that check also uses the plugins recorded in the CLI install state (`installer.InstalledPluginAgents`). It is used for agents that have no plugin manifest the CLI can read, which today means Codex and Copilot.

```diff
 for each detected agent:
   if agent has no plugin manifest reader (Codex, Copilot):
-    return false
+    if plugin not recorded in the CLI install state:
+      return false
+    continue
   if databricks plugin not active for the working directory:
     return false
 return true
```

- `HasDatabricksToolingInstalled` and `RecommendSkillsInstall` now take an `installedPlugins func(context.Context) []string` argument. The `agents` package can't import the installer, so this follows the pattern `MaybeVersionCheckTip` already uses.
- `apps init` passes `installer.InstalledPluginAgents` in both paths: the flags-mode tip and the interactive install prompt.
- `MaybeVersionCheckTip` passes its existing `installedPlugins` through. Its behavior doesn't change, because it already returned early when any plugin was recorded.
- Claude Code is unchanged. It still needs the plugin in its own manifest, active for the working directory. A recorded Claude Code plugin without a manifest entry doesn't count.
- Added a changelog fragment under `.nextchanges/cli/`.

## Why

When Codex or Copilot was detected, `HasDatabricksToolingInstalled` returned false unless skills were in the shared skills directory. The CLI can't read a plugin manifest for those agents, so it never saw their plugin. As a result, users who had already run `databricks aitools install` to get the Databricks plugin for Codex or Copilot were still told to install skills by `apps init`, and the tip was logged in telemetry.

The CLI install state is the only plugin source the CLI has for these agents. One gap remains: a plugin installed with the agent's own CLI, not with `databricks aitools install`, still isn't detected.

## Tests

- `TestHasDatabricksToolingInstalledRecordedPlugins`, a new table test with these cases:
  - Codex and Copilot both recorded: no recommendation.
  - Only Codex recorded while Copilot is also detected: recommendation shown.
  - Nothing recorded: recommendation shown.
  - Codex recorded alongside Cursor, which has no tooling: recommendation shown.
  - A Claude Code record without its manifest: recommendation shown.
- `TestRecommendSkillsInstallSkipsWhenPluginRecorded`: no prompt, no stderr output and no hint telemetry when Codex and Copilot plugins are recorded.
- Updated the existing `HasDatabricksToolingInstalled` and `RecommendSkillsInstall` tests for the new argument. They pass a stub that records no plugin, so their expectations don't change.
- Acceptance test `acceptance/cmd/aitools/telemetry/apps-init-tip`: a new case writes an install state that records Codex and Copilot plugins. It then checks that `apps init` prints no tip and logs no telemetry event.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 19, 2026 21:10
@github-actions github-actions Bot added the aitools AI tools (databricks aitools) related changes label Nov 19, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 231f85e5a | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (0f23a05) latest (v1.27.0)
TestAccept/cmd/aitools/telemetry/apps-init-tip ✅ ❌ ❌
TestAccept/cmd/aitools/telemetry/apps-init-tip: branch ✅ | main (0f23a05) ❌ | latest (v1.27.0) ❌

main (0f23a05):

acceptance_test.go:1268: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:39279" ("[DATABRICKS_HOST]")
    acceptance_test.go:1268: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:39279" ("[DATABRICKS_HOST]")
    acceptance_test.go:1400: Diff:
        --- cmd/aitools/telemetry/apps-init-tip/output.txt
        +++ /tmp/TestAcceptcmdaitoolstelemetryapps-init-tip1848425175/002/output.txt
        @@ -39,5 +39,7 @@
           Location: [TEST_TMP_DIR]/app-recorded
           Files: 3
         
        +Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
         
         >>> print_telemetry
        +{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}
        
    acceptance_test.go:1178: LOG.apps-init: 
    acceptance_test.go:1178: LOG.apps-init-plugin: 
    acceptance_test.go:1178: LOG.apps-init-recorded: 
FAIL

latest (v1.27.0):

acceptance_test.go:1268: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35679" ("[DATABRICKS_HOST]")
    acceptance_test.go:1268: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35679" ("[DATABRICKS_HOST]")
    acceptance_test.go:1400: Diff:
        --- cmd/aitools/telemetry/apps-init-tip/output.txt
        +++ /tmp/TestAcceptcmdaitoolstelemetryapps-init-tip4169723504/002/output.txt
        @@ -15,6 +15,7 @@
         >>> print_telemetry
         {"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
         {"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}
        +{"command":"apps_init","exit_code":0,"aitools_auto_install_event":{"outcome":"SKIPPED_CI","holdout_arm":"HOLDOUT","interactive":false}}
         
         === plugin installed: no hint, no tip, no event
         
        @@ -28,6 +29,7 @@
         
         
         >>> print_telemetry
        +{"command":"apps_init","exit_code":0,"aitools_auto_install_event":{"outcome":"SKIPPED_CI","holdout_arm":"TREATMENT","interactive":false}}
         
         === codex and copilot plugins recorded by aitools install: no tip, no event
         
        @@ -39,5 +41,8 @@
           Location: [TEST_TMP_DIR]/app-recorded
           Files: 3
         
        +Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
         
         >>> print_telemetry
        +{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}
        +{"command":"apps_init","exit_code":0,"aitools_auto_install_event":{"outcome":"SKIPPED_CI","holdout_arm":"TREATMENT","interactive":false}}
        
    acceptance_test.go:1178: LOG.apps-init: 
    acceptance_test.go:1178: LOG.apps-init-plugin: 
    acceptance_test.go:1178: LOG.apps-init-recorded: 
    acceptance_test.go:1190: Test produced unexpected files:
        plugin-home/.databricks/aitools/auto-install.json
        plugin-home/.databricks/aitools/auto-install.json.lock
        plugin-home/.databricks/aitools/auto-install.lock
        recorded-plugin-home/.databricks/aitools/auto-install.json
        recorded-plugin-home/.databricks/aitools/auto-install.json.lock
        recorded-plugin-home/.databricks/aitools/auto-install.lock
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 231f85e

Run: 44638893340

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 3:59
💚​ aws-windows-latest-4core-16gb 1 274 18 4:06
💚​ azure linux-2core-8gb 1 271 20 3:34
💚​ azure-windows-latest-4core-16gb 1 273 18 4:01
💚​ gcp linux-2core-8gb 1 272 20 3:47
💚​ gcp-windows-latest-4core-16gb 1 274 18 4:20
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:18 gcp-windows-latest-4core-16gb TestAccept
4:04 aws-windows-latest-4core-16gb TestAccept
3:59 azure-windows-latest-4core-16gb TestAccept

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at abbcede. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 8 non-blocking comment(s) and 1 question(s) below.

This PR makes apps init stop recommending skills when the Databricks plugin for detected Codex and Copilot agents is recorded in the CLI install state. Claude Code still needs its own manifest entry. The logic is correct, and the new acceptance case fails on main and passes here. Nothing blocking turned up. The main fix is to isolate the two new unit tests and the new acceptance case from host env vars such as CLAUDE_CONFIG_DIR and XDG_CONFIG_HOME: they fail when those point at a real agent config. There is also an open, non-blocking question about sharing the per-agent plugin check with DatabricksTooling, plus a few small test, comment and efficiency nits.

Questions

  1. HasDatabricksToolingInstalled now has its own copy of the per-agent rule that Agent.DatabricksTooling already applies: the manifest reader decides, otherwise the CLI record counts. This copy keys on pluginInstalls == nil rather than Plugin != nil. It also reads records through InstalledPluginAgents (global and project state plus the Claude cache), while MaybeHint and the aitools User-Agent use PluginRecordedFunc / installer.PluginRecorded(ctx, ScopeGlobal, name). If either side changes later (an agent gains a manifest reader, or the record source changes), the apps init tip and the per-agent hint can disagree about the same agent. Would you consider sharing one per-agent check with DatabricksTooling and injecting the existing PluginRecordedFunc? Or is the separate source intentional? (non-blocking)

Comments outside the changed lines

  • [nit] libs/aitools/agents/recommend_test.go:171: recordedCodexPlugin returns "codex_0.2.6", the InstalledTools format, but HasDatabricksToolingInstalled now matches entries against the agent name. The case passes only because len(installedPlugins(ctx)) > 0 short-circuits; slices.Contains(["codex_0.2.6"], "codex") is false. Return []string{NameCodex} so the fixture matches what InstalledPluginAgents returns.

Comment thread libs/aitools/agents/skills_test.go Resolved
Comment thread libs/aitools/agents/skills.go Resolved
Comment thread libs/aitools/agents/recommend.go Resolved
Comment thread libs/aitools/agents/skills_test.go Resolved
Comment thread libs/aitools/agents/recommend.go Resolved
…xture, read install state once

- Use testutil.SetAgentEnv in the new unit tests and unset the agent
  config-dir overrides in the acceptance script, so host agent dirs
  can't add a detected agent.
- Commit the recorded-plugin home as a fixture instead of creating it
  in the script, and explain why CLAUDECODE is unset.
- Read the CLI install state lazily in HasDatabricksToolingInstalled and
  once in MaybeVersionCheckTip.
- Note that a plugin removed with the agent's own CLI still counts while
  its record remains.
- Add a case with a Claude Code manifest plugin and a recorded Codex
  plugin, and make the recorded fixture in TestMaybeVersionCheckTip
  match what InstalledPluginAgents returns.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit c400ded Nov 20, 2026
29 checks passed
@anton-107
anton-107 deleted the apps-init-tip-recorded-plugins branch November 20, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aitools AI tools (databricks aitools) related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants