Skip to content

aitools: keep --skills-only agents on raw skills during update - #7831

Open
anton-107 wants to merge 10 commits into
mainfrom
aitools-update-keeps-skills-only
Open

anton-107 wants to merge 10 commits into
mainfrom
aitools-update-keeps-skills-only

Conversation

@anton-107

@anton-107 anton-107 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks aitools install --skills-only now records which agents the user kept on raw skills, and databricks aitools update leaves those agents alone instead of moving them to the Databricks plugin.

  • InstallState has a new skills_only_agents field: a sorted list of agent names, left out of the file when it is empty.
  • install --skills-only sets the new InstallOptions.SkillsOnly. InstallSkillsForAgents then adds every target agent that supports a plugin and has its CLI on PATH to that list.
  • An agent whose CLI is missing is not added. install suggests --skills-only in that case too, so the user didn't turn the plugin down, and update should still migrate the agent once the CLI is installed.
  • The deprecated experimental aitools skills install alias always adds --skills-only, so it does not record the choice.
  • pluginMigrationCandidatesForScope skips agents in the list, so update and update --check no longer migrate them or report that they would. Their raw skills still get refreshed as before.
  • An agent comes off the list when:
    • RecordPluginInstalls records the plugin for it;
    • install without --skills-only tries the plugin for it and the plugin install fails (ForgetSkillsOnlyChoice), so the next update tries the plugin again;
    • uninstall without --skills removes all of its skills. A plain uninstall clears the whole list, uninstall --agents only the named agents. An uninstall that could not remove a skill keeps the list, since the files are still there.
  • An install without --skills-only that does not try the plugin for an agent leaves the list as it is.
 pluginMigrationCandidatesForScope
   skip agents without a plugin
   skip agents already on the plugin
+  skip agents in state.SkillsOnlyAgents
   skip agents whose CLI is not on PATH
   skip agents without managed raw skills

Why

For Claude Code, Codex and GitHub Copilot, update replaces CLI-managed raw skills with the plugin whenever the agent's CLI is on PATH. The state file couldn't tell skills installed with --skills-only apart from skills written by an older CLI or while the agent's CLI was missing. So the next update undid the user's --skills-only choice. Now that choice is saved, and only raw skills the user didn't choose get migrated. To move an agent to the plugin, run databricks aitools install without --skills-only.

Tests

New unit tests:

  • TestInstallSkillsForAgentsRecordsSkillsOnlyAgents: only agents with a plugin and their CLI on PATH are recorded, and a later install without --skills-only keeps the list.
  • TestRecordPluginInstallsForgetsSkillsOnlyChoice: installing the plugin for an agent takes only that agent off the list.
  • TestForgetSkillsOnlyChoice, TestForgetSkillsOnlyChoiceWithoutState: only the named agents come off the list, and a scope without state is left alone.
  • TestExecutePlanFailedPluginForgetsSkillsOnlyChoice: a failed plugin install takes the agent off the list.
  • TestPluginMigrationCandidatesSkipSkillsOnlyAgents: agents in the list are not migration candidates, and other agents with raw skills still are.
  • TestUpdateKeepsSkillsOnlyAgentsOnRawSkills: update --check reports no migration, and update refreshes the raw skills without installing the plugin or cleaning up the skills.
  • TestUninstallTargetedAgentForgetsSkillsOnlyChoice, TestUninstallTargetedAgentFailureKeepsSkillsOnlyChoice, TestUninstallAllForgetsSkillsOnlyChoice: a targeted uninstall clears only the named agents and keeps the choice when a skill could not be removed, and a plain uninstall clears the list even when the state file stays for a plugin.
  • The existing tests for install --skills-only, the legacy skills install alias and state round-tripping now also check the new option and field.

The install-non-interactive acceptance test now also runs install --skills-only and update with a Claude Code CLI stand-in on PATH and checks that the skill files stay.

./task aitools-full passes.

Adds the changelog fragment .nextchanges/cli/aitools-update-keeps-skills-only.md.

## Changes

`databricks aitools install --skills-only` now records which agents the user kept on raw skills, and `databricks aitools update` leaves those agents alone instead of moving them to the Databricks plugin.

- `InstallState` has a new `skills_only_agents` field: a sorted list of agent names, left out of the file when it is empty.
- `install --skills-only` sets the new `InstallOptions.SkillsOnly`. `InstallSkillsForAgents` then adds every target agent that supports a plugin and has its CLI on PATH to that list.
- An agent whose CLI is missing is not added. `install` suggests `--skills-only` in that case too, so the user didn't turn the plugin down, and `update` should still migrate the agent once the CLI is installed.
- `pluginMigrationCandidatesForScope` skips agents in the list, so `update` and `update --check` no longer migrate them or report that they would. Their raw skills still get refreshed as before.
- `RecordPluginInstalls` takes an agent off the list once the plugin is installed for it. Running `install` again without `--skills-only` leaves the list as it is.

```diff
 pluginMigrationCandidatesForScope
   skip agents without a plugin
   skip agents already on the plugin
+  skip agents in state.SkillsOnlyAgents
   skip agents whose CLI is not on PATH
   skip agents without managed raw skills
```

## Why

For Claude Code, Codex and GitHub Copilot, `update` replaces CLI-managed raw skills with the plugin whenever the agent's CLI is on PATH. The state file couldn't tell skills installed with `--skills-only` apart from skills written by an older CLI or while the agent's CLI was missing. So the next `update` undid the user's `--skills-only` choice. Now that choice is saved, and only raw skills the user didn't choose get migrated. To move an agent to the plugin, run `databricks aitools install` without `--skills-only`.

## Tests

New unit tests:
- `TestInstallSkillsForAgentsRecordsSkillsOnlyAgents`: only agents with a plugin and their CLI on PATH are recorded, and a later install without `--skills-only` keeps the list.
- `TestRecordPluginInstallsForgetsSkillsOnlyChoice`: installing the plugin for an agent takes only that agent off the list.
- `TestPluginMigrationCandidatesSkipSkillsOnlyAgents`: agents in the list are not migration candidates, and other agents with raw skills still are.
- `TestUpdateKeepsSkillsOnlyAgentsOnRawSkills`: `update --check` reports no migration, and `update` refreshes the raw skills without installing the plugin or cleaning up the skills.
- The existing tests for `install --skills-only` and state round-tripping now also check the new option and field.

`go test ./cmd/aitools/... ./libs/aitools/installer/...` passes.

Adds the changelog fragment `.nextchanges/cli/aitools-update-keeps-skills-only.md`.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 19, 2026 21:44
@anton-107
anton-107 requested review from abdelrahmanomar1515 and simonfaltum and removed request for abdelrahmanomar1515 November 19, 2026 21:44
@github-actions github-actions Bot added the aitools AI tools (databricks aitools) related changes label Nov 19, 2026
Co-authored-by: Isaac <no-reply@databricks.com>

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at b6e70db. Each finding was verified against the code.

No blocking issues found. 2 non-blocking comment(s) below.

The PR adds a SkillsOnlyAgents field to the install state so that update no longer migrates agents the user explicitly put on raw skills via install --skills-only. The lifecycle is clear for the happy path: install --skills-only records the preference, RecordPluginInstalls removes it on a successful plugin install, and pluginMigrationCandidatesForScope skips listed agents. Two edge cases in the preference lifecycle are worth tightening: install without --skills-only does not clear entries even when it targets the agent (so a transient plugin failure leaves the user stuck), and an older CLI that saves state will silently drop the field.

Comment thread libs/aitools/installer/installer.go Resolved
Comment thread libs/aitools/installer/state.go Resolved
An install that tries the plugin for an agent the user had put on raw
skills now drops that agent from the --skills-only record even when the
plugin install fails, so update tries the plugin again. Also note on the
field that an older CLI rewriting the state drops it.

Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 06c8e20. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 7 non-blocking comment(s) and 1 question(s) below.

This PR records which agents install --skills-only put on raw skills, and makes update skip migrating those agents to the plugin. A recorded or failed plugin install clears the entry. The core change is small and well tested, and CI is green; I found nothing blocking. The main open point is that several CLI-suggested --skills-only runs (plugin-failure fallbacks, legacy-state rebuild advice, --skills cherry-picks, the deprecated alias) now pin agents to raw skills permanently. Smaller fixes: the changelog overstates the guarantee, a targeted uninstall leaves stale entries, and two existing tests can now rewrite a developer's real state file.

Questions

  1. --skills-only now also records "this agent declined the plugin", but the CLI itself tells users to run --skills-only for other reasons while the agent's CLI is on PATH: every plugin-failure fallback in pluginFailureText (recovery.go:436-461: CLI too old for plugin, timeout, shim error, failed install), the legacy-state rebuild advice (installer.go:379, uninstall.go:131), and --skills cherry-picks, which require --skills-only (install.go:206). Each of these now pins the agent to raw skills for every later update, even after the plugin problem is fixed; before this PR, update would migrate it. The CLI-missing exception is justified by the same reasoning ("install suggests --skills-only for that case too"). Should these runs also not be recorded? Or should --skills-only always count as a choice, with its help text and the fallback messages saying that update keeps it? (non-blocking)

Accepted answers

  • Should a failed plugin install during install clear the skills-only choice?: #7831 (comment)
  • An older CLI rewriting state drops skills_only_agents: #7831 (comment)
  • Why aren't agents whose CLI is missing recorded?

Comment thread cmd/aitools/install.go Resolved
Comment thread cmd/aitools/install.go Resolved
Comment thread libs/aitools/installer/state.go Resolved
Comment thread cmd/aitools/install.go Resolved
Comment thread cmd/aitools/update_test.go Resolved
- The deprecated experimental skills install alias no longer records the
  choice: it adds --skills-only itself, so its users did not decline the
  plugin.
- A targeted uninstall without --skills ends the choice of those agents.
- Scope the changelog entry to installs made from this release on.
- Keep the executePlan tests with a failing plugin off the real HOME, and
  stub agents.LookPath instead of writing a binary on PATH.
- Cover install --skills-only then update with an agent CLI on PATH in the
  install-non-interactive acceptance test.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 98cd70b52 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (8b7e509) latest (v1.27.0)
TestAccept/experimental/aitools/skills/install-non-interactive ✅ ❌ ❌
TestAccept/experimental/aitools/skills/install-non-interactive: branch ✅ | main (8b7e509) ❌ | latest (v1.27.0) ❌

main (8b7e509):

acceptance_test.go:1270: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:42193" ("[DATABRICKS_HOST]")
    acceptance_test.go:1402: Diff:
        --- experimental/aitools/skills/install-non-interactive/output.txt
        +++ /tmp/TestAcceptexperimentalaitoolsskillsinstall-non-interactive1993657606/002/output.txt
        @@ -49,6 +49,8 @@
         Installed 1 skill in [TEST_TMP_DIR]/home/.databricks/aitools/skills.
         
         >>> [CLI] aitools update --force
        +Installing databricks plugin for Claude Code...
        +Skipped Claude Code: claude-code: cli-not-on-path: fakeagent: claude does not take ["plugin" "--help"]
           updated test-stable v1.0.0 -> v1.0.0
         Updated 1 skill.
         Claude Code has skill files
        
FAIL

latest (v1.27.0):

acceptance_test.go:1270: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:36311" ("[DATABRICKS_HOST]")
    acceptance_test.go:1402: Diff:
        --- experimental/aitools/skills/install-non-interactive/output.txt
        +++ /tmp/TestAcceptexperimentalaitoolsskillsinstall-non-interactive1740191084/002/output.txt
        @@ -49,6 +49,8 @@
         Installed 1 skill in [TEST_TMP_DIR]/home/.databricks/aitools/skills.
         
         >>> [CLI] aitools update --force
        +Installing databricks plugin for Claude Code...
        +Skipped Claude Code: claude-code: cli-not-on-path: fakeagent: claude does not take ["plugin" "--help"]
           updated test-stable v1.0.0 -> v1.0.0
         Updated 1 skill.
         Claude Code has skill files
        
FAIL

Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 245f98f. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 6 non-blocking comment(s) below.

This PR records which agents install --skills-only put on skill files, so update stops moving them to the plugin. A plugin install for the agent, a failed attempt at one, or a targeted uninstall clears the entry. CI is green, the earlier review threads are addressed, and I found nothing blocking. Smaller points: a plain uninstall that can't remove a plugin leaves stale entries, plus a few nits (an acceptance step that can pass without a stand-in claude, test isolation, changelog wording, a duplicated helper, a stale comment, the outdated description). One earlier question, whether the CLI's own --skills-only suggestions should keep agents off the plugin, is still unanswered.

Comments outside the changed lines

  • [nit] cmd/aitools/install.go:421: The installOutputIsJSON comment says the legacy skills install alias "builds a NewInstallCmd", but newLegacySkillsInstallCmd now calls newInstallCmd(false), which deliberately turns off recording. Change the comment to name newInstallCmd.

Agreeing with existing threads

  • [nit] PR description still says install without --skills-only leaves the list as it is: #7831 (comment)

Comment thread libs/aitools/installer/uninstall.go Resolved
// CLI scope. Install calls it for agents whose plugin install it attempted and
// that failed, so update tries the plugin again. Agents not in the list, or a
// scope without state, are left as they are.
func ForgetSkillsOnlyChoice(ctx context.Context, cliScope string, agentNames []string) error {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] ForgetSkillsOnlyChoice repeats the skillsDir/LoadState/nil-check/change-detect/SaveState skeleton of ForgetPluginInstalls directly above it (plugin.go:544-563), and the same slices.DeleteFunc name filter also appears in RecordPluginInstalls and uninstall.go:125. An install with both successful and failed plugin agents now also writes the state twice. Optional: one helper that drops the named agents from a scope's state, or a single ForgetSkillsOnlyChoice call for all attempted plugin agents, would remove the duplication.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather leave this as is in this PR. A shared helper would mean refactoring ForgetPluginInstalls, which this PR doesn't otherwise touch, and the two functions filter different fields (a map and a sorted slice). The second state write only happens when one install has both successful and failed plugin agents, and it is a small local file. Happy to fold the three name filters into one helper in a follow-up if you think it's worth it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is here, for the reasons above. The shared helper would mean changing ForgetPluginInstalls, which is outside this change. I'll fold the name filters into one helper in a follow-up.

anton-107 and others added 3 commits November 20, 2026 08:20
Co-authored-by: Isaac <no-reply@databricks.com>
A plain uninstall now ends every --skills-only choice, also when the
state file stays for a plugin it could not remove. An uninstall that
could not remove a skill keeps the choice, since the files are still
there.

The install-non-interactive acceptance test stops when the agent CLI
stand-ins were not built, and sets FAKE_AGENT_LOG for them. Two
uninstall tests no longer follow the host's agent config dir
overrides. Reword the changelog fragment and fix a stale comment.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 1e1539b. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 4 non-blocking comment(s) below.

This PR records which agents install --skills-only put on skill files, and makes update stop moving them to the plugin. A plugin install, a failed plugin attempt or an uninstall clears the entry. The core change is small and well tested, CI is green, and I found nothing blocking. Two state gaps are worth closing: a plain install's CLI-missing fallback keeps the choice, and a multi-agent uninstall with one failure keeps every named agent's choice. There are also a few nits. The earlier question about the CLI's own --skills-only suggestions is still open.

Accepted answers

  • One shared helper or one state write for dropping agents from SkillsOnlyAgents: #7831 (comment)

Comments outside the changed lines

  • [nice to have] cmd/aitools/install.go:955: A plain databricks aitools install (no --agents) can find Claude Code's config dir without claude on PATH. The agent then becomes a fallback item and gets skill files, but nothing calls ForgetSkillsOnlyChoice for it. So an agent recorded by an earlier --skills-only run stays in skills_only_agents, and once claude is back on PATH, update never migrates it. The same install with --agents claude-code behaves differently: it attempts the plugin, fails, and clears the choice at line 1011. cliMissingOutcome already reports the fallback as a failed plugin attempt. Either call installer.ForgetSkillsOnlyChoice for fallbackItems next to forgetStalePluginRecords, or document that a fallback keeps the choice. In the second case, also adjust the SkillsOnlyAgents doc: it says update migrates raw skills that were written while the agent's CLI was missing.

Agreeing with existing threads

Comment thread libs/aitools/installer/uninstall.go Resolved
Comment thread libs/aitools/installer/state.go Resolved
Comment thread libs/aitools/installer/uninstall.go Resolved
anton-107 and others added 2 commits November 20, 2026 15:17
Co-authored-by: Isaac <no-reply@databricks.com>
With several --agents, a removal failure for one agent no longer keeps
every named agent's choice; agents whose entries were all removed lose
theirs. Trim the SkillsOnlyAgents doc comment to its contract and
restore an unrelated blank line.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aitools AI tools (databricks aitools) related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants