Skip to content

aitools: pin every registry agent in a contract test - #7822

Merged
anton-107 merged 3 commits into
mainfrom
aif-aitools-registry-contract-test
Nov 20, 2026
Merged

anton-107 merged 3 commits into
mainfrom
aif-aitools-registry-contract-test

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Changes

Adds TestRegistryContract to libs/aitools/agents/registry_test.go. It is a table test that pins, for every agent in Registry:

  • the display name and the binary looked up on PATH
  • the plugin installed for it (marketplace, ID, source, shared flag), or none
  • the marker file needed to confirm it is installed (MandatoryFile)
  • the global skills directory, derived from a temporary home directory
  • whether it supports project scope and, if so, its project skills directory

The test also checks that the registry holds exactly these agents, in this order. Per-agent override environment variables (CLAUDE_CONFIG_DIR, PI_CODING_AGENT_DIR, GEMINI_CLI_HOME, GOOSE_PATH_ROOT, XDG_CONFIG_HOME, APPDATA) are cleared so every path comes from the temporary home. The Goose global directory has its own expected value on Windows.

This PR only adds a test. No production code changes.

Why

Each registry entry decides where the CLI writes skills and which plugin it installs for that agent. Until now, nothing caught an accidental edit to an entry, such as a renamed directory, a dropped plugin or a flipped project-scope flag. With this test, any change to an agent's behavior also has to update the table, so the change shows up in review.

Tests

  • go test ./libs/aitools/agents/ -run TestRegistryContract -v passes, including one subtest for each of the 10 agents.

## Changes

Adds `TestRegistryContract` to `libs/aitools/agents/registry_test.go`. It is a table test that pins, for every agent in `Registry`:

- the display name and the binary looked up on `PATH`
- the plugin installed for it (marketplace, ID, source, shared flag), or none
- the marker file needed to confirm it is installed (`MandatoryFile`)
- the global skills directory, derived from a temporary home directory
- whether it supports project scope and, if so, its project skills directory

The test also checks that the registry holds exactly these agents, in this order. Per-agent override environment variables (`CLAUDE_CONFIG_DIR`, `PI_CODING_AGENT_DIR`, `GEMINI_CLI_HOME`, `GOOSE_PATH_ROOT`, `XDG_CONFIG_HOME`, `APPDATA`) are cleared so every path comes from the temporary home. The Goose global directory has its own expected value on Windows.

This PR only adds a test. No production code changes.

## Why

Each registry entry decides where the CLI writes skills and which plugin it installs for that agent. Until now, nothing caught an accidental edit to an entry, such as a renamed directory, a dropped plugin or a flipped project-scope flag. With this test, any change to an agent's behavior also has to update the table, so the change shows up in review.

## Tests

- `go test ./libs/aitools/agents/ -run TestRegistryContract -v` passes, including one subtest for each of the 10 agents.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 19, 2026 17:40
@github-actions github-actions Bot added aitools AI tools (databricks aitools) related changes test-only-fix PRs that update failing of flaky tests without changing CLI code. labels Nov 19, 2026

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at ad13778. Each finding was verified against the code.

No blocking issues found. 1 non-blocking comment(s) below.

This PR adds TestRegistryContract, a table-driven test that pins every field of all 10 agents in Registry. The test data is correct against the production Registry slice, and no production code is changed. The one real issue is that the pre-existing TestSkillAgentRegistryPaths is left in place, creating overlapping assertions for four agents that must now be kept in sync in two spots.

Comment thread libs/aitools/agents/registry_test.go Resolved

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at ad13778. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving. 3 non-blocking comment(s) below.

This PR adds a table test that pins each registry agent's display name, binary, plugin, marker file, skills directories and project scope. I checked every expected value against Registry, including the Windows branches, and they're all correct. Nothing blocks merging. The main suggestion is to delete the older TestSkillAgentRegistryPaths, which now duplicates four of the rows and fails when PI_CODING_AGENT_DIR or GOOSE_PATH_ROOT is set in the shell. The other two suggestions close gaps in what the table can catch: agent names come from the same constants on both sides, and clearing XDG_CONFIG_HOME/APPDATA means it only pins the fallback paths.

Accepted answers

  • Is a table that mirrors the registry literal worth having?

Agreeing with existing threads

  • [nice to have] TestSkillAgentRegistryPaths now duplicates part of the new table: #7822 (comment)

Comment thread libs/aitools/agents/registry_test.go Resolved
Comment thread libs/aitools/agents/registry_test.go Resolved
Comment thread libs/aitools/agents/registry_test.go Resolved
anton-107 and others added 2 commits November 19, 2026 23:08
Co-authored-by: Isaac <no-reply@databricks.com>
Remove TestSkillAgentRegistryPaths, which duplicated four rows of the
contract table. Write agent names as string literals so a renamed agent
fails the test. Point XDG_CONFIG_HOME and APPDATA at non-default
directories so the OpenCode and Goose rows pin that those overrides are
honored. Narrow the doc comment to the fields the table covers.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 added this pull request to the merge queue Nov 20, 2026
Merged via the queue into main with commit 3a11da3 Nov 20, 2026
23 checks passed
@anton-107
anton-107 deleted the aif-aitools-registry-contract-test branch November 20, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

aitools AI tools (databricks aitools) related changes test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants