You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: support catalog-installed external agent adapters - #4862
Support trusted, catalog-installed external AI-agent adapters without source-registry edits, pip installation, or copied core command inventories. A standalone archive containing root integration.yml and __init__.py exports a matching IntegrationBase subclass and renders Spec Kit's host templates through the existing integration bases.
The adapter-only descriptor requires identity, version, metadata, and host/tool requirements. Existing optional provides metadata remains compatible. Catalog identity/version/metadata, package descriptors, class configuration, checksums, source policy, download restrictions, and safe extraction are validated before installation.
Executable packages and provenance are stored separately from generated-file manifests. Fresh CLI processes load verified, trusted project-local implementations before rendering, registration, selection, status, or workflow dispatch; catalog metadata operations do not import adapter code. Project changes unload synthetic imports and refresh registry/configuration caches.
Lifecycle operations preserve existing built-in and generic behavior, active-integration handling, extension/preset contributions, script variants, and edited-file preservation. Failed operations restore only operation-owned writes, not independent workflow progress or unowned user files. Concurrent managed-file conflicts retain explicit recovery snapshots. Forced upgrade/uninstall can recover a damaged target using validated ownership metadata without bypassing source policy or replacement trust.
Tests use neutral sample-agent packages and loopback HTTP fixtures. Directly related integration design, catalog, contribution, and reference documentation describes the final contract and public commands. The branch is rebased onto upstream main.
Testing
Tested locally with uv run specify --help
Ran existing tests with uv sync && uv run pytest
Tested with a sample project (if applicable)
The full suite uses this worktree's own virtualenv, per CONTRIBUTING.md, rather than the literal uv run pytest checklist command, which can resolve an editable install from another checkout.
Command/check
Result
uv sync --extra test
Passed; local distribution metadata matches the rebased 1.1.2.dev0 manifest
ruff check --select F src/specify_cli/presets/_manager.py src/specify_cli/presets/_manager_commands.py src/specify_cli/presets/_manager_skills.py src/specify_cli/shared_infra.py
Passed
git diff upstream/main...HEAD --check
Passed
uv build --out-dir dist
Passed; source and wheel distributions built; temporary outputs removed
The 114 adapter cases cover adapter-only descriptors, actual catalog installation, trust denial/discovery-only sources, fresh-process loading, host command/skill rendering, extension/preset registration and cleanup, runtime dispatch with harmless process doubles, upgrades/uninstall, edited files, rollback, invalid metadata/classes/imports, built-in collisions, unsafe paths/symlinks, damaged-target recovery, and project/cache isolation.
Sixteen scope regressions were also run against the earlier implementation: all failed before the fixes and passed afterward. Sample-project scaffolding/lifecycle checks use temporary executable/process doubles, not authenticated model execution.
AI Disclosure
I did not use AI assistance for this contribution
I did use AI assistance (fill in the disclosure below)
AI disclosure: Implemented with GitHub Copilot, powered by GPT-6.1 Sol (gpt-6.1-sol), in autonomous agent mode; reasoning-effort setting was not explicitly selected. AI assistance covered investigation, implementation, regression tests, documentation, rebase, validation, and PR-description drafting. No human line-by-line review is claimed.
Use the rebased 1.1.2 development version as the minimum in external adapter examples rather than implying the new catalog-install capability ships in an earlier release.
Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Portable member validation now runs before filesystem access for adapter output paths and retained package entries, rejecting Win32 aliases, device names, invalid characters, and oversized components.
Authoritative consent is user-local in ~/.specify/integration-trust.json, bound to the canonical project root, adapter ID, and verified full-package digest. Project trusted metadata cannot authorize execution, and cache reuse rechecks consent. Managed ignore rules exclude executable packages and their provenance registry. Copied projects can reauthorize through a reviewed, install-enabled catalog using integration upgrade sample-agent --force --trust-integration.
Artifact list/info/lookup load trusted adapter configuration, resolve materialized extension/preset output in fresh processes, and retain a single JSON error envelope with useful adapter failure details.
Stable lifecycle locks are user-local and project-keyed rather than creating target scaffolding before init confirmation. Failed external initialization uses the scoped journal, removes only an empty newly created root, and preserves independent files.
Forced uninstall recovers an entirely absent package directory. Other filesystem errors remain explicit; an unchanged package is not destructively recopied during rollback after denied removal.
Added 41 adapter cases without removing existing tests. The initial targeted run reproduced 31 failures before fixes; final focused validation passed all 308 adapter/artifact/shared-ignore cases. Windows CI also exposed UTF-8 decoding errors in the adapter test harness; rendered-file reads and captured subprocess output now explicitly use UTF-8.
The new-head platform CI is separate from these local results. Reviewer conversations are left open for the reviewer.
Posted on behalf of @mnriem by GitHub Copilot, powered by GPT-6.1 Sol (gpt-6.1-sol), in autonomous mode; reasoning effort was not explicitly selected. AI authored the fixes, tests, documentation, and this response. No human line-by-line review is claimed.
Host writes now reject symlinked destinations and ancestors before directory creation or writes; snapshots and forced removal preserve/unlink owned leaf links without traversing their targets. Registry loading and registrar configuration snapshots are synchronized, recursive-load suppression is context-local, and runtime dispatch pins each project's adapter and verified imports without serializing independent agent processes.
Damaged-adapter cleanup uses user-local registrar/path ownership bound to the previously trusted package, rejecting edited project claims and overlap with other registered integrations, including legacy destinations. Copied projects and older grant-only stores preserve old-only artifacts with an explicit manual-cleanup warning. Ownership advances only after durable loading succeeds, so failed upgrades retain the previous recovery authority. Execute/resume reload OSError failures now produce one JSON failure envelope with no stderr.
Added 25 regressions, including overlapping prompt/command dispatch with lazy relative imports, cleanup after failed processes, forged recovery claims, missing local ownership, leaf-link snapshots, and failed durable-upgrade rollback. Before-fix runs reproduced the original eleven regressions; additional before/after checks reproduced the snapshot-traversal and recovery-ownership rollback gaps. Also corrected the prior Windows assertion to check directory entries rather than querying a Win32 alias for an invalid pathname. Existing fresh-process artifact/extension/preset registration coverage remains passing.
Source and wheel builds also passed, with build artifacts kept outside the checkout. New cross-platform CI is pending; Windows was not executed locally.
Posted on behalf of @mnriem by GitHub Copilot (model: GPT-6.1 Sol, autonomous mode). The implementation, regression tests, documentation, validation, and this response were AI-generated/executed; no human line-by-line review is claimed.
Addressed the six new findings in review 5435405580 in commit 8678402f3b066bde9ff921ba2e0e3ef548579177, pushed to the existing PR branch.
Validate optional legacy output directories for type, canonical project-relative paths, symlinks, and reserved roots; reject home-relative external output syntax.
Compare current and legacy output roots using case-folded path components, including other integrations' legacy roots.
Load and pin the project's installed adapters during native event refresh, including fresh-process event-only extension add/remove and enable/disable. Adapter-load failures are reported through EventRefreshError.
Atomically load and snapshot extension/preset registrar configuration. Extension candidate folders come from pinned integration metadata rather than stale global configuration. Preserve generic cleanup with missing/malformed settings and the existing exclusion of generic preset registration.
Catch missing workflow runs before broader I/O failures, preserving text/JSON diagnostics. Normalize adapter-loading filesystem failures at the engine boundary so a missing adapter file is not incorrectly reported as a missing run.
Added 30 regression cases. The corrected before-fix run reproduced 23 failures with one passing positive control. An additional regression caught the missing-adapter/missing-run classification conflict during remediation. Full-suite validation also caught three generic behavior regressions introduced by the initial root-aware snapshot change; those were fixed without removing or weakening the existing cases.
uv build --out-dir with a session-artifact directory outside the checkout
Source distribution and wheel built successfully
.venv/bin/specify --help
Passed
git diff --check
Passed
The earlier workflow-dispatch and artifact-resolution findings remain covered by the published scoped-dispatch/root-aware resolution changes and passing regressions. Scoped dispatch pins both project lookup and verified lazy-import namespaces without serializing parallel agent execution. Artifact list/info/lookup loads the project adapter, uses a root-aware registrar, and retains JSON error envelopes for damaged packages.
The PR head matches the pushed commit. Cross-platform CI is pending for this revision. Reviewer conversations remain unresolved for the reviewer to reassess.
AI disclosure: Posted on behalf of @mnriem by GitHub Copilot, powered by GPT-6.1 Sol, operating autonomously. The agent authored the fixes, tests, documentation, commit, and this response and executed the reported checks. No human line-by-line review is claimed.
The documented external-catalog workflow is still contradicted by integration search: for an install-enabled external entry, command_search.py:98-104 prints “Only built-in integration IDs can be installed,” and test_command_search.py:32-33 still asserts that obsolete behavior. Update search to advertise specify integration install <id> when the source permits installation, while keeping discovery-only entries blocked.
Rollback misses direct writes during transactional lifecycle commands
src/specify_cli/integrations/_lifecycle.py:234
The rollback journal only records writes that call these observer hooks, but lifecycle commands still perform unobserved mutations. For example, switching from an external adapter to Copilot can merge an existing .vscode/settings.json via a direct write_text (integrations/copilot/__init__.py:680); if the later package-registry commit fails, _restore_snapshots has no journal entry for that file, so the failed switch leaves the user's settings changed. Instrument every write reachable inside the transaction (including existing built-in and extension/preset paths), or make the transaction restore those touched scopes independently of hook coverage.
Addressed review 5436613967 in commit a72c6bc15a0a1a73a3d1105b89005ff3f4edd19a, pushed to the existing PR branch.
Workflow engine construction no longer loads executable adapters. workflow status (text, all-runs JSON, and individual-run JSON) and workflow info inspect metadata even when an adapter is damaged. Run/resume load adapters inside their existing structured-error boundary and reload immediately before execution. Existing dispatch, cross-project, and JSON failure regressions remain passing.
Catalog search now advertises specify integration install <id> for install-enabled entries, including external adapters. Discovery-only results do not advertise installation; search does not import candidate code. Updated the obsolete assertion and added real registered-catalog policy regressions.
Expanded the mutation journal coverage rather than restoring entire directories indiscriminately. Host settings merges/copies, native event writes/removals, legacy migrations, extension skill/dev caches, registrar dev caches, preset composition caches, and built-in post-processing now notify the journal. Existing built-in home-scoped output is permitted only for participating built-in destinations and receives the same safe-path and concurrent-edit checks. Project-local detection markers also roll back. Merged user settings remain unowned by uninstall; concurrent edits are preserved and reported with recovery snapshots.
Added 21 cases, including positive settings ownership behavior and negative rollback/metadata/discovery coverage. Before-fix evidence captured 11 failures for the initial findings. Running the rollback regressions against the published 8678402f source snapshot reproduced eight failures, and a separate comparison isolated the preset cache mutation and missing concurrent-merge warning. These failures are fixed in the final suite.
21 passed before the final marker-specific refinement; all 21 also pass in the final full suite
Affected integration/workflow/event/extension/preset and built-in test selection
2,000 passed before the final additional cases/refinements; covered again by the full suite
uvx --offline ruff@0.15.0 check src tests
Passed
uv build --out-dir with a session-artifact directory outside the checkout
Source distribution and wheel built successfully
.venv/bin/specify --help
Passed
git diff --check
Passed
The remaining event-refresh finding was already implemented in 8678402f: refresh loads/pins the project adapter itself, and fresh-process event-only add/remove/enable/disable regressions pass. This revision additionally journals the native configuration mutations during transactional integration lifecycle operations. Reviewer conversations remain unresolved for reassessment.
AI disclosure: Posted on behalf of @mnriem by GitHub Copilot, powered by GPT-6.1 Sol, operating autonomously. The agent authored the fixes, tests, documentation, commit, and this response and executed the reported checks. No human line-by-line review is claimed.
Validate invoke_separator and dev_no_symlink class attributes
src/specify_cli/integrations/installer.py:563
Class-level invoke_separator and dev_no_symlink bypass _validate_registrar_config, yet they are copied into agent configuration and persisted recovery metadata. An adapter can therefore pass installation with invoke_separator=None (later breaking command-reference rendering) or a truthy non-boolean dev_no_symlink. Validate these public class attributes alongside multi_install_safe.
Bind local recovery ownership to verified package hashes and require its local trust grant. Preserve legacy bindings and damaged-package cleanup while rejecting substituted recovery identities and invalid public adapter attributes.
Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The implementation validator now checks class-level invoke_separator (non-empty string) and dev_no_symlink (boolean), even when registrar configuration has optional overrides. Invalid values fail installation before they can enter rendering or persisted recovery configuration.
New local recovery records retain their verified package hashes. Recovery recomputes the package identity using those local hashes, the project root, and the adapter key, then requires the corresponding local trust grant. This deliberately does not compare against mutable project hashes: forced cleanup must still work when an installed package is missing, damaged, incompatible, or fails import. Substituted identities, hash mappings, and cross-project bindings fail explicitly. Legacy hash-less bindings remain supported with a local grant; revoked grants cause generated files to be preserved through the existing ownership-unavailable warning path. Added positive and negative coverage for both attribute propagation and recovery.
The event-refresh item remains addressed by the existing project_integrations(project_root) scope in refresh_integration_events, with fresh-process extension-event regression coverage included in the passing suite. No reviewer conversations were resolved.
Regression evidence: the initial 13 new cases produced 12 failures and one legacy-compatibility pass on a72c6bc before the fixes. All 17 new cases now pass within the complete adapter suite.
Source and wheel distribution builds also passed, with artifacts stored outside the checkout. Requesting another review after this update.
AI disclosure: posted on behalf of contributor @mnriem by GitHub Copilot, model GPT-6.1 Sol, in autonomous mode. The agent authored the implementation, regression tests, documentation changes, commit, and this response, and executed the validation and publication commands. No human line-by-line review is claimed.
Rollback prunes parent directories for every newly written path without recording whether those parents existed before the transaction. If an adapter writes into a pre-existing empty output tree and a later commit step fails, removing the generated leaf also removes the user's original empty directories, contrary to the operation-owned rollback contract. Journal parent existence (or restore from the initial directory snapshot) and stop pruning at the first pre-existing parent.
Validate overlap between an adapter's own root paths
src/specify_cli/integrations/installer.py:590
The documented adapter contract rejects overlap between the primary root and legacy_dir, but this validation only compares both roots against other integrations. An adapter can therefore declare legacy_dir equal to or nested under its own config.folder, causing registration and cleanup to treat the same tree as both current and legacy. Validate pairwise overlap within roots before the cross-integration check.
Trust containment now checks the actual user-local trust path against the canonical project root. A checkout rooted at ~/.specify, including a symlink alias, cannot supply execution grants. Candidate preparation validates the trust path and local state before importing Python, so a refused install does not execute the candidate first. Public-path regressions cover both installation and loading with a forged in-project grant, malformed local state, and valid home-directory projects that do not contain the trust path.
The file journal now records parent-directory existence and stops rollback pruning at the first original parent. It uses pre-operation scope snapshots when record_existing() runs after an adapter has created directories, rather than treating those new directories as original. Tests cover multiple writes, absent and pre-existing output trees at several depths, settings outside the adapter root, and built-in home-scoped output. Newly created empty parents are still removed; original empty directories remain.
Output validation now rejects pairwise overlap between the adapter's own primary and legacy roots before setup, independently of multi_install_safe. Coverage includes equal paths, ancestors, descendants, case-folded aliases, and distinct paths with a shared name prefix.
The event-refresh item already enters project_integrations(project_root) before looking up adapters in refresh_integration_events. The passing adapter suite includes fresh-process event-only extension add/remove and enable/disable regressions. No reviewer conversations were resolved.
Before these fixes, the initial 35 new cases produced 26 failures and 9 passes on b470eb1. The final suite includes 37 new cases, all passing.
Source and wheel builds passed, with artifacts stored outside the checkout. Requesting another review after this update.
AI disclosure: posted on behalf of contributor @mnriem by GitHub Copilot, model GPT-6.1 Sol, in autonomous mode. The agent authored the implementation, regression tests, documentation changes, commit, and this response, and executed the validation and publication commands. No human line-by-line review is claimed.
_pending_root is set and _refresh_configs() runs before the try/finally is entered. If refreshing a registrar cache raises, the context manager fails during __enter__, so its cleanup never runs; the unpersisted candidate remains registered and later commands see the project as still installing. Move the refresh into the protected block so _pending_root is always cleared and the candidate is unloaded.
Replace eager output-tree copies with metadata inventories and lazy bounded before-write snapshots. Preserve new and unchanged record_existing output while reporting unsupported unobserved overwrites explicitly. Protect candidate cache refresh with cleanup and align artifact error tests with the shared-operation contract used in CI.
Assisted-by: GitHub Copilot (model: GPT-6.1 Sol, autonomous)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Addressed review 5444897787 and the failing CI tests in 86f209a.
Lifecycle transactions no longer copy entire agent roots or .specify/integrations. Their initial census retains names and filesystem metadata without reading file contents; before-write observations create preimages lazily. Snapshot copying has aggregate budgets of 128 MiB of file content and 4,096 entries, including bounded reads if a file grows during copying. Tests measure zero eager snapshot bytes for repeated install, active use, and no-op switch, and verify budget refusal before overwriting an existing file, including an exact-limit success case.
New files and unchanged files still support record_existing() without an eager content copy. Overwriting an existing file must use IntegrationManifest.record_file() or the host before-write primitives. An overwrite performed outside that contract is reported explicitly as unrecoverable, and its resulting file is retained rather than deleted during rollback. The design and reference documentation now state this requirement. The filesystem-recovery test verifies that the previous package descriptor is retained for manual restoration, rather than expecting an eager copy of untouched registry metadata.
Candidate cache refresh now runs inside the cleanup-protected block. Failure during context entry clears the pending root, unloads the candidate and synthetic imports, and restores derived agent/registrar configuration. Both cache failure paths are exercised through public installation, followed by successful installation in another project.
The two CI failures were reproduced in an isolated merge of ee49e75 with upstream/main. The shared artifact.list operation intentionally sanitizes its resolution-error message. The fresh-process tests now accept that stable JSON contract as well as the prior detailed message, while still requiring exit 1, empty stdout, exactly one JSON error field, and no execution of the damaged adapter. The isolated merged adapter suite now passes. The existing Ruff job was successful; pinned Ruff was re-run successfully against both source trees rather than making unrelated lint edits.
The event-refresh item remains covered by the existing project_integrations(project_root) scope in refresh_integration_events and passing fresh-process event-only extension regressions. No reviewer conversations were resolved.
Before the fixes, the ten new review regression cases produced 8 failures and 2 passes. The isolated upstream merge separately reproduced the two CI artifact-list failures.
Adapter suite in the isolated upstream merge, using this worktree's interpreter and merged-source PYTHONPATH
295 passed
uvx --offline ruff@0.15.0 check src tests
Passed in the worktree and isolated upstream merge
.venv/bin/specify integration --help
Passed
git diff --check
Passed
Source and wheel builds passed. Isolated merged sources and temporary recovery-test snapshots were cleaned up; evidence logs remain outside the checkout. Requesting another review after this update.
AI disclosure: posted on behalf of contributor @mnriem by GitHub Copilot, model GPT-6.1 Sol, in autonomous mode. The agent authored the implementation, regression tests, documentation changes, commit, and this response, and executed the validation and publication commands. No human line-by-line review is claimed.
A zero exit from the wrapped upgrade is treated as a successful candidate install. When an external integration's generated-file manifest is missing, integration_upgrade() prints “Nothing to upgrade” and raises Exit(0), but this block sees the key still present in integration state and proceeds to persist_package(). The executable package is therefore upgraded without regenerating its files, despite the command saying it made no upgrade. Re-raise the no-op exit for candidate upgrades (or only persist after normal handler completion), and cover the missing-manifest case.
Persists class defaults instead of effective registrar configuration
src/specify_cli/integrations/installer.py:893
This overwrites explicit registrar-level values with the class defaults. The registrar's established effective semantics are that registrar_config.invoke_separator wins over the class attribute, while dev_no_symlink is enabled by either source (agents.py:34-44 and base.py:206-214). An external adapter that uses those supported registrar fields therefore renders correctly while healthy, but stores different recovery configuration; forced recovery can then unregister or restore artifacts using the wrong settings. Persist the effective values rather than the raw class defaults.
🧠 Review effort: Balanced
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Support trusted, catalog-installed external AI-agent adapters without source-registry edits, pip installation, or copied core command inventories. A standalone archive containing root
integration.ymland__init__.pyexports a matchingIntegrationBasesubclass and renders Spec Kit's host templates through the existing integration bases.The adapter-only descriptor requires identity, version, metadata, and host/tool requirements. Existing optional
providesmetadata remains compatible. Catalog identity/version/metadata, package descriptors, class configuration, checksums, source policy, download restrictions, and safe extraction are validated before installation.Executable packages and provenance are stored separately from generated-file manifests. Fresh CLI processes load verified, trusted project-local implementations before rendering, registration, selection, status, or workflow dispatch; catalog metadata operations do not import adapter code. Project changes unload synthetic imports and refresh registry/configuration caches.
Lifecycle operations preserve existing built-in and generic behavior, active-integration handling, extension/preset contributions, script variants, and edited-file preservation. Failed operations restore only operation-owned writes, not independent workflow progress or unowned user files. Concurrent managed-file conflicts retain explicit recovery snapshots. Forced upgrade/uninstall can recover a damaged target using validated ownership metadata without bypassing source policy or replacement trust.
Tests use neutral
sample-agentpackages and loopback HTTP fixtures. Directly related integration design, catalog, contribution, and reference documentation describes the final contract and public commands. The branch is rebased onto upstreammain.Testing
uv run specify --helpuv sync && uv run pytestThe full suite uses this worktree's own virtualenv, per
CONTRIBUTING.md, rather than the literaluv run pytestchecklist command, which can resolve an editable install from another checkout.uv sync --extra test1.1.2.dev0manifestuv run specify --helpLC_ALL=en_US.UTF-8 .venv/bin/python -m pytest tests -q --tb=shortruff check src/specify_cli/integrations/_file_changes.py src/specify_cli/integrations/_lifecycle.py src/specify_cli/integrations/installer.py tests/specify_cli/integrations/test_installed_adapters.pyruff check --select F src/specify_cli/presets/_manager.py src/specify_cli/presets/_manager_commands.py src/specify_cli/presets/_manager_skills.py src/specify_cli/shared_infra.pygit diff upstream/main...HEAD --checkuv build --out-dir distThe 114 adapter cases cover adapter-only descriptors, actual catalog installation, trust denial/discovery-only sources, fresh-process loading, host command/skill rendering, extension/preset registration and cleanup, runtime dispatch with harmless process doubles, upgrades/uninstall, edited files, rollback, invalid metadata/classes/imports, built-in collisions, unsafe paths/symlinks, damaged-target recovery, and project/cache isolation.
Sixteen scope regressions were also run against the earlier implementation: all failed before the fixes and passed afterward. Sample-project scaffolding/lifecycle checks use temporary executable/process doubles, not authenticated model execution.
AI Disclosure
AI disclosure: Implemented with GitHub Copilot, powered by GPT-6.1 Sol (
gpt-6.1-sol), in autonomous agent mode; reasoning-effort setting was not explicitly selected. AI assistance covered investigation, implementation, regression tests, documentation, rebase, validation, and PR-description drafting. No human line-by-line review is claimed.