Add /selftest.extension core extension to test other extensions

[dhilipkumars]

This PR adds an experimental testing sandbox under tests/extension-commands to evaluate how LLMs/Agents parse extension specifications and execute their mapped commands.

Sample Output from Gemini LLM

In my Gemini CLI if i entered below prompt

/speckit.selftest.run retrospective

It outputs the following.

 
    1 ============================= test session starts ==============================
    2 collected 3 items
    3
    4 test_selftest_discovery.py::test_catalog_search [PASS]
    5   Details: Found 1 extension(s): Retrospective Extension (v1.0.0).
    6
    7 test_selftest_installation.py::test_extension_add [PASS]
    8   Details: Extension installed successfully! Provided commands: speckit.retrospective.analyze.
    9
   10 test_selftest_registration.py::test_config_verification [PASS]
   11   Details: .specify/extensions.yml contains an enabled hook for the 'retrospective' extension.
   12
   13 ============================== [3] passed in 2.3s ==============================

AI Usage disclosure: Used Aniti-gravity to build this out.

Future Goal: Currently its manual to trigger these tests but at some point we can build a workflow for co-pilot to run these /selftests

[Copilot]

Pull request overview

Adds an experimental sandbox under tests/extension-commands/ intended to evaluate whether LLM/agent workflows can discover extension command definitions and “execute” their mapped markdown instructions.

Changes:

• Introduces a mock Python entrypoint (main.py) that prints timestamped outputs for --lint and --deploy.
• Adds a .specify/ sandbox containing command markdown files (lint.md, deploy.md) and an extensions.yml mapping.
• Adds TESTING.md with a copy/paste prompt for running the exercise in an LLM chat.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/extension-commands/main.py	Mock CLI script for lint/deploy command execution output
tests/extension-commands/TESTING.md	Manual LLM evaluation instructions and expected terminal-style output
tests/extension-commands/.specify/extensions.yml	Declares command mappings for the sandbox
tests/extension-commands/.specify/lint.md	Markdown “command file” instructing how to run the mock linter
tests/extension-commands/.specify/deploy.md	Markdown “command file” instructing how to run the mock deploy

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dhilipkumars]

@mnriem toying with some ideas for adding tests for extension. I'm starting with 'extension' commands. If you like it this pattern i can add more tests. this is manual for now but we can think of wiriting it up with the @copilot code review[agent] workflow at somepoint.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dhilipkumars]

@copilot code review[agent]
follow the instructions here tests/extension-commands/TESTING.md

[mnriem]

Wild idea, maybe as an extension with its own command /selftest "extension" ?

[dhilipkumars]

Interesting. Where do you think such a test extension should live? in this repo or outside as a community?

[mnriem]

Probably in the core repo under extensions/selftest ?

[dhilipkumars]

perfect let me re-work my PR

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mnriem]

Can you address the Copilot feedback where applicable. If not applicable, please explain why.

[dhilipkumars]

Can you address the Copilot feedback where applicable. If not applicable, please explain why.

@mnriem sure, on it. Have made some changes to address your previous comment was testing them locally last night. i should be get them changes in tonight or tomorrow.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dhilipkumars]

Can you address the Copilot feedback? Note this will be an official extension so it should be in the regular catalog.json and then refer to the full URL it will land on once we merge it in (extensions/selftest directory) as the download URL? Do we support referring to a directory instead of a tgz file? If we do NOT we should ;)

yes we do have that already as dev

specify extension add --help
                                                                                    
 Usage: specify extension add [OPTIONS] EXTENSION                                   
                                                                                    
 Install an extension.                                                              
                                                                                    
╭─ Arguments ──────────────────────────────────────────────────────────────────────╮
│ *    extension      TEXT  Extension name or path [required]                      │
╰──────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────╮
│ --dev               Install from local directory                                 │
│ --from        TEXT  Install from custom URL                                      │
│ --help              Show this message and exit.                                  │
╰──────────────────────────────────────────────────────────────────────────────────╯

apart from that i have addressed all copilots review comments and updated catalog.json (not the community one). PTAL @mnriem

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mnriem]

@dhilipkumars We probably should remove it from catalog.json then? And rely on the --dev mechanism for the self-test extension? Thoughts?

[dhilipkumars]

@dhilipkumars We probably should remove it from catalog.json then? And rely on the --dev mechanism for the self-test extension? Thoughts?

Happy to move it out if you think that's the right thing to do, but i feel this is a good candidate for core catalog, if not this what are the extensions you have in mind that would go in core catalog?

[mnriem]

OK you are right. Lets keep it in the catalog

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dhilipkumars]

@mnriem looks like this is ready to be merged.