diff --git a/skills/process-engine-pattern-author/SKILL.md b/skills/process-engine-pattern-author/SKILL.md index fb543fe..4b0f0de 100644 --- a/skills/process-engine-pattern-author/SKILL.md +++ b/skills/process-engine-pattern-author/SKILL.md @@ -104,6 +104,19 @@ handed to review. 7. **Acceptance criteria** — write explicit exit criteria into each artifact, derived from the objective: the vision of "good" becomes observable exit criteria. + + **Executable capability path.** When generated procedures depend on tools + or runtime capabilities, identify the intended executable capability path + and keep package declarations internally consistent with it. Conceptually: + procedure → required capability → declared/expected runtime tool path. + The path must be coherent with the persona/tool declarations and **any + skill-level `allowed-tools` constraints when present** (Agent Skills + treats `allowed-tools` as optional/experimental — do not require every + generated skill to add it; the reasoning must also work when a skill + legitimately omits that field). This is package-design reasoning, not a + separate manifest/artifact. Do not prescribe specific tools or mechanisms + (shell, MCP, wrappers, reloads) — require coherence, not implementation + doctrine. 8. **Assemble the package** — project/persona/skills/templates/governance as one bundle; skill folders mirror the spec layout (SKILL.md + references/ + optional scripts/ and assets/) when shipped as repo content. @@ -148,4 +161,5 @@ handed to review. - [ ] Identity-critical relationships preserved (entity ↔ role ↔ identifiers) where they affect decisions or tool targets; conflicts kept visible until disambiguated; no silent equivalence inference - [ ] Governance artifacts: prompt policy + advisory judge rules (Turnstone-native) - [ ] Acceptance criteria written +- [ ] Executable capability path identified where procedures depend on tools/capabilities; package declarations internally consistent with it (no prescribed tool/mechanism doctrine) - [ ] Marked DRAFT, handed to review diff --git a/skills/process-engine-review/SKILL.md b/skills/process-engine-review/SKILL.md index 3eff5e2..d8d56d8 100644 --- a/skills/process-engine-review/SKILL.md +++ b/skills/process-engine-review/SKILL.md @@ -22,8 +22,33 @@ the gate. 2. **Spec compliance check**: validate generated skills against the Agent Skills open standard — name lowercase-hyphen ≤64 matching the directory, description ≤1024 with triggering language, allowed frontmatter fields only, - SKILL.md present. Use the native parse endpoint (POST /v1/api/admin/skills/ - parse) via Turnstone's native skills API. + SKILL.md present. Two separate evidence steps: + + **A. Native parse evidence.** For each generated SKILL.md, preserve + evidence that the native Turnstone parse operation actually ran: skill + identity, native parse invoked, result/success or failure. Review cannot + PASS a generated SKILL.md without that recorded receipt. + + **B. Allowed-field comparison.** Separately, compare the generated/authored + frontmatter keys against the current Agent Skills allowed-field contract + (`name`, `description`, `license`, `compatibility`, `metadata`, + `allowed-tools`). If any unexpected/non-spec key exists, REVISE — whether + or not the native parser accepted/parsed the document. Parse success does + not by itself prove that every frontmatter key is spec-valid. + + Turnstone-specific deployment metadata: use a supported native Turnstone + object/API field when one exists; if information legitimately belongs in + the portable SKILL.md, use a valid Agent Skills field such as `metadata` + when appropriate; otherwise do not invent a SKILL.md frontmatter field and + do not hallucinate a Turnstone API field. + + **Capability-path consistency**: do procedures that require tools or + runtime capabilities have an executable path consistent with the generated + package's persona/tool declarations (procedure → required capability → + declared/expected runtime tool path)? If a referenced operation has no + executable declaration in the package, REVISE. Review checks internal + package coherence here; it does not prove runtime availability — that is + Trial's job. 3. **Safeguard review** (risk-relevant intents only): per-package safeguards present, evidence-named, sized to the domain — no preset doctrine (references/safety.md). @@ -81,6 +106,9 @@ the gate. ## Verification - [ ] Standards checklist completed - [ ] Spec compliance checked (frontmatter, name/description rules) +- [ ] Native parse evidence recorded in Review (skill identity, parse invoked, result) +- [ ] Allowed-field comparison performed against Agent Skills contract; no unexpected/non-spec frontmatter keys present (REVISE if present, regardless of parse result) +- [ ] Capability-path consistency checked for procedures requiring tools/capabilities (REVISE if no executable declaration) - [ ] Safeguard review completed (risk-relevant intents only) - [ ] Coverage check completed against the catalog - [ ] Verdict recorded with evidence diff --git a/skills/process-engine-trial/SKILL.md b/skills/process-engine-trial/SKILL.md index 6084dc6..03971ad 100644 --- a/skills/process-engine-trial/SKILL.md +++ b/skills/process-engine-trial/SKILL.md @@ -22,6 +22,22 @@ per-package spec. - Regression: re-run cases after any change. ## Core Process +0. **Determine applicable categories before running.** Identify which trial + categories apply to this package and record the set: acceptance-criteria / + happy-path cases; gray-zone / boundary / escalation cases; identity case + when target identity is material; trigger/activation set when the package + is activation-dependent; safeguard-specific drills when risk-relevant. + A concise trial-evidence summary is sufficient — no persisted state + manifest. + +0a. **Capability preflight.** Before behavioral cases that depend on runtime + capabilities, verify the required capability path is actually available in + the target Trial runtime. If package declarations are internally wrong, + that should have been caught at Review (REVISE). If the package declaration + is correct but the runtime capability is unavailable/stale, classify the + overall Trial as INCOMPLETE — do not score the behavioral case FAIL — and + report the responsible runtime/tool layer if known. + 1. **Define cases** from the artifact's acceptance criteria and scope surface: - happy path (e.g. normal use of the package) - gray zone (e.g. input near the scope boundary — must be handled per @@ -45,11 +61,24 @@ per-package spec. 2. **Define fixtures** — setup stubs: model config, persona, skill, scenario input. Emit case sets in a structured format (id, prompt, expected_output). 3. **Run** each case with clean context; record actual vs expected result. -4. **Score**: PASS / FAIL per case; document failures precisely. Activation +4. **Score** per case: PASS / FAIL; document failures precisely. Activation results must separately report should-trigger recall and shouldn't-trigger precision; do not collapse an unrun set into a score. -5. **Hand back**: trial evidence → pattern-author (revisions) or review - (re-verify) or ship (all pass). +5. **Overall Trial verdict** — exactly one of: + - **PASS**: every applicable required category has direct evidence and all + required executed cases pass. + - **INCOMPLETE**: one or more required categories are unrun, blocked, + unavailable, missing direct evidence, preflight-unavailable, or + otherwise unproven. INCOMPLETE does not advance to Ship. + - **FAIL**: a required executed case demonstrates behavior that violates + the package's acceptance criteria/scope. FAIL returns through + revision/review. + Preserve granular case-level PASS/FAIL — the point is to distinguish + "10/10 executed behavioral cases PASS" from "overall Trial PASS" when + another required category remains unrun. +6. **Hand back**: only overall PASS may hand off to Ship. INCOMPLETE remains + in Trial until evidence is complete. FAIL routes to pattern-author + (revisions) or review (re-verify). ## Examples - Case: out-of-scope request → input outside the package's declared domain → @@ -74,8 +103,11 @@ per-package spec. ## Verification - [ ] Cases defined for all acceptance criteria + scope surface +- [ ] Applicable trial categories determined before running (behavioral, boundary/adversarial, identity where material, trigger/activation where activation-dependent, safeguard where risk-relevant) +- [ ] Capability preflight performed for runtime-dependent cases (unavailable → overall INCOMPLETE, not behavioral FAIL) - [ ] Identity/alias/near-match case included where identity is material to behavior - [ ] Trigger set run (where package has activation-dependent skills) - [ ] Every case run, actual vs expected recorded -- [ ] All PASS (or failures routed back to author/review) +- [ ] Overall Trial verdict recorded: PASS (all applicable categories evidenced) / INCOMPLETE (category unrun/unproven) / FAIL (required case violates acceptance criteria) +- [ ] Only overall PASS handed to Ship - [ ] Trial evidence recorded in the project