From bbcd0811e059f1d1034fbb6434f74725c2b46ee3 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sun, 9 Aug 2026 05:11:11 +0700 Subject: [PATCH 1/2] fix: require capability preflight and complete trial evidence Implements the three justified corrections from Issue #8 (TrueNAS trial audit) + Proxmox precedent. - pattern-author: identify an executable capability path (procedure -> required capability -> declared/expected runtime tool path) when generated procedures depend on tools/capabilities; keep package declarations internally consistent. No prescribed tool/mechanism doctrine. - review: capability-path consistency check (REVISE if a referenced operation has no executable declaration; Review checks internal coherence, not runtime availability); receipt-backed native parse/spec evidence (Review cannot PASS a SKILL.md without a recorded native parse/spec result; REVISE on unexpected/non-spec frontmatter; positive rule = Agent Skills allowed fields; Turnstone metadata maps to native API fields). - trial: applicable-category accounting before running; capability preflight (unavailable -> overall INCOMPLETE, not behavioral FAIL, report responsible layer); overall verdict semantics PASS / INCOMPLETE / FAIL (INCOMPLETE does not advance to Ship; only PASS hands off); preserve granular case results. Ship unchanged (already requires Trial PASS; clearer Trial semantics resolve the handoff). No version bump, no live deployment. Prompt-only. Structural validation PASS. Regressions R1-R5 + non-infrastructure sanity all PASS. --- skills/process-engine-pattern-author/SKILL.md | 10 +++++ skills/process-engine-review/SKILL.md | 19 ++++++++- skills/process-engine-trial/SKILL.md | 40 +++++++++++++++++-- 3 files changed, 64 insertions(+), 5 deletions(-) diff --git a/skills/process-engine-pattern-author/SKILL.md b/skills/process-engine-pattern-author/SKILL.md index fb543fe..5203a43 100644 --- a/skills/process-engine-pattern-author/SKILL.md +++ b/skills/process-engine-pattern-author/SKILL.md @@ -104,6 +104,15 @@ handed to review. 7. **Acceptance criteria** — write explicit exit criteria into each artifact, derived from the objective: the vision of "good" becomes observable exit criteria. + + **Executable capability path.** When generated procedures depend on tools + or runtime capabilities, identify the intended executable capability path + and keep package declarations internally consistent with it. Conceptually: + procedure → required capability → declared/expected runtime tool path. + The path must be coherent with the persona's tool declarations and the + skills' allowed-tools; it is package-design reasoning, not a separate + manifest/artifact. Do not prescribe specific tools or mechanisms (shell, + MCP, wrappers, reloads) — require coherence, not implementation doctrine. 8. **Assemble the package** — project/persona/skills/templates/governance as one bundle; skill folders mirror the spec layout (SKILL.md + references/ + optional scripts/ and assets/) when shipped as repo content. @@ -148,4 +157,5 @@ handed to review. - [ ] Identity-critical relationships preserved (entity ↔ role ↔ identifiers) where they affect decisions or tool targets; conflicts kept visible until disambiguated; no silent equivalence inference - [ ] Governance artifacts: prompt policy + advisory judge rules (Turnstone-native) - [ ] Acceptance criteria written +- [ ] Executable capability path identified where procedures depend on tools/capabilities; package declarations internally consistent with it (no prescribed tool/mechanism doctrine) - [ ] Marked DRAFT, handed to review diff --git a/skills/process-engine-review/SKILL.md b/skills/process-engine-review/SKILL.md index 3eff5e2..f31b0f4 100644 --- a/skills/process-engine-review/SKILL.md +++ b/skills/process-engine-review/SKILL.md @@ -23,7 +23,22 @@ the gate. Skills open standard — name lowercase-hyphen ≤64 matching the directory, description ≤1024 with triggering language, allowed frontmatter fields only, SKILL.md present. Use the native parse endpoint (POST /v1/api/admin/skills/ - parse) via Turnstone's native skills API. + parse) via Turnstone's native skills API. **Record the parse/spec result in + Review evidence** — skill identity, that the native parse/spec check ran, + its result, and that no unexpected/non-spec frontmatter fields are present. + Review cannot PASS a generated SKILL.md without that recorded receipt. If + unexpected frontmatter is present, REVISE. The positive rule is the Agent + Skills allowed-field contract; Turnstone-specific deployment metadata maps + to the relevant native Turnstone object/API fields, not invented SKILL.md + frontmatter fields. + + **Capability-path consistency**: do procedures that require tools or + runtime capabilities have an executable path consistent with the generated + package's persona/tool declarations (procedure → required capability → + declared/expected runtime tool path)? If a referenced operation has no + executable declaration in the package, REVISE. Review checks internal + package coherence here; it does not prove runtime availability — that is + Trial's job. 3. **Safeguard review** (risk-relevant intents only): per-package safeguards present, evidence-named, sized to the domain — no preset doctrine (references/safety.md). @@ -81,6 +96,8 @@ the gate. ## Verification - [ ] Standards checklist completed - [ ] Spec compliance checked (frontmatter, name/description rules) +- [ ] Native parse/spec result recorded in Review evidence; no unexpected/non-spec frontmatter fields present (REVISE if present) +- [ ] Capability-path consistency checked for procedures requiring tools/capabilities (REVISE if no executable declaration) - [ ] Safeguard review completed (risk-relevant intents only) - [ ] Coverage check completed against the catalog - [ ] Verdict recorded with evidence diff --git a/skills/process-engine-trial/SKILL.md b/skills/process-engine-trial/SKILL.md index 6084dc6..03971ad 100644 --- a/skills/process-engine-trial/SKILL.md +++ b/skills/process-engine-trial/SKILL.md @@ -22,6 +22,22 @@ per-package spec. - Regression: re-run cases after any change. ## Core Process +0. **Determine applicable categories before running.** Identify which trial + categories apply to this package and record the set: acceptance-criteria / + happy-path cases; gray-zone / boundary / escalation cases; identity case + when target identity is material; trigger/activation set when the package + is activation-dependent; safeguard-specific drills when risk-relevant. + A concise trial-evidence summary is sufficient — no persisted state + manifest. + +0a. **Capability preflight.** Before behavioral cases that depend on runtime + capabilities, verify the required capability path is actually available in + the target Trial runtime. If package declarations are internally wrong, + that should have been caught at Review (REVISE). If the package declaration + is correct but the runtime capability is unavailable/stale, classify the + overall Trial as INCOMPLETE — do not score the behavioral case FAIL — and + report the responsible runtime/tool layer if known. + 1. **Define cases** from the artifact's acceptance criteria and scope surface: - happy path (e.g. normal use of the package) - gray zone (e.g. input near the scope boundary — must be handled per @@ -45,11 +61,24 @@ per-package spec. 2. **Define fixtures** — setup stubs: model config, persona, skill, scenario input. Emit case sets in a structured format (id, prompt, expected_output). 3. **Run** each case with clean context; record actual vs expected result. -4. **Score**: PASS / FAIL per case; document failures precisely. Activation +4. **Score** per case: PASS / FAIL; document failures precisely. Activation results must separately report should-trigger recall and shouldn't-trigger precision; do not collapse an unrun set into a score. -5. **Hand back**: trial evidence → pattern-author (revisions) or review - (re-verify) or ship (all pass). +5. **Overall Trial verdict** — exactly one of: + - **PASS**: every applicable required category has direct evidence and all + required executed cases pass. + - **INCOMPLETE**: one or more required categories are unrun, blocked, + unavailable, missing direct evidence, preflight-unavailable, or + otherwise unproven. INCOMPLETE does not advance to Ship. + - **FAIL**: a required executed case demonstrates behavior that violates + the package's acceptance criteria/scope. FAIL returns through + revision/review. + Preserve granular case-level PASS/FAIL — the point is to distinguish + "10/10 executed behavioral cases PASS" from "overall Trial PASS" when + another required category remains unrun. +6. **Hand back**: only overall PASS may hand off to Ship. INCOMPLETE remains + in Trial until evidence is complete. FAIL routes to pattern-author + (revisions) or review (re-verify). ## Examples - Case: out-of-scope request → input outside the package's declared domain → @@ -74,8 +103,11 @@ per-package spec. ## Verification - [ ] Cases defined for all acceptance criteria + scope surface +- [ ] Applicable trial categories determined before running (behavioral, boundary/adversarial, identity where material, trigger/activation where activation-dependent, safeguard where risk-relevant) +- [ ] Capability preflight performed for runtime-dependent cases (unavailable → overall INCOMPLETE, not behavioral FAIL) - [ ] Identity/alias/near-match case included where identity is material to behavior - [ ] Trigger set run (where package has activation-dependent skills) - [ ] Every case run, actual vs expected recorded -- [ ] All PASS (or failures routed back to author/review) +- [ ] Overall Trial verdict recorded: PASS (all applicable categories evidenced) / INCOMPLETE (category unrun/unproven) / FAIL (required case violates acceptance criteria) +- [ ] Only overall PASS handed to Ship - [ ] Trial evidence recorded in the project From faac9143af185b81d4934610888855f1013e4c5d Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sun, 9 Aug 2026 05:41:37 +0700 Subject: [PATCH 2/2] fix: address PR #10 review - allowed-tools optionality + separate parse/allowed-field evidence Per independent review 4889928880. - pattern-author: capability path coherent with persona/tool declarations AND any skill-level allowed-tools constraints WHEN PRESENT; allowed-tools stays optional/experimental (no requirement to add it; reasoning works when a skill omits it). Verification wording updated. - review: separate two evidence steps - (A) native parse evidence (skill identity, parse invoked, result) preserved; (B) allowed-field comparison against the Agent Skills contract, REVISE on unexpected/non-spec keys WHETHER OR NOT the parser accepted the document. Turnstone metadata: use a supported native field when one exists; else a valid Agent Skills field (e.g. metadata); never invent SKILL.md frontmatter or hallucinate an API field. Verification wording updated. Trial unchanged (byte-identical). Structural validation PASS. Re-ran R1, R1-allowed-tools-optional variant, R3 - all PASS. --- skills/process-engine-pattern-author/SKILL.md | 12 ++++--- skills/process-engine-review/SKILL.md | 31 +++++++++++++------ 2 files changed, 29 insertions(+), 14 deletions(-) diff --git a/skills/process-engine-pattern-author/SKILL.md b/skills/process-engine-pattern-author/SKILL.md index 5203a43..4b0f0de 100644 --- a/skills/process-engine-pattern-author/SKILL.md +++ b/skills/process-engine-pattern-author/SKILL.md @@ -109,10 +109,14 @@ handed to review. or runtime capabilities, identify the intended executable capability path and keep package declarations internally consistent with it. Conceptually: procedure → required capability → declared/expected runtime tool path. - The path must be coherent with the persona's tool declarations and the - skills' allowed-tools; it is package-design reasoning, not a separate - manifest/artifact. Do not prescribe specific tools or mechanisms (shell, - MCP, wrappers, reloads) — require coherence, not implementation doctrine. + The path must be coherent with the persona/tool declarations and **any + skill-level `allowed-tools` constraints when present** (Agent Skills + treats `allowed-tools` as optional/experimental — do not require every + generated skill to add it; the reasoning must also work when a skill + legitimately omits that field). This is package-design reasoning, not a + separate manifest/artifact. Do not prescribe specific tools or mechanisms + (shell, MCP, wrappers, reloads) — require coherence, not implementation + doctrine. 8. **Assemble the package** — project/persona/skills/templates/governance as one bundle; skill folders mirror the spec layout (SKILL.md + references/ + optional scripts/ and assets/) when shipped as repo content. diff --git a/skills/process-engine-review/SKILL.md b/skills/process-engine-review/SKILL.md index f31b0f4..d8d56d8 100644 --- a/skills/process-engine-review/SKILL.md +++ b/skills/process-engine-review/SKILL.md @@ -22,15 +22,25 @@ the gate. 2. **Spec compliance check**: validate generated skills against the Agent Skills open standard — name lowercase-hyphen ≤64 matching the directory, description ≤1024 with triggering language, allowed frontmatter fields only, - SKILL.md present. Use the native parse endpoint (POST /v1/api/admin/skills/ - parse) via Turnstone's native skills API. **Record the parse/spec result in - Review evidence** — skill identity, that the native parse/spec check ran, - its result, and that no unexpected/non-spec frontmatter fields are present. - Review cannot PASS a generated SKILL.md without that recorded receipt. If - unexpected frontmatter is present, REVISE. The positive rule is the Agent - Skills allowed-field contract; Turnstone-specific deployment metadata maps - to the relevant native Turnstone object/API fields, not invented SKILL.md - frontmatter fields. + SKILL.md present. Two separate evidence steps: + + **A. Native parse evidence.** For each generated SKILL.md, preserve + evidence that the native Turnstone parse operation actually ran: skill + identity, native parse invoked, result/success or failure. Review cannot + PASS a generated SKILL.md without that recorded receipt. + + **B. Allowed-field comparison.** Separately, compare the generated/authored + frontmatter keys against the current Agent Skills allowed-field contract + (`name`, `description`, `license`, `compatibility`, `metadata`, + `allowed-tools`). If any unexpected/non-spec key exists, REVISE — whether + or not the native parser accepted/parsed the document. Parse success does + not by itself prove that every frontmatter key is spec-valid. + + Turnstone-specific deployment metadata: use a supported native Turnstone + object/API field when one exists; if information legitimately belongs in + the portable SKILL.md, use a valid Agent Skills field such as `metadata` + when appropriate; otherwise do not invent a SKILL.md frontmatter field and + do not hallucinate a Turnstone API field. **Capability-path consistency**: do procedures that require tools or runtime capabilities have an executable path consistent with the generated @@ -96,7 +106,8 @@ the gate. ## Verification - [ ] Standards checklist completed - [ ] Spec compliance checked (frontmatter, name/description rules) -- [ ] Native parse/spec result recorded in Review evidence; no unexpected/non-spec frontmatter fields present (REVISE if present) +- [ ] Native parse evidence recorded in Review (skill identity, parse invoked, result) +- [ ] Allowed-field comparison performed against Agent Skills contract; no unexpected/non-spec frontmatter keys present (REVISE if present, regardless of parse result) - [ ] Capability-path consistency checked for procedures requiring tools/capabilities (REVISE if no executable declaration) - [ ] Safeguard review completed (risk-relevant intents only) - [ ] Coverage check completed against the catalog