[RedEyeNinja-BKK]

fix: require capability preflight and complete trial evidence

Base SHA: d0e0a2d431dc32deac4ed362c1cff8a3023be564 (post-PR #9 main)
Head SHA (current): faac9143af185b81d4934610888855f1013e4c5d
Governing: Issue #8 (TrueNAS trial audit + change definition 5228316036, gate 5228410356); evidence PR #9 (merged d0e0a2d4…)
Precedent: Issue #1 (Proxmox trial)
Independent review addressed: 4889928880 (allowed-tools optionality; separate native-parse vs allowed-field evidence; Turnstone metadata wording; reviewable regression receipts)

Problem statement

Two downstream infrastructure trials showed green executed cases can be over-credited as full Trial PASS while required coverage remains unrun, and a generated package can describe procedures that depend on capabilities with no actually-executable path. Three narrow corrections are implemented.

Corrections

1. Executable capability path (pattern-author): identify the intended executable capability path (procedure → required capability → declared/expected runtime tool path); coherent with persona/tool declarations and any skill-level allowed-tools constraints when present - allowed-tools remains optional/experimental (no requirement to add it). No prescribed tool/mechanism doctrine.
2. Capability-path consistency + receipt-backed spec review (review): (a) REVISE if a referenced operation has no executable declaration (internal coherence; not runtime availability); (b) two separate evidence steps - A: native parse evidence (skill identity, parse invoked, result) preserved; B: allowed-field comparison against Agent Skills contract, REVISE on unexpected/non-spec keys whether or not the parser accepted the document. Turnstone metadata: supported native field when one exists; else a valid Agent Skills field (e.g. metadata); never invent SKILL.md frontmatter or hallucinate an API field.
3. Trial completeness semantics (trial): applicable-category accounting; capability preflight (unavailable → INCOMPLETE, not behavioral FAIL, report layer); overall PASS / INCOMPLETE / FAIL (INCOMPLETE does not advance to Ship; only PASS hands off); granular results preserved.

Files changed (3 overall; correction commit touched Pattern + Review only; Trial byte-identical)

• skills/process-engine-pattern-author/SKILL.md
• skills/process-engine-review/SKILL.md
• skills/process-engine-trial/SKILL.md

Structural validation

python3 tools/validate.py --repo . → STRUCTURAL VALIDATION PASS (exact-head run on faac914…).

Regression receipts

Sanitized direct evidence for R1-R5 + non-infrastructure sanity: comment 5228543823 on this PR. (R1, R1-allowed-tools-optional variant, R3 re-run on corrected wording - all PASS; R2/R4/R5 receipts from prior run, contract unchanged - all PASS.)

Non-changes

Ship, standards, skill-anatomy, governance, core, persona, templates, intake, safety, evidence-library unchanged. No evaluator framework, no code runtime, no manifest/state machine, no version bump, no live deployment.

Why this remains prompts-only / Turnstone-native

Prompt instructions only; capability/spec verification uses Turnstone native parsing + runtime preflight + Agent Skills allowed-field contract; no reimplementation of enforcement.

Do not merge; awaiting independent re-review.

[RedEyeNinja-BKK]

Independent implementation review on exact head bbcd0811e059f1d1034fbb6434f74725c2b46ee3.

Overall architecture/scope is strong: exactly three prompt skill files, no Ship change, no runtime/evaluator/manifest machinery, and exact-head structural-validation run 31281117452 is PASS. The PASS/INCOMPLETE/FAIL distinction is a good small correction to the repeated Proxmox/TrueNAS over-crediting problem, and capability-path preflight is correctly separated between package coherence (Review) and target-runtime availability (Trial).

I am not yet recommending merge. Please address these two prompt ambiguities and one evidence gap:

1. Do not conflate native parsing with Agent Skills field validation. The Review wording currently says native parse/spec result, which can imply that POST /skills/parse itself proves the allowed-field contract. The TrueNAS evidence did not establish that the parser rejects all extra fields. Make the evidence contract explicit: preserve the native parse result and separately compare the parsed/authored frontmatter keys against the Agent Skills allowed-field contract. Unexpected fields => REVISE regardless of whether the native parser accepted the document. This remains prompt reasoning; do not build a custom parser/schema validator.

2. Keep allowed-tools optional. Pattern currently says the capability path must be coherent with the skills' allowed-tools. Agent Skills treats that field as optional/experimental. Reword to coherence with persona/tool declarations and any skill allowed-tools constraints when present. Do not make capability-path reasoning force every generated skill to add allowed-tools.

Also tighten the Review sentence about Turnstone metadata: use native Turnstone object/API fields when such supported fields exist; otherwise do not invent SKILL.md frontmatter or hallucinate an API field.

3. Expose reviewable R1–R5 receipts. GitHub currently contains only summarized PASS claims for R1–R5 + the customer-returns sanity run. Before merge, post a sanitized PR comment (or Issue #8 evidence comment linked here) containing enough direct evidence for independent review: fixture/prompt or concise setup, the material output/verdict, and what observation satisfied each expected result. Preserve local detailed receipts, but do not ask reviewers to trust an agent summary. No new evaluator or repo evidence framework is needed.

After those narrow corrections: rerun structural validation, rerun only the regressions materially affected by the wording changes (at minimum R3, plus R1 if allowed-tools handling changes), update exact head, and stop for re-review.

No broader product change requested.

[RedEyeNinja-BKK]

Regression receipts - R1-R5 + non-infrastructure sanity (head faac9143…)

Sanitized direct evidence for independent review. Detailed local receipts retained with the operator. Each case: fixture/setup → expected → actual (material excerpt) → assessment. Prompt/behavior-based; no custom evaluator.

R1 - generated tool-path mismatch

• Fixture: generated procedure "check appliance temperature" requires capability read_sensor; persona tool_allowlist and skill declarations provide no executable path to any sensor/read capability.
• Expected: Review = REVISE; no Trial.
• Actual: Review verdict REVISE. Finding (excerpt): "Procedure 'check appliance temperature' requires capability read_sensor, but neither the persona tool_allowlist nor the skill declarations provide an executable path to any sensor/read capability." Capability-path receipt: procedure inspected; required capability; persona declaration none; skill declaration none; consistency FAIL. Trial not run.
• Assessment: PASS - corrected contract detects mismatch and stops before Trial.

R1-variant - allowed-tools absent but persona path valid

• Fixture: procedure "look up customer order" requires lookup_order; SKILL.md has no allowed-tools (legitimately omitted); persona declares order-lookup (valid path).
• Expected: Review must not fail for missing allowed-tools; capability path coherent via persona declaration.
• Actual: "Fail merely because allowed-tools is absent? NO. Capability-path coherence: Coherent. Review verdict on capability-path grounds: PASS. REVISE for missing allowed-tools? NO."
• Assessment: PASS - optionality preserved; reasoning works without the optional field.

R2 - runtime capability unavailable (receipt from prior run, contract unchanged)

• Fixture: package declaration valid (procedure references read_sensor; allowed-tools declares sensor-reader); Trial runtime does not expose sensor-reader.
• Expected: Trial preflight unavailable → overall INCOMPLETE, not behavioral FAIL; responsible runtime/tool layer reported.
• Actual (excerpt): "Capability preflight: UNAVAILABLE - sensor-reader is not exposed by the trial runtime. Behavioral case score: Not scored as FAIL. Responsible layer: Runtime/tool layer. Overall Trial verdict: INCOMPLETE. Ship handoff: No."
• Assessment: PASS - unavailable capability → INCOMPLETE, not behavioral FAIL; layer reported.

R3 - non-spec frontmatter (re-run on corrected wording)

• Fixture: SKILL.md frontmatter contains allowed_tools (underscore), risk_tier, category, tags, version, activation. Native Turnstone parse runs and succeeds.
• Expected: Step A native parse receipt recorded; Step B allowed-field comparison independently identifies unexpected keys; REVISE regardless of parse success; no Trial; no custom parser.
• Actual (excerpt): "Step A - Native parse invoked: Yes; Result: Parse success; Receipt recorded: Yes. Step B - Allowed keys recognized: name, description; Unexpected/non-spec keys: allowed_tools (underscore; allowed is allowed-tools), risk_tier, category, tags, version, activation. Review verdict: REVISE. Native parse success does not establish that the frontmatter keys comply with the Agent Skills allowed-field contract. Trial allowed: No. Custom parser/schema validator: No."
• Assessment: PASS - the key regression for this review correction: parse success ≠ spec-valid keys.

R4 - incomplete activation coverage (receipt from prior run, contract unchanged)

• Fixture: activation-dependent package; behavioral cases 10/10 PASS; required trigger/activation + routing set unrun.
• Expected: behavioral PASS retained; activation UNPROVEN; overall INCOMPLETE; no Ship.
• Actual (excerpt): "Executed behavioral cases: 10/10 PASS retained. Required activation set: UNPROVEN. Should-trigger cases: Unrun. Should-not-trigger near-misses: Unrun. Parent → child routing: Unrun. Overall Trial verdict: INCOMPLETE. Ship handoff: No."
• Assessment: PASS - the TrueNAS regression; granular results preserved, overall INCOMPLETE.

R5 - complete package (receipt from prior run, contract unchanged)

• Fixture: all applicable categories evidenced + PASS (behavioral, trigger recall/precision, routing, identity).
• Expected: overall Trial PASS; eligible for operator Ship gate (not shipped).
• Actual (excerpt): "All applicable categories: Evidenced. Overall Trial verdict: PASS. Operator Ship-gate eligibility: Yes. Ship executed: No."
• Assessment: PASS - complete evidence → PASS; eligibility only.

Non-infrastructure sanity - customer returns routing

• Fixture: parent router + two child skills (returns-eligibility, refund-processing); capability lookup_order declared as order-lookup.
• Expected: PASS/INCOMPLETE/FAIL + capability-path semantics hold outside infrastructure.
• Actual: Scenario A (activation unrun, capability available) → overall INCOMPLETE, behavioral PASS retained, no Ship. Scenario B (lookup service down) → preflight unavailable → INCOMPLETE, not behavioral FAIL, responsible layer = runtime/lookup service. Scenario C (all categories evidenced) → overall PASS, Ship-gate eligible only.
• Assessment: PASS - semantics are domain-neutral. This is a narrow cross-domain sanity regression, not a third/fourth full Process Engine product trial.

[RedEyeNinja-BKK]

Independent re-review on exact head faac9143af185b81d4934610888855f1013e4c5d: PASS.

Verified the correction commit is exactly one commit ahead of prior reviewed head bbcd0811e059f1d1034fbb6434f74725c2b46ee3 and modifies only process-engine-pattern-author and process-engine-review; process-engine-trial is byte-unchanged from the prior implementation head. Full PR scope remains exactly three prompt skill files with no Ship/runtime/evaluator/manifest/version changes.

The three prior review concerns are resolved:

1. allowed-tools remains explicitly optional/experimental. Capability-path reasoning now applies to persona/tool declarations and any skill-level allowed-tools constraints only when present. The R1 variant directly demonstrates that a valid persona capability path is accepted when allowed-tools is absent.
2. Native parsing and Agent Skills field validation are now explicitly separate evidence steps. Review requires a native parse receipt, then independently compares authored frontmatter keys to the allowed-field contract. R3 demonstrates the intended behavior: native parse succeeds, unexpected keys are independently identified, Review = REVISE, no Trial.
3. Sanitized R1-R5 + customer-returns receipts are exposed in PR comment 5228543823 with fixture, expected result, material actual output, and assessment. They are sufficient for this bounded prompt change; no new evaluator/evidence framework is warranted.

The PASS/INCOMPLETE/FAIL Trial semantics remain a small, appropriate correction to the repeated Proxmox/TrueNAS evidence-overcredit problem. Capability coherence remains in Review; target-runtime availability remains in Trial. Ship correctly remains unchanged.

Exact-head structural-validation workflow run 31282283705 concluded success.

No blocking finding remains. Recommendation: READY FOR PR #10 MERGE GATE. Do not merge without the operator gate, and keep live Turnstone reconciliation / version-release decisions separate after merge.