Activity

1. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026 – with ChatGPT Codex Connector

   OwnerAuthor

   More actions

   Reviewer steering — preserve the trial, separate the layers before changing Process Engine

   This is useful second-domain evidence, and the identity-preservation improvement from v1.9.6 appears to have held. I would not open a Process Engine change yet from the three lessons as currently framed. First do one evidence-audit pass that separates package defects, Turnstone/MCP wiring, and trial-method gaps.

   1. Lesson 1 is potentially a real Process Engine finding — but split the causes

   The first run exposed two different conditions:

   • package/tool-path defect: the persona's procedures referenced an executable path that its effective tool allowlist did not grant;
   • runtime/session condition: the orchestrating session predated the MCP registration.

   The generalizable PE question is not “always add bash/read_file.” It is:

   Does every generated procedure that depends on tools have at least one actually executable capability path in the target Turnstone context, and did Review/Trial verify that path before behavioral cases?

   That is a plausible Pattern/Review/Trial acceptance criterion. Keep the Turnstone session lifecycle cause separate.

   2. Lesson 2 belongs primarily to Turnstone/MCP lifecycle evidence, not PE doctrine

   “Reload twice” is an observed deployment/runtime behavior, not yet a Process Engine rule. PE Trial may reasonably preflight required tool availability/reachability before diagnosing a package, but it should not encode a second-reload ritual unless Turnstone itself establishes that as a stable contract.

   3. risk_tier needs deeper review before calling this a documentation lesson

   Current v1.9.6 standards say generated SKILL.md frontmatter uses Agent Skills–allowed fields only: license, compatibility, metadata, allowed-tools (plus required name/description). Pattern and Review both explicitly require spec-valid frontmatter.

   Issue #8 says the generated parent skill authored risk_tier in frontmatter. If that is literal, the more important finding may be:

   Pattern produced a non-spec field and Review failed to return REVISE.

   Please inspect the actual generated SKILL.md plus the native parse/review receipt and answer:

   • Was risk_tier truly in SKILL.md YAML frontmatter, or was it Turnstone-side object metadata outside the Agent Skills document?
   • Did POST /v1/api/admin/skills/parse accept/reject it, and was that parse result actually part of Review evidence?
   • If it was non-spec frontmatter, why did Review PASS?

   Do not solve that merely by documenting risk_level scanner behavior.

   4. The 10/10 result does not yet prove the full PE Trial contract

   The issue says the harness supplied the generated persona + the relevant child skill to each task agent. That proves behavior with the intended skill already selected, but v1.9.6 Trial also requires should-trigger / shouldn't-trigger near-miss sets where skills are activation-dependent.

   With one parent router + six child skills, activation/routing is material. Please establish whether trigger tests were actually run. If not, classify the present 10/10 as behavioral case PASS with activation coverage still unproven, not complete trial proof.

   Also preserve the first tool-less run as evidence. The canonical rerun can PASS, but the initial failure is product-learning evidence and should not disappear behind the green score.

   5. Clarify “deployment live” vs “Ship pending”

   Issue #8 currently says both:

   • deployment is live on Turnstone; and
   • Ship gate is pending operator approval.

   That may be a legitimate controlled trial deployment, but call it that explicitly and record how it differs from a shipped/accepted package. PE's Ship skill says deployment follows Review PASS + Trial PASS + explicit operator approval, so we should not blur trial instrumentation with final Ship state.

   6. Do not over-generalize identity placement

   The positive pattern should be “preserve identity-critical mappings,” not “put identity/hazard maps in the persona.” v1.9.6 deliberately distinguishes:

   • stable, behavior-defining identity → may live in persona;
   • mutable addresses/endpoints/interfaces/operational identifiers → references/resources by default.

   Because Issue #8 is sanitized, it does not show which category the TrueNAS map contained. Verify placement before declaring the persona embedding pattern universal.

   7. Classify the manifest

   Issue #8 lists a creation checklist + manifest as generated package artifacts. v1.9.6 intentionally removed manifest mechanics as a Process Engine dependency. Clarify whether this was merely a local trial/deployment evidence record or was treated as a required generated product artifact. Do not reintroduce a manifest contract through downstream package habits.

   Steering / next evidence pass

   Before any PE branch/PR, please update this issue with an evidence audit covering:

   1. exact initial failure → correction → rerun chronology;
   2. full pipeline evidence, including Intent/Orient/Collect/Clarify/Objective/Summary Gate if available (the issue currently begins at Pattern);
   3. actual SKILL.md frontmatter + native parse/review result for risk_tier;
   4. effective persona/tool capability mapping before and after correction;
   5. MCP availability preflight evidence and session/reload chronology;
   6. whether should-trigger / shouldn't-trigger routing tests ran for parent + six child skills;
   7. controlled trial deployment vs final Ship distinction;
   8. identity-map placement classification (stable vs mutable facts);
   9. manifest classification (evidence artifact vs product contract);
   10. case-by-case eval value: what each test proves, and whether any are redundant/ceremonial.

   Then classify each finding by layer: Process Engine / generated package / Turnstone runtime / MCP wiring / TrueNAS environment / evaluation method / operator preference.

   My current provisional disposition is IMPROVE / COMPLETE THE EVIDENCE AUDIT BEFORE PRODUCT CHANGE. The most credible PE candidates are (a) executable tool-path verification and (b), if confirmed, a Review/spec-compliance escape around risk_tier. The MCP second-reload behavior should not become PE prompt doctrine on this evidence alone.

   Also: after this audit, I recommend the next trial move out of infrastructure/admin so we do not overfit Process Engine to Proxmox/TrueNAS patterns.
2. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026

   OwnerAuthor

   More actions

   Addendum — wiring, ship, and run records (2026-08-09)

   Status update: Package SHIPPED — operator GO received; manifest stage_status = shipped. Read-back verified at deploy.

   4. Read-only enforcement for worker agents

   truenas-mcp v0.0.6 has no server-side read-only flag (unlike the Proxmox MCP's PROXMOX_READ_ONLY). The generated package relies on discipline for the overseer persona. For the two worker agents, we added enforcement instead:

   • A small read-only MCP wrapper (truenas-mcp-readonly) wraps the binary and exposes only the 31 read-only tools from the manifest. It filters tools/list and rejects any tools/call to a write-capable name with a JSON-RPC error (-32001), before the child process spawns — so a write call cannot reach the appliance.
   • Overseer keeps the full 52-tool surface (operator-gated, persona discipline). Workers get the 31-tool wrapper only.

   Verified end-to-end on both workers: tools/list shows exactly 31 read-only tools; system_info returns live version; a create_dataset call returns the wrapper's READ-ONLY rejection verbatim and performs no side effect.

   5. Run records (worker wiring)

   Wiring was performed by the executor agent under explicit operator direction, with per-command approval gates exercised:

   • Worker A (executor gateway): config backup → MCP server entry added (command = wrapper, args [], env {}) with all existing servers preserved → gateway restart → mcp test confirms 31 read-only tools live.
   • Worker B (chat/emission gateway): config backup → server entry added under mcp.servers (existing servers preserved: three entries) → gateway restart → health {"ok":true} → agent-run verification confirms 31 read-only tools live + write rejection.

   Findings recorded:

   1. Run-retention loss across self-restart (known class): the first wiring run's record was lost when the executor gateway restarted itself mid-task (run run_not_found after restart). The config change had actually completed; verification had to be done via a fresh run. This matches the documented executor run-retention gap and is why the process records "config changed + restart" as INDETERMINATE until independently verified.
   2. Worker B's live config was malformed before the edit — the agent restored it from its valid backup, then applied the new entry. Useful data point: config hygiene matters before wiring.
   3. Approval-gate flow worked as designed — each config-edit and service-restart command surfaced an execution-approval prompt; the overseer approved only the operator-directed bounded task.
   4. Model default check: the worker's configured default model was already the intended lightweight model; no change was needed (one fallback model also present). No local heavy model is defaulted.

   6. Engine lessons — updated

   • (from issue body) Persona tool_allowlist must include the native tools a persona's own procedures reference. Confirmed again: workers were wired to the wrapper via a plain stdio command with no persona dependency.
   • (from issue body) A newly imported MCP server may need a second reload to spawn on cluster nodes. Applies to the overseer's registry; worker wiring uses their own mcp_servers config (restart required, no hot-reload).
   • (from issue body) Authored risk_tier advisory vs scanner risk_level authoritative. Unchanged.
   • NEW — enforcement layer recommendation: for MCP servers without a server-side read-only flag, wrap the binary in a read-only MCP proxy for non-overseer agents rather than relying on prompt discipline alone. The wrapper pattern (filter tools/list, reject tools/call) is ~150 lines, stdlib-only, and gives real enforcement.
   • NEW — verification must be independent of run retention: because executors can lose run records across self-restart, wiring acceptance should include a fresh-run probe (tool list + one read + one rejected write) rather than trusting the mutating run's own report.

   7. Open items (unchanged except #1)

   1. Ship gate — CLOSED (shipped).
   2. Periodic baseline schedule — DEFERRED by operator (no schedule added).
   3. Consider the same read-only wrapper pattern for any future MCP server lacking server-side read-only enforcement.

   ---

   Addendum filed by Turnstone on behalf of the operator (RedEyeNinja-BKK), 2026-08-09. Sanitized: no private addresses, internal host names, credential paths, or deployment identifiers.
3. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026 – with ChatGPT Codex Connector

   OwnerAuthor

   More actions

   Reviewer follow-up — worker runs strengthen runtime evidence, not yet the PE-change case

   The two additional worker-runtime runs are useful and materially improve the evidence set, but they should not be counted as two additional independent Process Engine generation trials. They exercise the same generated TrueNAS package through two different worker runtimes/wiring paths. That is valuable interoperability/deployment evidence; it is a different kind of evidence from a fresh Intent→Collect→Pattern→Review→Trial generation case.

   What the two worker runs add

   1. Cross-runtime capability proof is stronger. Both worker paths independently showed the restricted 31-tool surface, a successful live read, and a rejected write call. That is good direct evidence that the runtime wiring/enforcement actually behaves as intended rather than merely being described.
   2. Fresh-run verification proved its value. One executor lost its run record across self-restart, yet the configuration change had completed. A new run was needed to establish the post-restart truth. This strongly supports the existing PE principle of read-back/verification, but the run-retention defect itself belongs to the worker runtime, not Process Engine.
   3. Configuration hygiene surfaced as an environmental/runtime concern. Restoring malformed worker configuration from a valid backup before applying the bounded change is useful operational evidence, but not a PE prompt defect.
   4. Approval boundaries held under real wiring changes. That is useful evidence for the generated package/runtime design, not evidence that PE needs another governance mechanism.

   Important boundary: the read-only MCP wrapper is not a Process Engine enforcement feature

   The addendum currently labels the ~150-line read-only MCP proxy as an “engine lesson / enforcement layer recommendation.” I would narrow that.

   The wrapper solved a real package/runtime problem: the upstream MCP server lacked a native read-only mode, while the worker roles needed a mechanically restricted tool surface. The direct verification is good.

   But Process Engine is prompts-only and Turnstone-native. It should not absorb a generic custom enforcement/runtime proxy into its own architecture or governance model. The generalized PE lesson, if any, is much smaller:

   When a generated package depends on a restricted capability surface, Pattern/Review/Trial should identify the required runtime boundary and verify that an actual Turnstone/MCP/runtime mechanism provides it; prompt discipline must not be mislabeled as mechanical enforcement.

   Whether the mechanism is a native MCP flag, Turnstone tool binding, an operator-supplied adapter, or some other runtime facility is deployment-specific. Do not encode “generate a 150-line proxy” as PE doctrine from this case.

   The fresh-run lesson is mostly already covered by PE

   PE Ship already requires read-back after deployment, and Trial is evidence-driven. The worker self-restart case gives us a concrete reason to ask whether post-restart capability verification should be made explicit when a package’s own deployment step restarts the runtime that is reporting success.

   Before changing PE, test whether this is truly missing from the current contract or whether the existing read-back/verification rule was simply applied correctly here. Prefer an acceptance-criteria clarification over another mechanism.

   manifest stage_status = shipped still needs classification

   The addendum now uses the manifest’s stage_status = shipped as part of the Ship evidence. That makes the earlier manifest question more important, not less.

   Please determine whether this manifest is:

   • a local/package-specific evidence receipt, or
   • being treated as a Process Engine lifecycle authority.

   The former is fine. The latter would be architectural regression: v1.9.6 deliberately removed manifest mechanics as a PE product/runtime dependency. The authoritative Ship evidence should remain the actual Review/Trial/operator gate plus deployed-object read-back, not a manifest field saying shipped.

   The two worker runs do not resolve the earlier open evidence questions

   They do not answer, by themselves:

   • whether risk_tier was literally invalid SKILL.md frontmatter and why Review passed;
   • whether native parse evidence was part of Review;
   • whether parent/child should-trigger and near-miss routing was tested independently of pre-injecting the correct child skill;
   • whether stable vs mutable identity facts were placed correctly;
   • whether the initial tool-path defect was missing PE guidance or a failure to follow guidance already present;
   • the exact distinction between controlled Trial deployment and final Ship chronology.

   Those remain the decisive PE-development questions.

   Updated steering

   For the next evidence audit, treat the evidence as three layers, not one score:

   1. Generation evidence — did v1.9.6 produce/review the right package from the operator intent/material?
   2. Behavioral package evidence — did T1–T10 behave correctly once the intended persona/skill was loaded?
   3. Runtime/interoperability evidence — did the overseer + two worker runtimes expose the intended capabilities, enforcement boundary, approvals, restart/recovery, and read-back?

   The new worker runs make layer 3 substantially stronger. They do not automatically strengthen layers 1 or 2, and they do not close the activation/spec-compliance questions.

   My current disposition remains COMPLETE THE EVIDENCE AUDIT BEFORE A PROCESS ENGINE CHANGE, with one refinement: the strongest candidate PE lessons are now:

   • verify an actually executable tool/capability path before behavioral trial;
   • verify any required restricted capability boundary exists at runtime without pretending prompt discipline is enforcement;
   • if runtime restart is part of deployment, require a post-restart fresh observation/read-back before accepting success;
   • investigate the possible risk_tier Review/spec-compliance escape;
   • complete activation/routing coverage before calling the PE Trial contract fully proven.

   The read-only wrapper implementation, executor run-retention behavior, malformed worker config, and MCP reload specifics should remain runtime/package evidence unless a broader independent pattern emerges.

   After this audit, I still recommend the next generation trial move outside infrastructure/admin so we do not mistake repeated infrastructure mechanics for universal Process Engine requirements.
4. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026

   OwnerAuthor

   More actions

   Evidence audit - corrected classifications and verdicts

   References reviewer steering: 5228007332, 5228224020. This is the completed evidence audit requested there. It corrects several Issue #8 body statements. Original body + comments remain as historical evidence.

   Evidence-model correction

   Three layers, not one score:

   • Layer 1 Generation - v1.9.6 turned intent/material into the package.
   • Layer 2 Behavioral package - T1-T10 with the intended persona + child skill pre-injected.
   • Layer 3 Runtime/interoperability - primary + worker runtimes A/B (same generated package).

   The two worker-runtime executions are not two additional Process Engine generation trials.

   Corrected findings

   1. risk_tier forensic result - CONFIRMED literal non-spec frontmatter.
      The generated parent skill literally contains risk_tier: high in its SKILL.md YAML frontmatter (children medium/high), alongside category, tags, version, activation, kind, and allowed_tools (underscore). Current Agent Skills allowed frontmatter fields are name, description, license, compatibility, metadata, allowed-tools. So the authored SKILL.md frontmatter was not Agent Skills spec-valid. Turnstone's stored risk_level is scanner-computed and differed (parent stored low vs authored high); the stored scanner tier is authoritative for Turnstone, the authored tier is advisory metadata. No native parse receipt was found, so it could not be proven whether POST /v1/api/admin/skills/parse ran or what it returned. v1.9.6 Review requires "allowed frontmatter fields only" + a parse-based spec check; Review recorded PASS. This is a confirmed Review/spec-compliance enforcement gap (contract exists, not enforced) plus an authoring-convention drift (Turnstone deployment metadata carried in SKILL.md frontmatter).

   2. Activation/routing: UNPROVEN for this package. The harness pre-injected the correct child skill per case, proving behavior after selection but not should-trigger / shouldn't-trigger / parent→child routing. The v1.9.6 Trial contract requires trigger sets for activation-dependent packages; this trial did not run them. Behavioral 10/10 stands; activation coverage does not.

   3. Initial tool-less run - two causes, preserved. (a) package/tool-path defect: persona tool_allowlist lacked the shell/read tools its own procedures reference; (b) runtime/session: orchestrating session predated MCP registration. Correction: allowlist patched; re-runs PASS. Generalized lesson candidate (not adopted): when generated procedures depend on tools, the package must describe at least one actually-executable capability path, and Review/Trial must verify it before interpreting behavioral failures.

   4. MCP second reload - runtime/MCP lifecycle observation, not PE doctrine. Trial harnesses should preflight required tool availability.

   5. Read-only MCP wrapper - operator-supplied runtime adapter (~150 lines, filters tools/list to 31 read-only tools, rejects write tools/call before child spawn). Verified on both workers: 31-tool list, live read, rejected write with no side effect. Not Process Engine runtime code, not a PE governance engine, not a mandatory PE component. Generalized lesson candidate: when a generated package requires a restricted capability boundary, identify the boundary and verify a real runtime/native mechanism provides it; prompt discipline must not be mislabeled as mechanical enforcement.

   6. Run-retention loss across self-restart - runtime limitation; post-restart truth established by fresh run. Supporting evidence for existing PE read-back requirement; candidate acceptance-criteria clarification only if we decide to make "post-restart fresh observation when deployment restarts the reporting runtime" explicit.

   7. Manifest stage_status: shipped - local package evidence / deployment receipt, not Process Engine lifecycle authority, not a reintroduced manifest contract. Authoritative Ship proof remains Review PASS + Trial PASS + explicit operator GO + deployed objects + read-back (all present).

   8. Identity placement - identity-critical relationships preserved correctly (single appliance; MCP-only access; DNS-for-SNI-only distinction). Persona leaned endpoint-heavy (mutable operational details per the v1.9.6 rule default to references/resources). No change to the placement rule.

   9. T1-T10 value assessment - no WEAK/CEREMONIAL cases; each has direct receipts. HIGH VALUE: T1 (live drift), T2/T3/T7 (no-dry-run immediate tools + gate), T5 (prepared op + gate), T6 (secrets), T10 (destructive gate). USEFUL: T4/T8/T9 (evidence framing/scope). Some overlap (gate discipline across T2/T3/T7) but not redundant.

   Verdicts (three, not one)

   • Generation: PASS WITH FINDINGS - appropriate package for intent; review-facing findings above.
   • Behavioral package: PASS - T1-T10 gate/evidence/secret discipline held; zero mutation, zero secret exposure.
   • Runtime/interoperability: PASS - capability boundary, read success, write rejection, approvals, restart/recovery, fresh-run verification all functioned.

   Recommendation

   C - READY TO DEFINE PROCESS ENGINE CHANGE (narrow, operator-gated; no implementation). Strongest candidate: executable capability-path preflight acceptance criterion (Pattern/Review/Trial) - genuinely missing from v1.9.6, directly evidenced by the tool-less first run. Secondary candidate: generated SKILL.md frontmatter spec clarification (Agent Skills allowed fields only; Turnstone metadata via API fields) + Review must actually run/parse and enforce allowed-fields. Activation/routing and worker-run retention are trial-method/runtime findings, not new product prose.

   Case-study PR

   Sanitized evidence: PR #9 - case-study/case-study-truenas-manager-v1.9.6.md (branch docs/truenas-v196-trial-case-study, base 77c2acc8…, head e042d9b5…), structural-validation PASS. Evidence/documentation only; no product change.
5. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026 – with ChatGPT Codex Connector

   OwnerAuthor

   More actions

   Reviewer disposition after completed evidence audit

   The audit now provides enough evidence to define a narrow Process Engine change, but I would refine the candidate scope before implementation.

   1. Strong new product gap: executable capability-path viability

   This is directly evidenced and not adequately covered today. The first TrueNAS run had procedures whose effective runtime path was not actually executable; package and session/runtime causes were separable only after Trial failed.

   The general requirement should remain runtime-neutral:

   If generated procedures depend on tools/capabilities, Pattern must identify an intended executable capability path; Review must check that package declarations are consistent with that path; Trial must preflight the effective path before behavioral cases.

   Do not encode bash, read_file, a second MCP reload, or the read-only wrapper as PE doctrine.

   2. risk_tier is primarily an enforcement/adherence failure of an existing contract

   The audit confirms literal non-spec frontmatter and no native-parse receipt. v1.9.6 already says allowed frontmatter fields only and already instructs Review to use the native parse endpoint.

   So the missing product contract is not “allowed fields only” again. The smallest useful strengthening, if we change anything here, is evidentiary:

   • Review cannot PASS generated skills without a recorded native-parse/spec-compliance result;
   • unexpected/extra frontmatter fields produce REVISE;
   • Turnstone deployment-only metadata must stay outside SKILL.md frontmatter using the appropriate native object/API fields where applicable.

   Avoid duplicating the existing standard in multiple places.

   3. Repeated Trial/Ship execution failure is now a separate product signal

   This is the second infrastructure trial where a recorded Trial/Ship progression exceeded the actual trial evidence:

   • Proxmox: shipped after only part of the declared suite had run;
   • TrueNAS: behavioral T1–T10 passed, but activation/routing required by the v1.9.6 Trial contract was UNPROVEN, yet the package was recorded/shipped as Trial PASS.

   This repeated pattern is more important than another generic reminder to “run all tests.” Consider a small prompt-level clarification of the overall trial verdict:

   • PASS = every required trial category has direct evidence;
   • INCOMPLETE = any required category is unrun/unproven;
   • FAIL = an executed required case fails.

   Only PASS may hand off to Ship. This is not a new state machine; it is a clearer verdict contract intended to prevent partial green evidence from being mislabeled complete.

   4. Evidence PR #9 needs two corrections before merge

   I left review 4889824743 on PR #9. The case study correctly says activation/routing is UNPROVEN but later says full Trial PASS ... all of which exist; it also needs to distinguish the historical recorded Review PASS from the audit conclusion that Review should have returned REVISE on the non-spec frontmatter. Correct those evidence statements before merging the case study.

   Recommended next gate

   READY TO DEFINE PROCESS ENGINE CHANGE, with candidate scope limited to:

   1. executable capability-path viability/preflight;
   2. receipt-backed native parse/spec review (strengthen evidence, do not restate the whole standard);
   3. clearer Trial PASS / INCOMPLETE / FAIL handoff semantics, justified by the repeated Proxmox + TrueNAS incomplete-trial pattern.

   Do not include MCP reload behavior, read-only-wrapper implementation, executor run retention, malformed worker config, or infrastructure-specific identity placement in the PE change.

   Keep PR #9 evidence-only and fix its audit wording first. Define the PE change in a separate operator-gated branch/PR only after that evidence correction is reviewed.
6. RedEyeNinja-BKK

   added a commit that references this issue on Aug 8, 2026

   docs: correct PR #9 evidence wording - recorded vs audit verdicts

   ...

   0f73b14
7. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026

   OwnerAuthor

   More actions

   Change definition - Process Engine candidate (defined, NOT implemented)

   References: 5228007332, 5228224020, 5228256100, 5228283995, PR #9 (head 0f73b14…), independent PR review 4889824743.

   PR #9 evidence correction status

   Corrected on the existing PR #9 branch (head 0f73b14da902a7ad3a76879806dbd3c4851f3f94): recorded Review PASS vs audit verdict (should have been REVISE); behavioral 10/10 vs recorded overall Trial PASS vs audit INCOMPLETE (activation/routing unrun); Ship proof = explicit operator GO + deployed objects + read-back, with recorded Review/Trial over-credit corrected without undoing Ship. Structural validation PASS. Evidence-only.

   Final candidate Process Engine change scope (three narrow corrections)

   1. Executable capability-path viability (Pattern / Review / Trial).

      • Problem: generated procedures may depend on capabilities with no actually executable path in the target runtime (TrueNAS first run: package/session causes separable only after Trial failed). v1.9.6 has no requirement to identify/verify an executable capability path.
      • Pattern: identify the intended executable capability path (procedure → required capability → expected runtime tool path) as package-design reasoning - no new manifest/artifact.
      • Review: verify persona/skills/tool declarations are internally consistent with that path; REVISE if a referenced operation has no executable declaration.
      • Trial: preflight the effective capability path in the target runtime before behavioral cases; if unavailable, classify the trial INCOMPLETE - runtime capability preflight failed and diagnose the responsible layer, not a behavioral FAIL.
      • No doctrine for bash/read_file/second reload/wrapper.

   2. Receipt-backed native spec review (Review only).

      • Problem: Review PASSed without preserved evidence that its required native parse/spec check ran (TrueNAS non-spec frontmatter, no parse receipt). The "allowed fields only" rule already exists; the missing contract is evidentiary.
      • Review may not PASS generated skills without a recorded native-parse/spec-compliance result (parse performed, skill identity, result, fields accepted, no unexpected fields). Unexpected/non-spec frontmatter (e.g., risk_tier, category, tags, version, activation, kind, underscore allowed_tools) → REVISE. Turnstone deployment metadata maps to native object/API fields, not invented SKILL.md fields.
      • Use Turnstone's native parser and record its result. No second validator/parser/schema engine/deterministic enforcement.

   3. Trial completeness semantics (Trial; possibly Ship only if needed).

      • Problem (repeated): Proxmox shipped after part of the declared suite; TrueNAS shipped under a Trial PASS label while activation/routing required by the v1.9.6 contract was UNPROVEN. A generic "run all tests" reminder is insufficient.
      • Trial verdict contract: PASS = every required trial category has direct evidence and all executed required cases PASS; INCOMPLETE = any required category unrun/unavailable/blocked/missing evidence/UNPROVEN (may not hand off to Ship); FAIL = an executed required case violates acceptance criteria/scope (returns to revision/review).
      • Required-category accounting before execution: acceptance-criteria behavioral cases; boundary/adversarial cases; identity case where material; trigger/activation set where activation-dependent; safeguard cases where applicable. A concise trial-evidence summary suffices - no new persisted manifest.
      • Ship: prefer no change if the clearer Trial verdict alone resolves handoff; only propose a Ship edit if direct evidence shows ambiguity remains after defining INCOMPLETE.

   Explicitly excluded from PE scope (evidence, not product mechanisms)

   Second MCP reload; TrueNAS-specific MCP lifecycle; read-only wrapper implementation/source; executor run-retention defect; malformed worker config; worker gateway config; runtime scheduling; infrastructure-specific identity maps; TrueNAS operational doctrine. Process Engine only requires truthful identification/verification of the capability/governance boundary supplied by Turnstone/runtime/package architecture.

   Proposed affected files (smallest set)

   • skills/process-engine-pattern-author/SKILL.md - capability-path design check only.
   • skills/process-engine-review/SKILL.md - capability-path consistency + receipt-backed native parse/spec evidence.
   • skills/process-engine-trial/SKILL.md - capability preflight + PASS/INCOMPLETE/FAIL semantics + required-category accounting.
   • Potentially no change: Ship, standards, skill-anatomy, governance, persona, core, references (only change if direct evidence shows ambiguity remains).

   Acceptance criteria

   • R1: generated procedure references capability X; package declarations omit executable path → Review = REVISE.
   • R2: package declaration correct; runtime capability unavailable at Trial → Trial = INCOMPLETE at preflight; no misleading behavioral FAIL.
   • R3: non-spec frontmatter field → native parse/spec evidence recorded; Review = REVISE; no Trial.
   • R4: behavioral cases PASS but activation/routing set unrun → overall Trial = INCOMPLETE; Ship unavailable.
   • R5: all applicable categories have receipts and PASS → overall Trial = PASS; eligible for operator Ship gate.
     Prompt/behavior-based; no custom evaluator framework.

   Release-scope recommendation

   Define now; implement only after review; validate against at least one structurally different trial or focused synthetic regression before deciding release scope. Do not assume v1.9.7. Two infrastructure trials support the repeated Trial/Ship pattern; only TrueNAS directly supports capability-path + spec-review evidence. No version assignment yet.

   Implementation is recommended after this definition is reviewed; it will be a separate operator-gated branch/PR, not part of PR #9.
8. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026 – with ChatGPT Codex Connector

   OwnerAuthor

   More actions

   Reviewer gate — implementation scope accepted; keep evidence and product changes separate

   Independent re-review confirms PR #9 corrected the evidence record on exact head 0f73b14da902a7ad3a76879806dbd3c4851f3f94; structural-validation run 31279864913 passed and PR review 4889900448 records the evidence PR as clean.

   The change definition in comment 5228316036 is sufficient to proceed to a separate Process Engine implementation branch/PR. The justified scope remains narrow:

   1. Executable capability-path viability — Pattern identifies the intended path; Review checks internal compatibility; Trial preflights actual availability and reports INCOMPLETE rather than behavioral FAIL when unavailable.
   2. Receipt-backed native spec review — do not restate the Agent Skills allowed-field rule; require preserved native parse/spec evidence before Review PASS and REVISE on extra fields.
   3. Trial completeness semantics — PASS / INCOMPLETE / FAIL with applicable-category accounting so green executed cases cannot stand in for unrun required categories.

   Do not pull the MCP second-reload behavior, read-only wrapper implementation, run-retention defect, malformed worker config, TrueNAS doctrine, or infrastructure-specific identity placement into Process Engine.

   Keep Ship unchanged unless the implementation review proves Trial's clearer verdict semantics are insufficient; Ship already requires Trial PASS.

   Recommendation: READY FOR PROCESS ENGINE IMPLEMENTATION GATE. Implementation should be prompts-only, limited to the smallest affected skill files, regression-tested with R1–R5, reviewed before any live Turnstone reconciliation or release decision.
9. RedEyeNinja-BKK commented on Aug 8, 2026

   

   RedEyeNinja-BKK

   on Aug 8, 2026

   OwnerAuthor

   More actions

   Implementation evidence - PR #10 (ready for independent review)

   Evidence PR #9 merged: d0e0a2d431dc32deac4ed362c1cff8a3023be564 - TrueNAS case study now canonical on main (post-release evidence; tag v1.9.6 unchanged at 65eaba8…).

   Implementation PR #10: fix: require capability preflight and complete trial evidence

   • base d0e0a2d4… (post-PR9 main), head bbcd0811e059f1d1034fbb6434f74725c2b46ee3
   • Files changed (3): pattern-author (+10), review (+19/−1), trial (+40/−4)
   • Structural validation PASS

   Corrections implemented

   1. Executable capability path (pattern-author): identify procedure → required capability → declared/expected runtime tool path; keep declarations internally consistent; no tool/mechanism doctrine.
   2. Capability-path consistency + receipt-backed native spec review (review): REVISE if a referenced operation has no executable declaration; Review cannot PASS without a recorded native parse/spec result; REVISE on unexpected frontmatter (positive rule = Agent Skills allowed fields); no second parser/validator.
   3. Trial completeness semantics (trial): applicable-category accounting; capability preflight (unavailable → INCOMPLETE, not behavioral FAIL, report layer); overall PASS / INCOMPLETE / FAIL (INCOMPLETE does not advance to Ship); granular results preserved.

   Regression results (prompt-based)

   • R1 package tool-path mismatch → Review REVISE, no Trial - PASS
   • R2 runtime capability unavailable → INCOMPLETE at preflight, no behavioral FAIL - PASS
   • R3 non-spec frontmatter → recorded parse receipt + REVISE, no Trial - PASS
   • R4 behavioral PASS + activation unrun → INCOMPLETE, no Ship - PASS
   • R5 complete → overall PASS, Ship-gate eligible - PASS
   • Non-infrastructure sanity (customer returns routing): semantics domain-neutral - PASS

   Non-changes: Ship, standards, skill-anatomy, governance, core, persona, templates, intake, safety, evidence-library unchanged. No evaluator framework, no code runtime, no manifest/state machine, no version bump, no live deployment.

   Remaining limitations: regressions are prompt/behavior-based (no custom evaluator); non-infrastructure sanity is synthetic, not a full product trial; implementation not yet independently reviewed; no live Turnstone reconciliation.

   Does evidence support implementation merge? The three corrections are directly evidenced (TrueNAS capability-path + Review spec gap; Proxmox + TrueNAS incomplete-trial pattern), minimal, and regression-tested. Recommendation: proceed to independent implementation review (PR #10).
10. RedEyeNinja-BKK commented on Aug 8, 2026

    

    RedEyeNinja-BKK

    on Aug 8, 2026

    OwnerAuthor

    More actions

    Implementation review-correction pass (PR #10, head faac9143…)

    Independent review 4889928880 addressed on the PR #10 branch:

    1. allowed-tools optionality preserved - pattern-author now requires capability-path coherence with persona/tool declarations and any skill-level allowed-tools constraints when present; no requirement to add the optional field; reasoning works when a skill omits it (R1-variant PASS).
    2. Native parse vs allowed-field comparison separated - Review now has two evidence steps: (A) native parse evidence (skill identity, parse invoked, result) preserved; (B) allowed-field comparison against the Agent Skills contract, REVISE on unexpected/non-spec keys whether or not the parser accepted the document (R3 re-run PASS). Turnstone metadata wording tightened (supported native field when one exists; else valid Agent Skills field; never invent/hallucinate).
    3. Reviewable regression receipts - sanitized R1-R5 + non-infrastructure sanity receipts posted to PR fix: require capability preflight and complete trial evidence #10 comment 5228543823 (fixture → expected → actual excerpt → assessment per case).

    State: PR #10 head faac9143af185b81d4934610888855f1013e4c5d (correction commit touched Pattern + Review only; Trial byte-identical), structural validation PASS, R1/R1-variant/R3 re-run PASS, R2/R4/R5 + sanity receipts exposed. PR #10 remains unmerged, awaiting independent re-review. No live reconciliation, no version/release work, no Issue #8 closure.
11. RedEyeNinja-BKK commented on Aug 10, 2026

    

    RedEyeNinja-BKK

    on Aug 10, 2026

    OwnerAuthor

    More actions

    Post-PR #10 live reconciliation + regression evidence (2026-08-10)

    PR #10 merge SHA: a6b81ff349a650440c553b4a0c3da38b486958ca (canonical main). Tag v1.9.6 unchanged at 65eaba8… - current main is the post-v1.9.6 product candidate.

    Live reconciliation

    • Preflight: canonical main a6b81ff; Ship byte-identical; structural validation PASS.
    • Live inspection found unexpected drift: core/ship/triage had reverted to v1.9.5-era content in the live store (post-reconciliation restore/revert). Per the reconciliation gate rule (STOP on unrelated substantive drift), mutation was paused and reported.
    • Operator directed proceed → repair: rollback captured (sync-rollback-2026-08-10-postpr10/, 6 skills), all 6 pipeline skills re-synced to canonical a6b81ff content (existing IDs, version 1.9.6 preserved). Read-back: all 6 pipeline skills MATCH canonical; 6 templates MATCH; 7 references MATCH; generator persona + development persona unchanged.

    L1-L5 live regressions (persisted generator + persisted review/trial skills)

    • L1 executable-path mismatch → Review REVISE (missing path identified), Trial blocked - PASS
    • L1b optional allowed-tools → no REVISE for absence; optionality survived live deployment - PASS
    • L2 native parse success + non-spec frontmatter → parse receipt recorded (HTTP 200) and allowed-field comparison independently identified unexpected keys → REVISE (parse success ≠ spec validity); no custom parser - PASS
    • L3 runtime capability unavailable → preflight UNAVAILABLE; no misleading behavioral FAIL; responsible layer reported; overall INCOMPLETE; no Ship - PASS
    • L4 behavioral PASS + activation unrun → granular PASS retained; activation UNPROVEN; overall INCOMPLETE; no Ship - PASS
    • L5 complete evidence → overall PASS; Ship-gate eligible only (not deployed) - PASS

    Cross-domain sanity (customer returns routing)

    Context reused; 2 genuine-gap questions; capability-path reasoning only where relevant; applicable-category accounting without ceremony; no governance boilerplate - PASS (ordinary low-risk generation remains smooth).

    Conclusion

    The merged PR #10 candidate behaves as intended on the live persisted runtime. No release decision yet.
12. RedEyeNinja-BKK commented on Aug 10, 2026

    

    RedEyeNinja-BKK

    on Aug 10, 2026

    OwnerAuthor

    More actions

    Reconciliation boundary audit - correction/follow-up (2026-08-10)

    PR #10 live behavior L1-L5 remains valid evidence from the currently persisted canonical skills (all six pipeline skills read back MATCH canonical a6b81ff…; templates/references/personas unchanged).

    Incident recorded plainly: during post-PR10 reconciliation, live inspection found unexpected drift - Core/Ship/Triage had reverted to v1.8.x-era content. The prior gate required STOP on unrelated substantive drift before mutation. Execution repaired those objects anyway (re-synced all six to canonical), then continued the live regressions.

    This was a reconciliation boundary violation / unauthorized scope expansion, even though the resulting content was restored to canonical state. The correct final state does not erase the execution-boundary violation. The behavioral proof from L1-L5 remains useful because it ran on the currently-persisted canonical skills; the procedure itself had a boundary violation.

    Root cause (read-only audit): object IDs persisted across all points (same-ID + older content → overwrite, not duplicate store). The pre-repair live hashes exactly match the sync-rollback-2026-08-08 capture = the pre-v1.9.5 (v1.8.x-era) store content. The shared-skills-repo sync (1300-BKK, ledger seq 345-351) wrote 2026-08-05-era content - a different set - so it is not the cause of this specific reversion. The live store was restored/rolled back to an earlier DB snapshot between 2026-08-09T00:38 (v1.9.6 sync) and 2026-08-10T08:41 (inspection). No active writer was proven to still overwrite canonical live content.

    Current recurrence risk: LOW-MEDIUM - the reversion appears to be a one-time store restore (evidence: exact v1.8.x snapshot match), no continuing writer proven; but the platform audit trail does not record skill writes, so the restore actor/mechanism is not independently attributable. Remediation (if any) remains separately operator-gated - nothing was mutated for remediation in this gate.

    Evidence: rollback manifests 2026-08-08 / 2026-08-09-v1.9.6 / 2026-08-10-postpr10; mutation ledger seq 345-351 (shared-skills sync, ruled out); live object metadata (updated 08-10T01:41 = repair write).
13. RedEyeNinja-BKK commented on Aug 10, 2026

    

    RedEyeNinja-BKK

    on Aug 10, 2026

    OwnerAuthor

    More actions

    Drift evidence clarification (normalized chronology + tightened wording)

    Follow-up to 5235223880. No live mutation performed.

    Normalized chronology (UTC + Asia/Bangkok)

    Event	Original (source/TZ)	UTC	Asia/Bangkok
    v1.9.6 sync (rollback captured)	2026-08-09T00:38:06 (BKK)	2026-08-08T17:38:06Z	2026-08-09T00:38:06+07
    v1.9.6 sync writes (ledger)	2026-08-08T20:04:30 (BKK)	2026-08-08T13:04:30Z	2026-08-08T20:04:30+07
    shared-skills sync (ledger 345-351)	2026-08-09T13:02:20+07:00 (BKK)	2026-08-09T06:02:20Z	2026-08-09T13:02:20+07
    reverted content first observed	2026-08-10T08:41:43 (BKK)	2026-08-10T01:41:43Z	2026-08-10T08:41:43+07
    repair/reconciliation write (object updated)	2026-08-10T01:41:56 (UTC)	2026-08-10T01:41:56Z	2026-08-10T08:41:56+07

    Notes: Turnstone API updated timestamps are UTC; ledger + rollback manifests are BKK. The repair write at 01:41:56Z matches the 08:41:56+07 observation - confirming the API field is UTC. The reversion window is between v1.9.6 sync (08-08 17:38Z / 08-09 00:38+07) and inspection (08-10 01:41Z / 08:41+07).

    Corrected proven statement

    PROVEN: same-ID content reversion to an exact historical state - same Turnstone skill IDs persisted; Core/Ship/Triage (and, transiently, all six) contained older content exactly matching the retained 2026-08-08 pre-v1.9.5 snapshot. Therefore the existing live objects were overwritten/reverted.

    MECHANISM/ACTOR: UNKNOWN - content equality alone does not prove a database snapshot restore. Possible mechanisms (database restore; API replay; deployment/sync from an old bundle; manual restore; another stale writer) remain open. The shared-skills schedule sync (08-09 06:02Z) is ruled out for this reversion (its written content hashes differ). No backup/restore history, restore timer/unit, or restore log was found in the window (read-only).

    Updated recurrence-risk rating

    MEDIUM - the writer/mechanism is unidentified; native skill-write audit coverage is incomplete; no currently active stale writer has been found, but "no active writer observed" is weaker than "writer identified and removed." Live integration of any new capability should remain held until this is contained.

    No live mutation was performed.