fix: preserve provenance and qualification scope

case-study/case-study-etsy-realignment-2026-08-17.md

@@ -0,0 +1,254 @@
+# Case Study — Etsy Package Realignment Through Process Engine (2026-08-17)
+
+> **What this is:** a recorded downstream use of Process Engine to realign an
+> existing Turnstone-native Etsy package with new source material. It is a
+> usage, generation, review, trial, and later deployment/read-back evidence
+> record—not a claim that Process Engine provides Etsy API access or that the
+> downstream package is fully runtime-qualified.
+>
+> **Privacy:** the shop identity and source URLs are retained only as the
+> operator's supplied context. Buyer data, credentials, private worksheet
+> contents, host data, and internal identifiers are not published here.
+
+---
+
+## Executive result
+
+The operator asked Process Engine to update and refine an existing Etsy skills
+family and persona so it could assist with online-business management and Etsy
+SEO. The engine collected and synthesized the supplied material, revised the
+package, and was later recorded as having a bounded native Turnstone object
+deployment. The evidence does not prove that deployment satisfied the Ship gate.
+
+The resulting package is useful for **analysis, drafting, translation,
+worksheet-export analysis, prioritization, trend frameworks, and advisory
+policy heads-ups**. The following capabilities were deliberately not claimed:
+
+- authenticated Etsy Seller App/API access;
+- live shop, listing, order, or storefront reads;
+- live seller-taxonomy lookup;
+- live Google Sheets reads or writes;
+- publishing listings or changing shop settings;
+- sending customer messages;
+- purchases, inventory commitments, or any Etsy mutation.
+
+The correct overall runtime-dependent Trial classification remains
+**INCOMPLETE**, not FAIL: the executed cases passed, but unavailable required
+capability categories were not proven.
+
+## Operator intent and material collected
+
+The interaction supplied the existing Etsy package plus the following material
+and context classes:
+
+1. an online-business-manager skills article;
+2. the operator's ChummyThailand storefront URL as target context;
+3. an article describing an autonomous Etsy operating model;
+4. the shared `Online_Business_Worksheet_2026_Master-V3.0` as business-analysis
+   context;
+5. Etsy developer documentation;
+6. Etsy Dev MCP documentation;
+7. the `getSellerTaxonomyNodes` reference.
+
+The engine treated these differently rather than copying them wholesale:
+available source material supplied techniques and requirements, the operator's
+storefront URL identified the intended shop/target context but did not thereby
+prove that storefront page content was retrieved, the worksheet supplied
+point-in-time analytical inputs, and the developer references informed
+capability-path design. Unavailable material remained unavailable, and partial
+material contributed only its usable portion. The Etsy Dev MCP path was recorded
+as **documentation/specification discovery**, not Etsy API execution.
+
+The objective was refined into an assistant-manager package that could help
+with customer-message drafting, listing and SEO analysis, business and
+financial analysis, trend interpretation, weekly prioritization, and
+operator-controlled next steps. The target was retained as an aspirational
+north star rather than a guarantee.
+
+## Process Engine pipeline evidence
+
+The recorded path was:
+
+**Intent → Orient → Collect → Clarify → Objective → Summary Gate → Pattern →
+Review → Trial → Ship**
+
+A companion assessment was used during development but is excluded from the
+proposed canonical commit; its generalized rationale and evaluation matrix
+belong in the eventual issue/PR record rather than a permanent evaluation
+subtree.
+
+| Stage | Evidence and result |
+|---|---|
+| Collect | Existing package and seven new source/context inputs were collected and classified. |
+| Clarify / Objective | The desired day-to-day behavior was clarified as an OBM + Etsy SEO assistant, not an autonomous store operator. |
+| Summary Gate | Material basis, intent, and intended package direction were confirmed before authoring. |
+| Pattern | Existing persona and skills were realigned; new supporting skills and governance posture were authored. |
+| Review | Initial draft returned **REVISE** with seven findings. The revised draft returned **PASS** after identity, completeness, coexistence, capability-path, financial-source, privacy/commitment, and trend-evidence corrections. |
+| Trial | Executed behavioral and activation cases passed; runtime-dependent categories remained unproven. Overall Trial: **INCOMPLETE**. |
+| Ship | A later record describes a narrowed analysis/drafting deployment and native read-back, but the inspected evidence does not prove a new exact-scope Review PASS + Trial PASS chain after the overall-INCOMPLETE result; classify this as a pipeline-contract failure, not a successful narrow Ship. |
+
+### Trial → Ship chronology
+
+The original Trial record explicitly classified the overall result as
+**INCOMPLETE** and stated that INCOMPLETE does not advance to Ship. A later
+deployment record described a narrowed analysis/drafting scope as having a
+separate Trial PASS claim and recorded native deployment/read-back, but the
+evidence inspected here contains no separate Pattern revision, exact-scope
+Review PASS, exact-scope Trial PASS receipt, or operator decision establishing
+a new PASS chain for that narrowed artifact.
+
+Accordingly, the deployment read-back proves object persistence, not Ship-gate
+compliance. This case study records the outcome as a **Process Engine
+pipeline-contract failure** unless a primary receipt for the missing narrowed
+Review/Trial chain is later produced. It must not be rewritten as a successful
+narrow Ship.
+
+The initial Review did not silently pass the draft. It identified seven concrete
+findings, including unproven destination-skill absence, incomplete persona and
+governance artifacts, ambiguous treatment of existing skills, abstract
+capability declarations, missing financial-source precedence, insufficient
+customer-message safeguards, and an under-specified trend-evidence standard.
+The revised package resolved those findings and was re-reviewed PASS.
+
+## Downstream package trial evidence
+
+The recorded trial evidence exercised **12 behavioral cases**, all passing. It
+was later described as supporting the narrowed scope, but the evidence
+inspected here does not prove that these cases were rerun against a separately
+revised exact narrowed artifact before Ship.
+
+- customer-message uncertainty and missing-order-fact handling;
+- SEO boundary behavior without ranking guarantees or unauthenticated API use;
+- financial arithmetic and source/freshness labeling;
+- trend escalation into a bounded, testable recommendation;
+- policy and external-action boundaries;
+- wrong shop/listing identity;
+- Etsy capability escalation;
+- financial-source conflict;
+- buyer-data minimization;
+- commitment safeguards;
+- external-emission safeguards; and
+- adversarial medical-claim handling.
+
+Activation cases also passed for the OBM assistant, customer-message skill,
+financial-analysis skill, and the combined SEO/taxonomy and trend-analysis
+path. The recorded activation observations were:
+
+- OBM assistant: should-trigger 2/2; shouldn't-trigger 3/3;
+- customer messages: should-trigger 3/3; shouldn't-trigger 2/2;
+- financial analysis: should-trigger 3/3; shouldn't-trigger 2/2;
+- SEO/taxonomy and trend analysis combined: should-trigger 4/4;
+  shouldn't-trigger 1/1.
+
+One buyer-privacy case passed with an evidence-path caveat: the safeguard
+behavior was supported, but a delegated context did not consistently resolve
+the direct reference path. This is retained rather than erased.
+
+## Runtime and deployment evidence
+
+The later native Turnstone deployment record reports read-back of the
+following objects:
+
+- the existing Etsy manager persona was updated in place;
+- six supporting skills were created and read back;
+- each created skill was enabled and scanner-rated safe;
+- a content-only prompt policy was created and read back;
+- the advisory judge helper remained **not deployed** because its native
+  creation path was not exposed in the execution context;
+- existing Etsy skills were retained and not silently overwritten, disabled,
+  or superseded;
+- a reversible rollback path was recorded and not executed.
+
+This proves native object persistence and read-back for the objects named in
+that deployment record, not Trial/Ship gate compliance and not the unavailable
+Etsy integrations listed above.
+
+## Reference-retrieval limitation
+
+The re-review explicitly recorded a reference-retrieval limitation. The
+following operator-supplied or identified references were **not retrievable
+through the available fetch/preview path** during the run:
+
+- the ChummyThailand storefront page;
+- the Medium autonomous-Etsy article;
+- the `getSellerTaxonomyNodes` reference.
+
+The package correctly did **not** assert these as proven evidence, and trial
+fixtures must not represent them as verified sources without a later authorized
+retrieval.
+
+The root cause of each failure is **not proven**. Each could have been any of:
+
+- unsupported page/URL behavior;
+- authentication or permission scope;
+- a transient network failure;
+- a tool/adapter limitation;
+- a malformed or redirected URL;
+- source-side blocking;
+- a retrieval timeout; or
+- incomplete page extraction.
+
+The strongest development signal from the run is that Process Engine should
+keep a compact conversational per-material receipt/evidence note: distinguish
+supplied, retrieved, partial, unavailable, and intentionally-not-attempted
+states as applicable; preserve what usable content actually informed generation;
+and carry source-dependent limitations forward. Record a retrieval cause only
+when supported; otherwise keep the cause unknown. Do not infer inaccessible
+source contents. This is the implemented evidence-handling principle, not a new
+ledger, manifest, schema, persisted receipt object, or failure taxonomy.
+
+## Usage and performance receipt
+
+The Process Engine workstream's captured session evidence records:
+
+| Metric | Recorded value |
+|---|---:|
+| Workstream messages | 277 |
+| User messages | 12 |
+| System messages | 27 |
+| Assistant messages | 85 |
+| Tool messages | 153 |
+| Natural requests in the measured routing window | 250 |
+| Successful HTTP responses | 250/250 (200) |
+| Efficient-route requests | 185 (74.0%) |
+| Capable-route requests | 65 (26.0%) |
+| Failures / fallbacks / route unavailable | 0 / 0 / 0 |
+| `reasoning_content` recurrence | 0 |
+| Context-window errors / truncation | 0 |
+| Model-call count | **UNAVAILABLE** in the authoritative receipt |
+| Input/output/cache/reasoning tokens | **UNAVAILABLE** in the authoritative receipt |
+| Authoritative total token usage | **UNAVAILABLE** |
+| Defensible API-equivalent cost | **UNAVAILABLE** |
+
+The routing counts are usage telemetry, not a quality score. No token or cost
+estimate is derived from message length, and bundled/local model usage is not
+represented as zero cost.
+
+## Evidence classification and next trial
+
+| Claim | State |
+|---|---|
+| Process Engine collected and classified the supplied material | **PROVEN** by the recorded interaction |
+| Revised package addressed the seven Review findings | **PROVEN** by the re-review record |
+| Executed package behavior passed 12/12 cases | **PROVEN** by the trial record |
+| Activation cases recorded in the trial passed | **PROVEN** for the executed set |
+| Authenticated Etsy API path | **UNPROVEN / excluded** |
+| Live Google Sheets path | **UNPROVEN / excluded** |
+| Storefront and live taxonomy execution | **UNPROVEN / excluded** |
+| Full end-to-end Etsy runtime qualification | **INCOMPLETE** |
+| Narrowed native object persistence/read-back | **PROVEN** for the named objects; Ship-gate compliance for the narrowed deployment: **NOT PROVEN** |
+| Full Etsy package ship | **NOT PROVEN** |
+
+A future completion trial would require an operator-controlled Etsy Seller App
+read path, an authorized Google Sheets read path, exact taxonomy-operation
+receipts, and re-execution of the affected cases through those intended paths.
+Until then, the package should be described as a narrowed analysis/drafting
+assistant, not an autonomous Etsy manager.
+
+## Source and evidence boundary
+
+This case study is derived from the recorded Etsy Skills Refinement
+interaction and its associated local review, trial, and ship evidence. It
+records observed outcomes and known gaps; it does not reproduce private
+transcripts or source documents. The Process Engine repository remains the
+canonical product source. Live Turnstone objects are deployment evidence only.

references/intake.md

@@ -65,6 +65,17 @@ clarification — the material sharpens the questions. Run conversationally:
    doc), fetch the page and extract what's useful. Record the exact source
    URL.
 
+   For every item, distinguish: supplied; retrieval attempted or intentionally
+   not attempted; usable content retrieved; partial extraction; or retrieval
+   unavailable. Supplied is not thereby retrieved, and retrieved is not thereby
+   completely extracted. Record retrieval path/date when known. Record a
+   failure cause only when supported; otherwise say failure cause unknown.
+   Retrieval failure does not prove source content false. When a materially
+   useful source cannot be retrieved, offer one practical substitute such as
+   paste/upload once; if declined, do not ask again. Continue from available
+   evidence and carry the limitation forward. Inaccessible material cannot
+   silently become an inferred or verified source fact.
+
    **Text dumps / files (no link)** — no fetch. Provenance is whatever the
    user provides or states; if none is given, record "user-provided, source
    unknown". Never assume a source for pasted content.
@@ -86,9 +97,12 @@ clarification — the material sharpens the questions. Run conversationally:
    - general input (notes, constraints, ideas) → incorporate into the
      intent understanding
 4. **Extract** — pull the techniques, constraints, domain specifics, and
-   intent from ALL provided material. Multiple sources combine into a
-   best-of-all-worlds understanding. That synthesis never drops what the
-   package will later need to tell entities apart:
+   intent from ALL material whose content is actually available. For supplied
+   but unavailable or partially extracted material, preserve the limitation and
+   use only claims supported by the usable portion or another named source.
+   Multiple sources combine into a best-of-all-worlds understanding, but
+   unavailable material cannot silently be replaced by inference. The synthesis
+   never drops what the package will later need to tell entities apart:
    - preserve identity-critical relationships (entity ↔ role ↔
      identifier/alias — machines, accounts, people, products, records,
      services) when they are material to later decisions or tool targets;
@@ -123,9 +137,13 @@ clarification — the material sharpens the questions. Run conversationally:
 9. **Sweep** — the engine's own output must stay sweep-clean (the zero-
    tolerance language rule applies to what the engine WRITES, not to what a
    user pastes; we cannot control user input).
-10. **Provenance record** — source URL(s) or "user-provided" status, item
-    type, license info if visible, fetched date, extraction summary. This
-    record is evidence.
+10. **Material receipt** — for each item, record a compact conversational
+    receipt: stable identity, material class, supplied/retrieval status,
+    retrieval path/date when known, usable or partial extraction, extracted and
+    used material, excluded/not-used material, and downstream claims or
+    requirements left unproven. Record known failure cause only when supported;
+    otherwise record unknown. This is an evidence note in the existing intake
+    flow, not a new persisted subsystem or mandatory manifest.
 11. **Operator gate** — the generated package then passes through the normal
     Pattern → Review → Trial → Ship gates; the operator's sign-off is the
     gate (the engine's core identity, unchanged).
@@ -166,6 +184,10 @@ clarification — the material sharpens the questions. Run conversationally:
 
 ## Verification
 
+- [ ] Material receipt distinguishes supplied from retrieved, complete from partial, and unavailable from not attempted
+- [ ] Known failure causes are supported; unknown causes remain unknown
+- [ ] One practical substitute was offered for materially useful unavailable material; a declined offer was not repeated
+- [ ] Source-dependent claims left unproven are carried forward
 - [ ] Item type + source recorded ("user-provided, source unknown" if no source)
 - [ ] Techniques / domain specifics extracted from all provided material
 - [ ] Original instructions authored — no verbatim copying

skills/process-engine-core/SKILL.md

@@ -57,8 +57,11 @@ and governance surfaces.
    note the objective as underspecified and propose one in the draft for
    correction at review — never interrogate.
 5. **Summary gate** — before generation, present: "Working from: N links +
-   M text blocks (k sources unknown). Intent: X. Good looks like: <vision>.
-   Generate?" — proceed only on confirmation.
+   M text blocks (k sources unknown)." Also disclose any material limitations
+   that affect generation in plain language: supplied-but-unretrieved,
+   partially extracted, or source-dependent facts still unproven. Keep it
+   conversational, not a provenance dump. State intent and good looks like,
+   then ask "Generate?" — proceed only on confirmation.
 6. **Route** — classify the request:
    - author an artifact → `process-engine-pattern-author`
      (eligibility gate decides shape: project / persona / skill(s) /
@@ -103,12 +106,14 @@ and governance surfaces.
 7. **Load standards checklist** (references/standards.md) and the generation
    basis (references/best-practices.md — full Osmani catalog index) and apply
    them to every step.
-8. **Gate** — nothing proceeds past authoring without a review step.
-   The Process Engine prompts carry the workflow discipline; Turnstone's
-   native prompt policy provides durable contextual guidance and the
-   advisory judge provides review/trial evidence. Neither silently replaces
-   operator approval. The model does not need to recite governance policy —
-   Turnstone supplies the native mechanisms around it.
+8. **Qualification continuity** — Review and Trial evidence applies only to the
+   artifact and deployable scope it identifies. A change that can affect behavior,
+   acceptance criteria, artifact membership, target/entity identity, capability or
+   tool boundaries, safeguards/risk behavior, or deployable scope is material and
+   returns through Pattern → Review → Trial as applicable. Typographical, formatting,
+   or equivalent non-behavioral cleanup may retain prior evidence when its
+   non-material judgment is recorded. Neither prior evidence nor native governance
+   silently replaces operator approval.
 
 ## Examples
 - "I want a skill that writes release notes" → collect (any material?) →