Skip to content

Attribute implementation cost to XMD workflow runs and lanes #1160

Closed as not planned

Description

@taras

Story

As a person running four implementation lanes, I want every measured prompt and retry associated with the feature and role that caused it, so I can distinguish higher throughput from higher cost per completed feature.

For example, a feature runs through architect/planner exchanges, reaches a handoff limit, and succeeds in a later attempt. Its account includes both attempts and any recorded human intervention. Work in another lane remains separate even when sessions share an agent runtime.

Current gap

Provider-session totals do not identify feature outcomes, workflow stages or the reason a handoff occurred. XMD already owns canonical Prompt execution and host publication lifecycles, but their associations do not yet form a usable implementation-cost account.

Required outcome

  • Associate available usage with feature/task, trial, attempt, workflow run, lane, role, canonical Prompt, logical/native session and provider request identities. A trial distinguishes an independent execution of a task from another experimental repetition.
  • Use workflow ownership for role/lane labels and actual request/session identities for joins. Descriptive session names, process environment, PID or timestamps alone do not establish ownership.
  • Record the workflow/prompt version, starting repository revision, configured handoff/budget settings and actual concurrency needed to interpret a run.
  • Keep execution termination separate from feature acceptance. Record acceptance evidence, retries/rework and supplied human-intervention observations without inferring them from a successful process exit.
  • Link each handoff to its source and destination prompts where known. Capture its reason, payload identity/size, requested next action, relevant artifact references and position against the configured limit. Unknown reasons remain unknown.
  • Preserve historical spend when a workflow is reopened or replayed. Reusing a completed effect adds no new agent-spend record.
  • Failed/cancelled work and interruptions before a successful journal publication remain visible. Reconciliation can recover available provider measurements; gaps are reported explicitly.
  • Provide a documented local, versioned export for analysis. JSONL is a suitable initial candidate; the storage and export surface are design decisions.
  • State coverage for direct Agent.prompt() calls that do not pass through the canonical journal-owned Prompt boundary.

Acceptance and evidence

  1. A deterministic multi-lane fixture with distinct request identities attributes every record correctly, including several sessions sharing a runtime and repeated descriptive session names.
  2. An accepted feature report includes a preceding failed or cap-exhausted attempt and links the acceptance evidence to the correct trial.
  3. Replaying or resuming completed work does not duplicate its spend. A newly executed turn after resume adds only its own consumption.
  4. Failure/cancellation before publication preserves observed work and marks incomplete accounting. A missing measurement is neither a zero nor a reason to remove the attempt from the cohort.
  5. An interrupted export can be retried or re-imported without duplicating records; the chosen record identity and completion/coverage semantics are documented.
  6. Demonstrate one accepted implementation and one unsuccessful implementation using the documented export path, with enough provenance to trace a reported total back to its source.

Design and relationships

Depends on #1159 for verified prompt usage.

Use the existing host Prompt publication lifecycle and journal live/replay boundary where applicable. The publication hook does not itself provide usage today; connect the existing owners without assuming a public middleware sees provider-private values.

Settle how the implementation workflow supplies feature identity and acceptance/human-effort observations before finalizing the interface. Collection failure behavior must preserve authoritative execution results and make coverage loss visible.

This story does not change process/session allocation, authentication directories, scheduling or handoff policy to make attribution easier. #894 owns general Effection-tree inspection; this account concerns usage and implementation outcomes.

Disposition — 2026-11-08

Closed as not planned at the user's direction. The current cost-tracking goal is per-message/turn token counts, completed entry/round summaries, and a running Sidekick total, tracked in #1201. This broader analytics story is not required for that delivery and is removed from the queue, with no automatic follow-on implementation. Closing this issue does not claim its requirements were implemented. The merged usage-capture foundation in #1166 remains in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions