Skip to content

Report cost per accepted feature in XMD workflows #1162

Closed as not planned
Closed as not planned

Description

@taras

Story

As a person improving an XMD implementation workflow, I want a local report that relates token consumption to accepted features and shows the most expensive patterns, so I can choose useful optimization experiments.

For example, I compare one lane with four lanes. The report shows completed work, time, token consumption and accounting coverage. It also includes failed attempts and rework, so faster token consumption is not mistaken for worse efficiency and unfinished work is not mistaken for savings.

Current gap

Usage events and session totals do not by themselves answer whether a workflow is becoming more efficient. A usable analysis needs feature/trial outcomes, correct aggregation, visible missing data and evidence for possible improvements.

Required outcome

Provide a documented local analysis entrypoint over exported workflow records. DuckDB queries over versioned JSONL are the preferred initial implementation to evaluate; a dashboard service is unnecessary for the first report.

The report includes:

  • Tokens and cache-aware estimated cost per accepted feature/trial, including all associated failed attempts, retries and rework.
  • Acceptance rate, elapsed time, accepted-work throughput, role/stage consumption, handoff distribution and cap hits, recorded human effort/intervention, and accounting completeness.
  • Separate provider-reported amounts, API-equivalent estimates and actual billed amounts where available, with pricing/version provenance. Subscription percentages remain account/window capacity data.
  • Independent identities for experiment, variant and task repetition when comparing configurations.
  • Diagnostic views of session growth, internal request counts and repeated retrieval when deeper telemetry is available. Unsupported measurements are shown as unavailable.
  • A short ranked list of optimization hypotheses tied to the observed evidence, required follow-up measurement and quality criteria.

The primary cost metric is total cost across the assigned trial cohort divided by accepted trials. State the cohort and observation period, and show unfinished trials and missing accounting. Comparisons use all assigned outcomes, not only successful traces.

Accounting and interpretation

Aggregate usage and attempt-level costs separately before joining them. Multiple usage records cannot multiply one attempt's cost. Preserve the complete outcome cohort when measurements are missing.

Distinguish total work duration from elapsed time across concurrent lanes. Repeated reads are candidates for investigation, not proof of waste. The acceptance rate among tasks reaching handoff k describes that population; it does not establish the causal benefit of another handoff.

Acceptance and evidence

  1. Provide a recognizable, documented report/query entrypoint and a small known-total fixture with accepted, failed, retried, unfinished and partially observed trials.
  2. A fixture with one attempt costing 1 unit and six usage observations reports 1 unit, not 6. Adding a duplicate observation from another source does not increase consumption; adding a distinct real request does.
  3. A feature accepted after a failed attempt includes both attempts' costs. A failed feature still contributes to the cohort's expenditure. Missing usage does not remove its outcome or silently become zero.
  4. Repeated experimental executions of the same feature remain distinct trials. Moving an attempt between lanes does not create another accepted feature.
  5. Re-import and replay leave original historical totals unchanged; new live work increases them once.
  6. Produce a reconciled report for an accepted run and an unsuccessful run using real observations, and identify the strongest next optimization hypothesis without claiming unmeasured savings.
  7. A basic report works without internal-agent diagnostics; the richer views become available when that data is present.

Design and relationships

Depends on #1160, which depends on #1159. #1161 enriches the report but is not a prerequisite for its first useful delivery.

Settle the user-facing report/export invocation and initial host support before implementation. The report is an analysis of existing execution records, not a replacement for XMD's execution persistence.

Automatic prompt optimization, GEPA integration, shared context packs, model changes, session resets and handoff-cap changes are follow-up experiments selected from the evidence. This story measures and explains; it does not apply those changes.

Disposition — 2026-11-08

Closed as not planned at the user's direction. The current cost-tracking goal is per-message/turn token counts, completed entry/round summaries, and a running Sidekick total, tracked in #1201. This broader analytics story is not required for that delivery and is removed from the queue, with no automatic follow-on implementation. Closing this issue does not claim its requirements were implemented. The merged usage-capture foundation in #1166 remains in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions