Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions BACKLOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ This backlog tracks public repository governance work that is safe to discuss in

## Current Governance Queue

- Review Dependabot PR #6 for `@types/node` patch updates. Current required check `verify` is passing, but merging remains a maintainer action.
- Keep issue #5 open as the public v0.1.0 feedback collection point until launch feedback has been triaged into concrete issues.
- Keep main branch protection active with required PR review, stale review dismissal, strict status checks, and the `verify` required check.
- Review current dependency proposals using the [dependency review guide](docs/dependency-review.md); recheck exact-head CI rather than relying on old PR numbers or green checks.
- Triage public feedback into reproducible findings, expected behavior and focused fixes before changing issue state.
- Recheck branch protection and required checks before a merge; record the current state instead of treating this backlog as a live configuration report.

## Launch Hygiene

Expand All @@ -16,5 +16,5 @@ This backlog tracks public repository governance work that is safe to discuss in

## Later

- Add a short Dependabot review checklist for patch, minor, and major dependency updates.
- Add a release checklist that links launch copy, distribution notes, smoke output, and npm package metadata.
- Keep the dependency guide aligned with runtime engines, Action runner requirements and current maintainer rules.
- Use the existing [release readiness](docs/release-readiness.md) and [distribution checklist](docs/launch/distribution-checklist.md); verify package metadata and links before a separately authorized release.
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Changelog

## Unreleased

- Count nested snake_case usage tokens, support case-insensitive JSONL exports and UTF-8 BOMs, and warn when malformed JSON records leave partial cost totals.
- Add an exact-head dependency/Action review guide and replace stale maintenance and pre-release response placeholders with usable public documentation.

## 0.1.0

- Added the first productized CLI: `scan`, `doctor`, and `init`.
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ The repository includes a launch kit so maintainers can prepare public posts, de
- [Community responses](docs/launch/community-responses.md)
- [Channel rules](docs/launch/channel-rules.md)
- [Distribution checklist](docs/launch/distribution-checklist.md)
- [Dependency and Action review](docs/dependency-review.md)
- [Demo GIF script](docs/launch/demo-gif-script.md)
- [Product Hunt draft](docs/launch/product-hunt.md)
- [DEV article draft](docs/launch/devto-article.md)
Expand Down
5 changes: 5 additions & 0 deletions docs/ai-cost-guard.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ JSONL:
```json
{"provider":"openai","model":"gpt-5.2","inputTokens":1000,"outputTokens":500,"costUsd":0.25}
{"provider":"anthropic","model":"claude-sonnet","usage":{"prompt_tokens":2000,"completion_tokens":1000},"cost_usd":0.75}
{"provider":"synthetic-provider","model":"synthetic-model","usage":{"input_tokens":1000,"output_tokens":500,"total_tokens":1500},"costUsd":0.25}
```

JSON:
Expand All @@ -41,4 +42,8 @@ JSON:

## Scope

Token fields can use camelCase or the shown snake_case names, at the event root or inside `usage`. Existing `prompt_tokens`/`completion_tokens` aliases remain supported. An explicit total takes precedence over the input/output sum. The tool summarizes supplied cost fields; it does not infer prices or cost from token counts.

JSON/JSONL file extensions are case-insensitive, and a UTF-8 BOM at the start of an export is accepted. Empty JSONL lines are ignored. If a JSON document or nonempty JSONL line has invalid syntax, valid records are still summarized, but an alert marks the totals as potentially incomplete and `cost-report` returns 1 even if the parsed cost is below budget. The invalid input text is not copied into reports. Repair the trace and rerun before treating the totals as complete. This syntax check does not certify arbitrary event schemas or provider billing data.

This is a local cost guard, not a provider billing source of truth. It is designed for agent run traces, budget alerts, and "which model burned the most tokens?" debugging.
26 changes: 26 additions & 0 deletions docs/dependency-review.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Dependency and Action review

Review the proposed head, not a previous commit's successful run. Dependency installation and verification are preparation; they do not grant permission to merge, publish a package or change repository protection.

## Classify the proposal

| Change | Review focus | Merge gate |
| --- | --- | --- |
| npm patch | Manifest/lock agreement, resolved package and transitive changes, supported runtime | A low-risk Dependabot patch may use auto-merge only when current protection and required checks are active and repository rules permit it. |
| npm minor | New APIs, transitive changes, runtime/engine requirements, relevant behavior | Apply current maintainer rules; a minor label alone does not establish compatibility. |
| npm major | Migration notes, changed defaults, supported Node versions, build and CLI behavior | Maintainer review is required. |
| GitHub Action | Official source/ref, runner runtime, permissions, triggers, token use and output/artifact scope | Maintainer review is required, even when the application Node version is unchanged. |

For grouped updates, assess every package rather than treating the group as one version bump. Development type packages can expose APIs absent from an older supported runtime; a passing typecheck alone does not prove runtime compatibility.

## Prepare evidence

1. Check out the exact proposal in isolation and preserve current main fixes and unrelated work. Inspect the manifest/lock diff and release or migration notes from the upstream project.
2. Run `npm ci`, `npm run check` and `npm run smoke`. For a runtime-sensitive change, exercise the minimum supported Node version as well as the current development version.
3. Run `npm audit` and distinguish actionable findings from unsupported-platform skips or network failures. Zero audit findings do not replace behavior tests or code review.
4. Recheck GitHub CI on the exact final head, current mergeability, required checks and review requirements. A later edit invalidates evidence that covered only the old head.
5. Record versions, commands, results and remaining limits in the PR. Keep private paths, credentials and raw private logs out of shared evidence.

Do not dispatch an issue-writing digest, release workflow or privileged job just to obtain a green badge. Inspect existing workflow triggers before deciding which verification is appropriate.

For a separately authorized release, continue with [release readiness](release-readiness.md) and the [distribution checklist](launch/distribution-checklist.md). This guide does not create or modify auto-merge, branch protection or workflow configuration.
24 changes: 6 additions & 18 deletions docs/launch/community-responses.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Agent Reliability Kit is a local-first CLI that checks whether a repository is r

Links:

- <PUBLIC_REPO_URL>
- <NPM_PACKAGE_URL>
- <DOCS_URL>
- [Public repository](https://github.com/aolingge/agent-reliability-kit)
- [npm package](https://www.npmjs.com/package/agent-reliability-kit)
- [Documentation](https://aolingge.github.io/agent-reliability-kit/)

## Is this another AI coding agent?

Expand Down Expand Up @@ -38,7 +38,7 @@ No. It makes repo readiness issues visible and repeatable. It does not guarantee

Yes, that is one intended workflow. The CLI writes human-readable and machine-readable outputs so maintainers can use it before PRs, in CI, or during release preparation.

Example after public package availability:
Example using the public package:

```bash
npx agent-reliability-kit scan . --min-score 85
Expand All @@ -58,23 +58,11 @@ MCP and local toolchains can expand what an agent can do. The scanner looks for

## Is the npm package available?

Use the current launch state. If <NPM_PACKAGE_URL> is not live yet, say:

> The project is in pre-release local mode. The `npx` path will be the public install path once the npm package is published.

If <NPM_PACKAGE_URL> is live, use:

> The npm package is available here: <NPM_PACKAGE_URL>
Yes. The package is available on [npm](https://www.npmjs.com/package/agent-reliability-kit). Check the registry version before quoting a version number; installation instructions are in the [README](../../README.md#quick-start).

## Is the GitHub repo public?

Use the current launch state. If <PUBLIC_REPO_URL> is not live yet, say:

> The public repository link will be shared when the launch surface is ready.

If <PUBLIC_REPO_URL> is live, use:

> The public repository is here: <PUBLIC_REPO_URL>
Yes. The source, public fixtures and contributor instructions are in the [GitHub repository](https://github.com/aolingge/agent-reliability-kit). These response examples are drafts for maintainers to adapt; no message is sent automatically.

## How should I report a false positive?

Expand Down
43 changes: 25 additions & 18 deletions src/cost/costReport.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,9 @@ export interface CostReport {
export function buildCostReport(options: CostReportOptions): CostReport {
const root = resolveRoot(options.root);
const traceFiles = resolveTraceFiles(root, options.tracePath);
const events = traceFiles.flatMap((file) => readCostEvents(file));
const parsedFiles = traceFiles.map((file) => readCostEvents(file));
const events = parsedFiles.flatMap((parsed) => parsed.events);
const malformedRecords = parsedFiles.reduce((count, parsed) => count + parsed.malformedRecords, 0);
const byModel = summarizeByModel(events);
const total = byModel.reduce<Omit<CostBucket, "provider" | "model">>((acc, bucket) => ({
calls: acc.calls + bucket.calls,
Expand All @@ -55,6 +57,7 @@ export function buildCostReport(options: CostReportOptions): CostReport {

if (traceFiles.length === 0) alerts.push("No trace files found. Pass --trace or write JSON/JSONL traces under .agent-reliability/traces.");
if (events.length === 0 && traceFiles.length > 0) alerts.push("Trace files were found, but no token or cost events could be parsed.");
if (malformedRecords > 0) alerts.push(`Skipped ${malformedRecords} malformed JSON record(s); totals may be incomplete. Repair the trace input and rerun.`);
if (options.budgetUsd !== undefined && total.costUsd > options.budgetUsd) {
alerts.push(`Cost ${formatMoney(total.costUsd)} is above budget ${formatMoney(options.budgetUsd)}.`);
}
Expand Down Expand Up @@ -131,12 +134,24 @@ function resolveTraceFiles(root: string, tracePath?: string): string[] {
return files;
}

function readCostEvents(file: string): CostEvent[] {
const text = fs.readFileSync(file, "utf8");
const records = file.endsWith(".jsonl")
? text.split(/\r?\n/).map((line) => line.trim()).filter(Boolean).map(parseJson)
: [parseJson(text)];
return records.flatMap((record) => normalizeRecord(record)).filter((event): event is CostEvent => event !== null);
function readCostEvents(file: string): { events: CostEvent[]; malformedRecords: number } {
const text = fs.readFileSync(file, "utf8").replace(/^\uFEFF/, "");
const records = path.extname(file).toLowerCase() === ".jsonl"
? text.split(/\r?\n/).map((line) => line.trim()).filter(Boolean)
: [text];
const events: CostEvent[] = [];
let malformedRecords = 0;
for (const record of records) {
let parsed: unknown;
try {
parsed = JSON.parse(record) as unknown;
} catch {
malformedRecords += 1;
continue;
}
for (const event of normalizeRecord(parsed)) events.push(event);
}
return { events, malformedRecords };
}

function normalizeRecord(record: unknown): CostEvent[] {
Expand All @@ -149,9 +164,9 @@ function normalizeRecord(record: unknown): CostEvent[] {
const usage = typeof item.usage === "object" && item.usage !== null ? item.usage as Record<string, unknown> : {};
const provider = stringField(item.provider) ?? stringField(item.modelProvider) ?? "unknown";
const model = stringField(item.model) ?? stringField(item.modelName) ?? "unknown";
const inputTokens = numberField(item.inputTokens) ?? numberField(item.input_tokens) ?? numberField(item.prompt_tokens) ?? numberField(usage.inputTokens) ?? numberField(usage.prompt_tokens) ?? 0;
const outputTokens = numberField(item.outputTokens) ?? numberField(item.output_tokens) ?? numberField(item.completion_tokens) ?? numberField(usage.outputTokens) ?? numberField(usage.completion_tokens) ?? 0;
const totalTokens = numberField(item.totalTokens) ?? numberField(item.total_tokens) ?? numberField(usage.totalTokens) ?? inputTokens + outputTokens;
const inputTokens = numberField(item.inputTokens) ?? numberField(item.input_tokens) ?? numberField(item.prompt_tokens) ?? numberField(usage.inputTokens) ?? numberField(usage.input_tokens) ?? numberField(usage.prompt_tokens) ?? 0;
const outputTokens = numberField(item.outputTokens) ?? numberField(item.output_tokens) ?? numberField(item.completion_tokens) ?? numberField(usage.outputTokens) ?? numberField(usage.output_tokens) ?? numberField(usage.completion_tokens) ?? 0;
const totalTokens = numberField(item.totalTokens) ?? numberField(item.total_tokens) ?? numberField(usage.totalTokens) ?? numberField(usage.total_tokens) ?? inputTokens + outputTokens;
const costUsd = numberField(item.costUsd) ?? numberField(item.cost_usd) ?? numberField(item.usd) ?? 0;

if (inputTokens + outputTokens + totalTokens + costUsd === 0) return [];
Expand Down Expand Up @@ -181,14 +196,6 @@ function summarizeByModel(events: CostEvent[]): CostBucket[] {
return [...buckets.values()].sort((left, right) => right.costUsd - left.costUsd);
}

function parseJson(text: string): unknown {
try {
return JSON.parse(text) as unknown;
} catch {
return null;
}
}

function numberField(value: unknown): number | undefined {
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
}
Expand Down
65 changes: 65 additions & 0 deletions tests/costReport.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { buildCostReport } from "../src/cost/costReport.js";
import { runCli } from "../src/cli.js";

function fixture(name: string, content: string): { root: string; tracePath: string } {
const root = fs.mkdtempSync(path.join(os.tmpdir(), "ark-cost-parsing-"));
fs.writeFileSync(path.join(root, name), content, "utf8");
return { root, tracePath: name };
}

describe("cost trace parsing", () => {
it("summarizes a large valid event array without argument-list limits", () => {
const input = fixture("trace.json", JSON.stringify({ events: Array.from({ length: 150_000 }, () => ({ inputTokens: 1 })) }));
const report = buildCostReport({ ...input, outDir: ".reports" });
expect(report.total.calls).toBe(150_000);
expect(report.total.inputTokens).toBe(150_000);
expect(report.status).toBe("pass");
});

it("counts nested snake_case usage and preserves the explicit total", () => {
const input = fixture("trace.json", JSON.stringify({
provider: "synthetic-provider", model: "synthetic-model",
usage: { input_tokens: 100, output_tokens: 20, total_tokens: 150 }, costUsd: 0.25
}));
const report = buildCostReport({ ...input, outDir: ".reports" });
expect(report.total).toMatchObject({ calls: 1, inputTokens: 100, outputTokens: 20, totalTokens: 150, costUsd: 0.25 });
expect(report.status).toBe("pass");
});

it("reads multiple records from an uppercase JSONL extension", () => {
const input = fixture("trace.JSONL", '{"inputTokens":10,"costUsd":0.1}\n{"inputTokens":20,"costUsd":0.2}\n');
const report = buildCostReport({ ...input, outDir: ".reports" });
expect(report.total.calls).toBe(2);
expect(report.total.inputTokens).toBe(30);
expect(report.total.costUsd).toBeCloseTo(0.3);
expect(report.status).toBe("pass");
});

it("accepts a UTF-8 BOM in a JSON export", () => {
const input = fixture("trace.json", '\uFEFF{"inputTokens":12,"costUsd":0.2}');
const report = buildCostReport({ ...input, outDir: ".reports" });
expect(report.total.inputTokens).toBe(12);
expect(report.status).toBe("pass");
});

it("warns on partial parsing without exposing the malformed record", () => {
const input = fixture("trace.jsonl", '{"inputTokens":10,"costUsd":0.1}\nsynthetic-private-invalid-record\n');
const report = buildCostReport({ ...input, outDir: ".reports", budgetUsd: 1 });
expect(report.total.calls).toBe(1);
expect(report.status).toBe("warn");
expect(report.alerts.join(" ")).toContain("incomplete");
expect(JSON.stringify(report)).not.toContain("synthetic-private-invalid-record");
const stdout: string[] = [];
const stderr: string[] = [];
const code = runCli(["cost-report", input.root, "--trace", input.tracePath, "--budget-usd", "1", "--out", ".reports"], {
cwd: input.root, stdout: (value = "") => stdout.push(value), stderr: (value = "") => stderr.push(value)
});
expect(code).toBe(1);
expect(stdout.join(" ")).toContain("incomplete");
expect(stderr).toEqual([]);
});
});
Loading