Repository navigation
v0.12.0: money-aware tool policies, testable policies and oversight reporting - #17
Merged
Merged
Conversation
…rsight reporting - max_per_day on numeric tool arguments: a running total per tool argument per UTC day; a call past the cap is handled like a mismatch. - review_above sends values above it to a human; with max it gives allowed, asked and refused bands. - draftcat gate check|test decides tool calls offline with the same evaluation as the endpoint; test exits 1 when a decision differs. - draftcat doctor --online confirms the Telegram token and chat, the provider key and relay reachability with read-only calls. - draftcat report summarizes decisions by pipeline, step, person and policy, skip reasons, tool outcomes and receipt integrity. - fixtures/<pipeline>/_expect.json turns draftcat test into a regression check. - budget status splits the day by provider model and by pipeline. - tool_gate.notify_outcomes tells the operator what allowed actions did. - Error notices for a halted run name the steps that did not run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v0.12.0: money-aware tool policies, testable policies and oversight reporting
Bounds for money-shaped tool arguments, policies and pipelines you can test in CI, a preflight that checks the credentials themselves, and a report of who decided what.
Changes, in review order
max_per_dayon a numeric tool argument: a running total per tool argument per UTC day. A call that would pass the cap is handled like a mismatch (on_mismatch: denyrefuses it asrule: max_per_day.<arg>,approveasks a human). Totals live in the state store and hold across restarts; capped decisions are made one at a timemaxcannot be bypassed by splitting one large amount into many allowed callsdraftcat gate check <tool> --args JSONanddraftcat gate test cases.yaml: decide tool calls againsttool_gatewithout the engine, with the same evaluation the endpoint uses.testexits 1 when a decision (or the deciding rule) differs from the casedraftcat doctor --online: read-only checks that Telegram accepts the token (getMe) and can reach the operator chat (getChat), the provider accepts the key, and the relay answers. Sends nothing and spends no tokens; credentials are redacted from every detail linedraftcat report: decisions per pipeline and step over a period, decisions by person vs. policy, per-operator counts, skip reasons, reported tool executions and receipt signature status, ending in a digest that matches the receipts of the same window. Markdown or--json; exits 1 on a tampered receiptreview_aboveon a numeric argument: values above it go to a human, even when the rule would allow the call. Withmaxit gives three bands: allowed, asked, refusedfixtures/<pipeline>/_expect.jsonfordraftcat test: assert the outcome, values in the final data, and text in approval drafts and operator messages. Unmet expectations are listed and the run exits 1; unknown keys are refusedbudget statusby model and by pipeline: each admitted model call records its provider model and pipeline in the admission row; status and--json(by_model,by_pipeline) split the daytool_gate.notify_outcomes: failed|all: the operator hears what an allowed action did once the harness reports it, once per outcomehalted at: <step>andnot run: <steps>Prompts for capped tools show the running total (
today: amount: 1200 allowed today, max_per_day 10000). Validation coversreview_above(must be belowmax),max_per_day(not negative) andnotify_outcomes. Guides: amount bands and daily totals, checking a policy offline, running Draftcat unattended, fixture expectations.Compatibility
All new settings are optional, and configurations without them decide exactly as before. Rules without
review_aboveormax_per_daykeep their policy hashes, so permits bound before the upgrade still match (pinned by a test). The state store gains atool_action_amountstable andmodel/pipelinecolumns onbudget_callsat first start;budget statuson a store the engine has not opened yet shows totals without the split.doctorwithout--onlinemakes no network calls. The stored run error and the log keep each step's own error text; only the operator notice names the halted steps.budget status --jsonadds two arrays. Versions are synchronized to 0.12.0 inversion.go,package.jsonandpackage-lock.json.Verification
go build ./...,go vet ./...,gofmt: cleango test -short -count=1 ./...: 16 packages ok; 495 top-level tests pass (469 onmaster), 0 failuresgo test -count=1 ./...(non-short),go test -race -short ./...,go test -short -tags voice ./...: okgolangci-lint run --new-from-rev=master(v2.5.0, repo config): 0 issues; the full run adds no finding tomaster's setgo run . validate: 0 errors;npm test: pass;npm pack --dry-run:draftcat-0.12.0.tgzmax_per_day: 1000through the live/gate/tool-callendpoint, then consume and a reported failure;reportcounts match;SIGTERMexits cleanly; no configured credential appears in the engine log.doctor --onlinewith deliberately invalid credentials reports the Telegram token and the provider key as rejected without printing eitherNew tests cover the bands and daily totals (including revoked permits, restarts and several capped arguments), the store-less fail-closed path, the prompt text,
gate check/gate test(pass, failing case, usage errors), doctor probes against a local Bot API and provider stand-in (accepted, rejected, transport errors redacted, no calls without--online), fixture expectations (pass, drift, skip outcome, unknown keys), the report (window, operators, skip reasons, digest, tampered receipt), outcome notices per mode with retry dedup, usage attribution (including older stores) and halted-run notes.Review notes
master@448be5a. No other open PRs.draftcat gate), tool rule constraints (internal/config), state store (amount totals, receipt queries, budget attribution), doctor,draftcat test, receipts verification helper, budget status, notices.tool_policy.go,gate_cmd.go,report_cmd.go,doctor_online.go,test_expect.go,halted_run.go,internal/state/{tool_amounts,report,budget_usage}.go.max_per_dayare stored next to the action so totals can be summed; all other arguments remain hashed only.