Repository navigation
feat(report): one summary over cli_output.v1 documents, the leaves it needs, and actions/report (#749) - #850
Conversation
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi report summary`
+ argument `inputs`
+ option `--format`
+ option `--help/-h`
+ option `--out`
~ `dokimi catalog check` parameters changed:
+ added option `--json`
~ `dokimi catalog eval` parameters changed:
+ added option `--json`
~ `dokimi runs list` parameters changed:
~ changed option `--kind`: choices: ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'report_summary'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]
~ `dokimi skill routing` parameters changed:
+ added option `--json`Updated on every push; the |
|
OpenCodeReview: Commit fea499c was not reviewed. This PR has reached its review limit (6 reviews). CI at `fea499c`: 1 failing, 30 passing, 3 skipped
To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (4eea587): 1 medium</summary> Open threads from the bot: 0 </details> |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved validation and aggregation defects can crash summaries or hide failures and missing results, while output escaping and the required versioned schema remain incomplete.
6 open issues
What changed in this PR
6 findings.
Adds dokimi report summary, the renderer layer of #749, to combine CLI output documents without consumer-specific parsing.
Changes:
- Adds Markdown, JSON and annotation output with per-check results, joined subjects and spend totals.
- Registers the new CLI command with file/directory inputs and optional output-file writing.
- Adds focused tests, updates the CLI reference and documents the feature in a changelog fragment.
| File | Description |
|---|---|
dokimi/tests/test_report_summary.py |
Tests aggregation, invalid inputs, escaping and CLI output. |
dokimi/dokimi/report/summary.py |
Implements input loading, aggregation and rendering. |
dokimi/dokimi/commands/report.py |
Adds the command handler and arguments. |
dokimi/dokimi/cli_routes.py |
Registers summary as a free command. |
dokimi/docs/reference/cli-interface.json |
Records the new command’s interface. |
dokimi/changelog.d/20261116_120000_vchamarthi_report-summary.md |
Announces the renderer and its behavior. |
Review effort: Balanced
| for s in summary["subjects"]: | ||
| for kind, verdict in s["verdicts"].items(): | ||
| if verdict in _BAD: | ||
| out.append(f"::warning title={_escape(str(kind))}::{_escape(s['subject'])}: {verdict}") |
There was a problem hiding this comment.
Already holds on the head: Subject annotation titles go through _prop in render_annotations (dokimi/report/summary.py), pinned by tests/test_report_summary.py::test_subject_annotation_title_escapes_its_delimiters.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #850 +/- ##
==========================================
+ Coverage 95.75% 95.77% +0.02%
==========================================
Files 424 430 +6
Lines 56425 57143 +718
Branches 9720 9853 +133
==========================================
+ Hits 54029 54731 +702
- Misses 1487 1500 +13
- Partials 909 912 +3
🚀 New features to boost your workflow:
|
… needs, and actions/report (intel#749) - report summary: merges cli_output.v1 documents into markdown, json (report_summary.v1) or annotations. Envelope keys only; missing, unreadable or foreign inputs stay rows (not_run / invalid). No verdict computed, exit 0 once written. Stdlib only, no GITHUB_* read. - catalog check, catalog eval and skill routing take --json. A subject may carry findings (level, code, message, file, line); catalog check, catalog eval, repo validate and repo scan fill them. - actions/report: renders out-dir into the job summary and annotations, optional sticky PR comment, uploads out-dir. repo-eval.yml renders repo validate and repo scan with the same renderer. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
3dc9c78 to
2b70eeb
Compare
Fixes the review threads on report summary and its leaves: - an empty input directory stays a not_run row - malformed verdict, status, subjects or cost make an invalid row, not a TypeError - two documents of one kind keep their own subject verdicts - the subject table header is escaped like the check table - report_summary.v1 gets a bundled schema and a registry entry - skill routing --json names the loaded skill, carries selection_rate and still prints its document when the cap or a model error stops the run - catalog check --json refuses --jsonl, which would leave it no rows - actions/report finds the sticky comment of the token's own user and uploads out-dir after a failed comment step - the CLI interface snapshot lists the report_summary kind of runs list --kind Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
|
🟢 Ready for re-review · 1 commit on Next: a re-review of the new head. The author: correct the PR description (below). Changes
**Needs **
Threads and comments
1 thread left as it is, and why
Verified on the new head: ✅ 9/9 tests fail before the fix, pass after · ✅ full suite with main merged in: 22327 passed · ✅ second reader: nothing raised What each test shows
|
- summary: a non-string reason or error message, or a finding without string level, code and message, is an invalid row instead of output that breaks report_summary.v1. - summary: one column key per input document, so a subject missing from an earlier same-kind document is no longer filed under another key. - actions/report: a token other than the default whose login cannot be read is refused instead of assumed to be the Actions bot, which made the sticky comment duplicate on every run. Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 2137cb6. The renderer and the new --json leaves work, and the earlier review round is fixed. Some states still read better than they are: a run stopped by the cap can leave the summary at pass, and with --changed-from the skills nobody touched show as failing. The details are in the line comments.
Agreed with @exai-bot: the same-kind column is still picked per subject; _malformed does not check reason or the finding items; the sticky lookup fails with a GitHub App token.
What I ran, at 2137cb6:
make ci-local --offline: 22,298 passed, 0 failed, 72 skipped; every check passed or was skipped offline.- Repros of
summarizeon hand-madecli_output.v1documents: abudget_stoppedrouting run next to a passing check, two same-kind documents with different subjects, a malformedreasonand finding, and a foreign JSON next to the documents in one directory. catalog eval --changed-from HEAD~1 --jsonon a two-skill git catalog where only the skill outside the change is broken, thenreport summary --format annotationson its output.
Checked, no issues
- Schema:
cli_output.v1is only widened (an optionalfindingson a subject); the rest of its diff is reformatting.report_summary.v1has a schema, a fixture and a registry entry. skill routing --json: a run stopped by the cap or a model error prints its document and keeps its exit code; the subject is the loaded skill's name.catalog check --json: subject verdicts follow the same rule as the scan's exit,--strictincluded;--jsonland--format jsonare refused.- Subject annotation titles: escaped with
_prop(the open Copilot thread on this is fixed). - Changelog and CLI snapshot: they cover the new command, the three new
--jsonflags and theruns list --kindchoice.
|
🟢 Ready for re-review · 1 commit on Next: a re-review of the new head. Changes
Threads and comments
1 thread left as it is, and why
Verified on the new head: ✅ 4/4 tests fail before the fix, pass after · ✅ full suite with main merged in: 22335 passed · ✅ second reader: nothing raised What each test shows
|
- A document with status budget_stopped, partial or cancelled is that state in the summary, not a pass, and gets a warning annotation. - An unpriced spend prints as unpriced and leaves the total empty; skill routing --json gives priced null for no calls and no remainder for an unpriced run. - catalog eval --json gives a skill outside --changed-from verdict null and no findings, and points findings at SKILL.md under the given root. - The Markdown lists each finding (first 100) and escapes <, > and \r. - actions/report cuts the sticky comment below 65,536 characters and warns instead of failing when it cannot be posted; out-dir documents that its top-level *.json must be cli_output.v1. Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
|
🟢 Ready for re-review · 1 commit on Next: @vlad-perevezentsev: re-review the new head. PR description updated
Threads and comments
Verified on the new head: ✅ 9/9 tests fail before the fix, pass after · What each test shows
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at e2387d8. Every point from my first round is fixed: a stopped run is no longer a pass, skills outside --changed-from are left unjudged, the job summary lists the findings again, and an unpriced spend shows as unpriced. Three small gaps remain, all in @exai-bot's new threads.
Agreed with @exai-bot: cost.priced is not validated; task findings point at a SKILL.md that does not exist; a failed sticky lookup posts a duplicate comment.
What I ran, at e2387d8:
make ci-local --offline: 22,312 passed, 0 failed, 72 skipped. The one FAIL row,schema_compat, compares against the currentmain, which addedrepo_event.v1.jsonafter this PR's base; this PR changes onlycli_output.v1andreport_summary.v1.- Repros of
summarizeon hand-made documents: abudget_stoppedrun next to a passing check (overall: budget_stopped, one::warning, totalnullfor an unpriced spend), the same document given twice (keyskind (src)andkind (src) #3),reason: 123(aninvalidrow), andcost.priced: "false". catalog eval's_emiton a report with anundeclared-taskfinding.
Checked, no issues
- Sticky comment: a failed POST or PATCH is now a warning, the body stays below the 65,536-character limit, and a token whose login cannot be read is refused instead of guessed.
report_summary.v1: the newpricedfield and the three stopped states are new in this PR, so changing the schema in place is fine.- Changelog: it covers the stopped states,
unpriced, the findings table and the--changed-fromscope.
- actions/report no longer posts a second comment when the lookup of the existing one fails (even after printing an id); it warns and skips. - catalog eval --json points a task finding at its task.toml, task.yaml or task.md instead of a nonexistent <task>/SKILL.md. - report summary treats a cost.priced that is not a boolean or null as an invalid row, so the JSON stays valid against report_summary.v1. Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
|
🟢 Ready for re-review · 1 commit on Next: a re-review of the new head. Threads and comments
Verified on the new head: ✅ 3/3 tests fail before the fix, pass after · ✅ full suite with main merged in: 22680 passed · ✅ second reader: nothing raised What each test shows
|
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The renderer keeps every input as a row, computes no verdict, and the --json leaves, the schemas and actions/report match the description. The earlier review rounds are fixed at this head. One nit inline.
What I ran, on a Linux review host at 4eea587:
make ci-local: 34 passed, 0 failed, 14 skipped (offline or Docker-only jobs); 22,369 tests; changed-line coverage 95%.- A reproduction of the
repo scanjob summary with the LLM transport mocked and no network: a scan payload with onewarningsentry renders a summary without that warning. - All five commits carry a
Signed-off-by.
Checked, no issues
- Schemas:
report_summary.v1has a schema, a fixture and a registry entry;cli_output.v1is only widened with an optionalfindings. - Annotations: titles and files go through
_prop, messages through_escape;test_subject_annotation_title_escapes_its_delimiterspins it. - Sticky comment: a failed lookup, POST or PATCH is a warning and the body stays under the 65,536-character limit.
- Docs and CLI surface: the changelog fragment and
cli-interface.jsoncover the new command and the three--jsonflags.
Bot points I checked that no longer apply at this head (1)
- Copilot's point on unescaped subject annotation titles does not apply:
render_annotationspasses them through_prop(summary.py:243), andtests/test_report_summary.py:289pins it.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to fea499c, which fix the nit from my review:
fea499c:dokimi/dokimi/commands/repo_scan.py:171:scan_subjectsadds each payload warning as ascan-warningfinding on the first subject.
make ci-local at fea499c: 34 passed, 0 failed, 14 skipped.

Closes #749 except step 5 (paid leaves opting in) and the switch in intel/skills.
What this changes
dokimi report summary PATH... [--format markdown|json|annotations] [--out FILE]: mergescli_output.v1files, or directories of*.jsoncatalog check,catalog evalandskill routingtake--json. A subject may carryfindings(level,code,message,file,line);catalog check,catalog eval,repo validateandrepo scanfill themactions/report: rendersout-dirinto the job summary and annotations, optionally keeps one sticky PR comment, and uploadsout-dir.repo-eval.ymlrendersrepo validateandrepo scanwith it instead of two inline Python blocksRenderer rules, from the issue:
kind,status,verdict,reason,error,summary,subjects,cost. It has no per-leaf code.cli_output.v1input becomes anot_runorinvalidrow and setsoverall. It is never dropped.verdict: nullshows asreport_only. It exits 0 once it has written its output.unpriced, the scalarsummaryvalues, reason), a subject table (verdict and finding count per check) and a findings table (the first 100). Annotations have one line per finding, withfile=andline=when the finding has them.GITHUB_*variable.Leaf details:
catalog check --json: one subject per skill and per task (task:<name>). A subject fails when it alone fails the exit: a load error, or under--stricta spec error or unrestricted Bash with a risky command. The exit code is unchanged.--jsonwith--format jsonis refused.catalog eval --json: one subject per skill with its static findings.dataholds the evaluation state of each skill and the findings that belong to no skill. Receipts are written as before.skill routing --json: report only (verdict: null), withqueries_self_derivednext to the recall scores and cost against--max-usd. The human summary goes to stderr.repo scan --json: one subject per guidance file. Each failing claim is a finding at its file and line. Claim counts go insummary.Not changed:
Select tasksstep inrepo-eval.yml. It is selection logic, not a summary.Summarystep.eval checkemits nocli_output.v1yet.catalog routing(all-vs-all) and the paid leaves get no--jsonhere.Tests
tests/test_report_summary.py: the renderer (verdicts per document, subject join,not_runandinvalidrows, cost, escaping, findings as annotations, the summary column), each new--jsonleaf validated against the schema,repo scansubjects,actions/report(renderer only, no parsing), andrepo-eval.yml.cli-interface.jsonis regenerated.make ci-local --offlineon Linux: 22 jobs pass, including mypy, the growth ratchet and the CLI surface.test / Run testsfails only on the 3 tests intests/test_repo_compare_interval.py, which fail the same way onorigin/main.