Skip to content

feat(report): one summary over cli_output.v1 documents, the leaves it needs, and actions/report (#749) - #850

Merged
antonwolfy merged 6 commits into
intel:mainfrom
vchamarthi:feat/report-summary
Nov 17, 2026
Merged

antonwolfy merged 6 commits into
intel:mainfrom
vchamarthi:feat/report-summary

Conversation

@vchamarthi

@vchamarthi vchamarthi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #749 except step 5 (paid leaves opting in) and the switch in intel/skills.

What this changes

Layer Change
1. renderer dokimi report summary PATH... [--format markdown|json|annotations] [--out FILE]: merges cli_output.v1 files, or directories of *.json
2. leaves catalog check, catalog eval and skill routing take --json. A subject may carry findings (level, code, message, file, line); catalog check, catalog eval, repo validate and repo scan fill them
3. action actions/report: renders out-dir into the job summary and annotations, optionally keeps one sticky PR comment, and uploads out-dir. repo-eval.yml renders repo validate and repo scan with it instead of two inline Python blocks

Renderer rules, from the issue:

  • It reads envelope keys only: kind, status, verdict, reason, error, summary, subjects, cost. It has no per-leaf code.
  • A missing, unreadable or non-cli_output.v1 input becomes a not_run or invalid row and sets overall. It is never dropped.
  • It computes no verdict. verdict: null shows as report_only. It exits 0 once it has written its output.
  • Markdown has a check table (state, verdict, spent / cap or unpriced, the scalar summary values, reason), a subject table (verdict and finding count per check) and a findings table (the first 100). Annotations have one line per finding, with file= and line= when the finding has them.
  • It is stdlib only and reads no GITHUB_* variable.

Leaf details:

  • catalog check --json: one subject per skill and per task (task:<name>). A subject fails when it alone fails the exit: a load error, or under --strict a spec error or unrestricted Bash with a risky command. The exit code is unchanged. --json with --format json is refused.
  • catalog eval --json: one subject per skill with its static findings. data holds the evaluation state of each skill and the findings that belong to no skill. Receipts are written as before.
  • skill routing --json: report only (verdict: null), with queries_self_derived next to the recall scores and cost against --max-usd. The human summary goes to stderr.
  • repo scan --json: one subject per guidance file. Each failing claim is a finding at its file and line. Claim counts go in summary.

Not changed:

  • The Select tasks step in repo-eval.yml. It is selection logic, not a summary.
  • The tasks-check Summary step. eval check emits no cli_output.v1 yet.
  • catalog routing (all-vs-all) and the paid leaves get no --json here.

Tests

  • tests/test_report_summary.py: the renderer (verdicts per document, subject join, not_run and invalid rows, cost, escaping, findings as annotations, the summary column), each new --json leaf validated against the schema, repo scan subjects, actions/report (renderer only, no parsing), and repo-eval.yml.
  • cli-interface.json is regenerated.
  • make ci-local --offline on Linux: 22 jobs pass, including mypy, the growth ratchet and the CLI surface. test / Run tests fails only on the 3 tests in tests/test_repo_compare_interval.py, which fail the same way on origin/main.

@vchamarthi
vchamarthi requested a review from a team as a code owner November 17, 2026 02:05
Copilot AI balanced review requested due to automatic review settings November 17, 2026 02:05
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi report summary`
    + argument `inputs`
    + option `--format`
    + option `--help/-h`
    + option `--out`
~ `dokimi catalog check` parameters changed:
    + added option `--json`
~ `dokimi catalog eval` parameters changed:
    + added option `--json`
~ `dokimi runs list` parameters changed:
    ~ changed option `--kind`: choices: ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"] -> ["'answers'", "'arms'", "'catalog_eval'", "'checklist'", "'cli_routes'", "'cross_judge'", "'dokimi_config'", "'effective_instructions'", "'eval'", "'eval_coverage'", "'explore_runs'", "'grounding'", "'guidance_reach'", "'headroom'", "'judge_validity'", "'judge_wording'", "'leak'", "'matrix_rollup'", "'need_graph'", "'novelty'", "'pairwise'", "'persona_analysis'", "'personas'", "'plan'", "'plugin_delta'", "'question_panel'", "'question_quality'", "'questions'", "'ragas'", "'reach_matrix'", "'redteam'", "'repo_budget'", "'repo_environment'", "'repo_hits_judged'", "'repo_ledger'", "'repo_readiness'", "'report_qa'", "'report_summary'", "'result'", "'run_context'", "'security_report'", "'skill_coverage'", "'skill_discovery'", "'skill_evidence'", "'subject_scorecard'", "'task_inspect'", "'task_runs'", "'tool_preference'"]
~ `dokimi skill routing` parameters changed:
    + added option `--json`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit fea499c was not reviewed. This PR has reached its review limit (6 reviews).

CI at `fea499c`: 1 failing, 30 passing, 3 skipped

33 of 34 checks are passing or skipped; Tests (py3.14) fails on one SIGINT/cancellation lifecycle test. The failure is consistent with a timing-sensitive test race in unchanged batch execution code, not a regression in the report-summary changes; no change-specific fix is supported by the evidence.

Check Status Assessment
Tests (py3.14) failing at step Run tests Infrastructure or flaky. The logged test relies on a 0.3-second sleep after sending SIGINT for the main thread to cancel queued work; this run started repetitions 3 and 4 before cancellation. Neither the test, the executor cancellation code, nor the test workflow is changed by this PR, supporting a...

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (4eea587): 1 medium</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved validation and aggregation defects can crash summaries or hide failures and missing results, while output escaping and the required versioned schema remain incomplete.

6 open issues
What changed in this PR

6 findings.

Adds dokimi report summary, the renderer layer of #749, to combine CLI output documents without consumer-specific parsing.

Changes:

  • Adds Markdown, JSON and annotation output with per-check results, joined subjects and spend totals.
  • Registers the new CLI command with file/directory inputs and optional output-file writing.
  • Adds focused tests, updates the CLI reference and documents the feature in a changelog fragment.
File Description
dokimi/​tests/​test_report_summary.py Tests aggregation, invalid inputs, escaping and CLI output.
dokimi/​dokimi/​report/​summary.py Implements input loading, aggregation and rendering.
dokimi/​dokimi/​commands/​report.py Adds the command handler and arguments.
dokimi/​dokimi/​cli_routes.py Registers summary as a free command.
dokimi/​docs/​reference/​cli-interface.json Records the new command’s interface.
dokimi/​changelog.d/​20261116_120000_vchamarthi_report-summary.md Announces the renderer and its behavior.

Review effort: Balanced

Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Outdated
for s in summary["subjects"]:
for kind, verdict in s["verdicts"].items():
if verdict in _BAD:
out.append(f"::warning title={_escape(str(kind))}::{_escape(s['subject'])}: {verdict}")

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already holds on the head: Subject annotation titles go through _prop in render_annotations (dokimi/report/summary.py), pinned by tests/test_report_summary.py::test_subject_annotation_title_escapes_its_delimiters.

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.65176% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.77%. Comparing base (b524769) to head (fea499c).
⚠️ Report is 19 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/report/summary.py 95.34% 3 Missing and 5 partials ⚠️
dokimi/dokimi/commands/report.py 50.00% 6 Missing ⚠️
dokimi/dokimi/commands/catalog.py 88.88% 2 Missing and 2 partials ⚠️
dokimi/dokimi/commands/novelty.py 92.30% 2 Missing and 1 partial ⚠️
dokimi/dokimi/commands/catalog_eval.py 96.29% 0 Missing and 1 partial ⚠️
dokimi/dokimi/commands/repo_scan.py 93.33% 0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #850      +/-   ##
==========================================
+ Coverage   95.75%   95.77%   +0.02%     
==========================================
  Files         424      430       +6     
  Lines       56425    57143     +718     
  Branches     9720     9853     +133     
==========================================
+ Hits        54029    54731     +702     
- Misses       1487     1500      +13     
- Partials      909      912       +3     
Files with missing lines Coverage Δ
dokimi/dokimi/artifacts.py 95.52% <ø> (ø)
dokimi/dokimi/cli_output.py 100.00% <100.00%> (ø)
dokimi/dokimi/cli_routes.py 97.47% <ø> (ø)
dokimi/dokimi/commands/repo.py 98.98% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/commands/routing_parser.py 100.00% <100.00%> (ø)
dokimi/dokimi/runs/kinds.py 100.00% <ø> (ø)
dokimi/dokimi/commands/catalog_eval.py 98.87% <96.29%> (-0.48%) ⬇️
dokimi/dokimi/commands/repo_scan.py 92.66% <93.33%> (+0.10%) ⬆️
dokimi/dokimi/commands/novelty.py 99.07% <92.30%> (-0.67%) ⬇️
dokimi/dokimi/commands/catalog.py 98.11% <88.88%> (-1.01%) ⬇️
... and 2 more

... and 53 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… needs, and actions/report (intel#749)

- report summary: merges cli_output.v1 documents into markdown, json
  (report_summary.v1) or annotations. Envelope keys only; missing,
  unreadable or foreign inputs stay rows (not_run / invalid). No verdict
  computed, exit 0 once written. Stdlib only, no GITHUB_* read.
- catalog check, catalog eval and skill routing take --json. A subject may
  carry findings (level, code, message, file, line); catalog check, catalog
  eval, repo validate and repo scan fill them.
- actions/report: renders out-dir into the job summary and annotations,
  optional sticky PR comment, uploads out-dir. repo-eval.yml renders repo
  validate and repo scan with the same renderer.

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi vchamarthi changed the title feat(report): report summary merges cli_output.v1 documents into one summary (#749) feat(report): one summary over cli_output.v1 documents, the leaves it needs, and actions/report (#749) Nov 17, 2026
Comment thread actions/report/action.yml Resolved
Comment thread actions/report/action.yml Resolved
Comment thread dokimi/dokimi/commands/catalog.py Resolved
Comment thread dokimi/dokimi/commands/novelty.py Resolved
Comment thread dokimi/dokimi/commands/novelty.py Resolved
Comment thread dokimi/dokimi/commands/novelty.py Resolved
Fixes the review threads on report summary and its leaves:
- an empty input directory stays a not_run row
- malformed verdict, status, subjects or cost make an invalid row, not a TypeError
- two documents of one kind keep their own subject verdicts
- the subject table header is escaped like the check table
- report_summary.v1 gets a bundled schema and a registry entry
- skill routing --json names the loaded skill, carries selection_rate and
  still prints its document when the cap or a model error stops the run
- catalog check --json refuses --jsonl, which would leave it no rows
- actions/report finds the sticky comment of the token's own user and uploads
  out-dir after a failed comment step
- the CLI interface snapshot lists the report_summary kind of runs list --kind

Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
@vchamarthi

Copy link
Copy Markdown
Contributor Author

🟢 Ready for re-review · 1 commit on feat/report-summary (2b70eeb..2137cb6)

Next: a re-review of the new head. The author: correct the PR description (below).

Changes

  • An empty results directory stays a not_run row.
  • Malformed envelope fields give an invalid row.
  • Same-kind documents keep their subject verdicts.
  • skill routing --json prints its document when the run is stopped.

**Needs **

  • PR description says "catalog check --json: ... --json with --format json is refused."; it should read: --json with --format json or --jsonl is refused.
  • PR description says "skill routing --json: report only (verdict: null), with queries_self_derived next to the recall scores and cost against --max-usd."; it should read: skill routing --json: report only (verdict: null), with queries_self_derived next to the lexical recall and the model selection rate, cost against --max-usd; a run stopped by the cap prints its document too and exits non-zero.

Threads and comments

  • 16 fixed; each answered in its thread.
1 thread left as it is, and why
  • dokimi/dokimi/report/summary.py:135 (copilot-pull-request-reviewer) – Subject annotation titles already go through _prop() on the head (render_annotations); ….

Verified on the new head: ✅ 9/9 tests fail before the fix, pass after · ✅ full suite with main merged in: 22327 passed · ✅ second reader: nothing raised

What each test shows
  • test the json summary conforms to its schema: fails before, passes after.
  • test an empty directory stays a not run row: fails before, passes after.
  • test malformed envelope fields are invalid rows not a crash: fails before, passes after.
  • test two documents of one kind keep each subject verdict: fails before, passes after.
  • test markdown subject header escapes the kind: fails before, passes after.
  • test the report action uploads after a failed comment and finds its own comment: fails before, passes after.
  • test catalog check json refuses jsonl: fails before, passes after.
  • test skill routing json names the loaded skill and carries the model rate: fails before, passes after.
  • test skill routing json of a capped run keeps its document and its exit: fails before, passes after.
  • subject annotation titles stay escaped: passes.
  • Also changed, outside this PR's files: dokimi/dokimi/artifacts.py, dokimi/dokimi/runs/kinds.py.
  • Left to GitHub's own checks: agent-quality.yml:external-catalog-smoke, ci.yml:static / pre-commit, ci.yml:static / Install gitleaks, ci.yml:static / Scan commits, ci.yml:fidelity / Install the pinned Claude Code, ci.yml:test-latest, ci.yml:package-smoke, ci.yml:windows-smoke.

Comment thread actions/report/action.yml Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
- summary: a non-string reason or error message, or a finding without
  string level, code and message, is an invalid row instead of output
  that breaks report_summary.v1.
- summary: one column key per input document, so a subject missing from
  an earlier same-kind document is no longer filed under another key.
- actions/report: a token other than the default whose login cannot be
  read is refused instead of assumed to be the Actions bot, which made
  the sticky comment duplicate on every run.

Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 2137cb6. The renderer and the new --json leaves work, and the earlier review round is fixed. Some states still read better than they are: a run stopped by the cap can leave the summary at pass, and with --changed-from the skills nobody touched show as failing. The details are in the line comments.

Agreed with @exai-bot: the same-kind column is still picked per subject; _malformed does not check reason or the finding items; the sticky lookup fails with a GitHub App token.

What I ran, at 2137cb6:

  • make ci-local --offline: 22,298 passed, 0 failed, 72 skipped; every check passed or was skipped offline.
  • Repros of summarize on hand-made cli_output.v1 documents: a budget_stopped routing run next to a passing check, two same-kind documents with different subjects, a malformed reason and finding, and a foreign JSON next to the documents in one directory.
  • catalog eval --changed-from HEAD~1 --json on a two-skill git catalog where only the skill outside the change is broken, then report summary --format annotations on its output.

Checked, no issues

  • Schema: cli_output.v1 is only widened (an optional findings on a subject); the rest of its diff is reformatting. report_summary.v1 has a schema, a fixture and a registry entry.
  • skill routing --json: a run stopped by the cap or a model error prints its document and keeps its exit code; the subject is the loaded skill's name.
  • catalog check --json: subject verdicts follow the same rule as the scan's exit, --strict included; --jsonl and --format json are refused.
  • Subject annotation titles: escaped with _prop (the open Copilot thread on this is fixed).
  • Changelog and CLI snapshot: they cover the new command, the three new --json flags and the runs list --kind choice.

Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread dokimi/dokimi/commands/catalog_eval.py Resolved
Comment thread .github/workflows/repo-eval.yml Resolved
Comment thread actions/report/action.yml Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
Comment thread actions/report/action.yml Resolved
Comment thread dokimi/dokimi/commands/catalog_eval.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved
@vchamarthi

Copy link
Copy Markdown
Contributor Author

🟢 Ready for re-review · 1 commit on feat/report-summary (2137cb6..a10e74b)

Next: a re-review of the new head.

Changes

  • summary.py: a non-string reason, a non-string error.message, or a finding without string level, code, message makes the row invalid.
  • summary.py: the column key is chosen once per input document, outside the subject loop.
  • actions/report/action.yml: a token other than github.token whose gh api user fails is refused.

Threads and comments

  • 3 fixed; each answered in its thread.
1 thread left as it is, and why
  • dokimi/dokimi/report/summary.py:135 (copilot-pull-request-reviewer) – Subject titles go through _prop on the head, pinned by ….

Verified on the new head: ✅ 4/4 tests fail before the fix, pass after · ✅ full suite with main merged in: 22335 passed · ✅ second reader: nothing raised

What each test shows
  • same-kind documents keep one key each: fails before, passes after.
  • wrong-typed reason and findings are invalid: fails before, passes after.
  • unreadable-login token refused: fails before, passes after.
  • default token still posts without a user lookup: fails before, passes after.
  • Left to GitHub's own checks: agent-quality.yml:external-catalog-smoke, ci.yml:static / pre-commit, ci.yml:static / Install gitleaks, ci.yml:static / Scan commits, ci.yml:fidelity / Install the pinned Claude Code, ci.yml:test-latest, ci.yml:package-smoke, ci.yml:windows-smoke.

- A document with status budget_stopped, partial or cancelled is that
  state in the summary, not a pass, and gets a warning annotation.
- An unpriced spend prints as unpriced and leaves the total empty;
  skill routing --json gives priced null for no calls and no remainder
  for an unpriced run.
- catalog eval --json gives a skill outside --changed-from verdict null
  and no findings, and points findings at SKILL.md under the given root.
- The Markdown lists each finding (first 100) and escapes <, > and \r.
- actions/report cuts the sticky comment below 65,536 characters and
  warns instead of failing when it cannot be posted; out-dir documents
  that its top-level *.json must be cli_output.v1.

Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
@vchamarthi

Copy link
Copy Markdown
Contributor Author

🟢 Ready for re-review · 1 commit on feat/report-summary (a10e74b..e2387d8)

Next: @vlad-perevezentsev: re-review the new head.

PR description updated

  • Markdown has a check table (state, verdict, spent / cap or unpriced, the scalar summary values, reason), a subject table (verdict and finding count per check) and a findings table (the first 100).

Threads and comments

  • 8 fixed; each answered in its thread.
  • 1 thread already settled on the head; answered with what settles it.
  • Holds on this head (comment by @vlad-perevezentsev): same-kind column still picked per subject – tests/test_report_summary.py::test_same_kind_documents_with_different_subjects_keep_one_key_each: the key is chosen once per document.
  • Holds on this head (comment by @vlad-perevezentsev): _malformed does not check reason or finding items – tests/test_report_summary.py::test_reason_and_findings_of_the_wrong_type_are_invalid_rows.
  • Holds on this head (comment by @vlad-perevezentsev): sticky lookup fails with a GitHub App token – tests/test_report_summary.py::test_sticky_comment_refuses_a_token_whose_login_is_unreadable: the step refuses it with a message instead of guessing a login.

Verified on the new head: ✅ 9/9 tests fail before the fix, pass after · ⚠️ 1 answered as fixed without a test that shows it · ✅ full suite with main merged in: 22542 passed · ✅ second reader: nothing raised

What each test shows
  • stopped status is not a pass: fails before, passes after.
  • unpriced spend: fails before, passes after.
  • routing unpriced remainder: fails before, passes after.
  • unselected skills unjudged: fails before, passes after.
  • eval finding file: fails before, passes after.
  • findings listed and escaped: fails before, passes after.
  • findings capped: fails before, passes after.
  • sticky warning: fails before, passes after.
  • sticky size: fails before, passes after.
  • Also changed, outside this PR's files: actions/setup/action.yml.
  • Left to GitHub's own checks: agent-quality.yml:external-catalog-smoke, ci.yml:static / pre-commit, ci.yml:static / Install gitleaks, ci.yml:static / Scan commits, ci.yml:fidelity / Install the pinned Claude Code, ci.yml:test-latest, ci.yml:package-smoke, ci.yml:windows-smoke.

Comment thread actions/report/action.yml Resolved
Comment thread dokimi/dokimi/commands/catalog_eval.py Resolved
Comment thread dokimi/dokimi/report/summary.py Resolved

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at e2387d8. Every point from my first round is fixed: a stopped run is no longer a pass, skills outside --changed-from are left unjudged, the job summary lists the findings again, and an unpriced spend shows as unpriced. Three small gaps remain, all in @exai-bot's new threads.

Agreed with @exai-bot: cost.priced is not validated; task findings point at a SKILL.md that does not exist; a failed sticky lookup posts a duplicate comment.

What I ran, at e2387d8:

  • make ci-local --offline: 22,312 passed, 0 failed, 72 skipped. The one FAIL row, schema_compat, compares against the current main, which added repo_event.v1.json after this PR's base; this PR changes only cli_output.v1 and report_summary.v1.
  • Repros of summarize on hand-made documents: a budget_stopped run next to a passing check (overall: budget_stopped, one ::warning, total null for an unpriced spend), the same document given twice (keys kind (src) and kind (src) #3), reason: 123 (an invalid row), and cost.priced: "false".
  • catalog eval's _emit on a report with an undeclared-task finding.

Checked, no issues

  • Sticky comment: a failed POST or PATCH is now a warning, the body stays below the 65,536-character limit, and a token whose login cannot be read is refused instead of guessed.
  • report_summary.v1: the new priced field and the three stopped states are new in this PR, so changing the schema in place is fine.
  • Changelog: it covers the stopped states, unpriced, the findings table and the --changed-from scope.

- actions/report no longer posts a second comment when the lookup of the
  existing one fails (even after printing an id); it warns and skips.
- catalog eval --json points a task finding at its task.toml, task.yaml
  or task.md instead of a nonexistent <task>/SKILL.md.
- report summary treats a cost.priced that is not a boolean or null as
  an invalid row, so the JSON stays valid against report_summary.v1.

Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
@vchamarthi

Copy link
Copy Markdown
Contributor Author

🟢 Ready for re-review · 1 commit on feat/report-summary (e2387d8..4eea587)

Next: a re-review of the new head.

Threads and comments

  • 3 fixed; each answered in its thread.

Verified on the new head: ✅ 3/3 tests fail before the fix, pass after · ✅ full suite with main merged in: 22680 passed · ✅ second reader: nothing raised

What each test shows
  • a failing lookup posts and patches nothing: fails before, passes after.
  • a task finding points at the task definition: fails before, passes after.
  • a string cost.priced is an invalid row: fails before, passes after.
  • Left to GitHub's own checks: agent-quality.yml:external-catalog-smoke, ci.yml:static / pre-commit, ci.yml:static / Install gitleaks, ci.yml:static / Scan commits, ci.yml:fidelity / Install the pinned Claude Code, ci.yml:test-latest, ci.yml:package-smoke, ci.yml:windows-smoke.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The renderer keeps every input as a row, computes no verdict, and the --json leaves, the schemas and actions/report match the description. The earlier review rounds are fixed at this head. One nit inline.

What I ran, on a Linux review host at 4eea587:

  • make ci-local: 34 passed, 0 failed, 14 skipped (offline or Docker-only jobs); 22,369 tests; changed-line coverage 95%.
  • A reproduction of the repo scan job summary with the LLM transport mocked and no network: a scan payload with one warnings entry renders a summary without that warning.
  • All five commits carry a Signed-off-by.

Checked, no issues

  • Schemas: report_summary.v1 has a schema, a fixture and a registry entry; cli_output.v1 is only widened with an optional findings.
  • Annotations: titles and files go through _prop, messages through _escape; test_subject_annotation_title_escapes_its_delimiters pins it.
  • Sticky comment: a failed lookup, POST or PATCH is a warning and the body stays under the 65,536-character limit.
  • Docs and CLI surface: the changelog fragment and cli-interface.json cover the new command and the three --json flags.
Bot points I checked that no longer apply at this head (1)
  • Copilot's point on unescaped subject annotation titles does not apply: render_annotations passes them through _prop (summary.py:243), and tests/test_report_summary.py:289 pins it.

Comment thread dokimi/dokimi/commands/repo_scan.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I pushed a commit per nit, up to fea499c, which fix the nit from my review:

  • fea499c: dokimi/dokimi/commands/repo_scan.py:171: scan_subjects adds each payload warning as a scan-warning finding on the first subject.

make ci-local at fea499c: 34 passed, 0 failed, 14 skipped.

@antonwolfy
antonwolfy merged commit 7125813 into intel:main Nov 17, 2026
42 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI summaries are written by hand in each consumer: one report renderer over cli_output.v1, and actions/report

6 participants