Repository navigation
feat(report): one summary over cli_output.v1 documents, the leaves it needs, and actions/report (#749) #850
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
2b70eeb
feat(report): one summary over cli_output.v1 documents, the leaves it…
vchamarthi 2137cb6
fix(report): keep every input, validate envelopes, add summary schema
vchamarthi a10e74b
fix(report): validate findings, key columns once, refuse app tokens
vchamarthi e2387d8
fix(report): stopped runs, unpriced spend, eval scope, comment limits
vchamarthi 4eea587
fix(report): sticky lookup failure, task finding paths, cost.priced type
vchamarthi fea499c
Show repo scan warnings as findings in the report summary
antonwolfy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| name: dokimi report | ||
| description: >- | ||
| Places the summary of every cli_output.v1 document in out-dir: runs | ||
| `dokimi report summary`, appends its Markdown to the job summary, prints its | ||
| annotations, optionally keeps one sticky PR comment, and uploads out-dir. No | ||
| parsing and no verdict in YAML: the gate stays the exit code of each dokimi | ||
| step. Needs `contents: read`; `pull-requests: write` only with comment: sticky. | ||
| Run it after actions/setup, with `if: always()`. | ||
|
|
||
| inputs: | ||
| out-dir: | ||
| description: >- | ||
| Directory holding the cli_output.v1 documents (`*.json`, not recursive), usually | ||
| the out-dir output of actions/setup. Every top-level `*.json` there must be a | ||
| cli_output.v1 document, else it is an `invalid` row; other files (a receipt, a | ||
| `--out` result) go in a subdirectory. The rendered summary goes to its `summary/` | ||
| subdirectory. | ||
| required: true | ||
| comment: | ||
| description: "false, or sticky: create or update one comment on the pull request" | ||
| required: false | ||
| default: 'false' | ||
| github-token: | ||
| description: Token for the sticky comment (needs pull-requests write) | ||
| required: false | ||
| default: ${{ github.token }} | ||
| artifact-name: | ||
| description: Name of the uploaded out-dir artifact; empty skips the upload | ||
| required: false | ||
| default: dokimi-report | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - name: Render the summary | ||
| shell: bash | ||
| env: | ||
| OUT: ${{ inputs.out-dir }} | ||
| run: | | ||
| if ! command -v dokimi >/dev/null; then | ||
| echo "::error title=dokimi report::dokimi is not on PATH; run intel/dokimi/actions/setup first" | ||
| exit 1 | ||
| fi | ||
| mkdir -p -- "$OUT/summary" | ||
| dokimi report summary --format markdown --out "$OUT/summary/summary.md" -- "$OUT" | ||
| dokimi report summary --format json --out "$OUT/summary/summary.json" -- "$OUT" | ||
| dokimi report summary --format annotations -- "$OUT" | ||
|
vchamarthi marked this conversation as resolved.
Resolved
|
||
| cat -- "$OUT/summary/summary.md" >> "$GITHUB_STEP_SUMMARY" | ||
| - name: Sticky pull request comment | ||
| if: ${{ inputs.comment == 'sticky' && github.event.pull_request.number }} | ||
| shell: bash | ||
| env: | ||
| OUT: ${{ inputs.out-dir }} | ||
| GH_TOKEN: ${{ inputs.github-token }} | ||
| DEFAULT_TOKEN: ${{ github.token }} | ||
| PR: ${{ github.event.pull_request.number }} | ||
| REPO: ${{ github.repository }} | ||
| run: | | ||
| marker='<!-- dokimi-report -->' | ||
| # A comment body is capped at 65,536 characters: cut the summary below it and point to the job summary. | ||
| { echo "$marker"; head -c 60000 -- "$OUT/summary/summary.md" | ||
| if [ "$(wc -c < "$OUT/summary/summary.md")" -gt 60000 ]; then | ||
| printf '\n\n_Summary cut here; the full one is in the job summary._\n' | ||
| fi; } > "$OUT/summary/comment.md" | ||
| # The comment is the token's own: a user token posts as that user, the default token as the bot. | ||
| # Any other token whose login cannot be read (an app installation token) is refused, not guessed. | ||
| if [ "$GH_TOKEN" = "$DEFAULT_TOKEN" ]; then | ||
| me='github-actions[bot]' | ||
| else | ||
| me=$(gh api user --jq .login 2>/dev/null) || { | ||
| echo "::error title=dokimi report::github-token has no readable user login (an app installation token?); use the default token or a user token" | ||
| exit 1 | ||
| } | ||
| fi | ||
| # A failed lookup is not proof that no comment exists: skip publishing instead of posting a duplicate. | ||
| ids=$(gh api --paginate "repos/$REPO/issues/$PR/comments" \ | ||
| --jq ".[] | select(.user.login == \"$me\" and (.body | startswith(\"$marker\"))) | .id") || { | ||
| echo "::warning title=dokimi report::the sticky comment lookup failed; the summary is in the job summary" | ||
| exit 0 | ||
| } | ||
| id=$(printf '%s\n' "$ids" | head -n1) | ||
| # A token that cannot write (a pull request from a fork) must not turn a green job red. | ||
| if [ -n "$id" ]; then | ||
| gh api -X PATCH "repos/$REPO/issues/comments/$id" -F "body=@$OUT/summary/comment.md" >/dev/null || | ||
| echo "::warning title=dokimi report::the sticky comment could not be updated; the summary is in the job summary" | ||
| else | ||
| gh api -X POST "repos/$REPO/issues/$PR/comments" -F "body=@$OUT/summary/comment.md" >/dev/null || | ||
| echo "::warning title=dokimi report::the sticky comment could not be posted; the summary is in the job summary" | ||
| fi | ||
| - name: Upload out-dir | ||
| if: ${{ always() && inputs.artifact-name != '' }} | ||
| uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 | ||
| with: | ||
| name: ${{ inputs.artifact-name }} | ||
| path: ${{ inputs.out-dir }} | ||
| if-no-files-found: ignore | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
6 changes: 6 additions & 0 deletions
6
dokimi/changelog.d/20261116_120000_vchamarthi_report-summary.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| ### Added | ||
|
|
||
| - **`dokimi report summary` merges `cli_output.v1` documents into one summary** (#749, the renderer). It takes files or directories of `*.json` and writes `--format markdown` (a check table and a subject table, usable as a job summary, a PR comment or terminal output), `json` (`report_summary.v1`, with a bundled schema) or `annotations` (`::error`/`::warning` lines). It reads envelope keys only, so a new leaf needs no change in it. A missing, unreadable, malformed (including a `cost.priced` that is not a boolean) or non-`cli_output.v1` input, and a directory with no `*.json` file, stays a row (`not_run` or `invalid`) and makes the overall state bad; it is never dropped or shown as a pass. A document whose status is `budget_stopped`, `partial` or `cancelled` is that state, not a pass, and gets a `::warning`; an unpriced spend shows as `unpriced` and leaves the total empty; the Markdown lists each finding (file:line, code, message, the first 100). It computes no verdict and exits 0 once written: the gate stays each leaf's own exit code. Stdlib only, and it reads no `GITHUB_*` variable. | ||
| - **`catalog check`, `catalog eval` and `skill routing` take `--json`** (#749, #94 F2): one `cli_output.v1` document on stdout, with one subject per skill (and per task for `catalog check`). Exit codes are unchanged. `catalog eval --json` leaves a skill outside `--changed-from` unjudged (`verdict: null`, no findings) and points a finding at its `SKILL.md` (a task finding at its `task.toml`, `task.yaml` or `task.md`) under the given root. `skill routing --json` is report only (`verdict: null`) and carries `queries_self_derived` next to its scores; its human summary goes to stderr. `skill routing --json` still prints its document when `--max-usd` or a model error stops the run (status `budget_stopped` or `partial`) and then exits non-zero as before; its subject is the loaded skill's name. `catalog check --json` refuses `--format json` and `--jsonl`. | ||
| - **A `cli_output.v1` subject may carry `findings`** (`level`, `code`, `message`, `file`, `line`). `catalog check`, `catalog eval`, `repo validate` and `repo scan` fill them; `repo scan --json` puts each failing claim at its file and line and adds the claim counts to `summary`. `report summary --format annotations` gives each finding a `file=`/`line=` annotation, and the Markdown shows the scalar `summary` values of each document. | ||
| - **`actions/report`** places the summary: it runs `dokimi report summary` over `out-dir`, appends the Markdown to the job summary, prints the annotations, keeps one sticky PR comment with `comment: sticky`, cut below the comment limit, only a warning when it cannot be posted, and no comment at all when the lookup of the existing one fails, and uploads `out-dir`. No parsing and no verdict in YAML. `repo-eval.yml` renders `repo validate` and `repo scan` with the same renderer instead of two inline Python blocks. |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.