Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 13 additions & 41 deletions .github/workflows/repo-eval.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,22 +166,9 @@ jobs:
status=0
dokimi repo validate . --manifest "$MANIFEST" || status=$?
dokimi repo validate . --manifest "$MANIFEST" --json > "$RUNNER_TEMP/validate.json" || true
python - <<'PY' >> "$GITHUB_STEP_SUMMARY"
import json, os
path = os.path.join(os.environ["RUNNER_TEMP"], "validate.json")
try:
d = json.load(open(path, encoding="utf-8"))["data"] # cli_output.v1: the report under data
d["tasks"] # a run that failed has data null
except (OSError, ValueError, KeyError, TypeError):
print("### repo validate", "", "The manifest could not be read; see the job log.", sep="\n")
raise SystemExit
tasks = d["tasks"]
print("### repo validate", "", f"{'ok' if d['ok'] else 'FAILED'}: `{d['manifest']['manifest']}`, "
f"{tasks['count']} task(s), {len(d['guidance_files'])} guidance file(s)", sep="\n")
for kind in ("errors", "warnings", "notes"):
for line in d[kind]:
print(f"- {kind[:-1]}: {line}")
PY
# One renderer over the cli_output.v1 document (#749): the job summary and an annotation per finding.
dokimi report summary "$RUNNER_TEMP/validate.json" >> "$GITHUB_STEP_SUMMARY"
dokimi report summary "$RUNNER_TEMP/validate.json" --format annotations
exit "$status"
- name: Select tasks
id: select
Expand Down Expand Up @@ -335,36 +322,21 @@ jobs:
env:
MANIFEST: ${{ inputs.manifest }}
run: |
dokimi repo scan . --manifest "$MANIFEST" --out "$RUNNER_TEMP/repo-readiness.json"
python - <<'PY' >> "$GITHUB_STEP_SUMMARY"
import json, os
d = json.load(open(os.path.join(os.environ["RUNNER_TEMP"], "repo-readiness.json"), encoding="utf-8"))
c = d["counts"]

def cell(text):
return str(text).replace("|", "\\|").replace("\n", " ")

print("### repo scan (L0, report only)", "",
f"{len(d['files'])} guidance file(s), adapters: {', '.join(d['adapters']) or 'none'}", "",
"| claims | ok | unresolved | fail |", "|---|---|---|---|",
f"| {c['total']} | {c['ok']} | {c['unresolved']} | {c['fail']} |", sep="\n")
for w in d["warnings"]:
print(f"- warning: {cell(w)}")
fails = [x for x in d["claims"] if x["status"].split("_", 1)[0] == "fail"]
if fails:
print("", "| file | line | kind | claim | why |", "|---|---|---|---|---|", sep="\n")
for x in fails[:100]:
print(f"| {cell(x['file'])} | {x['line']} | {cell(x.get('kind'))} | "
f"`{cell(str(x.get('claim'))[:80])}` | {cell(x.get('why'))} |")
if len(fails) > 100:
print(f"\n{len(fails) - 100} more in the repo-readiness.json artifact")
PY
status=0
dokimi repo scan . --manifest "$MANIFEST" --out "$RUNNER_TEMP/repo-readiness.json" \
--json > "$RUNNER_TEMP/repo-scan.json" || status=$?
# The claim counts, and each failing claim at its file and line, from the one renderer (#749).
dokimi report summary "$RUNNER_TEMP/repo-scan.json" >> "$GITHUB_STEP_SUMMARY"
dokimi report summary "$RUNNER_TEMP/repo-scan.json" --format annotations
Comment thread
vchamarthi marked this conversation as resolved.
Resolved
exit "$status"
- name: Upload the scan report
if: ${{ !cancelled() }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: repo-eval-scan-${{ needs.validate.outputs.id }}
path: ${{ runner.temp }}/repo-readiness.json
path: |
${{ runner.temp }}/repo-readiness.json
${{ runner.temp }}/repo-scan.json
if-no-files-found: ignore

tasks-check:
Expand Down
99 changes: 99 additions & 0 deletions actions/report/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

name: dokimi report
description: >-
Places the summary of every cli_output.v1 document in out-dir: runs
`dokimi report summary`, appends its Markdown to the job summary, prints its
annotations, optionally keeps one sticky PR comment, and uploads out-dir. No
parsing and no verdict in YAML: the gate stays the exit code of each dokimi
step. Needs `contents: read`; `pull-requests: write` only with comment: sticky.
Run it after actions/setup, with `if: always()`.

inputs:
out-dir:
description: >-
Directory holding the cli_output.v1 documents (`*.json`, not recursive), usually
the out-dir output of actions/setup. Every top-level `*.json` there must be a
cli_output.v1 document, else it is an `invalid` row; other files (a receipt, a
`--out` result) go in a subdirectory. The rendered summary goes to its `summary/`
subdirectory.
required: true
comment:
description: "false, or sticky: create or update one comment on the pull request"
required: false
default: 'false'
github-token:
description: Token for the sticky comment (needs pull-requests write)
required: false
default: ${{ github.token }}
artifact-name:
description: Name of the uploaded out-dir artifact; empty skips the upload
required: false
default: dokimi-report

runs:
using: composite
steps:
- name: Render the summary
shell: bash
env:
OUT: ${{ inputs.out-dir }}
run: |
if ! command -v dokimi >/dev/null; then
echo "::error title=dokimi report::dokimi is not on PATH; run intel/dokimi/actions/setup first"
exit 1
fi
mkdir -p -- "$OUT/summary"
dokimi report summary --format markdown --out "$OUT/summary/summary.md" -- "$OUT"
dokimi report summary --format json --out "$OUT/summary/summary.json" -- "$OUT"
dokimi report summary --format annotations -- "$OUT"
Comment thread
vchamarthi marked this conversation as resolved.
Resolved
cat -- "$OUT/summary/summary.md" >> "$GITHUB_STEP_SUMMARY"
- name: Sticky pull request comment
if: ${{ inputs.comment == 'sticky' && github.event.pull_request.number }}
shell: bash
env:
OUT: ${{ inputs.out-dir }}
GH_TOKEN: ${{ inputs.github-token }}
DEFAULT_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
REPO: ${{ github.repository }}
run: |
marker='<!-- dokimi-report -->'
# A comment body is capped at 65,536 characters: cut the summary below it and point to the job summary.
{ echo "$marker"; head -c 60000 -- "$OUT/summary/summary.md"
if [ "$(wc -c < "$OUT/summary/summary.md")" -gt 60000 ]; then
printf '\n\n_Summary cut here; the full one is in the job summary._\n'
fi; } > "$OUT/summary/comment.md"
# The comment is the token's own: a user token posts as that user, the default token as the bot.
# Any other token whose login cannot be read (an app installation token) is refused, not guessed.
if [ "$GH_TOKEN" = "$DEFAULT_TOKEN" ]; then
me='github-actions[bot]'
else
me=$(gh api user --jq .login 2>/dev/null) || {
echo "::error title=dokimi report::github-token has no readable user login (an app installation token?); use the default token or a user token"
exit 1
}
fi
# A failed lookup is not proof that no comment exists: skip publishing instead of posting a duplicate.
ids=$(gh api --paginate "repos/$REPO/issues/$PR/comments" \
--jq ".[] | select(.user.login == \"$me\" and (.body | startswith(\"$marker\"))) | .id") || {
echo "::warning title=dokimi report::the sticky comment lookup failed; the summary is in the job summary"
exit 0
}
id=$(printf '%s\n' "$ids" | head -n1)
# A token that cannot write (a pull request from a fork) must not turn a green job red.
if [ -n "$id" ]; then
gh api -X PATCH "repos/$REPO/issues/comments/$id" -F "body=@$OUT/summary/comment.md" >/dev/null ||
echo "::warning title=dokimi report::the sticky comment could not be updated; the summary is in the job summary"
else
gh api -X POST "repos/$REPO/issues/$PR/comments" -F "body=@$OUT/summary/comment.md" >/dev/null ||
echo "::warning title=dokimi report::the sticky comment could not be posted; the summary is in the job summary"
fi
- name: Upload out-dir
if: ${{ always() && inputs.artifact-name != '' }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: ${{ inputs.artifact-name }}
path: ${{ inputs.out-dir }}
if-no-files-found: ignore
3 changes: 2 additions & 1 deletion actions/setup/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ inputs:
description: >-
Directory for the run output, created by this action; empty means
`$RUNNER_TEMP/dokimi-out`. The caller writes its output under it, and uploads the
whole directory as an artifact.
whole directory as an artifact. When actions/report summarizes it, its top-level
`*.json` files must be cli_output.v1 documents; other files go in subdirectories.
required: false
default: ''

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
### Added

- **`dokimi report summary` merges `cli_output.v1` documents into one summary** (#749, the renderer). It takes files or directories of `*.json` and writes `--format markdown` (a check table and a subject table, usable as a job summary, a PR comment or terminal output), `json` (`report_summary.v1`, with a bundled schema) or `annotations` (`::error`/`::warning` lines). It reads envelope keys only, so a new leaf needs no change in it. A missing, unreadable, malformed (including a `cost.priced` that is not a boolean) or non-`cli_output.v1` input, and a directory with no `*.json` file, stays a row (`not_run` or `invalid`) and makes the overall state bad; it is never dropped or shown as a pass. A document whose status is `budget_stopped`, `partial` or `cancelled` is that state, not a pass, and gets a `::warning`; an unpriced spend shows as `unpriced` and leaves the total empty; the Markdown lists each finding (file:line, code, message, the first 100). It computes no verdict and exits 0 once written: the gate stays each leaf's own exit code. Stdlib only, and it reads no `GITHUB_*` variable.
- **`catalog check`, `catalog eval` and `skill routing` take `--json`** (#749, #94 F2): one `cli_output.v1` document on stdout, with one subject per skill (and per task for `catalog check`). Exit codes are unchanged. `catalog eval --json` leaves a skill outside `--changed-from` unjudged (`verdict: null`, no findings) and points a finding at its `SKILL.md` (a task finding at its `task.toml`, `task.yaml` or `task.md`) under the given root. `skill routing --json` is report only (`verdict: null`) and carries `queries_self_derived` next to its scores; its human summary goes to stderr. `skill routing --json` still prints its document when `--max-usd` or a model error stops the run (status `budget_stopped` or `partial`) and then exits non-zero as before; its subject is the loaded skill's name. `catalog check --json` refuses `--format json` and `--jsonl`.
- **A `cli_output.v1` subject may carry `findings`** (`level`, `code`, `message`, `file`, `line`). `catalog check`, `catalog eval`, `repo validate` and `repo scan` fill them; `repo scan --json` puts each failing claim at its file and line and adds the claim counts to `summary`. `report summary --format annotations` gives each finding a `file=`/`line=` annotation, and the Markdown shows the scalar `summary` values of each document.
- **`actions/report`** places the summary: it runs `dokimi report summary` over `out-dir`, appends the Markdown to the job summary, prints the annotations, keeps one sticky PR comment with `comment: sticky`, cut below the comment limit, only a warning when it cannot be posted, and no comment at all when the lookup of the existing one fails, and uploads `out-dir`. No parsing and no verdict in YAML. `repo-eval.yml` renders `repo validate` and `repo scan` with the same renderer instead of two inline Python blocks.
Loading
Loading