Skip to content

Comment on PRs with their coverage against main - #7849

Draft
denik wants to merge 2 commits into
mainfrom
denik/coverage-comment
Draft

denik wants to merge 2 commits into
mainfrom
denik/coverage-comment

Conversation

@denik

@denik denik commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Changes

A new workflow, coverage-comment.yml, runs after the build workflow of a PR. It acts only when that build uploaded coverage: on Linux, for a PR whose head commit message has with-coverage. It then posts one comment on the PR, and updates the same comment on later builds. The comment shows:

  • the coverage of the PR's changed lines: combined, acceptance and unit;
  • the totals, and their change against the newest main build that the PR's change starts from;
  • the changed lines that no test ran, per file;
  • the packages that the PR changes and whose coverage changed. When the base build is behind the PR's starting point, the comment says by how many commits.

coverpack -compare computes it from the packed files of the two builds and from git:

  • The lines are those of GitHub's merge commit that the build tested. The PR's change is that commit against its first parent.
  • A changed file is left out when the build measured another version of it, checked by its git blob sha.
  • The baseline is the newest completed main run whose commit is an ancestor of that first parent and still has coverage artifacts. Without one, the comment shows the PR's numbers alone.

Security

  • The workflow is a workflow_run, so it runs the default branch's code and never checks out the PR.
  • It reads the PR build's artifacts only as data:
    • artifacts are listed through the API: at most 8, plain coverage-linux-<cell> names, each under 16 MB;
    • decompression is capped at 64 MB and the decoder at 2M blocks;
    • the decoder rejects blocks past the end of their file, and bitmap or profile counts that the remaining bytes can't hold;
    • only the changed lines of a block are walked, and the job times out after 15 minutes;
    • a failed listing or download fails the job instead of producing a partial comparison;
    • the base must have the same cells as the PR's build;
    • the merge commit named in the artifact must have the run's head sha as its second parent, and its first parent must be on main;
    • every path printed in the markdown is reduced to [A-Za-z0-9_./+=-] and put in backticks, and cell names must be plain lowercase words.
  • The PR is looked up by the run's head owner and branch, and must still have the run's head sha.
  • Permissions: actions: read, contents: read, pull-requests: write, and id-token: write for JFrog.

Fork PRs get the comment too, because the token here is the base repository's.

Tests

  • Unit tests in tools/coverpack/compare_test.go cover:
    • changed lines, totals and per-package rows against a hand-made git repo;
    • the no-base case;
    • a stale file version;
    • diff parsing and sanitizing.
  • A local run of -compare on the artifacts of runs 44739969429 and 44746083519:
    • totals 79.3% / 54.9% / 71.9% (combined / acceptance / unit), matching the CI dashboard;
    • the uncovered lines it reports in bundle/configsync/permissions.go match -dump;
    • it takes 0.1 s.
  • A dry run of .github/scripts/coverage-comment.sh (DRY_RUN=1) against run 44739969429, both without a base and with one. Its first job, looking up the open PR, was bypassed because that PR is merged.
  • workflow_run workflows only run from the default branch, so the first real comment comes after this merges: on the next PR with with-coverage.

This pull request and its description were written by Isaac.

A new workflow runs after the build of a PR and, when the build uploaded coverage (Linux, a head
commit message with "with-coverage"), posts or updates one comment: the coverage of the PR's
changed lines, the totals against the newest main build its change starts from, the changed
lines that no test ran, and the packages whose coverage changed.

`coverpack -compare` computes it from the packed files of the two builds and git, reading the
PR's files as untrusted data. The workflow runs from the default branch, so fork PRs get the
comment too.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
From a review:
- decode rejects blocks past the end of their file, bitmaps and profiles that the bytes left
  cannot hold, and checks the block count before allocating; compare walks only the changed
  lines of a block, and the job has a 15 minute timeout;
- the script lists the artifacts through the API (plain names, under 16 MB, at most 8), fails
  when a listing or download fails, and takes as the base only a build with the same cells;
- cell names are plain words in backticks; the diff is read hunk by hunk, so a changed line
  that looks like a header is not taken for one;
- the package table lists only packages the PR changes, and the comment says how many commits
  of main the base build is behind.

with-coverage

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 2b35cb2

Run: 44773568777

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 19 6:48
💚​ aws-windows-latest-4core-16gb 1 274 17 4:57
💚​ azure linux-2core-8gb 1 271 19 6:06
💚​ azure-windows-latest-4core-16gb 1 273 17 5:20
💚​ gcp linux-2core-8gb 1 272 19 6:18
💚​ gcp-windows-latest-4core-16gb 1 274 17 6:02
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:08 azure-windows-latest-4core-16gb TestAccept
4:00 aws-windows-latest-4core-16gb TestAccept
3:55 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants