Skip to content

Add .codecov.yml with threshold-based coverage status checks#529

Open
khatchad wants to merge 1 commit into
wala:masterfrom
ponder-lab:add-codecov-config-upstream
Open

Add .codecov.yml with threshold-based coverage status checks#529
khatchad wants to merge 1 commit into
wala:masterfrom
ponder-lab:add-codecov-config-upstream

Conversation

@khatchad
Copy link
Copy Markdown
Collaborator

Summary

Adds a minimal .codecov.yml so Codecov's project/patch status checks tolerate small base-vs-head coverage drift instead of failing on every fractional drop. Mirrors the configuration in ponder-lab/ML.

Why

Without an explicit .codecov.yml, Codecov uses default target: auto with zero tolerance: any project-coverage decrease at all triggers codecov/project: FAILURE. Base-vs-head test-environment drift or flakiness can shave 1-2 lines off the project total and fail the gate even on PRs that don't touch Java code.

Concrete case: #516 (the POM cleanup mirror of ponder-lab/ML#272) has codecov/patch: SUCCESS (no Java changed, so the patch is trivially 100% covered) but codecov/project: FAILURE from a -0.03% project delta (-2 hits / +2 misses) that can't be attributed to the diff. The default Codecov policy fails on any non-zero negative delta; this configuration absorbs the noise.

What This Changes

coverage:
  status:
    project:
      default:
        target: auto
        threshold: 1%
        informational: false
    patch:
      default:
        target: auto
        threshold: 2%
        informational: false
github_checks:
  annotations: true
  • project threshold 1%—tolerates project-coverage drops of up to 1% before failing. Real regressions still fail.
  • patch threshold 2%—same idea for changed-line coverage.
  • github_checks.annotations: true—surfaces uncovered changed lines as inline annotations in the PR's "Files changed" view.

informational: false keeps the checks as real build signal rather than unconditionally green—the thresholds are the absorptive layer. The per-PR Codecov comment remains enabled by default (Codecov's default behavior with comment unspecified).

History Of The Equivalent ponder-lab/ML Config

For reference, the four commits that produced this state on the fork:

  • f7f610ff (#243): added .codecov.yml with informational: true (unconditionally green).
  • fd19bb91 (#263): disabled per-PR comment, enabled inline annotations.
  • ee247f0b (#264): flipped informational: false with the 1%/2% thresholds—"Now that there's no comment, we need to see the build failure."
  • 3eac854a (#266): re-enabled the per-PR comment.

This PR ports the current combined state in one commit.

Test Plan

🤖 Generated with Claude Code

Without an explicit `.codecov.yml`, Codecov uses default `target: auto` with zero tolerance: any project-coverage decrease at all triggers `codecov/project: FAILURE`. Base-vs-head test-environment drift or flakiness can shave 1-2 lines off the project total and fail the gate even on PRs that don't touch Java code (e.g., POM-only fixes—see PR wala#516 where a -0.03% project delta fails the gate despite the patch having no Java changes to cover).

Add a minimal `.codecov.yml` mirroring the configuration in `ponder-lab/ML`'s current state (commits `f7f610ff` / `fd19bb91` / `ee247f0b` / `3eac854a`):

* `coverage.status.project.default.threshold: 1%`—tolerate up to 1% project-coverage decrease before failing. Real regressions still fail; noise doesn't.
* `coverage.status.patch.default.threshold: 2%`—tolerate up to 2% patch-coverage decrease.
* `github_checks.annotations: true`—surface uncovered changed lines as inline diff annotations in the PR's "Files changed" view (alongside the per-PR Codecov comment, which remains enabled by default).

`informational: false` keeps the checks as real build signal rather than unconditionally-green—the thresholds are the absorptive layer.
Copilot AI review requested due to automatic review settings May 12, 2026 16:56
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a repository-level Codecov configuration to make codecov/project and codecov/patch status checks tolerant to small coverage fluctuations, reducing CI noise while keeping coverage regressions as real failures.

Changes:

  • Add .codecov.yml configuring project and patch coverage statuses to use target: auto with threshold tolerances (1% / 2%).
  • Enable github_checks.annotations so uncovered changed lines appear as inline PR annotations.

@khatchad khatchad enabled auto-merge (squash) May 12, 2026 16:58
@codecov
Copy link
Copy Markdown

codecov Bot commented May 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.90%. Comparing base (ad96543) to head (ab803d8).

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #529      +/-   ##
============================================
- Coverage     57.93%   57.90%   -0.03%     
+ Complexity      625      624       -1     
============================================
  Files           111      111              
  Lines          7671     7671              
  Branches        856      856              
============================================
- Hits           4444     4442       -2     
- Misses         3049     3051       +2     
  Partials        178      178              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants