Skip to content

Compare benchmarks against the base branch in pull requests - #40

Open
pavog wants to merge 1 commit into
masterfrom
compare-benchmarks-in-pull-requests
Open

pavog wants to merge 1 commit into
masterfrom
compare-benchmarks-in-pull-requests

Conversation

@pavog

@pavog pavog commented Nov 13, 2026

Copy link
Copy Markdown
Member

CI only checked that the benchmarks still run, so slower pull requests went unnoticed.

This adds a workflow that runs the benchmarks for pull requests twice on the same runner, once with the library code of the base branch and once with the code of the pull request, and writes the comparison to the job summary.
Both runs use the benchmarks of the pull request, so new or changed benchmarks are compared too.

Decisions:

  • Same job for both runs, because shared runners differ too much to compare against stored results.
  • Job summary instead of a pull request comment, because comments need write access that fork pull requests do not get safely.
  • No failure on regressions yet, because differences below about 10% are mostly noise.

Not covered: a benchmark for a class that only exists in the pull request fails on the base branch run.

Changes

  • add a workflow that compares benchmarks against the base branch in pull requests
  • remove the benchmark smoke step from the test workflow
  • document the comparison in the readme

@pavog
pavog requested a review from matthi4s November 13, 2026 12:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant