Skip to content

fix(rpm): a rebuild only matched a release from the same checkout path - #156

Merged
jamesconwaydev merged 1 commit into
mainfrom
fix/rpm-reproducible-anywhere
Nov 15, 2026
Merged

jamesconwaydev merged 1 commit into
mainfrom
fix/rpm-reproducible-anywhere

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Owner

I rebuilt v0.2.10 to check that it matches the release byte for byte. It didn't, for three reasons:

  1. The checkout path ends up in the packages. rpmbuild records its _topdir in the SRPM and RPM headers, so only a clone at /__w/sentry-v2/sentry-v2, the release runner's path, could match.
  2. Host Python gzipped the tarball. Without rpmbuild on the host, source-tarball.py ran on macOS. The tar was identical, but macOS zlib compresses differently from EL10's.
  3. The bind mount ate a chmod. dist-info/INSTALLER came out 0600 instead of 0644 when the build tree sat on a bind mount.

The fix: rpmbuild always builds in /var/tmp/sentry-v2-rpmbuild and copies the results to rpmbuild/, so the release workflow and itest.sh see the same layout. On a host without rpmbuild, the sources go into the container over stdin, and the whole build, tarball included, runs there. Not mounting the checkout also stops :Z from relabelling it on an SELinux host.

Checked on the v0.2.10 tree with this Makefile: the Mac container route and a native EL build from a different directory give identical SRPM and RPMs. Compared with the release, every file inside matches (identical rpm2cpio payloads). The headers differ only by architecture, because I'm on aarch64 and a noarch package still records the build arch and compiler flags. docs/installation.md now says so.

I couldn't run an x86_64 build. The test host is shared, and building there outside sentry-test.slice was blocked. The 0.2.11 release run will be the first x86_64 build with this change.

Rebuilding v0.2.10 to compare with the release turned up three leaks:

- rpmbuild records its _topdir in the SRPM and RPM headers, so a clone
  anywhere but /__w/sentry-v2/sentry-v2 gave different bytes.
- Without rpmbuild on the host, scripts/source-tarball.py ran on the
  host's Python. macOS zlib compresses differently from EL10's, so the
  tar was identical but the .tar.gz was not.
- The container built on a bind-mounted tree, where pip's chmod of
  dist-info/INSTALLER didn't take and it came out 0600.

rpmbuild now always works in /var/tmp/sentry-v2-rpmbuild and copies the
results to rpmbuild/. Without rpmbuild, the sources are piped into the
container and the whole build runs there; piping instead of mounting
also avoids relabelling the checkout on an SELinux host.

On v0.2.10, the Mac/container route and a native build from another
directory now give the same SRPM and RPMs. Every file inside matches
the release; the headers differ only by build architecture, which a
noarch package still records.

Signed-off-by: James Conway <148910373+jamesconwaydev@users.noreply.github.com>

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read 8c76c9b and checked it against the bug it fixes.

It really fixes it. I cloned the repo into two different paths (a/checkout-one and b/somewhere/else/checkout-two) and ran make rpm in each through the container route (no rpmbuild on my host), first on main, then on this branch:

main,  clone A vs clone B:   DIFF  sentry-v2, sentry-v2-cockpit, sentry-v2-selinux RPMs, and the SRPM
#156,  clone A vs clone B:   same  all four, byte for byte

So on main the checkout path makes every package differ, and with this change it doesn't. All four builds exited 0. That's on x86_64, which also covers the arch you couldn't build on, though for path independence, not against the release itself.

Against the checklist:

  1. Worth having: yes, a rebuild that can match a release is the point of the reproducibility section.
  2. Tested: no automated test, but the two-path comparison above fails on main and passes here. If you want it kept, a CI job that runs make rpm from two paths and cmps the results would hold it.
  3. Safe: see the one thing below. Nothing touches a host's config, and the build no longer bind-mounts the checkout, so :Z stops relabelling it.
  4. Small: yes. rpm-here is shared by both routes rather than duplicated.
  5. Words match: the CHANGELOG and docs/installation.md say what changed, including the aarch64 header caveat.

One thing for a shared build host, not blocking. /var/tmp/sentry-v2-rpmbuild is one fixed path, and rpm-here starts with rm -rf on it and leaves it behind afterwards. Two builds at once on one machine, such as two CI jobs on a self-hosted runner, would wipe each other's tree. And a tree left by one user, mode 0700 in sticky /var/tmp, stops the next user's rm -rf until it's cleaned. The path has to stay fixed for reproducibility, so the fix would be an flock around rpm-here, and removing the tree once the results are copied out. Since mkdir (not -p) fails if something appears in between, a race only ever makes the build fail, never write somewhere else.

Approving.

@jamesconwaydev
jamesconwaydev merged commit 29f5660 into main Nov 15, 2026
6 checks passed
@jamesconwaydev
jamesconwaydev deleted the fix/rpm-reproducible-anywhere branch November 15, 2026 20:09
jamesconwaydev added a commit that referenced this pull request Nov 15, 2026
Follow-up to your note on #156. `rpm-here` now removes
`/var/tmp/sentry-v2-rpmbuild` once the results are copied to
`rpmbuild/`, so one user's finished build doesn't block the next user's.
A failed build still leaves the tree, for debugging.

I left out the `flock`. Two `make rpm` runs at once on one machine isn't
something we do: CI and the release run in a fresh container each time.
And as you said, a race only makes a build fail, never write somewhere
else.

Checked: `make rpm` through the container route exits 0, with the
packages in `rpmbuild/` and the tree gone.

Signed-off-by: James Conway <148910373+jamesconwaydev@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants