Repository navigation
Keep the parts of the old compaction work that still help - #82
Open
venkat1701 wants to merge 3 commits into
Open
venkat1701 wants to merge 3 commits into
venkat1701 wants to merge 3 commits into
Conversation
Each inline posting list went through a stream, a list that allows nulls and then a second copy in Postings.Inline. It is now built once with List.of, which Inline keeps as it is.
Codecs that hold tree references only had mapRefs, which builds a new record even when every reference maps to itself. The reference counter walks every leaf value of every commit's changed nodes this way, and so does the check that sets a leaf's flag for page references. mapRefs now returns the same record when nothing changed, and a new forEachRef visits the references without building anything; the counter and the flag check use it.
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#73 to #76 sped up the old compaction, which walked the trees and rewrote the paths to moved pages. #81 replaced that with moving pages through the page directory and counting references, so most of that stack no longer applies and it is being closed. This PR keeps the three pieces that still help on the current code.
What changes
List.of.mapRefsbuilt a new record for every leaf value even when every reference mapped to itself. It now returns the same record in that case, and a newforEachRefvisits the references without building anything. The reference counter's walk (TreeWalker.references) and the check that sets a leaf'sREFERS_TO_PAGESflag now useforEachRef, so the counter no longer builds one record per edge or promoted posting list it passes.Not carried over
#76 also moved the catalog write and the WAL sync of a checkpoint out of the commit lock. #81 already syncs the segments before taking the lock, but the catalog publish and WAL sync still run under it. While writing the test I saw what that costs: with checkpoints running back to back, a writer gets very few commits in. Doing that part again on top of the reference count journal is a separate change.
Testing
AtomCodecTestandPostingsCodecTestcheck thatmapRefskeeps the same record when nothing changes and thatforEachRefsees the same referencesmapRefsdoes.CrashRecoveryTest.checkpointsTakenWhileCommitsRunLoseNothingruns in both WAL modes, about 13 s together.