Repository navigation
Copy relocated leaf pages as bytes during compaction - #74
Closed
venkat1701 wants to merge 3 commits into
Closed
venkat1701 wants to merge 3 commits into
venkat1701 wants to merge 3 commits into
Conversation
Compaction decoded every moved leaf into objects, copied them and encoded them again. A leaf of a tree without nested references can be copied as it is: verify it against its old id, re-stamp the header with the new page id and checksum, and write it. Only branches and leaves holding nested trees are still rebuilt. Moves are paired after the copies are written, since copied pages never go through the node cache.
A corrupted membership leaf makes relocation fail instead of being re-sealed with a valid checksum; the test fails if verification is skipped.
This was referenced Oct 7, 2026
copyPage wrote the new page id straight into the image that PageStore.read handed back. That only works while read returns a private buffer. Once pages are read through the segment map they come back read-only, and every compaction failed with "Attempt to write a read-only segment". Copy the page into its own array first.
Collaborator
Author
|
Closing this. #81 replaced tree-rewriting compaction with moving pages through the page directory, so most of this stack no longer applies. A review also found two data-loss bugs in #73 (a retirement generation older than a commit that still used the moved pages, and bulk-loaded pages left behind in a victim). The pieces that still help, the posting list decode, records reused by mapRefs, and the checkpoint crash test, are in #82. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #50. Stacked on #73: merge that first; this PR's base then moves to
mainonce the #73 branch is deleted.Compaction decoded every moved leaf into objects, copied them, and encoded them again. That was most of the roughly 2 GiB a pass allocated, and the GC pauses that came with it were what was left of the commit tail after #73.
Now a moved page from a tree without nested references is copied as raw bytes:
MaterializerdoesBranches and leaves that hold nested trees are still rebuilt. This also covers nested single-leaf trees (most small edges' membership trees), which are reached through a reference whose height the walker doesn't know; the page header tells it. Old and new pages are now paired after the copies are written, since copied pages never go through the node cache.
Results, HStore only, 3 alternating runs each, #73 against #73 plus this PR, load 9 to 21, all checksums identical:
Space reclaimed and disk size after compaction are unchanged.
I also tried capping background passes at 64 MiB of live data. With 16 MiB segments that made passes bigger (16 to 17 segments instead of 1), and the
mixed.writetail got worse. Real bounding needs moving part of a segment per pass, so I dropped that change.Tested:
./mvnw installpasses (103 tests); each commit compiles on its ownCompactionTestcases pass, including the snapshot-to-swap merge, two branches with reclaim and reopen, and the crash matrix of the swapaCorruptPageIsNotCopiedWithAFreshChecksum: a membership leaf corrupted on disk (while its decoded copy is still cached) makes relocation fail. I checked that it fails if verification is skipped.