Repository navigation
Cache the fingerprint and size of inline posting lists - #24
Merged
Merged
Conversation
Freezing a reverse-index leaf recomputed its summary by re-hashing every posting list in it, touched or not, and every copy re-measured their encoded size the same way. Both are now worked out once when an inline list is built or decoded. Fingerprints are unchanged, so nothing on disk moves.
venkat1701
force-pushed
the
perf/edge-ingest
branch
from
October 4, 2026 08:21
4f2092b to
b242be3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #10
I profiled edge ingest before following the plan in the issue, and the plan turned out to be wrong. A transaction's write scope already owns the nodes it copies, so touching the same reverse-index leaf twice in one commit doesn't copy it twice. Sorting the updates wouldn't change the pages written.
The real cost was elsewhere. About 90% of ingest time is commit materialization, and the biggest single piece, around 36%, was recomputing leaf summaries. Freezing a reverse-index leaf re-hashed every inline posting list in it, including the ones the commit never touched, so hub nodes with long posting lists got re-hashed again and again.
Leaf.remeasuredid the same for encoded sizes on every copy.Postings.Inlinenow carries its fingerprint and encoded size. Both are computed once inPostingsCodec.inlinewhen the list is built or decoded, which is already an O(n) step because of the array copy. Summaries and size accounting just read them. The fingerprint formula hasn't changed, so pages and summaries on disk are identical and there's no format change.Numbers: 60 commits of 1,000 edges on a 50k node database, main and this branch run back to back three times. Wall time was too noisy on this machine (load average around 19 from other containers), so I measured CPU time on the committing thread:
Pages and bytes written per commit are identical in every run (1,135 pages, 993 KiB), which is also a nice check that the output didn't change.
The write volume itself, around 1 KiB of pages per edge at this batch size, is what copy-on-write costs for random updates to the reverse index. Bringing that down would need a write buffer in front of the index, which is a bigger design change than this issue.
Full
./mvnw installpasses locally, including the crash matrix.