Skip to content

Reduce SKStore dependency storage overhead - #1585

Merged
mbouaziz merged 1 commit into
mainfrom
skstore-compact-dependencies
Nov 19, 2026
Merged

mbouaziz merged 1 commit into
mainfrom
skstore-compact-dependencies

Conversation

@pikatchu

@pikatchu pikatchu commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Store singleton dependency readers directly and pack the inverse directory-to-path index into immutable sorted blocks. Reuse the existing reader block implementation for paths, preserving ordering, equal-key representatives and snapshot semantics. Both improvements apply unconditionally; the existing multi-reader policies still select arrays, trees or blocks for larger reader sets.

Add regressions against an independent relation model for insertion, deletion, block boundaries, directory removal, reinsertion and retained snapshots.

Validation: 127 prelude tests pass in all 16 combinations of the existing storage options. All 21 incremental compiler regressions pass with both default and packed settings. The final compiler self-build produces 266 LLVM modules byte-identical to the tested prototype, and its generated compiler passes the smoke test.

On the controlled prototype self-build, peak RSS fell from 20.21 to 18.67 GiB and reachable store memory from 13.32 to 12.40 GiB (23 million fewer objects). Total user+system CPU fell from 323.73 to 315.47 seconds. These are single cold-build trials; warm-update performance has not been benchmarked.

The unconditional implementation's confirmation build retains the same memory result (18.67 GiB peak, 12.40 GiB reachable), with 329.16 seconds of CPU. It uses a rebuilt compiler; these timings do not establish a reliable speedup.

@pikatchu
pikatchu force-pushed the skstore-compact-dependencies branch from c1b71fa to 3eaa41e Compare November 15, 2026 12:31
Store singleton dependency readers directly and pack the inverse directory-to-path
index into immutable sorted blocks. Reuse the existing reader block implementation
for paths, preserving ordering, equal-key representatives and snapshot semantics.
Both improvements apply unconditionally; the existing multi-reader policies still
select arrays, trees or blocks for larger reader sets.

Add regressions against an independent relation model for insertion, deletion,
block boundaries, directory removal, reinsertion and retained snapshots.

Validation: 127 prelude tests pass in all 16 combinations of the existing storage
options. All 21 incremental compiler regressions pass with both default and packed
settings. The final compiler self-build produces 266 LLVM modules byte-identical
to the tested prototype, and its generated compiler passes the smoke test.

On the controlled prototype self-build, peak RSS fell from 20.21 to 18.67 GiB and
reachable store memory from 13.32 to 12.40 GiB (23 million fewer objects). Total
user+system CPU fell from 323.73 to 315.47 seconds. These are single cold-build
trials; warm-update performance has not been benchmarked.

The unconditional implementation's confirmation build retains the same memory
result (18.67 GiB peak, 12.40 GiB reachable), with 329.16 seconds of CPU. It uses
a rebuilt compiler; these timings do not establish a reliable speedup.

Co-authored-by: mbouaziz <mbouaziz@users.noreply.github.com>
@mbouaziz
mbouaziz force-pushed the skstore-compact-dependencies branch from 3eaa41e to e421659 Compare November 19, 2026 21:48
@mbouaziz
mbouaziz enabled auto-merge November 19, 2026 21:49
@mbouaziz
mbouaziz merged commit 3c5dfc4 into main Nov 19, 2026
22 checks passed
@mbouaziz
mbouaziz deleted the skstore-compact-dependencies branch November 19, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants