Skip to content

Give pages permanent numbers and keep their addresses in a directory - #79

Merged
venkat1701 merged 8 commits into
mainfrom
feat/page-directory
Oct 11, 2026
Merged

venkat1701 merged 8 commits into
mainfrom
feat/page-directory

Conversation

@venkat1701

Copy link
Copy Markdown
Collaborator

Stacked on #77, since both change PageStore and the directory uses the same mapping approach.

Why

A page id was the page's address. Moving a page gave it a new id, so every node on the path to it had to be
rewritten too. That is why a full compaction at scale 4 allocated 6 to 9 GB and ran out of a 4 GB heap: it
rebuilds large parts of every tree just to move pages out of a segment.

This PR separates the two. A page id is now a number handed out in order, and a page directory maps each number
to where the page sits. Compaction still works the old way here; the next PR changes it to move pages by
copying their bytes and updating one directory entry.

What changes

  • PageId is a 48-bit index plus a 16-bit reuse count (always 0 for now). PageAddress is segment, offset and
    length in 64-byte units.
  • PageDirectory is data/segments/pages.dir: a 64-byte header, then one 8-byte address per number,
    memory-mapped in 8 MiB chunks. Lookups are a load from the page cache and use no heap.
  • Page and PageRef WAL records carry the address. The directory is forced only at checkpoints, and recovery
    writes every logged address back before it checks or replays a page, so an entry lost in a crash comes back
    from the log.
  • Checkpointer now reads its LSN before syncing pages. Before, a spill running outside the commit lock could
    write a page after the sync and log it before the LSN was read, and the checkpoint would drop a record whose
    page was never forced. That's the first commit, on its own.
  • Storage format goes to 4. Older directories fail to open with the usual export and reload message.

Testing

  • Full suite: 103 tests pass, and each commit builds and passes on its own.
  • New PageDirectoryTest: entries and the next number survive a reopen across more than one mapped chunk,
    reserved numbers are never handed out, and a file in another format is refused.
  • New crash test, for both WAL modes: after a halt it puts pages.dir back to its last checkpointed state, which
    is what losing unsynced directory writes looks like. Every logged commit has to come back, and later
    allocations must not reuse the restored numbers. Taking the address step out of recovery makes it fail.
  • Native image: built the server binary, loaded examples/clinical-claims.hql, and hstore check verified all
    105 pages. Then a SIGKILL partway through a 200,000-insert load: on reopen it replayed 813 commits from the log
    and check passed.
  • Timed on an Azure D8ds_v5 (8 vCPU, 32 GiB, local SSD), HStore only, Read pages through a memory map of their segment #77 against this branch, with the default
    node cache and with --cache-mb 64 (about 4,000 nodes, so most reads miss the cache and go through the
    directory). Each cell is this branch's throughput divided by Read pages through a memory map of their segment #77's, as medians of 3 runs at scale 1 and 2 at
    scale 4. Runs of the same setup varied by 1 to 15% on most rows, so read anything within about 10% of 1.00 as no
    change:
Workload scale 1, default cache scale 1, 64 MiB scale 4, default cache scale 4, 64 MiB
read.incidence 1.09 0.95 0.96 0.86
read.members 1.03 1.03 0.89 0.95
read.comembership 1.09 1.01 1.01 1.03
read.twohop 1.02 0.98 0.96 1.04
latency.read 1.04 0.86 1.00 1.05
read.incidence.cold 1.00 0.98 1.07 1.06
mixed.read 0.94 1.02 1.00 0.95
write.update 1.04 1.02 1.02 0.95
latency.commit 1.05 0.98 0.90 1.03
churn.delete 1.02 0.97 0.97 0.99
recover 0.93 0.94 0.94 0.99

The reads that miss the cache most, read.members and read.twohop at 64 MiB, are where the directory lookup
happens on almost every page, and they didn't move. A few cells are lower: read.incidence at scale 4 with
64 MiB (0.86), latency.read at scale 1 with 64 MiB (0.86, 22% spread), read.members at scale 4 with the
default cache (0.89, 9 and 12% spread), and read.incidence.churned at scale 4 with the default cache (0.83,
not in the table, 38% spread). In the first one, #77 itself ran faster with the small cache than with the full
one, so I read these as run-to-run variation, but they're worth watching when phase 2 is timed. Disk after loading is about 10 MB larger at scale 1, which is the directory
file.

What's next

Phase 2 moves pages during compaction by copying bytes and updating the directory, and removes relocate and
adopt.

The checkpoint took its LSN after pages.sync(). A spill can run outside
the commit lock, so it could write a page after the sync and append its
record before the LSN was read. The checkpoint then dropped a record whose
page was never forced.

Reading the LSN first means every record below it was appended after its
page was written, and the sync that follows covers that page.
…n a directory

A page id used to be the page's address: segment, offset and the low bits
of its generation. Moving a page meant giving it a new id, and so
rewriting every node on the path to it. That is most of what compaction
spends its time and memory on.

A page id is now a number handed out in order, and pages.dir maps each
number to its segment, offset and length in 64-byte units. The directory
is memory-mapped in 8 MiB chunks, so lookups cost no heap. Only PageStore
deals in addresses.

Page and PageRef records carry the address as well. The directory is
forced only at checkpoints, and recovery writes every logged address back
before it checks or replays a page, so entries a crash lost come back
from the WAL. Compaction is unchanged for now; it just gets new numbers
for the pages it rewrites.

The storage format goes to 4. Directories written by older builds fail to
open with the usual export-and-reload message.
PageDirectoryTest checks that entries and the next number survive a
reopen across more than one mapped chunk, that a reserved number is never
handed out, and that a file in another format is refused.

PageStoreTest checks that numbers stay in order across segments, that a
number with no entry reads as corrupt, and that a page can't be written
past what was allocated for it.

The new crash test puts pages.dir back to its last checkpointed state
after a halt, which is what losing unsynced directory writes looks like,
and expects every logged commit to come back and later allocations not to
reuse the restored numbers. It fails if recovery stops placing addresses.
pages.md explains numbers, addresses and pages.dir, with the hex example
regenerated for a numbered page and a way to follow an entry by hand.
wal-and-recovery.md gets the larger Page and PageRef records, the address
step in recovery and the new checkpoint order. The other pages pick up the
format change and the extra file.
…blank one

Mapping the new file extended it to 8 MiB, but the header was only forced at
the first checkpoint. A power loss before that left a file of zeros that every
later open refused as an unknown format. The header is now forced as soon as it
is written, and a header that is still all zeros is treated as a new file.
@venkat1701
venkat1701 changed the base branch from perf/mapped-reads to main October 11, 2026 09:14
# Conflicts:
#	docs/architecture/overview.md
#	docs/operations/configuration.md
#	engine/src/main/java/io/hstore/engine/page/PageStore.java
@venkat1701
venkat1701 merged commit e290f74 into main Oct 11, 2026
@venkat1701
venkat1701 deleted the feat/page-directory branch October 11, 2026 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant