Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
36fffab
Rebase on upstream hourly, add AI/LLM PR review
gburd Mar 10, 2026
33d0690
dev setup v32
gburd Apr 29, 2026
62b1398
Add tests to cover a variety of heap HOT update behaviors
gburd Mar 10, 2026
d036549
Identify modified indexed attributes in the executor on UPDATE
gburd Apr 1, 2026
9137add
Replace TU_UpdateIndexes with per-index bitmapset tracking
gburd Apr 1, 2026
3a8b537
Replace MODIFIED_IDX_ATTRS_ALL_IDX sentinel with TM_IndexUpdateInfo s…
gburd May 6, 2026
5781006
Phase 1 spike: HOT-indexed tombstone on-disk format
gburd May 6, 2026
ca2cb96
Add RelationGetIndexedAttrs() per-index attribute bitmap accessor
gburd May 6, 2026
f3dba5f
Make ExecSetIndexUnchanged precise for SIU via RelationGetIndexedAttrs
gburd May 7, 2026
2849062
Add heap_build_hot_indexed_tombstone builder for SIU tombstones
gburd May 7, 2026
0cc0f61
Introduce HeapUpdateHotMode tri-state (preparation for SIU)
gburd May 7, 2026
a51403e
Teach pruneheap/vacuumlazy to skip HOT-indexed tombstones
gburd May 7, 2026
80afe3e
Implement HOT-indexed (Selective Index Update) write path
gburd May 7, 2026
95d34d5
Add SIU reader path: chain-walk recheck + direct heap-only lookup
gburd May 7, 2026
4d6e047
Reclaim orphaned SIU tombstones during prune
gburd May 7, 2026
de91ba5
SIU: reader-side fixes for CLUSTER, unique checks, and stale dedup
gburd May 7, 2026
46c8ab5
BitmapHeapScan: force recheck on HOT-indexed chain walks
gburd May 7, 2026
3759d3d
SIU: reach full 246/246 with hot_indexed_updates=on
gburd May 8, 2026
32da879
Flip hot_indexed_updates default to on (Phase 3.1g)
gburd May 8, 2026
0ce2828
heap_update: harden SIU fit check; genam: filter SIU-stale catalog sc…
gburd May 8, 2026
6dfc208
heap_update: palloc the SIU tombstone scratch buffer (P2.3)
gburd May 8, 2026
4ea527d
pruneheap: allow SIU tombstones in LP_UNUSED assertion; document P7 s…
gburd May 8, 2026
3e68b15
Remove hot_indexed_updates GUC; SIU is always on (catalogs excepted)
gburd May 8, 2026
a92143f
heap_update: reserve a 2nd LP slot when asking for a SIU-eligible buffer
gburd May 8, 2026
8add1dd
Add hot_indexed_update_threshold GUC (percent cap on SIU eligibility)
gburd May 9, 2026
c890e91
Add src/test/benchmarks/siu: the HOT-indexed A/B harness
gburd May 9, 2026
acfb794
Expose SIU metrics: n_tup_siu_upd counter + pg_relation_siu_stats()
gburd May 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Empty file added .bash_profile
Empty file.
Empty file added .bashrc
Empty file.
43 changes: 43 additions & 0 deletions .clangd
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
Diagnostics:
MissingIncludes: None
InlayHints:
Enabled: true
ParameterNames: true
DeducedTypes: true
CompileFlags:
CompilationDatabase: build/ # Search build/ directory for compile_commands.json
Remove: [ -Werror ]
Add:
- -DDEBUG
- -DLOCAL
- -DPGDLLIMPORT=
- -DPIC
- -O2
- -Wall
- -Wcast-function-type
- -Wconversion
- -Wdeclaration-after-statement
- -Wendif-labels
- -Werror=vla
- -Wextra
- -Wfloat-equal
- -Wformat-security
- -Wimplicit-fallthrough=3
- -Wmissing-format-attribute
- -Wmissing-prototypes
- -Wno-format-truncation
- -Wno-sign-conversion
- -Wno-stringop-truncation
- -Wno-unused-const-variable
- -Wpointer-arith
- -Wshadow
- -Wshadow=compatible-local
- -fPIC
- -fexcess-precision=standard
- -fno-strict-aliasing
- -fvisibility=hidden
- -fwrapv
- -g
- -std=c11
- -I.
- -I../../../../src/include
39 changes: 39 additions & 0 deletions .envrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
watch_file flake.nix
use flake

# libumem: pass -DUSE_LIBUMEM to enable palloc integration at compile time
export CFLAGS="${CFLAGS:+$CFLAGS }-DUSE_LIBUMEM"

# libumem: uncomment to enable malloc replacement via LD_PRELOAD
#export LD_PRELOAD="$LIBUMEM_HOME/lib/libumem.so"

# UMEM_DEBUG controls debug features:
# default - audit+contents+guards (comprehensive)
# audit[=N] - allocation auditing, optional stack frames
# guards - redzone + deadbeef (detect overruns/use-after-free)
# verbose - error messages to stderr
# firewall=N - guard pages for objects >= N bytes
# nosignal - abort if called from signal handler
# checknull - abort if allocation returns NULL
#export UMEM_DEBUG="default"
#export UMEM_DEBUG="audit=10,guards"
#export UMEM_DEBUG="guards,verbose"
export UMEM_DEBUG="audit=10,guards,checknull,nosignal"

# UMEM_OPTIONS controls allocator behavior:
# backend=sbrk|mmap - memory backend
# allocator=best|first|next - allocation strategy
# perthread_cache=SIZE - per-thread cache (e.g. 2m)
# nomagazines - disable caching (simplifies debugging)
# concurrency=N - max CPUs
# numa=1 - NUMA-aware allocation
#export UMEM_OPTIONS="backend=mmap"
#export UMEM_OPTIONS="nomagazines"
export UMEM_OPTIONS="backend=mmap"

# UMEM_LOGGING controls logging:
# transaction[=SIZE] - audit log (default 64k)
# contents[=SIZE] - freed object contents (default 256 bytes)
# fail[=SIZE] - failed allocations
#export UMEM_LOGGING="transaction=1m,fail=64k"
export UMEM_LOGGING="transaction=1m,fail=64k"
156 changes: 156 additions & 0 deletions .gdbinit
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# HOT Indexed Updates — GDB breakpoints for code review
#
# Usage: gdb -x .gdbinit <postgres-binary>
# Or from gdb: source .gdbinit
#
# These breakpoints cover the major code paths introduced or modified by
# the HOT indexed updates patch series. They are organized by subsystem
# to make it easy to enable/disable groups during debugging.
#
# Tip: to skip to a specific subsystem, disable all then enable selectively:
# disable breakpoints
# enable 1 2 3 # just the update-decision group

# =========================================================================
# 1. UPDATE DECISION — heap_update() HOT/HOT-indexed/non-HOT choice
# src/backend/access/heap/heapam.c
# =========================================================================

# Main entry: heap_update
break heapam.c:3210

# HOT decision block: pure HOT vs HOT indexed vs non-HOT
# Line 4019: pure HOT (no indexed columns changed)
# Line 4024: HOT indexed path (non-catalog, some indexed columns changed)
# Line 4031: predict augmented tuple size
# Line 4033: size+space check before creating augmented tuple
break heapam.c:4019
break heapam.c:4024
break heapam.c:4033

# Set HEAP_INDEXED_UPDATED flag on new tuple before page insertion
break heapam.c:4101

# Restore HEAP_INDEXED_UPDATED on old tuple (only if it previously had it)
break heapam.c:4147

# =========================================================================
# 2. TUPLE CREATION — building the augmented tuple with embedded bitmap
# src/backend/access/heap/heapam.c
# =========================================================================

# Predict augmented tuple size (returns 0 if t_hoff would overflow)
break heap_hot_indexed_tuple_size

# Create augmented tuple with embedded modified-column bitmap
break heap_hot_indexed_create_tuple

# Serialize Bitmapset into raw bytes in tuple header
break heap_hot_indexed_serialize_bitmap

# =========================================================================
# 3. BITMAP UTILITIES — raw bitmap operations for chain following
# src/backend/access/heap/heapam.c
# =========================================================================

# Compute raw bitmap byte size from natts
break heap_hot_indexed_bitmap_raw_size

# Check if tuple header has room for bitmap between null bitmap and data
break heap_hot_indexed_has_bitmap_space

# Read HOT indexed bitmap from tuple header (returns Bitmapset)
break heap_hot_indexed_read_bitmap

# Fast overlap check: does tuple's raw bitmap overlap with indexed_attrs?
break heap_hot_indexed_bitmap_overlaps_raw

# OR a tuple's raw bitmap into an accumulator buffer
break heap_hot_indexed_bitmap_or_raw

# Check if accumulated raw bitmap overlaps with indexed_attrs
break heap_hot_indexed_accum_overlaps

# Merge bitmaps from dead tuples into a target tuple on the page
break heap_hot_indexed_merge_bitmaps_raw

# Deserialize raw bytes back to Bitmapset
break heap_hot_indexed_deserialize_bitmap

# =========================================================================
# 4. INDEX SCAN — HOT chain following with stale-entry detection
# src/backend/access/heap/heapam_indexscan.c
# =========================================================================

# Main HOT chain search with indexed update awareness
break heap_hot_search_buffer

# Redirect-with-data: initialize bitmap accumulator from collapsed redirect
break heapam_indexscan.c:182

# Accumulate bitmap from INDEXED_UPDATED tuple in chain
break heapam_indexscan.c:250

# Stale entry detection: accumulated bitmap overlaps this index's attrs
break heapam_indexscan.c:297

# =========================================================================
# 5. INDEX SCAN SETUP — indexed_attrs bitmap computation
# src/backend/access/index/indexam.c
# =========================================================================

# Compute indexed_attrs for HOT indexed update chain following
break indexam.c:299

# =========================================================================
# 6. INDEX INSERTION — skip unchanged indexes for HOT indexed updates
# src/backend/executor/execIndexing.c
# =========================================================================

# Entry: insert/update index tuples
break ExecInsertIndexTuples

# Index skip decision: skip indexes whose attrs don't overlap modified set
break execIndexing.c:370

# =========================================================================
# 7. PRUNING — chain collapsing and redirect-with-data
# src/backend/access/heap/pruneheap.c
# =========================================================================

# Main prune function
break heap_page_prune_and_freeze

# Per-chain pruning entry
break heap_prune_chain

# Chain collapsing: collect bitmaps from dead INDEXED_UPDATED intermediates
break pruneheap.c:1802

# OR dead tuple bitmaps into combined bitmap
break pruneheap.c:1836

# Record redirect-with-data for execute phase
break pruneheap.c:1863

# Execute phase: apply redirect-with-data entries on the page
break pruneheap.c:1287

# =========================================================================
# 8. WAL REPLAY — recovery of HOT indexed updates
# src/backend/access/heap/heapam_xlog.c
# =========================================================================

# WAL replay for XLOG_HEAP2_INDEXED_UPDATE
break heap_xlog_indexed_update

# =========================================================================
# 9. WAL LOGGING — writing HOT indexed update records
# src/backend/access/heap/heapam.c
# =========================================================================

# WAL logging for heap updates (handles indexed_update flag)
break log_heap_update

# Serialize redirect-with-data into WAL record (pruneheap.c)
break pruneheap.c:2936
Empty file added .gitconfig
Empty file.
18 changes: 18 additions & 0 deletions .github/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Node modules
scripts/ai-review/node_modules/
# Note: package-lock.json should be committed for reproducible CI/CD builds

# Logs
scripts/ai-review/cost-log-*.json
scripts/ai-review/*.log

# OS files
.DS_Store
Thumbs.db

# Editor files
*.swp
*.swo
*~
.vscode/
.idea/
Loading
Loading