Skip to content
Draft

Undo #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
a928518
Rebase on upstream hourly, add AI/LLM PR review
gburd Mar 10, 2026
01646b2
dev setup v32
gburd Apr 29, 2026
d0034b8
Add UNDO WAL logging infrastructure with physical rollback
gburd Mar 21, 2026
6972618
Add checkpoint phase timing instrumentation and UNDO checkpoint hook
gburd Apr 25, 2026
93ba921
Optimize UNDO write path: defer fsync, cache fds, atomic alloc, reuse…
gburd Apr 26, 2026
c12bb9c
Add UNDO flush daemon with group commit
gburd Apr 27, 2026
3d5a3e7
Add UNDO log segment rotation and lifecycle management
gburd Apr 28, 2026
0faa9e0
Add bulk UNDO hints for reduced per-row overhead in DML
gburd Apr 28, 2026
d70cdef
Add transactional file operations (FILEOPS) framework
gburd Apr 17, 2026
535ef94
Add FILEOPS CREATE operation
gburd Apr 17, 2026
72a6618
Add FILEOPS DELETE operation
gburd Apr 17, 2026
0c78271
Add FILEOPS RENAME operation
gburd Apr 17, 2026
8c81e87
Add FILEOPS WRITE operation
gburd Apr 17, 2026
d53caf7
Add FILEOPS TRUNCATE operation
gburd Apr 17, 2026
8c8e009
Add FILEOPS CHMOD and CHOWN operations
gburd Apr 17, 2026
4f4329d
Add FILEOPS MKDIR and RMDIR operations
gburd Apr 17, 2026
34e17c3
Add FILEOPS SYMLINK and LINK operations
gburd Apr 17, 2026
8159120
Add FILEOPS SETXATTR and REMOVEXATTR operations
gburd Apr 17, 2026
7a5cadb
FILEOPS: add PendingFileOps abort-time reversal, remove enable_transa…
gburd May 9, 2026
80141b3
Add opt-in UNDO support to the HEAP table AM
gburd Mar 21, 2026
97ea4d0
Add nbtree UNDO for zero-VACUUM aborted transaction cleanup
gburd Apr 21, 2026
1bad1b2
Add skiplist and sparsemap libraries with test modules
gburd Apr 29, 2026
23c4495
Replace sLog hash tables with skip-list and sparsemap
gburd Apr 29, 2026
fed2184
UNDO subsystem: batch WAL, buffer management, and reduced per-row ove…
gburd Apr 30, 2026
b49331f
UNDO-in-WAL: embed UNDO records in the standard WAL stream
gburd Apr 30, 2026
05cd102
UNDO: remove RELUNDO, add subtransaction support, make Tier 2 AM-agno…
gburd May 5, 2026
8008052
Add UNDO record support for HASH indexes
gburd May 9, 2026
41672b7
Remove UNDO from Heap AM; add adversarial crash tests
gburd May 10, 2026
941d79b
RECNO: Add RECNO table access method with dynamic sLog sizing
gburd May 10, 2026
71b0278
RECNO: Fix WAL replay bugs in cross-page defrag and consistency checking
gburd May 11, 2026
bb188e6
RECNO: Fix use-after-scope bug in overflow WAL logging for INSERT
gburd May 11, 2026
48f6401
RECNO: zero entire t_flags in recno_mask() for WAL consistency
gburd May 11, 2026
bc395f0
RECNO: force FPI for DEFRAG WAL records to fix standby divergence
gburd May 11, 2026
7512417
RECNO: Unify per-tuple sLog into shared slog.c; fix standby visibilit…
gburd May 11, 2026
c82c222
RECNO: Enable autoconf build by unconditionally defining USE_RECNO
gburd May 11, 2026
51d1f14
RECNO: make sLog overflow graceful instead of crashing
gburd May 11, 2026
e04b1d4
RECNO: fix benchmark script for sLog overflow and server crashes
gburd May 11, 2026
83ff2f6
RECNO: fix benchmark 6 index scan to use pgbench \set variables
gburd May 11, 2026
6998fb1
RECNO: add PD_ALL_VISIBLE fast path and max_logical_revert_workers GUC
gburd May 11, 2026
78fee54
RECNO: fix sLog overflow leaving tuples permanently UNCOMMITTED
gburd May 11, 2026
e5b90e5
RECNO: eliminate per-tuple buffer pin overhead in sequential scan
gburd May 11, 2026
7487b98
RECNO: fix logical revert worker SIGSEGV after WAL recycling
gburd May 11, 2026
f3942fe
RECNO: implement Large Transaction Mode to fix sLog overflow catastrophe
gburd May 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Empty file added .bash_profile
Empty file.
Empty file added .bashrc
Empty file.
43 changes: 43 additions & 0 deletions .clangd
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
Diagnostics:
MissingIncludes: None
InlayHints:
Enabled: true
ParameterNames: true
DeducedTypes: true
CompileFlags:
CompilationDatabase: build/ # Search build/ directory for compile_commands.json
Remove: [ -Werror ]
Add:
- -DDEBUG
- -DLOCAL
- -DPGDLLIMPORT=
- -DPIC
- -O2
- -Wall
- -Wcast-function-type
- -Wconversion
- -Wdeclaration-after-statement
- -Wendif-labels
- -Werror=vla
- -Wextra
- -Wfloat-equal
- -Wformat-security
- -Wimplicit-fallthrough=3
- -Wmissing-format-attribute
- -Wmissing-prototypes
- -Wno-format-truncation
- -Wno-sign-conversion
- -Wno-stringop-truncation
- -Wno-unused-const-variable
- -Wpointer-arith
- -Wshadow
- -Wshadow=compatible-local
- -fPIC
- -fexcess-precision=standard
- -fno-strict-aliasing
- -fvisibility=hidden
- -fwrapv
- -g
- -std=c11
- -I.
- -I../../../../src/include
39 changes: 39 additions & 0 deletions .envrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
watch_file flake.nix
use flake

# libumem: pass -DUSE_LIBUMEM to enable palloc integration at compile time
export CFLAGS="${CFLAGS:+$CFLAGS }-DUSE_LIBUMEM"

# libumem: uncomment to enable malloc replacement via LD_PRELOAD
#export LD_PRELOAD="$LIBUMEM_HOME/lib/libumem.so"

# UMEM_DEBUG controls debug features:
# default - audit+contents+guards (comprehensive)
# audit[=N] - allocation auditing, optional stack frames
# guards - redzone + deadbeef (detect overruns/use-after-free)
# verbose - error messages to stderr
# firewall=N - guard pages for objects >= N bytes
# nosignal - abort if called from signal handler
# checknull - abort if allocation returns NULL
#export UMEM_DEBUG="default"
#export UMEM_DEBUG="audit=10,guards"
#export UMEM_DEBUG="guards,verbose"
export UMEM_DEBUG="audit=10,guards,checknull,nosignal"

# UMEM_OPTIONS controls allocator behavior:
# backend=sbrk|mmap - memory backend
# allocator=best|first|next - allocation strategy
# perthread_cache=SIZE - per-thread cache (e.g. 2m)
# nomagazines - disable caching (simplifies debugging)
# concurrency=N - max CPUs
# numa=1 - NUMA-aware allocation
#export UMEM_OPTIONS="backend=mmap"
#export UMEM_OPTIONS="nomagazines"
export UMEM_OPTIONS="backend=mmap"

# UMEM_LOGGING controls logging:
# transaction[=SIZE] - audit log (default 64k)
# contents[=SIZE] - freed object contents (default 256 bytes)
# fail[=SIZE] - failed allocations
#export UMEM_LOGGING="transaction=1m,fail=64k"
export UMEM_LOGGING="transaction=1m,fail=64k"
156 changes: 156 additions & 0 deletions .gdbinit
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# HOT Indexed Updates — GDB breakpoints for code review
#
# Usage: gdb -x .gdbinit <postgres-binary>
# Or from gdb: source .gdbinit
#
# These breakpoints cover the major code paths introduced or modified by
# the HOT indexed updates patch series. They are organized by subsystem
# to make it easy to enable/disable groups during debugging.
#
# Tip: to skip to a specific subsystem, disable all then enable selectively:
# disable breakpoints
# enable 1 2 3 # just the update-decision group

# =========================================================================
# 1. UPDATE DECISION — heap_update() HOT/HOT-indexed/non-HOT choice
# src/backend/access/heap/heapam.c
# =========================================================================

# Main entry: heap_update
break heapam.c:3210

# HOT decision block: pure HOT vs HOT indexed vs non-HOT
# Line 4019: pure HOT (no indexed columns changed)
# Line 4024: HOT indexed path (non-catalog, some indexed columns changed)
# Line 4031: predict augmented tuple size
# Line 4033: size+space check before creating augmented tuple
break heapam.c:4019
break heapam.c:4024
break heapam.c:4033

# Set HEAP_INDEXED_UPDATED flag on new tuple before page insertion
break heapam.c:4101

# Restore HEAP_INDEXED_UPDATED on old tuple (only if it previously had it)
break heapam.c:4147

# =========================================================================
# 2. TUPLE CREATION — building the augmented tuple with embedded bitmap
# src/backend/access/heap/heapam.c
# =========================================================================

# Predict augmented tuple size (returns 0 if t_hoff would overflow)
break heap_hot_indexed_tuple_size

# Create augmented tuple with embedded modified-column bitmap
break heap_hot_indexed_create_tuple

# Serialize Bitmapset into raw bytes in tuple header
break heap_hot_indexed_serialize_bitmap

# =========================================================================
# 3. BITMAP UTILITIES — raw bitmap operations for chain following
# src/backend/access/heap/heapam.c
# =========================================================================

# Compute raw bitmap byte size from natts
break heap_hot_indexed_bitmap_raw_size

# Check if tuple header has room for bitmap between null bitmap and data
break heap_hot_indexed_has_bitmap_space

# Read HOT indexed bitmap from tuple header (returns Bitmapset)
break heap_hot_indexed_read_bitmap

# Fast overlap check: does tuple's raw bitmap overlap with indexed_attrs?
break heap_hot_indexed_bitmap_overlaps_raw

# OR a tuple's raw bitmap into an accumulator buffer
break heap_hot_indexed_bitmap_or_raw

# Check if accumulated raw bitmap overlaps with indexed_attrs
break heap_hot_indexed_accum_overlaps

# Merge bitmaps from dead tuples into a target tuple on the page
break heap_hot_indexed_merge_bitmaps_raw

# Deserialize raw bytes back to Bitmapset
break heap_hot_indexed_deserialize_bitmap

# =========================================================================
# 4. INDEX SCAN — HOT chain following with stale-entry detection
# src/backend/access/heap/heapam_indexscan.c
# =========================================================================

# Main HOT chain search with indexed update awareness
break heap_hot_search_buffer

# Redirect-with-data: initialize bitmap accumulator from collapsed redirect
break heapam_indexscan.c:182

# Accumulate bitmap from INDEXED_UPDATED tuple in chain
break heapam_indexscan.c:250

# Stale entry detection: accumulated bitmap overlaps this index's attrs
break heapam_indexscan.c:297

# =========================================================================
# 5. INDEX SCAN SETUP — indexed_attrs bitmap computation
# src/backend/access/index/indexam.c
# =========================================================================

# Compute indexed_attrs for HOT indexed update chain following
break indexam.c:299

# =========================================================================
# 6. INDEX INSERTION — skip unchanged indexes for HOT indexed updates
# src/backend/executor/execIndexing.c
# =========================================================================

# Entry: insert/update index tuples
break ExecInsertIndexTuples

# Index skip decision: skip indexes whose attrs don't overlap modified set
break execIndexing.c:370

# =========================================================================
# 7. PRUNING — chain collapsing and redirect-with-data
# src/backend/access/heap/pruneheap.c
# =========================================================================

# Main prune function
break heap_page_prune_and_freeze

# Per-chain pruning entry
break heap_prune_chain

# Chain collapsing: collect bitmaps from dead INDEXED_UPDATED intermediates
break pruneheap.c:1802

# OR dead tuple bitmaps into combined bitmap
break pruneheap.c:1836

# Record redirect-with-data for execute phase
break pruneheap.c:1863

# Execute phase: apply redirect-with-data entries on the page
break pruneheap.c:1287

# =========================================================================
# 8. WAL REPLAY — recovery of HOT indexed updates
# src/backend/access/heap/heapam_xlog.c
# =========================================================================

# WAL replay for XLOG_HEAP2_INDEXED_UPDATE
break heap_xlog_indexed_update

# =========================================================================
# 9. WAL LOGGING — writing HOT indexed update records
# src/backend/access/heap/heapam.c
# =========================================================================

# WAL logging for heap updates (handles indexed_update flag)
break log_heap_update

# Serialize redirect-with-data into WAL record (pruneheap.c)
break pruneheap.c:2936
Empty file added .gitconfig
Empty file.
18 changes: 18 additions & 0 deletions .github/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Node modules
scripts/ai-review/node_modules/
# Note: package-lock.json should be committed for reproducible CI/CD builds

# Logs
scripts/ai-review/cost-log-*.json
scripts/ai-review/*.log

# OS files
.DS_Store
Thumbs.db

# Editor files
*.swp
*.swo
*~
.vscode/
.idea/
Loading
Loading