fix: avoid using uninitialized memory

[buriedpot]

Fix crash in RUM index scan when VACUUM removes all items from a posting tree leaf page

During a RUM index scan, if a concurrent VACUUM operation completes and removes all entries from a posting tree leaf page, the scan entry’s nlist field may become zero. In this state, the curItem pointer can end up referencing uninitialized or invalid memory, leading to a backend crash.

This commit resolves the issue by ensuring that curItem is properly handled (or not dereferenced) when nlist == 0, thus preventing undefined behavior during index scans under concurrent vacuuming.

Reproduction Steps

1. Prepare test data

CREATE EXTENSION rum;
ALTER SYSTEM SET enable_seqscan = off;
ALTER SYSTEM SET enable_indexscan = off;
ALTER SYSTEM SET enable_bitmapscan = on;
SELECT pg_reload_conf();

DROP TABLE IF EXISTS t;
CREATE TABLE t (id int, body tsvector);
ALTER TABLE t SET (autovacuum_enabled = FALSE);

INSERT INTO t
SELECT i, to_tsvector('rare word')
FROM generate_series(1, 100000) AS i;

CREATE INDEX ON t USING rum (body rum_tsvector_ops);

DELETE FROM t;  -- Leaves index entries pointing to dead tuples

2. Trigger the race condition using GDB

• Backend 1 (VACUUM):
  Attach GDB and set a breakpoint at rumVacuumPostingTreeLeaves.
  Run:

  VACUUM t;

  Let it pause at the breakpoint.

• Backend 2 (Query):
  Attach another GDB instance and set a breakpoint at startScanEntry.
  Run:

  SELECT * FROM t WHERE body @@ 'rare'::tsquery;

  Let it pause at the breakpoint.

• Resume execution:
  First, continue Backend 1 to completion (so VACUUM fully removes all items from the leaf page).
  Then, continue Backend 2.

  → The query backend will crash due to accessing invalid memory via entry->curItem when entry->nlist == 0.

This fix ensures safe handling of empty posting lists during scans, eliminating the crash under concurrent VACUUM and query workloads. The related issue is #168 and #99.

[buriedpot]

I strongly recommend prioritizing the review and applying of this fix.

[buriedpot]

You can observe that a concurrent VACUUM may set a page’s maxoff to zero in rumvacuum.c:

			if (newMaxOff > 0)
				memcpy(RumDataPageGetData(newPage), cleaned, newSize);

			pfree(cleaned);
			RumPageGetOpaque(newPage)->maxoff = newMaxOff;
			updateItemIndexes(newPage, attnum, &gvs->rumstate);

When newMaxOff == 0, the page ends up with no valid items.
Later, during a RUM index scan, the query backend reads this maxoff as entry->nlist. If nlist == 0, the subsequent logic in rumget.c still attempts to initialize entry->curItem:

			entry->nlist = maxoff;

			ptr = RumDataPageGetData(page);

			for (i = FirstOffsetNumber; i <= maxoff; i = OffsetNumberNext(i))
			{
				ptr = rumDataPageLeafRead(ptr, entry->attnum, &item, true,
										  rumstate);
				entry->list[i - FirstOffsetNumber] = item;
			}

			LockBuffer(entry->buffer, RUM_UNLOCK);
			entry->isFinished = setListPositionScanEntry(rumstate, entry);
			if (!entry->isFinished)
				entry->curItem = entry->list[entry->offset];

Since the loop body is never executed when maxoff == 0, entry->list remains uninitialized. Yet, if setListPositionScanEntry() returns false (which can happen even with an empty list under certain scan states), entry->curItem will point to garbage memory — leading to a crash.