Skip to content

Fix core lookup and add selectable RFF methods with benchmarks - #24

Draft
SWORDIntel wants to merge 3 commits into
mainfrom
fix/core-search-correctness-20261114
Draft

SWORDIntel wants to merge 3 commits into
mainfrom
fix/core-search-correctness-20261114

Conversation

@SWORDIntel

@SWORDIntel SWORDIntel commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Why

The original scalar Hilbert/RFF lookup could return a candidate without checking the original record. A controlled reproduction returned index 2 for all 32 present queries (only 1/32 matched) and for absent query 999. Query features did not determine the result, and the smoke test counted non-sentinel results as accuracy.

This PR repairs that path and the related numerical defects, then adds selectable core methods and a reproducible benchmark so future optimizations can be judged against typed exact search.

Implementation

  • Verify original-record equality after signed RFF candidate scoring. Preserve full int64/uint64 equality, bounded record strides and correct absence handling. Default verification recovers failed candidates through a complete scan.
  • Canonicalize public math layouts and state implementation ownership. Correct normalization, fidelity, sampling, amplification convergence/round counts/probabilities, dimension budgets and explicitly sized verification. Unsupported historical stubs return ENOTSUP instead of false success.
  • Correct deterministic RFF construction, CPU projection strides/tails/cosines, SIMD norm arithmetic and heterogeneous partition remainders.
  • Rerank the complete admitted INT8/binary candidate pool. Filter clearance/SCI before admission in both selectors and recheck before payload access. A hidden approximate winner can no longer consume a pool of one and suppress the authorized result.
  • Add meaningful correctness regressions and synchronize headers, API references and migration guidance. Four AGENTS.md additions concern shared definitions, synchronized docs/bindings, migrations and incremental upgrades. No CI or additional safety rules added.

Selectable core methods

The additive, size/version checked qihse_search_ex and qihse_batch_search_ex APIs return status separately from result indices. They preserve existing qihse_config_t and kernel layouts.

Method Execution
QIHSE_SEARCH_EXACT Original-record equality scan; no feature allocation; projection configuration is irrelevant.
QIHSE_SEARCH_RFF_COSINE Existing random-phase cosine map and typed verification.
QIHSE_SEARCH_RFF_PAIRED D/2 frequencies, D interleaved sine/cosine coordinates, and typed verification.

Options select query tiles of 1–64. A batch constructs one basis and shares row projections within each tile; legacy batch calls use phase cosine with tile 16. Scratch is O(TD), independent of record count; dot products remain O(nQD).

Per-call metrics report candidate hits, fallback work, successful/failed projections, original-record comparisons, basis builds, active tiles, scratch and stage times. Failure injection checks allocations and clocks. No benchmark winner is chosen automatically.

The additive opaque qihse_rff_map_t owns its method, preventing constructor/projector mismatch. Its descriptor includes basis version, method, dimensions, gamma and seed; legacy phase outputs are checked bit-for-bit.

Measured reference

The checked-in reference and source manifest cover 45 comparisons across present, absent, duplicate, nearby-double and large-integer fixtures, with an independent equality oracle. Every case passed.

For 512 records, 32 present queries, 128 output coordinates, GCC 14.2.0/O3, one warmup and three timed repeats:

Method Tile Median complete batch Row projections per batch Peak owned scratch
Exact 1 0.007862 ms 0 0 B
Phase cosine 1 47.231098 ms 16,384 4,200 B
Phase cosine 16 3.898890 ms 1,024 19,560 B
Paired sine/cosine 1 32.206445 ms 16,384 2,664 B
Paired sine/cosine 16 3.183138 ms 1,024 18,024 B

These are resident synthetic scalar lookups on an unpinned Linux x86-64 workspace, including setup and instrumentation. Exact scan wins these fixtures. Lower kernel approximation error does not guarantee exact candidate discrimination: the nearby-double fixture needed fallback for most present queries. This is not a VectorDB or production throughput claim.

The runner supports method, tile, dimensions, dataset, records, queries, repeats, seed, gamma and JSONL/CSV selection. A compiler-derived 19-file source/header SHA256 manifest identifies the measured dirty tree precisely.

Dependency and integration follow-up

LuaJIT development/runtime dependencies and the remaining native, Python and Rust test dependencies were supplied locally. The shared library, server and administrative executable now build.

  • Discover LuaJIT through pkg-config with explicit overrides for local prefixes.
  • Use tree-sitter's already-vendored Unicode headers through its target-specific include path.
  • Make lib-ctypes use the canonical library build, avoiding two link recipes overwriting the same artifact; align launcher descriptions.
  • Repair two test launchers' shared-library search paths.
  • Replace the Python controller fixture's stale C configuration mirror with a helper compiled against the canonical header; fix its Thread._stop name collision and close fixture pipes.
  • Declare Python test-only dependencies and document reproducible environment setup.
  • Give the CodeGraph reader test an accurate early-exit diagnostic.

Validation

Passed:

  • Core lookup regressions and the new methodology regressions, normally and with ASan/UBSan/float-cast-overflow.
  • Explicit exact mode with allocation disabled; method and tile parity; tail tiles; padded strides; candidate-only tests; deterministic projection collisions; fallback accounting; every allocation and clock failure position.
  • Direct map tests: paired identities, descriptor/storage accounting, failure contracts, deterministic streams, concurrent use and legacy bitwise parity, normally and with sanitizers.
  • Existing core math and runtime regressions for scalar, AVX2 and AVX512.
  • 45/45 reference benchmark cases; 25/25 smaller sanitizer cases; CSV/JSONL and CLI boundary checks.
  • Installed C/C++ header checks, source declaration counts and git diff --check.
  • Actual canonical make lib-ctypes after the gate: unchanged library hash, native Python extension import, expected CPython/search exports and ctypes fidelity 1.0 for identical vectors.

LeakSanitizer is disabled because of the workspace tracing restriction; address and undefined-behaviour checks ran.

Final complete gate: make test TEST_JOBS=32 — 254 targets executed: 253 passed; test-codegraph failed (command exit status 2).

The library-backed candidate regression passed using real principals and cryptographic providers, including the INT8 and binary pool-of-one boundary. Controller validation executed all 61 tests with no skips, including all four native E2E cases. Browser unit/integration targets passed (41 unit and 36 integration checks). The gold pack completed 74 workloads in 11 areas with zero failures.

The PostgreSQL application-compatibility target ran 12 self-contained client/numeric tests successfully and skipped 15 cases that require the optional external DIRECTEYE source corpus. These are the same optional skips as the first complete run; they are not included in the controller's zero-skip result.

CodeGraph's named reader cannot start here: independent C and Python probes fail at socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0) with EPERM, before bind. The aggregate target remains visibly failed. All 18 downstream recipe commands were also executed separately: 16 passed; query-index and runtime-spill reader tests hit the same socket boundary. The incremental suite within those commands ran 15 tests successfully.

The validation follow-up records the complete outcome, repaired failures and environmental limitation. Original benchmark samples and their 19-file source manifest remain unchanged.

This remains a draft PR for reconciliation with ongoing local additions and the remaining named-reader run on a host that provides Unix sockets.

Compatibility and integration

Rebuild consumers against the canonical headers. Replace unsized float verification with _sized calls. The original repair changes the RFF RNG/basis, so regenerate retained features from that older implementation. The additive paired method is explicit; do not mix its features with phase-cosine features.

There is no stored database schema, .qdb, INT8 or binary format migration. Full dependency-complete gate: make clean && make lib && make test TEST_JOBS=32.

This stays on a separate PR branch for reconciliation with ongoing local additions. The complete repository audit and GPU/CUDA/AMX equivalence remain outside this focused CPU work.

Verify original typed records, stream O(D) feature scratch, unify public math layouts, correct CPU numerical helpers and candidate reranking, and synchronize API/migration documentation and focused regressions.
…benchmarks

Expose exact, phase-cosine and paired sine/cosine methods through additive sized APIs. Share one basis and row projections across configurable query tiles, report measured work per call, and include independently checked benchmark fixtures, source fingerprints and synchronized API documentation.
@SWORDIntel SWORDIntel changed the title Fix core Hilbert/RFF correctness and synchronize numerical APIs Fix core lookup and add selectable RFF methods with benchmarks Nov 15, 2026
Resolve native dependency discovery and canonical ctypes build behavior; replace the stale controller config mirror with a canonical-header helper. Filter INT8 candidates before pool admission and extend the real-principal regression.

Synchronize API/setup docs and record the complete 254-target run: 253 passed; CodeGraph remains blocked by named AF_UNIX socket creation returning EPERM. All 74 gold workloads passed; 15 optional DIRECTEYE corpus cases were skipped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant