Repository navigation
Fix core lookup and add selectable RFF methods with benchmarks - #24
Draft
SWORDIntel wants to merge 3 commits into
Draft
SWORDIntel wants to merge 3 commits into
SWORDIntel wants to merge 3 commits into
Conversation
Verify original typed records, stream O(D) feature scratch, unify public math layouts, correct CPU numerical helpers and candidate reranking, and synchronize API/migration documentation and focused regressions.
…benchmarks Expose exact, phase-cosine and paired sine/cosine methods through additive sized APIs. Share one basis and row projections across configurable query tiles, report measured work per call, and include independently checked benchmark fixtures, source fingerprints and synchronized API documentation.
Resolve native dependency discovery and canonical ctypes build behavior; replace the stale controller config mirror with a canonical-header helper. Filter INT8 candidates before pool admission and extend the real-principal regression. Synchronize API/setup docs and record the complete 254-target run: 253 passed; CodeGraph remains blocked by named AF_UNIX socket creation returning EPERM. All 74 gold workloads passed; 15 optional DIRECTEYE corpus cases were skipped.
SWORDIntel
added a commit
that referenced
this pull request
Nov 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The original scalar Hilbert/RFF lookup could return a candidate without checking the original record. A controlled reproduction returned index 2 for all 32 present queries (only 1/32 matched) and for absent query 999. Query features did not determine the result, and the smoke test counted non-sentinel results as accuracy.
This PR repairs that path and the related numerical defects, then adds selectable core methods and a reproducible benchmark so future optimizations can be judged against typed exact search.
Implementation
Selectable core methods
The additive, size/version checked
qihse_search_exandqihse_batch_search_exAPIs return status separately from result indices. They preserve existingqihse_config_tand kernel layouts.QIHSE_SEARCH_EXACTQIHSE_SEARCH_RFF_COSINEQIHSE_SEARCH_RFF_PAIREDOptions select query tiles of 1–64. A batch constructs one basis and shares row projections within each tile; legacy batch calls use phase cosine with tile 16. Scratch is O(TD), independent of record count; dot products remain O(nQD).
Per-call metrics report candidate hits, fallback work, successful/failed projections, original-record comparisons, basis builds, active tiles, scratch and stage times. Failure injection checks allocations and clocks. No benchmark winner is chosen automatically.
The additive opaque
qihse_rff_map_towns its method, preventing constructor/projector mismatch. Its descriptor includes basis version, method, dimensions, gamma and seed; legacy phase outputs are checked bit-for-bit.Measured reference
The checked-in reference and source manifest cover 45 comparisons across present, absent, duplicate, nearby-double and large-integer fixtures, with an independent equality oracle. Every case passed.
For 512 records, 32 present queries, 128 output coordinates, GCC 14.2.0/O3, one warmup and three timed repeats:
These are resident synthetic scalar lookups on an unpinned Linux x86-64 workspace, including setup and instrumentation. Exact scan wins these fixtures. Lower kernel approximation error does not guarantee exact candidate discrimination: the nearby-double fixture needed fallback for most present queries. This is not a VectorDB or production throughput claim.
The runner supports method, tile, dimensions, dataset, records, queries, repeats, seed, gamma and JSONL/CSV selection. A compiler-derived 19-file source/header SHA256 manifest identifies the measured dirty tree precisely.
Dependency and integration follow-up
LuaJIT development/runtime dependencies and the remaining native, Python and Rust test dependencies were supplied locally. The shared library, server and administrative executable now build.
lib-ctypesuse the canonical library build, avoiding two link recipes overwriting the same artifact; align launcher descriptions.Thread._stopname collision and close fixture pipes.Validation
Passed:
make lib-ctypesafter the gate: unchanged library hash, native Python extension import, expected CPython/search exports and ctypes fidelity 1.0 for identical vectors.LeakSanitizer is disabled because of the workspace tracing restriction; address and undefined-behaviour checks ran.
Final complete gate:
make test TEST_JOBS=32— 254 targets executed: 253 passed;test-codegraphfailed (command exit status 2).The library-backed candidate regression passed using real principals and cryptographic providers, including the INT8 and binary pool-of-one boundary. Controller validation executed all 61 tests with no skips, including all four native E2E cases. Browser unit/integration targets passed (41 unit and 36 integration checks). The gold pack completed 74 workloads in 11 areas with zero failures.
The PostgreSQL application-compatibility target ran 12 self-contained client/numeric tests successfully and skipped 15 cases that require the optional external DIRECTEYE source corpus. These are the same optional skips as the first complete run; they are not included in the controller's zero-skip result.
CodeGraph's named reader cannot start here: independent C and Python probes fail at
socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0)withEPERM, before bind. The aggregate target remains visibly failed. All 18 downstream recipe commands were also executed separately: 16 passed; query-index and runtime-spill reader tests hit the same socket boundary. The incremental suite within those commands ran 15 tests successfully.The validation follow-up records the complete outcome, repaired failures and environmental limitation. Original benchmark samples and their 19-file source manifest remain unchanged.
This remains a draft PR for reconciliation with ongoing local additions and the remaining named-reader run on a host that provides Unix sockets.
Compatibility and integration
Rebuild consumers against the canonical headers. Replace unsized float verification with
_sizedcalls. The original repair changes the RFF RNG/basis, so regenerate retained features from that older implementation. The additive paired method is explicit; do not mix its features with phase-cosine features.There is no stored database schema, .qdb, INT8 or binary format migration. Full dependency-complete gate:
make clean && make lib && make test TEST_JOBS=32.This stays on a separate PR branch for reconciliation with ongoing local additions. The complete repository audit and GPU/CUDA/AMX equivalence remain outside this focused CPU work.