From 1dc057b3d28ccbc5b27a72908f386f48e81b65fb Mon Sep 17 00:00:00 2001 From: Minsu Lee <1964421+amondnet@users.noreply.github.com> Date: Mon, 9 Nov 2026 06:37:59 +0900 Subject: [PATCH 1/2] chore: update agent memory --- .../please-please-code-explorer/MEMORY.md | 1 + .../project_csp_parity_findings_2026-10.md | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) create mode 100644 .claude/agent-memory/please-please-code-explorer/MEMORY.md create mode 100644 .claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md diff --git a/.claude/agent-memory/please-please-code-explorer/MEMORY.md b/.claude/agent-memory/please-please-code-explorer/MEMORY.md new file mode 100644 index 0000000..2c3c5c7 --- /dev/null +++ b/.claude/agent-memory/please-please-code-explorer/MEMORY.md @@ -0,0 +1 @@ +- [csp parity findings 2026-10](project_csp_parity_findings_2026-10.md) — which upstream semble #260-#295 behaviors csp already has vs real gaps (512-token truncation, serial chunking) diff --git a/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md b/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md new file mode 100644 index 0000000..c8d3ce0 --- /dev/null +++ b/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md @@ -0,0 +1,19 @@ +--- +name: csp-parity-findings-2026-10 +description: Non-obvious csp-vs-semble facts found in the 2026-10-14 parity pass (#260-#295); where csp already matches vs real gaps +metadata: + type: project +--- + +Facts verified by reading code on 2026-10-14 (upstream semble HEAD 0e5e028, #295): + +- csp `Model::encode` (crates/csp/src/indexing/dense.rs:107) calls model2vec-rs `encode()`, which hardcodes `max_length=Some(512)` + batch 1024. Upstream #264 passes `max_length=None` for chunk embedding only (queries stay truncated). Fix = `StaticModel::encode_with_args(texts, None, 1024)` in the chunk-embedding path only. +- csp embeds fresh rows in ONE batch (create.rs `embed_fresh_rows`) so the #260 double-embedding bug never existed. +- csp `chunk_source` (chunking/source.rs) already resolves line numbers in one forward pass (#266 is moot), but `byte_to_char` in chunking/core.rs:102 rescans `text[..b]` per boundary (quadratic on huge files; upstream has same pattern). +- csp has no direct rayon dep (only transitive via tokenizers/model2vec-rs); all file read+hash+chunk+tokenize is serial in `create_index_from_path` (#295 analog). +- Cache validity = sha256 over every source file on each `load_or_build_index` (cache_orchestrator.rs `source_fingerprint`) and again per-file in `create_index_from_path` on a stale rebuild (double read+hash). ADR-0005 accepts this cost. +- `Bm25Index::set_doc_order` (indexing/sparse.rs) is lenient; upstream #284 made it raise unless the order lists every indexed doc exactly once. +- csp `find_related` (indexing/index.rs:259) passes `selector=None`; upstream restricts to same-language chunks (since upstream #29). Unverified whether csp intends this. + +**Why:** avoids re-deriving these when triaging future upstream sync batches. +**How to apply:** re-verify file:line before citing; code may have moved. From 4bdd6190d0bfc96f715a2ed98309c7dbe7a4217b Mon Sep 17 00:00:00 2001 From: Minsu Lee <1964421+amondnet@users.noreply.github.com> Date: Mon, 9 Nov 2026 07:08:18 +0900 Subject: [PATCH 2/2] chore: apply AI code review suggestions --- .claude/agent-memory/please-please-code-explorer/MEMORY.md | 2 +- .../project_csp_parity_findings_2026-10.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.claude/agent-memory/please-please-code-explorer/MEMORY.md b/.claude/agent-memory/please-please-code-explorer/MEMORY.md index 2c3c5c7..a42c21b 100644 --- a/.claude/agent-memory/please-please-code-explorer/MEMORY.md +++ b/.claude/agent-memory/please-please-code-explorer/MEMORY.md @@ -1 +1 @@ -- [csp parity findings 2026-10](project_csp_parity_findings_2026-10.md) — which upstream semble #260-#295 behaviors csp already has vs real gaps (512-token truncation, serial chunking) +- [csp parity findings 2026-10](project_csp_parity_findings_2026-10.md) — which upstream semble #260-#295 behaviors csp already has vs real gaps (512-token chunk-embedding truncation, lenient BM25 doc order) diff --git a/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md b/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md index c8d3ce0..5ba24c3 100644 --- a/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md +++ b/.claude/agent-memory/please-please-code-explorer/project_csp_parity_findings_2026-10.md @@ -10,10 +10,10 @@ Facts verified by reading code on 2026-10-14 (upstream semble HEAD 0e5e028, #295 - csp `Model::encode` (crates/csp/src/indexing/dense.rs:107) calls model2vec-rs `encode()`, which hardcodes `max_length=Some(512)` + batch 1024. Upstream #264 passes `max_length=None` for chunk embedding only (queries stay truncated). Fix = `StaticModel::encode_with_args(texts, None, 1024)` in the chunk-embedding path only. - csp embeds fresh rows in ONE batch (create.rs `embed_fresh_rows`) so the #260 double-embedding bug never existed. - csp `chunk_source` (chunking/source.rs) already resolves line numbers in one forward pass (#266 is moot), but `byte_to_char` in chunking/core.rs:102 rescans `text[..b]` per boundary (quadratic on huge files; upstream has same pattern). -- csp has no direct rayon dep (only transitive via tokenizers/model2vec-rs); all file read+hash+chunk+tokenize is serial in `create_index_from_path` (#295 analog). +- Resolved (#126): csp has a direct rayon dep, and `create_index_from_path` (create.rs) parallelizes file read+hash+chunk+tokenize in `INDEX_BATCH_SIZE` batches, then applies index mutations serially in walk order (#295 analog). - Cache validity = sha256 over every source file on each `load_or_build_index` (cache_orchestrator.rs `source_fingerprint`) and again per-file in `create_index_from_path` on a stale rebuild (double read+hash). ADR-0005 accepts this cost. - `Bm25Index::set_doc_order` (indexing/sparse.rs) is lenient; upstream #284 made it raise unless the order lists every indexed doc exactly once. -- csp `find_related` (indexing/index.rs:259) passes `selector=None`; upstream restricts to same-language chunks (since upstream #29). Unverified whether csp intends this. +- Resolved (#123): csp `find_related` (indexing/index.rs) restricts candidates to the non-empty seed language and searches unrestricted for absent or empty languages, matching upstream #29. **Why:** avoids re-deriving these when triaging future upstream sync batches. **How to apply:** re-verify file:line before citing; code may have moved.