fix(search): bound workspace queries with cancellable multi-root snapshots

docs/workspace-search-design.md

@@ -0,0 +1,125 @@
+# Native workspace search
+
+Search surfaces never enumerate an entire workspace in the renderer. The
+desktop and HTTP commands rank natively and return at most 200 entries. The
+ranking tiers, UTF-16 offsets, deterministic tie order, ignore rules and
+authorized linked-directory behavior match the previous file search.
+
+## Discovery belongs to a query
+
+Mounting a composer or picker starts no discovery. The legacy prepare command
+is a compatibility no-op. Empty queries return a bounded traversal prefix using
+two listing permits, independently of index discovery and nonempty scoring.
+
+Nonempty queries share a process-wide `IndexService`. A short-held registry
+mutex protects an LRU of complete snapshots, in-progress discovery jobs, and
+temporary failures. Traversal, file I/O and scoring run on blocking workers
+outside this lock. Discovery has two permits, scoring four, and empty listings
+two: one large discovery cannot block a small-root discovery or an empty
+listing. Permit acquisition is cancellable and bounded by the query deadline.
+There are no background jobs competing with foreground requests.
+
+Queries for the same root attach to one discovery job. Each waiting query owns
+a lease. Closing one consumer removes that lease without interrupting another
+consumer; when the last leaves, the discovery cancellation token is signalled.
+Queued work exits without acquiring a permit. Running work checks cancellation
+between entries and never publishes a partial snapshot. A cancelled worker
+cannot publish over a newer job for the same root: publication checks the job
+identity under the registry lock. Dropping an async search also drops its lease.
+
+Consumer IDs and generations reject stale requests and handle cancellation that
+arrives before its query. The consumer registry and failure backoff map are
+bounded to 128 entries. A five-minute cooperative deadline covers admission,
+discovery and scoring; the transport allows 330 seconds for the response.
+
+## Multiple roots share total budgets
+
+The LRU holds at most eight complete roots. Keys use the canonical workspace
+path, current authorized link names/targets, and root/linked-root ignore-file
+metadata. Aliases share a snapshot. Authorization and root-ignore changes select
+a different snapshot immediately; a configuration change during discovery
+invalidates that build. Other filesystem changes become visible after a
+five-minute TTL or explicit refresh. Refresh publishes only after a complete
+successful discovery and does not move the timestamp on each query.
+
+Compact in-memory snapshots deduplicate basenames and store integer parent
+references. All roots share a 128 MiB retained-index budget. Each of the two
+discovery workers can use at most half the 192 MiB compact-builder budget,
+including its interning table. The cache reserves space before publication.
+Evicted snapshots still held by active queries remain charged until their final
+lease drops, so eviction cannot admit more than the total budget. When memory
+admission is unavailable, the complete disk snapshot remains usable instead.
+
+Scoring retains only top-k candidates. The compact scorer's basename memoization
+is limited to 4,096 entries per worker, rather than allocating one score for
+every unique name in every concurrent query. The limits describe index-owned
+allocations; filesystem walkers, OS page cache, thread stacks and the app's other
+subsystems have separate overhead and are not represented as a total RSS limit.
+
+## Oversized roots reuse a complete disk snapshot
+
+Discovery writes a length-prefixed binary snapshot in traversal order while
+optionally building the compact memory index. Once the compact budget is
+exhausted, it drops that builder and continues the same walk to disk. Every
+entry remains searchable, including late/deep exact matches. There is no
+full-workspace payload or fallback list allocated in the renderer.
+
+The disk format contains UTF-8 basename/path lengths, directory kind and their
+bytes. A query reads records in 64 KiB buffered pages and keeps only the best
+matches. Independent positional-read cursors allow concurrent queries on one
+snapshot without shared-seek races. No database dependency, migration, persistent
+index directory or full-path array is needed.
+
+All roots and active disk builds share a 2 GiB disk quota, with up to 1 GiB
+reserved for each discovery. Successful snapshots shrink their reservation to
+their actual bytes. LRU eviction releases ownership; active leases continue to
+count until close. A record is capped at 1 MiB. Quota or I/O failure yields an
+error and a five-second retry backoff. It never publishes an incomplete index or
+silently rescans the filesystem for each keystroke after a failed build.
+
+Snapshots use `tempfile::tempfile`: unlinked at creation on Unix and
+delete-on-close on Windows. Cancellation, construction errors, eviction and
+process exit close them automatically, including abrupt termination without an
+application-managed stale-file sweep. Successful compact-index publication
+closes the temporary disk copy as well.
+
+Disk scoring remains O(N) in snapshot records; initial discovery still walks
+the filesystem. This change removes repeated filesystem discovery within the
+TTL and bounds retained data, rather than claiming constant-time search in
+multi-million-entry workspaces. Incremental updates and a secondary text index
+can be evaluated separately using measured latency.
+
+## UI lifecycle
+
+Keyboard and pointer navigation pin the suggestion group. Selection follows
+reference identity across progressive updates, with a valid fallback after
+deletion. Delayed file results cannot turn a selected session into a file or
+invalidate Enter. Focus clears lazy session/commit caches for the next query
+while keeping the current in-flight search valid.
+
+Picker closure, root changes, newer typing and hook unmount abort queries.
+Frontend state retains only the current bounded results. There is no
+Array.at dependency beyond the project's ES2020 target.
+
+## Regression coverage
+
+`workspace_file_search/index_cache/tests.rs` uses independent services and
+controlled workers, including:
+
+- A blocked large-root discovery while a small-root query and empty listing
+  complete, plus empty listing with both discovery permits held.
+- Same-root coalescing, one of two consumers closing, last-consumer and queued
+  cancellation, dropped queries, deadlines, and cancelled old-generation
+  completion racing its successor.
+- A -> B -> A reuse, canonical aliases, authorization changes, ignore metadata,
+  TTL, refresh and LRU eviction.
+- Tiny forced memory budgets with complete late matches from disk, one walk
+  across repeated queries, and concurrent positional reads.
+- Memory and disk quotas with active evicted leases, disk overflow cleanup,
+  retry backoff, cancelled disk scoring, Unicode/path ranking and stable ties.
+
+The frontend tests cover bounded/cancelled transport requests, no eager
+prewarm, picker closure, delayed groups, selection reorder/deletion and focus.
+An ignored `large_workspace_memory_probe` can be run with
+`CODEG_MEMORY_WORKSPACE` to report counts, response bytes, memory/disk accounting,
+walk counts and query timings without printing workspace filenames.

src-tauri/src/commands/folders.rs

@@ -4872,7 +4872,7 @@ pub async fn list_directory_with_files(
 /// Only *authorized* links are returned (see [`crate::folder_links`]): a
 /// symlink that merely happens to sit in the tree — the `secrets -> ~/.ssh` a
 /// cloned repo might ship — is not one of them and stays unfollowed.
-fn authorized_links_in(root: &Path) -> Vec<(String, PathBuf)> {
+pub(super) fn authorized_links_in(root: &Path) -> Vec<(String, PathBuf)> {
     let Ok(canonical_root) = std::fs::canonicalize(root) else {
         return Vec::new();
     };
@@ -5174,16 +5174,24 @@ fn build_file_tree(
 }
 
 /// Flat, gitignore-aware listing of every file and directory under `path`, for
-/// fuzzy file search. Unlike [`get_file_tree`], ignored directories
-/// (`node_modules`, `target`, `dist`, …) are pruned *during* the walk, so no
-/// depth cap is needed: deep files stay reachable while the heavy trees are
-/// never descended and the payload stays small. Gitignore handling that used to
-/// run client-side now happens here at native speed in a single pass.
+/// compatibility with clients that need enumeration. Search surfaces use
+/// `workspace_file_search` so an unbounded listing never reaches the renderer.
 #[cfg_attr(feature = "tauri-runtime", tauri::command)]
 pub async fn list_workspace_files(
     path: String,
 ) -> Result<Vec<WorkspaceFileEntry>, AppCommandError> {
-    let root = PathBuf::from(&path);
+    let mut entries = Vec::new();
+    visit_workspace_files(&path, |entry| {
+        entries.push(entry);
+        true
+    });
+    Ok(entries)
+}
+
+/// Visit the search-visible entries without collecting the whole workspace.
+/// Returning false stops both the main walk and any authorized linked roots.
+pub(super) fn visit_workspace_files(path: &str, mut visit: impl FnMut(WorkspaceFileEntry) -> bool) {
+    let root = PathBuf::from(path);
     // Linked directories are walked separately below, rooted at the link so
     // their own `.gitignore` applies. Excluded from the main pass because the
     // walker reports a symlink as a leaf file.
@@ -5220,7 +5228,6 @@ pub async fn list_workspace_files(
         })
         .build();
 
-    let mut entries: Vec<WorkspaceFileEntry> = Vec::new();
     for result in walker {
         // Skip unreadable entries (permission errors, transient races) rather
         // than failing the whole search.
@@ -5244,36 +5251,44 @@ pub async fn list_workspace_files(
             .to_string_lossy()
             .replace('\\', "/");
 
-        entries.push(WorkspaceFileEntry {
+        if !visit(WorkspaceFileEntry {
             name,
             path: rel_path,
             kind: if is_dir {
                 WorkspaceEntryKind::Dir
             } else {
                 WorkspaceEntryKind::File
             },
-        });
+        }) {
+            return;
+        }
     }
 
     for (link_name, target) in linked {
-        entries.push(WorkspaceFileEntry {
+        if !visit(WorkspaceFileEntry {
             name: link_name.clone(),
             path: link_name.clone(),
             kind: WorkspaceEntryKind::Dir,
-        });
+        }) {
+            return;
+        }
         // Rooted at the resolved target so the linked project's own ignore
         // files apply, then re-prefixed with the link name so every path stays
         // relative to the workspace root and resolves back through the link.
-        entries.extend(list_files_under(&target, &link_name));
+        if !visit_files_under(&target, &link_name, &mut visit) {
+            return;
+        }
     }
-
-    Ok(entries)
 }
 
 /// Flat listing of `root`, with every path prefixed by `prefix/`. Shares the
 /// ignore configuration of [`list_workspace_files`]; nested symlinks are not
 /// followed, so this cannot recurse.
-fn list_files_under(root: &Path, prefix: &str) -> Vec<WorkspaceFileEntry> {
+fn visit_files_under(
+    root: &Path,
+    prefix: &str,
+    visit: &mut impl FnMut(WorkspaceFileEntry) -> bool,
+) -> bool {
     let walker = WalkBuilder::new(root)
         .hidden(false)
         .parents(false)
@@ -5295,7 +5310,6 @@ fn list_files_under(root: &Path, prefix: &str) -> Vec<WorkspaceFileEntry> {
         })
         .build();
 
-    let mut entries = Vec::new();
     for result in walker {
         let Ok(entry) = result else { continue };
         let entry_path = entry.path();
@@ -5308,17 +5322,19 @@ fn list_files_under(root: &Path, prefix: &str) -> Vec<WorkspaceFileEntry> {
         let is_dir = entry
             .file_type()
             .is_some_and(|t| entry_presents_as_dir(&t, entry_path));
-        entries.push(WorkspaceFileEntry {
+        if !visit(WorkspaceFileEntry {
             name: entry.file_name().to_string_lossy().to_string(),
             path: format!("{prefix}/{}", rel.to_string_lossy().replace('\\', "/")),
             kind: if is_dir {
                 WorkspaceEntryKind::Dir
             } else {
                 WorkspaceEntryKind::File
             },
-        });
+        }) {
+            return false;
+        }
     }
-    entries
+    true
 }
 
 #[cfg_attr(feature = "tauri-runtime", tauri::command)]

src-tauri/src/commands/mod.rs

@@ -83,6 +83,7 @@ pub mod version_control;
 #[cfg(feature = "tauri-runtime")]
 pub mod windows;
 pub mod work_task;
+pub mod workspace_file_search;
 pub mod workspace_state;
 #[cfg(feature = "tauri-runtime")]
 pub mod workspace_windows;