Skip to content

Option to index git submodules as part of the parent project #4593

Closed

Description

@goughjo02

Summary

Since 836c366 ("prune nested repositories from the parent source walk"), the source walker skips every non-root directory that contains a .git entry. That fixed a real problem (linked worktrees inside a checkout being indexed many times over), but it also drops git submodules from the parent project's index, with no setting to bring them back.

I'd like an opt-in way to index submodules as part of the parent, while still skipping linked worktrees and unrelated nested clones.

Use case

A "super repo" that pulls in the main application as a submodule:

super-repo/            <- where I start every agent session
  .gitmodules
  dev/                 <- scripts that launch and wrap the app
  docs/
  app/                 <- git submodule, ~1,000 tracked files

I always start agent sessions from super-repo/. Today:

  • The super-repo index covers only its own files; nothing under app/ is indexed, and nothing reports that it was skipped.
  • To search the app I have to tracedecay init it as a separate project and pass project_selector on every call, or use multi_root_* scope sets.
  • Even then there are no call or import edges between the two, so a path from dev/ into app/ never appears as one chain in callers / impact / call_chain.

For this layout the submodule is effectively part of the project, and one index is what I want.

Telling submodules apart from worktrees

With current git, both a linked worktree and a checked-out submodule have a .git file, not a directory, so the presence check can't distinguish them. Some signals that do:

  • The gitdir: target. A linked worktree points to <common-dir>/worktrees/<name>; a submodule points to <parent>/.git/modules/<name>.
  • The parent's index. A submodule is a gitlink entry (mode 160000) at that path, and is usually listed in .gitmodules.
  • Linked worktrees also show up in git worktree list for the parent.

Possible shape

Something like an index setting (name up to you):

[index]
include_submodules = true   # default false keeps today's behaviour

When enabled, the walker would descend into a directory only if it is a registered submodule of the root (gitlink plus .gitmodules entry), and keep pruning linked worktrees and other nested repositories exactly as now.

Related

  • Because grep, ast-grep, and unmounted-files share this walker, they also skip submodule contents today.
  • It would help if status or unmounted_files reported "skipped nested repository: app/", so users know why that code is missing.

Environment

  • tracedecay 1.0.0-beta.91, macOS, Claude Code plugin install
  • git 2.54

Activity

  1. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    Owner

    Implementation investigation (2026-11-08 UTC), Refs #4593 — issue remains open; no fix is claimed.

    A walker-only include_submodules switch cannot satisfy this issue without silently disagreeing with the indexed graph. I checked master at 91c72aba7750a398dd28e83d64bf0c6d6fef4172, root AGENTS.md (no nested AGENTS entries in the complete tree), the issue (zero comments before this investigation), both currently open PRs, and prior closed/unmerged #4719 (which explicitly withdrew a walker-only implementation).

    The additional integration points that must change together are:

    • crates/tracedecay-code-index/src/source_walk.rs:167-176 excludes every nonroot .git directory; live grep, ast-grep, and unmounted-file auditing reuse this.
    • crates/tracedecay-code-index-runtime/src/code_index_scheduler/classification.rs builds candidates from the parent Git index and explicitly selects .index_worktree_submodules(None). The parent indexes the gitlink path, not the child source files.
    • crates/tracedecay-code-index-runtime/src/code_index_scheduler/git_tree_capture.rs explicitly skips entry.mode.is_commit(), so the clean-HEAD/exact-tree fast path drops gitlinks. reconcile.rs:4011+ uses that fast path to publish clean generations; merely changing the filesystem walker cannot affect the index.
    • freshness_witness.rs::source_candidates derives its roster from the parent classification, and identity.rs::GitMetadataFingerprintV1 samples only the parent Git metadata. A changed child revision or same-size content rewrite could otherwise remain falsely fresh.

    An independent disposable native Git probe executed five nonvacuous assertions: the parent records 160000 ... app but not app/feature.rs; an initialized child has .git -> ../.git/modules/app; a linked worktree has a different .git -> .../worktrees/... target; and the child file really exists despite absence from the parent index. This is Git/filesystem behavior, not repository Rust/Bazel, CI, or native macOS validation.

    Acceptance needs a single explicit, restart-applied opt-in from the canonical project configuration; Git-index gitlink plus .gitmodules membership and bound child repository/worktree identity; consistent nested-source admission, sanitized capture, sealed graph/retrieval, and both clean and dirty snapshot authorities; freshness invalidation on child changes and removals; fail-closed linked-worktree/untrusted clone/metadata substitution handling. Tests must exercise default-deny, opt-in parent→child call edges and grep/AST parity, commit/dirty change, restart/reuse, removal, denied/nested clone, and macOS/native behavior. No secondary walker-only authority.

    No production changes or PR were published: the environment providing this investigation has no repository checkout or Bazel/native macOS toolchain, so it cannot legitimately compile, run nonzero repository tests, or establish a safe completed implementation. The issue-specific owner remains active; this comment is a concrete implementation boundary, not a closure recommendation.

  2. 7 remaining items

  3. ScriptedAlchemy commented on Nov 16, 2026

    @ScriptedAlchemy
    Owner

    Closing as completed. #4801 added the submodule opt-in, and #4905 adds tests that pin its boundaries.

    How each request is met:

    • Opt-in setting. index.include_submodules.v1 defaults to false and takes effect after a daemon restart. With the default, submodule contents stay out of the index, as before.
    • Registered submodules only. The walker descends into a child only when the parent's staged index has a gitlink at that path for the pinned commit and .gitmodules registers it. Linked worktrees, unrelated nested clones, deeper gitlinks, and uninitialized children stay excluded. test(code-index): pin submodule opt-in boundaries #4905 adds a test that creates a linked worktree inside the parent with the opt-in on. The worktree stays pruned and app/first.rs is still walked.
    • One index for the super repo. With the opt-in on, child files are indexed under the parent with parent-relative paths, so parent-to-child call edges are in one graph. Child commits, additions, edits, and removals invalidate freshness.
    • grep, ast-grep, unmounted files. All three share the walker and follow the same setting. test(code-index): pin submodule opt-in boundaries #4905 adds grep and ast-grep tests on one fixture. By default each finds parent.rs and nothing under app/, so the default check can't pass on an empty result. With the opt-in on, each finds app/first.rs. The MCP grep tool searches the resolved project's root with that project's own path policy.
    • Report skipped code. unmounted_files adds a "Skipped nested repositories" note listing the skipped paths (EcosystemAudit.note). The issue asked for this in status or unmounted_files. It is in unmounted_files.

    Unit suites run locally on the #4905 change: code-index 415 passed, configuration 41 passed, code-index-runtime 746 passed with 2 ignored. The three new tests passed at bec33fbc. The merged head b8212e8 differs from that commit only in rustfmt line wrapping, and all required checks passed on it.

    This issue set no benchmark or remote performance criterion. None was added, and none is claimed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions