fix: an unmodeled diff path degrades to every journey; verify's UNKNOWN NONE is not a pass (#117, #113)

README.md

@@ -13,7 +13,7 @@ python3 -m testgraph.verify --repo . --base origin/main -- tests/ -q
 # exit 0  every selected journey ran and passed
 # exit 1  a journey's tests failed
 # exit 2  refused: the index can't be trusted
-# exit 3  a selected journey has no journey-level test — named, not hidden
+# exit 3  a selected journey has no journey-level test, or the selection is UNKNOWN — named, not hidden
 ```
 
 That last case is not hypothetical. Run on itself, testgraph found that the 20
@@ -176,7 +176,10 @@ It runs `codegraph sync` first. Seeds come from *line ranges*, so an index built
 before the code moved resolves the diff against stale spans, and nothing else
 refreshes these indexes on commit. If a changed file's bytes still disagree with the
 indexed copy afterwards, the answer degrades to `RECALL DEGRADED` and names the file
-instead of quietly trusting the wrong line numbers.
+instead of quietly trusting the wrong line numbers. So does a change to a file the
+index does not model at all — SQL, a `Dockerfile`, CI YAML, CSS, config — unless it is
+a test, docs or a type declaration: not indexed is not "no impact" (TECHNICAL.md
+"Unmodeled Paths Degrade Too").
 
 Each run appends one `selection` row to `~/.local/share/testgraph/ledger.jsonl`
 (`{kind, ts, repo, base, head, commit, status, n_journeys, journey_ids, duration_ms,

TECHNICAL.md

@@ -69,7 +69,7 @@ leaving edges wrong) — only a full `codegraph index` did. Three checks:
 - `testgraph/registry.py` — loads the journey JSON, resolves entry `name`+`file`
   to node ids at run time (ids are stable, names drift), maps every matching
   node to its journey.
-- `testgraph/select.py` — diff parsing (test/e2e files excluded as seeds),
+- `testgraph/select.py` — diff parsing (test/e2e/docs files exempt, unmodeled paths degrade — #117),
   orchestration, ranking, human + `--json` output, the CLI.
 - `testgraph/propose.py` — drafts a registry for a repo that has none: ast scan
   for route-decorated handlers, index resolution, spot-check derivation, declared
@@ -649,6 +649,54 @@ byte-identical with the degrade reverted, because no labeled commit exercises th
 zero-seed path (`0b4135f` already degraded via a whole-file change). The guard is
 justified by the reproduced failure in #29, not by a metric move.
 
+## Unmodeled Paths Degrade Too (issue #117)
+
+The parser used to drop every path outside `PRODUCT_EXT` before anything looked
+at it, so an external trial's migration-only diff reported **0 changed files, 0
+journeys and no warning**. A SQL migration, a `Dockerfile`, CI YAML or a
+stylesheet can change runtime behaviour without ever becoming a CodeGraph node;
+"not indexed" is not "no impact".
+
+Every path in the diff is now accounted for before filtering
+(`_parse_unified_diff(diff, paths)`: both sides of a rename, the `---` side of a
+deletion, binary and mode-only blocks) and lands in one of three buckets on the
+result:
+
+| bucket | field | effect |
+|---|---|---|
+| product (`_is_product`) | `changed_files` | seeded as before |
+| exempt | `exempt_files` `{path: reason}` | none; reason is `test` (under a `tests/`, `e2e/` or `__tests__/` directory — not `_is_test`'s name rules, which exempted `test_and_deploy.yml`), `type declaration` (`.d.ts` family) or `docs` (`.md`/`.rst`/`.adoc`, a top-level `docs/` or `doc/`, or exactly `LICENSE`/`COPYING`/`CHANGELOG`/`NOTICE`/`AUTHORS` with no extension or a prose one — a prefix match exempted `db/authors.sql`) |
+| unmodeled (anything else) | `unmodeled_files` | joins `unmapped`: warns, `recall_degraded: true`, every journey listed `verify_manually` |
+
+`diff_files` lists every raw path, and the text renderer prints
+`changed files: N product of M in diff (E exempt, U unmodeled)` plus one
+`unmodeled:` line per path. The hook gives unmodeled paths their own capped NOTE
+line, like `closure_confined`, so the capped warnings cannot hide them.
+
+All-journeys rather than UNKNOWN-only was chosen deliberately. UNKNOWN exists
+only for an EMPTY list, so a mixed product + unmodeled diff would have needed a
+new "partial" signal in select, the hook, MCP and verify; the #29 fallback
+already covers all four, and verify then runs every journey and exits 3 unless
+each one passes. The exemption list is short on purpose: `requirements.txt`,
+`package.json`, `.dockerignore` and the like all change what runs.
+
+The cost, measured on the last 40 first-parent commits of each dogfooded repo:
+22/40 (testgraph), 16/36 (honeyslate) and 21/36 (signedintake) touch an
+unmodeled path, mostly `.json`; 12, 4 and 7 of them touch nothing else, and those
+used to answer NONE. A per-registry exemption list is the lever if that noise
+bites; it is not built.
+
+verify follows the same rule from the other side: an empty selection whose
+`unknown_because` is non-empty prints `journeys selected: NONE — but UNKNOWN, not
+verified-safe:` as its headline (#113) and exits 3, not 0. UNKNOWN is not a pass.
+One reason is left out of the exit decision: an entry no parser covers
+(`entries_unchecked`) never clears, so counting it would hold every NONE in such a
+repo at exit 3 for good. The headline still lists it.
+
+`git diff` also runs with `--submodule=short --ignore-submodules=none`: with
+`diff.ignoreSubmodules`, a `.gitmodules` `ignore`, or `diff.submodule=log`, a
+submodule bump produced no `diff --git` block and answered a clean NONE.
+
 ## Re-verification After the Extractor Fix (2026-09-09)
 
 Both harnesses were re-run after `codegraph` was rebuilt at extraction version

skills/testgraph-verify/SKILL.md

@@ -56,6 +56,7 @@ below means *unknown*, never *none*:
 | You **added** a `.py` function or file | Run `select` (below). A new symbol has no row by definition. |
 | You **deleted** or **renamed** a `.py` file | Run `select`. If the file is still in the index it seeds that file's symbols precisely and returns a *narrow* list — correct, not a malfunction. Only when the file has already left the index does it report unbounded impact and list every journey. |
 | You changed **any non-`.py` file** (`.js`, `.jsx`, `.ts`, `.tsx`, `.svelte`, `.vue`, …) | Run `select` — it seeds these paths (issue #21; it used to answer `NONE` for all of them). If your extension is not in the index it prints `RECALL DEGRADED` and lists every journey: that is *unknown*, not none. A journey reached only through frontend code often arrives on a weak edge — expect `!` and verify by hand. |
+| You changed a file the index does not model — SQL, `Dockerfile`, CI YAML, CSS, JSON/toml config, anything that is not code, a test, docs or a `.d.ts` | Run `select`. It names the path as `unmodeled:`, prints `RECALL DEGRADED` and lists every journey (issue #117): its impact is *unknown*, not none. The map can never answer for it. |
 | The file has no `###` section at all | Unknown. The indexer may not cover it. Escalate and say so. |
 | `generated from commit` is far behind HEAD | The map under-reports. Regenerate (below) or escalate. |
 | The header says the **registry was not human-approved** | A different failure from the one below, with a different fix. The index may be fine; what is unverified is whether the registry lists every journey this app has. Absence of a symbol from every journey is *unknown*, not *unaffected*. **Do not re-index** — it cannot clear this. Escalate, or review and approve the registry (`skills/testgraph-propose`). |

testgraph/hook.py

@@ -109,6 +109,15 @@ def render(result, repo, more_cmd=None):
             f"  NOTE: impact for {shown} did not leave the file it started "
             f"in — that file's own contribution is UNKNOWN, not verified-safe"
         )
+    # Same reasoning as the confined line: a path the index does not model
+    # forced the all-journeys fallback (issue #117), and the reader should see
+    # which one without depending on the capped warnings below.
+    unmodeled = result.get("unmodeled_files", [])
+    if unmodeled:
+        shown = ", ".join(unmodeled[:MAX_CONFINED])
+        if len(unmodeled) > MAX_CONFINED:
+            shown += f", … {len(unmodeled) - MAX_CONFINED} more"
+        lines.append(f"  NOTE: not modeled by the index: {shown} — impact UNKNOWN")
     # Warnings are the channel that carries an unapproved registry and entry
     # drift — both mean "this answer may be understated", so they are worth the
     # lines. Capped: an unbounded warning block is the noise problem again.

testgraph/select.py

@@ -1,8 +1,9 @@
 """testgraph select — git diff -> ranked affected journeys.
 
 Recall-first: the closure over-selects on shared-symbol changes rather than
-risk dropping a truly-affected journey. Test/e2e file changes are ignored as
-seeds (they don't change product behavior).
+risk dropping a truly-affected journey. Test, docs and type-declaration changes
+are exempt (they don't change product behavior); any other path the index does
+not model degrades the answer to every journey (issue #117).
 
 Usage:
     python -m testgraph.select --repo <path> [--base SHA] [--head SHA]
@@ -46,6 +47,18 @@
 # directory entirely, so root-level test files were seeded as product code.
 TEST_DIRS = frozenset({"tests", "e2e", "__tests__"})
 
+# Paths exempt from the unmodeled-path fallback (issue #117) because they cannot
+# change runtime behavior: prose, by extension, by a top-level docs directory,
+# or by the conventional file names — matched EXACTLY (optionally with a prose
+# extension), because a prefix match exempted `db/authors.sql` and
+# `config/licenses.json`. Deliberately short. Everything else outside
+# PRODUCT_EXT -- SQL, Dockerfile, CI YAML, CSS, JSON, toml, requirements.txt,
+# anything unrecognized -- is UNKNOWN impact, not "safe".
+DOC_EXT = (".md", ".rst", ".adoc")
+DOC_DIRS = frozenset({"docs", "doc"})
+DOC_BASENAMES = frozenset({"LICENSE", "COPYING", "CHANGELOG", "NOTICE", "AUTHORS"})
+DOC_BASENAME_EXT = ("", ".txt", ".md", ".rst", ".adoc")
+
 
 def _is_product(path):
     return (
@@ -55,6 +68,30 @@ def _is_product(path):
     )
 
 
+def _exemption(path):
+    """Why a non-product path cannot change behavior, or None when it might.
+
+    Only a test DIRECTORY exempts a non-code file. `_is_test`'s name rules
+    (`test_*`, `*.test.*`, `*.spec.*`) are conventions for code; on other files
+    they exempted `test_and_deploy.yml` and `openapi.spec.yaml`."""
+    parts = path.split("/")
+    if TEST_DIRS.intersection(parts[:-1]):
+        return "test"
+    if path.endswith(DECLARATION_EXT):
+        return "type declaration"
+    stem, ext = os.path.splitext(parts[-1])
+    if (path.endswith(DOC_EXT) or (len(parts) > 1 and parts[0] in DOC_DIRS)
+            or (stem.upper() in DOC_BASENAMES and ext.lower() in DOC_BASENAME_EXT)):
+        return "docs"
+    return None
+
+
+def _unmodeled_kind(path):
+    base = path.rsplit("/", 1)[-1]
+    ext = os.path.splitext(base)[1]
+    return f"{ext} file" if ext else base
+
+
 def _is_test(path):
     parts = path.split("/")
     base = parts[-1]
@@ -126,7 +163,7 @@ def _diff_git_path(rest):
     return None
 
 
-def _parse_unified_diff(diff):
+def _parse_unified_diff(diff, paths=None):
     """Parse a `git diff --unified=0` into
     `(ranges, whole_files)` where
 
@@ -152,11 +189,21 @@ def _parse_unified_diff(diff):
     unknown, not absent. The one exception is a block git marks as a new or
     deleted file without a header, which is an empty file — no content to seed,
     and a mode-only change (`old mode`/`new mode`, no '+++'), which is a chmod.
+
+    `paths`, when given, is a set that collects EVERY path in the diff, product
+    or not, before any filtering (issue #117): the caller accounts for each one
+    instead of inferring "nothing changed" from what survived the filter.
     """
     ranges, whole_files, cur, prev = {}, {}, None, None
     blk = None  # state of the current `diff --git` block
 
+    def note(path):
+        if paths is not None and path:
+            paths.add(path)
+
     def close_block():
+        if blk:
+            note(blk["path"])
         if (blk and not blk["header"] and not blk["rename"] and not blk["empty"]
                 and not blk["mode"]):
             path = blk["path"]
@@ -183,12 +230,14 @@ def close_block():
                 blk["empty"] = True
         elif line.startswith("rename from "):
             p = _unquote_git_path(line[len("rename from "):])
+            note(p)
             if blk:
                 blk["rename"] = True
             if _is_product(p):
                 whole_files[p] = "renamed from"
         elif line.startswith("rename to "):
             p = _unquote_git_path(line[len("rename to "):])
+            note(p)
             if blk:
                 blk["rename"] = True
             if _is_product(p):
@@ -202,12 +251,14 @@ def close_block():
                 blk["header"] = True
             if path == "/dev/null":
                 # whole-file deletion: the surviving path is on the '---' line
+                note(prev)
                 if prev and _is_product(prev):
                     whole_files[prev] = "deleted"
                 cur = None
                 continue
             if path.startswith("b/"):
                 path = path[2:]
+            note(path)
             cur = path if _is_product(path) else None
             if cur is not None:
                 ranges.setdefault(cur, [])
@@ -228,7 +279,7 @@ def close_block():
     return {f: r for f, r in ranges.items() if r}, whole_files
 
 
-def changed_ranges(repo, base, head):
+def changed_ranges(repo, base, head, paths=None):
     diff = subprocess.run(
         # Every flag pins something a user's git config could otherwise change
         # under us (audit C1): diff.noprefix / mnemonicPrefix drop the `b/` the
@@ -241,14 +292,19 @@ def changed_ranges(repo, base, head):
         # --end-of-options: `base`/`head` are never parsed as git options (M2).
         ["git", "-C", repo, "-c", "core.quotePath=false", "diff", "--no-color",
          "--no-ext-diff", "--no-textconv", "--src-prefix=a/", "--dst-prefix=b/",
-         "--unified=0", "-M", "--end-of-options", f"{base}..{head}"],
+         "--unified=0", "-M",
+         # diff.ignoreSubmodules, a .gitmodules `ignore`, or diff.submodule=log
+         # drop or reshape a submodule bump so no `diff --git` block names it
+         # (#117 review): pin both so the path is always accounted for.
+         "--submodule=short", "--ignore-submodules=none",
+         "--end-of-options", f"{base}..{head}"],
         capture_output=True,
         text=True,
         encoding="utf-8",
         errors="surrogateescape",
         check=True,
     ).stdout
-    return _parse_unified_diff(diff)
+    return _parse_unified_diff(diff, paths)
 
 
 def select(repo, base, head, db_path, registry_path, strict_registry=True):
@@ -335,7 +391,8 @@ def select(repo, base, head, db_path, registry_path, strict_registry=True):
             result["blocking"] = blocking
             return result
 
-        ranges, whole_files = changed_ranges(repo, base, head)
+        diff_paths = set()
+        ranges, whole_files = changed_ranges(repo, base, head, diff_paths)
 
         # A changed file whose hunks resolve to NO node is the same failure as an
         # unmappable whole-file change, and it used to pass silently: seeds stayed
@@ -391,6 +448,23 @@ def _mark_unmapped(path, detail):
             else:
                 _mark_unmapped(path, reason)
 
+        # A changed path the index does not model at all — SQL, a Dockerfile, CI
+        # YAML, CSS — never becomes a seed, and used to vanish in the parser: a
+        # migration-only diff answered "0 changed files, NONE" with no warning
+        # (issue #117). Not indexed is not "no impact". Unless the path is
+        # exempt (tests, docs, type declarations), it degrades the answer to
+        # every journey, exactly like a code file with no symbols (#29).
+        exempt_files, unmodeled_files = {}, []
+        for path in sorted(diff_paths):
+            if _is_product(path):
+                continue
+            why = _exemption(path)
+            if why:
+                exempt_files[path] = why
+            else:
+                unmodeled_files.append(path)
+                _mark_unmapped(path, f"not modeled by the index: {_unmodeled_kind(path)}")
+
         # A changed file that is NEWER than its index row is a third way to be
         # unmappable, and the quietest. The other two resolve to no node and are
         # obvious; this one resolves to the WRONG node. Seeds come from line ranges
@@ -541,6 +615,9 @@ def _mark_unmapped(path, detail):
         result.update(
             status="OK",
             changed_files=sorted(ranges),
+            diff_files=sorted(diff_paths),
+            exempt_files=exempt_files,
+            unmodeled_files=unmodeled_files,
             whole_file_changes=whole_files,
             recall_degraded=bool(unmapped),
             closure_confined=confined_files,
@@ -623,12 +700,21 @@ def _render(result):
         for b in result["blocking"]:
             lines.append(f"  x {b}")
         return "\n".join(lines)
+    changed = f"changed files: {len(result['changed_files'])}"
+    if "diff_files" in result:
+        changed += (
+            f" product of {len(result['diff_files'])} in diff "
+            f"({len(result.get('exempt_files', {}))} exempt, "
+            f"{len(result.get('unmodeled_files', []))} unmodeled)"
+        )
     lines.append(
-        f"changed files: {len(result['changed_files'])} | "
+        f"{changed} | "
         f"seeds: {result['seed_symbols']} | impacted symbols: {result['impacted_symbols']}"
     )
     for path, reason in sorted(result.get("whole_file_changes", {}).items()):
         lines.append(f"  whole-file: {path} ({reason})")
+    for path in result.get("unmodeled_files", []):
+        lines.append(f"  unmodeled: {path} — not in the index, impact UNKNOWN")
     if result.get("recall_degraded"):
         lines.append("  RECALL DEGRADED — unbounded impact, all journeys listed")
     for f in result.get("closure_confined", []):