perf(extract): resolve and parse each path once in the id remap and call tie-break by rohit-jsfreaky · Pull Request #4195 · Graphify-Labs/graphify

rohit-jsfreaky · 2026-10-07T11:46:29Z

What does this PR do?

Fixes #4194.

extract(): the id remap now resolves each distinct path once through a local memo (_resolved), instead of once per input, per target_file-stamped edge and per node. A hit returns the same Path the call would; a path that raises is never stored, so every existing try/except behaves as before. The memo lives for one extract() call only.
paths._path_proximity_winner: the parent-directory parse of each candidate path is memoized (_parent_parts, functools.lru_cache(maxsize=65536)). Equal part-tuples are exactly equal PurePosixPath(...).parent values, so the same-file, same-directory and longest-common-prefix tiers decide as before.

No other code changed; #3500 already memoized the symbol-resolution passes, this covers what it did not touch.

Type of change

(performance; no behaviour change)

Verification & Invariants

Invariant: graph.json is identical; only the wall time changes.

graphify's own repo (18,830 nodes), python -m graphify update <repo> --force (clustered), v8 and this PR alternated:

run	v8 `f765dcb`	this PR	saved
cold build	28.1 s	26.3 s	1.8 s
no-change update	24.8 s	22.5 s	2.3 s
no-change update	20.9 s	18.0 s	2.9 s

graph.json identical between v8 and this PR (all node and edge attributes), cold and warm.

Work removed: Path.resolve() from extract()'s remap 23,819 calls -> one per distinct path (~1k on graphify's repo); PurePosixPath objects from the tie-break 226,809 per update -> one parse per distinct candidate path.

Old vs new _path_proximity_winner and disambiguate_ambiguous_candidates compared on 200,000 random cases (graphify's file list plus backslash, absolute, root-level, .., ./ and empty paths): 0 differences.

Read the CONTRIBUTING.md guide.
Reproduced the issue and identified the invariant.
Made the smallest fix necessary.
Added a regression test (if bug fix) or isolated boundary test.
Kept the PR description synchronized with the final implementation.
Documented any limitations / unsupported cases explicitly.

How was this tested?

pytest tests/test_extract_path_memo.py         # 7 passed; the resolve-count test fails on f765dcb
                                               # (204 remap resolve calls for 7 paths, now 1 per path);
                                               # 6 tie-break equality cases pass on both
pytest tests/test_paths.py tests/test_extract.py   # 336 passed, 1 failed: test_python_external_calls_survive_real_incremental_context,
                                               # which also fails on clean v8 (Windows)
pytest tests -q                                # 6679 passed, 18 failed: every one also fails on clean v8 (0 new failures)
ruff check graphify/extract.py graphify/paths.py tests/test_extract_path_memo.py   # All checks passed
pyright graphify/extract.py graphify/paths.py  # 120 errors, same as v8
bandit graphify/extract.py graphify/paths.py   # same findings as v8

Graphify-specific checklist

I updated generated skill artifacts (uv run python -m tools.skillgen --bless) when changing their source fragments. (n/a, no skill sources touched)
I confirmed that AST/structural extraction remains deterministic (no ambient state dependencies like ENV variables).
I reviewed changes for security implications (no unsafe interpolation into shell/Python).
I confirmed no API keys or local-only graph data are included.
(If applicable) I disclosed AI authorship in my commit messages.

🤖 Generated with Claude Code

…all tie-break extract()'s id remap called Path.resolve() on the same file paths once per input, once per target_file-stamped edge and once per node: ~24k calls for ~1k distinct paths on graphify's own repo. Resolve each distinct path once per extract() call through a local memo; a hit returns the same Path the call would, and a path that raises is never stored. _path_proximity_winner (the Graphify-Labs#1553 god-node tie-break) re-parsed every candidate's parent directory with PurePosixPath on every ambiguous call site (~227k Path objects per update on graphify's repo), although the same candidate files come back for every site of a common name. Memoize the parent-directory parts per path string; equal tuples are exactly equal PurePosixPath parents, so every tier decides as before. graphify's own repo (18,830 nodes), `graphify update --force`, alternated: cold 28.1 -> 26.3 s, no-change update 24.8 -> 22.5 s and 20.9 -> 18.0 s. graph.json identical (all attributes) cold and warm; old vs new tie-break identical on 200,000 random cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-07T11:46:37Z

Thanks for the pull request, @rohit-jsfreaky. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

graphify-labs

Graphify reviewed this change.

Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).

Formal verification. PR-changed functions: 1/2 verified (0 proven, 1 may-equivalent, 0 distinguished) · 1 not verified (1 vacuous).

Not verified on this run: extract (vacuous: never exercised).

Graphify review — findings

Memoizes path resolution in extract's id-remap pass so each distinct file path is resolved once per call via a local _resolved helper, instead of once per input, stamped edge and node; paths that raise are never cached, so they keep falling through to the existing except branches. _path_proximity_winner now gets parent-directory segments from an lru_cached _parent_parts (backslashes normalized). That stops re-parsing the same candidate directories on every ambiguous call site, and the chosen winners stay the same.

No blocking issues surfaced.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 3563 functions depend on the 313 functions this change touches.

Health — this change adds coupling hotspots:

new: extract() — 787 callers, 51 callees
new: _rebuild_code() — 151 callers, 56 callees
new: build_from_json() — 233 callers, 20 callees
new: detect() — 121 callers, 16 callees
new: build_merge() — 85 callers, 14 callees
new: save_semantic_cache() — 65 callers, 9 callees
new: to_obsidian() — 41 callers, 14 callees
new: save_manifest() — 44 callers, 13 callees
…and 98 more — each is listed as a finding

Verification — 3563 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

verification_scope: 2939 function(s) in the blast radius were not formally verified this run

Test selection

247 of 338 test file(s) selected (73%) via static blast radius.

tests/test_affected_cli.py — impact
tests/test_agents_platform.py — impact
tests/test_analyze.py — impact
tests/test_astro_extraction.py — impact
tests/test_astro_import_ids.py — impact
tests/test_atomic_canvas_export.py — impact
tests/test_atomic_version_stamp.py — impact
tests/test_atomic_writes.py — impact
tests/test_benchmark.py — impact
tests/test_benchmark_raw_graph.py — impact
tests/test_blade_extractor.py — impact
tests/test_build.py — impact
tests/test_build_located_semantic_identity.py — impact
tests/test_build_merge_dedup_scope.py — impact
tests/test_build_merge_hyperedges_and_prune.py — impact
tests/test_build_merge_shrink_guard.py — impact
tests/test_builtin_global_type_refs.py — impact
tests/test_cache.py — impact
tests/test_callflow_html.py — impact
tests/test_carried_hyperedge_remap.py — impact
tests/test_case_sensitive_resolution.py — impact
tests/test_charmap_encoding.py — impact
tests/test_chunking.py — impact
tests/test_cjs_module_extension.py — impact
tests/test_cli_export.py — impact
tests/test_cluster.py — impact
tests/test_cobol_extractor.py — impact
tests/test_codebuddy.py — impact
tests/test_community_labels_skill.py — impact
tests/test_confidence.py — impact
tests/test_corrupt_graph_json.py — impact
tests/test_cpp_method_declarations.py — impact
tests/test_cpp_nested_and_cli.py — impact
tests/test_cpp_objc_cross_file_calls.py — impact
tests/test_cross_extension_reexport_self_cycle.py — impact
tests/test_cross_language_call_resolution.py — impact
tests/test_cross_repo_external_call_guards.py — impact
tests/test_cross_repo_member_calls.py — impact
tests/test_csharp_call_site_generic_args.py — impact
tests/test_csharp_enum_members.py — impact
tests/test_csharp_field_generic_args.py — impact
tests/test_csharp_generic_callsites.py — impact
tests/test_csharp_interface_dispatch.py — impact
tests/test_csharp_member_calls.py — impact
tests/test_csharp_member_nodes.py — impact
tests/test_csharp_object_creation.py — impact
tests/test_csharp_partial_classes.py — impact
tests/test_csharp_tuple_type_refs.py — impact
tests/test_csharp_type_resolution.py — impact
tests/test_dedup.py — impact
… and 197 more

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

Docs that may be stale (advisory)

graphify/skill-agents.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 564-665): references changed symbols _corpus
graphify/skill-aider.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 684-764): references changed symbols _corpus
graphify/skill-amp.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 564-665): references changed symbols _corpus
graphify/skill-claw.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 567-668): references changed symbols _corpus
graphify/skill-codex.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 564-665): references changed symbols _corpus
graphify/skill-copilot.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 567-668): references changed symbols _corpus
graphify/skill-devin.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 802-883): references changed symbols _corpus
graphify/skill-droid.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 564-665): references changed symbols _corpus
graphify/skill-kilo.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 567-668): references changed symbols _corpus
graphify/skill-kiro.md § Step 9 - Save manifest, update cost tracker, clean up, and report (lines 567-668): references changed symbols _corpus

…and 10 more.

Formal verification

Could not verify: Could not verify extract.

The verifier did not have enough to check extract, so it is saying so rather than guessing. No false assurance is the whole point.

Guarantee: No guarantee either way, this is an honest abstention, not a pass.

Note: Reason: no capturable inputs from the test suite; property tier: not verifiable: all 90 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly TypeError — names the real obstacle, not a sampling gap)

No difference found (not proven): No behavior difference found in \_path\_proximity\_winner (not a proof).

The verifier ran both versions of \_path\_proximity\_winner on many inputs and saw identical behavior every time. Strong evidence the change is safe, but evidence, not a proof.

Guarantee: Empirical: differential testing (both versions run on many generated inputs). A divergence on an untested input remains possible, so this is 'no counterexample found', not 'proven equivalent'.

Note: An input the sampler did not try could still differ.

· 106 more finding(s) on lines outside this diff (see the check run).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* upstream/v8: (184 commits) release: 0.9.80 — include Graphify-Labs#4195, Graphify-Labs#4198 feat(python): resolve obj.method() through an annotated/constructor-bound local (Graphify-Labs#4198) perf(extract): resolve and parse each path once in the id remap and call tie-break release: 0.9.80 fix(deps): raise tree-sitter runtime floor to >=0.25 for ABI 15 grammars (Graphify-Labs#4148) fix(build): give --no-cluster graph.json the same endpoint rules as build_from_json fix(watch): keep extract()'s run-only keys out of --no-cluster graph.json fix(watch): drop an external import stub once no edge references it fix(detect): drop a previous checkout's absolute manifest keys (Graphify-Labs#4175) fix(python): keep TYPE_CHECKING-only imports out of import cycles (Graphify-Labs#3159) fix(python): resolve self.<attr>.<method>() calls through the attribute's constructor type (Graphify-Labs#2860) fix(cache): treat an AST hit whose import target is gone as a miss fix(extract): preserve cross-drive syntax warnings fix(extract): let the Windows console script use the extraction pool fix(resolution): keep re_exports from both files whose ids collide fix(markdown): resolve links to file names that contain a space fix(java): capture methods and calls inside anonymous class bodies test: skip optional-extra tests when the extra is not installed (Graphify-Labs#4190) fix(serve): show the graph's build commit in graph_stats perf(serve): build the MCP traversal view and shortest-path graphs once per graph ... # Conflicts: # CHANGELOG.md # graphify/extract.py # graphify/extractors/engine.py # graphify/serve.py # pyproject.toml # uv.lock

rohit-jsfreaky requested a review from safishamsi as a code owner October 7, 2026 11:46

graphify-labs Bot reviewed Oct 7, 2026

View reviewed changes

safishamsi added a commit that referenced this pull request Oct 7, 2026

release: 0.9.80 — include #4195, #4198

6478eb7

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Navigation Menu

Repository navigation

perf(extract): resolve and parse each path once in the id remap and call tie-break - #4195

perf(extract): resolve and parse each path once in the id remap and call tie-break#4195
rohit-jsfreaky wants to merge 1 commit into
Graphify-Labs:v8from
rohit-jsfreaky:perf/extract-path-memo

rohit-jsfreaky commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

graphify-labs Bot left a comment •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

rohit-jsfreaky commented Oct 7, 2026

What does this PR do?

Type of change

Verification & Invariants

How was this tested?

Graphify-specific checklist

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

graphify-labs Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Graphify review — findings

Impact & health

Graphify review

Gate & verification

graphify gate

Test selection

Test selection

Docs that may be stale (advisory)

Formal verification

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

graphify-labs Bot left a comment •

edited

Loading