Skip to content

perf: skip impossible lexical field matches - #8439

Merged
huangruiteng merged 1 commit into
mainfrom
codex/semantic-lexical-scan-fast-rejection-20261109
Nov 10, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/semantic-lexical-scan-fast-rejection-20261109

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

The semantic retirement inventory repeatedly searches each source for each legacy field, including sources with no literal occurrence. Reject those impossible matches with a string containment check before the existing escaped standalone-token regex.

The metric still counts raw source modules, retains its original ASCII boundary semantics and consumes one-shot iterables. AST reader/writer classification and the repository-wide computed-key unknown population are unchanged. There is no cache, new parser or budget increase.

Written by an OpenAI GPT-6 coding agent. Tested head: 0d6f37d07d1d12b9a95c4da37d8fa43cd931b5ec; base: 6787fc8d51a24dc7ab7713c8602d46226d2bafae.

Validation:

  • 81 field-use tests passed, including Unicode adjacency, prefix/suffix identifiers, comments, repeated tokens, escaped spelling, invalid syntax and source suffix separation. Ruff and diff checks passed.
  • On one fixed tracked corpus (1,825 Python/TypeScript sources, six registered fields), three alternating samples per arm returned identical count vectors. Helper median: 2.925 s before, 0.121 s after. These are local helper timings, not end-to-end latency, model tokens or accepted-outcome savings.
  • Exact-head standard premerge passed with no manual hold. Its complete semantic vocabulary smoke took 83.872 s under the original 120 s limit.

This was selected while diagnosing two C1 semantic-inventory timeouts at the existing 120 s limit. Those failures remain recorded. Profiling identified lexical scanning as one contributor among AST and other source scans; this result does not establish the sole cause of those timeouts or a matched full-command speedup. C1 integration and its broader benchmark/adoption qualification remain separate.

Scope-fit assessment: the active consumer is the existing semantic retirement smoke. The change is independently reversible and stays within its current metric owner; wider scanner restructuring is unnecessary for this slice. No product UI, authority store or runtime scheduling behavior changes. Public-boundary scan and DCO sign-off passed.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

English verdict: APPROVE at 0d6f37d. No blocking finding was found; matched native-consumer counts and the existing semantic gate passed. This does not certify an end-to-end speedup or C1 adoption.

动机 / Motivation

运行语义词表检查的贡献者,需要保留完整退役计数,同时减少重复扫描工作。旧实现对每个字段逐一执行 regex,即使源码没有该字面量;当前实现只跳过这些必定不匹配的来源,保留原计数、ASCII 边界和一次性迭代器。

已验证结果是固定1,825个来源、六个注册字段的12组计数不变,实际 regex search 调用从10,950降到125。它交付的是现有检查入口中的计算量减少,不是新 parser、缓存或预算调整;不认证原 C1 timeout 的唯一根因、整条命令加速、模型收益或安装采用。

改动思路 / Approach

最强的不发布理由是 substring 本身不能定义 standalone token:只留字面量检查,会把 goal_boundary_repair 也算进来。这里保留原 escaped regex 作为最终判定,literal containment 只是必要条件。相比缓存整棵源码、缩小 scan roots 或重构 AST scanner,这个局部预筛选直接复用现有 owner,不新增失效规则或第二种计数语义。

当前 PR 的边界是现有 Python field-use 指标函数与其原生 smoke consumer。它没有改变字段退役权限、预算、computed-key unknown 或默认检查范围。Python 保留现有文本/AST 专门计算职责;为这一谓词加入 TypeScript bridge 或新能力没有必要。已在同一真实 consumer 的源码载入路径验证,不用另一个测试侧计数器冒充生产输出。

具体改动 / Changes

依据修改前 accepted RFC:Semantic Vocabulary Convergence,spec_ref=docs/architecture/rfcs/semantic-vocabulary-convergence-v0.md;spec_revision=6787fc8d51a24dc7ab7713c8602d46226d2bafae。I5 要求计数与预算保持同一字段集合、原 role partition;B3 区分 token modules、重叠 reader/writer/binding facts 与 unresolved,而不是把词法缺席当成没有 reader。现有完整 semantic smoke 和新旧 consumer 对照覆盖这些当前约束,未扩大为整个词表收敛计划已完成。

  • lexical_module_count 仍先过滤 suffix、再按模块计一次;只有字面量存在时才执行原 token.search,既不按 occurrence 累加,也不提前消费 sources。
  • _token_pattern 未改,继续 re.escape 与显式 ASCII identifier 边界。因此汉字相邻仍可匹配,prefix/suffix identifier 仍拒绝;不是换成 Unicode 的 \b。
  • count_identifier_modules 是未变的实际 consumer。check_reader_metric 仍核对原 population 与 AST roles;扫描全部 TypeScript/Python computed-key sites 的路径没有改成按字面量裁剪。

全量 diff 仅2文件 +20/-1:生产5行增加/1行删除,其余为既有测试文件中的 import 与 regression cases。没有新增模块、CLI 参数、schema、registry entry 或持久状态。现有 Sources/regex/consumer 已足够,相邻整理无需扩大为 scanner 重构。

对主干的风险 / Risks

没有 blocking finding。独立经原生 count_identifier_modules 运行同一21组输入:comments、重复 token、复合 identifier、Unicode 邻接、escaped spelling、无效语法、regex metacharacter、空字段、空源码及 suffix 分离,base/head 均符合独立 ASCII 邻接 oracle;sources 使用一次性 iter。真实 load_sources 读取的同一1,825来源 corpus 与 fixture 的摘要在两端相同,12组计数向量一致。search 计数包装器调用真实 compiled regex,没有提供假匹配结果。

故意在实际 consumer 的调用接线处移除 regex boundary,独立 oracle 在错误计数4处失败;正常 head 对同一例计数3。另经实际 retirement budget gate 验证:无关同容器来源不改变输出,新符合条件来源正常计入,重命名同 suffix 不隐藏 token;超过原预算仍拒绝,移除违规新增 carrier 后恢复原结果,未提高预算。

当前 head 的81项 field-use 回归、仓库规定 Ruff、MyPy17、改动路径 Ruff 均通过。standard premerge 的6项 direct、2项 selected 均通过,包含完整 semantic-vocabulary-drift smoke,保留原120秒限制。其输出继续显式报告2,429个 Python computed-key sites、1,410个 TypeScript dynamic member sites,以及未证明安全的 producer sites,没有把这些 unknown 归零。此处只报告当前通过和计算工作量,不把单次 smoke 耗时与 author 的其他环境样本拼成性能对照。

输入是公开源码与合成 fixture;没有数据库、安装态服务或模型调用边界需要模拟。没有查询、轮询或等待 GitHub CI。本轮 source premerge 不冒充 active Goal 的完整 strict CQR 或正式合并授权;author 的2.925s→0.121s helper测量及其原超时解释保持作者来源,本轮未独立认证那些秒数或全部历史超时的原因。

我的整体评价 / Verdict

APPROVE:原生使用方得到相同指标并减少必定无效的 regex 工作,局部交付有明确价值,owner 和规模合适。长程与用户体验保持原约束:不增加操作、确认、调度、执行权限或人工同步;仍需根据 reader、writer、unknown 与原合同判断退役,不能据一个更快的零计数删除字段。

当前 PR 的边界是现有 Python field-use 指标函数与其原生 smoke consumer;保留该边界即可,不需要缓存、bridge 或预算放宽。队列中的 recovery-RFC 经验只支持跟到实际有用 consumer;本次有已验证的内部维护边界,不额外要求 UI 或 C1 全面交付。已检查 scanner、既有 CLI smoke 与测试,没有呈现层变化,visual review 不适用。未验证范围:匹配条件下的整命令耗时、全部原 timeout 归因、跨平台性能、安装采用及长期/model收益。此代码结论不授予合并、安装或其他任务执行权限。

@huangruiteng
huangruiteng merged commit 6cd55ce into main Nov 10, 2026
30 of 37 checks passed
@huangruiteng
huangruiteng deleted the codex/semantic-lexical-scan-fast-rejection-20261109 branch November 10, 2026 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants