Skip to content

fix(canary): keep Git's pathname identity in the remaining tracked-file readers - #8678

Open
gcl-coder wants to merge 1 commit into
loopx-project:mainfrom
gcl-coder:fix/git-path-identity-analysis
Open

gcl-coder wants to merge 1 commit into
loopx-project:mainfrom
gcl-coder:fix/git-path-identity-analysis

Conversation

@gcl-coder

Copy link
Copy Markdown
Contributor

Goal And Delivered Outcome

  • Outcome basis: two remaining git ls-files -z readers rebuilt each NUL record through a text-mode decode, so the path they produced named no file and the tracked file silently left a shipped check.
  • Goal/source and gap: the ratchet and the public-boundary scan both need the identity Git recorded, not a display rendering of it. text=True applies universal-newline translation (a CR inside a legal name became LF) and errors="replace" collapses an undecodable record to U+FFFD; the subsequent is_file()/suffix filter then dropped the record without any failure.
  • Observable before → after: a tracked carriage\rreturn.py was absent from tracked_python_paths() and a tracked venv/carriage\rreturn.py was absent from iter_scan_files(venv); both now appear. The validation rows below are the two new tests, which fail at eba8a7f95 and pass at this head.
  • Issue/task and intended base: no issue; this is a self-contained repair found while auditing the remaining ls-files -z consumers after fix(canary): preserve exact changed Git paths #8585 and fix(integration-branch): preserve worktree paths with line breaks #8603. Intended base eba8a7f95f54fe09cb3dcc28bb1f55443898bbed.

Author Declaration

  • Written by: model_agent — cursor-model (cursor).

Implemented against

  • Specification and revision: no written specification; the request in this PR is the basis. The documented behavior it preserves is the existing one-record-per-path rule already stated in loopx/canary/maintainability_ratchet.py and the existing scan enumeration contract in tests/test_contract_public_boundary_prefilter.py.
  • Criteria:
Criterion (spec clause) Disposition Symbol / path Test or command
One NUL-framed record is one pathname, including a CR inside the name implemented loopx/canary/maintainability_ratchet.py::tracked_python_paths tests/canary/test_git_path_framing.py::test_tracked_python_paths_keeps_a_carriage_return_name
A pruned scan root enumerates its tracked files from Git, so none may be dropped implemented loopx/contract.py::_tracked_scan_files tests/test_contract_public_boundary_prefilter.py::test_pruned_scan_root_keeps_a_carriage_return_tracked_name
The bounded Git probe keeps its timeout, session isolation and helper retirement implemented loopx/contract.py::_git_query_bytes tests/test_contract_git_probe_budget.py, tests/test_contract_git_probe_cancel.py
Scan rules, suffix filters, budgets and selection policy are unchanged implemented same modules existing tests/test_contract_*, tests/canary/test_maintainability_ratchet.py
  • Self-check before submission: read both readers and the probe budget/cancel tests; ran the two new tests against the base commit to confirm they fail there for the reported reason; ran the surrounding suites, ruff, and canary premerge --from-git-diff. Did not add a fixture for a filesystem that stores an undecodable byte in a name, because macOS APFS refuses to create one; that branch is os.fsdecode's documented round trip and is unexercised here.

Scope And Continuation

  • Completed scope and remaining work: complete within this scope. The other ls-files -z consumers already read bytes (loopx/canary/runner.py, loopx/capabilities/change_quality/scope.py, packages/loopx-core/prepare.py, loopx/semantics/development_probe.py); the two changed here were the ones that dropped a record silently instead of failing.
  • Slice boundary / successor: N/A — no successor task is required for this repair.

Validation

  • Tested revision: a208bc6ac48e46de20bed8d08b2cd69ec1a83c91
  • Run state: finished
Check Command Result
Focused suites python3 -m pytest tests/canary/test_git_path_framing.py tests/canary/test_maintainability_ratchet.py tests/test_contract_public_boundary_prefilter.py tests/test_contract_git_probe_budget.py tests/test_contract_git_probe_cancel.py tests/test_contract_scan_missing_roots.py tests/test_contract_scan_unreadable_directories.py tests/test_contract_scan_unreadable_files.py tests/test_contract_credential_pattern_precision.py -q 166 passed
New tests at base same two test ids at eba8a7f95 both fail: the ratchet set omits the CR module, the scan list is empty
Lint ruff check on the four changed files pass
Premerge canary premerge --from-git-diff --git-diff-base eba8a7f95 diff hygiene, py_compile, maintainability-ratchet smoke and all 8 risk-profile smokes pass; 5 catalog canaries fail identically at the base commit without this diff
Unrelated local failures tests/canary/test_rfc_bilingual.py, tests/canary/test_telemetry_isolation.py 12 failures at both revisions; local environment, not this diff

The five catalog canaries that fail here (first-connect-contract-smoke, quota-plan-smoke, quota-work-lane-policy-smoke, monitor-scheduler-contract-smoke, semantic-vocabulary-drift-smoke) fail with the same commands on a clean eba8a7f95 checkout in this environment; the hosted lanes are the authority for them.

Risk / 风险

这两个读取器此前把“显示编码”当成了路径身份:文本模式会做通用换行转换(合法名字里的 CR 变成 LF),errors="replace" 又把无法解码的记录折成 U+FFFD,重建出来的路径不指向任何文件,后续 is_file() 直接把该记录丢掉,既不报错也不计数。改动只在两处 Git 读取边界保留字节身份并用 os.fsdecode 还原(POSIX 与 PEP 529 的 Windows 均为原始字节的无损反解),扫描规则、后缀过滤、超时预算与选择策略一律未动。未验证:在文件名里存放无法解码字节的文件系统(APFS 不允许创建),该分支只有 os.fsdecode 的文档往返保证。

`tracked_python_paths` and the contract scan's tracked-file enumeration both
read `git ls-files -z` in text mode. Universal-newline translation turns a CR
inside a legal name into LF and `errors="replace"` turns an undecodable record
into U+FFFD, so the rebuilt path names no file. A tracked module then leaves
the maintainability ratchet, and a tracked file under a pruned scan root (the
only enumeration a virtualenv root gets) leaves the public-boundary scan.

Both reads now stay in bytes and rebuild each record with `os.fsdecode`, which
reverses exactly the bytes Git wrote. The ratchet keeps its one-record-per-path
framing comment; the contract scan gains a bytes twin of the bounded probe so
the timeout, session isolation and helper retirement stay unchanged.

Found while auditing the remaining `ls-files -z` readers after the canary
pathname work. No selection policy, scan rule or budget changed.

Signed-off-by: ChengLong GU <88359731+gcl-coder@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max

评审提交 a208bc6。GitHub 目标与 merge base 为 eba8a7f,本轮读取的最新主干 7bb1b01 未改动这组变更路径。

[P2] Linux 文件名进入发现后会破坏 JSON 输出。 在 Linux Git 索引中加入一个名称含不可解码字节的文件,让该文件生成一条检查发现,再请求 JSON 输出。 两个命令均已返回失败退出码,但 stdout 混入原始字节,严格 UTF-8 解码在 JSON 解析前失败,机器调用者无法读取违规位置。 问题位置

动机 / Motivation

用 JSON 消费预合并检查结果并按位置修复发现的维护者和自动化。 被扫描器跳过的目录里仍可能有 Git 跟踪文件。旧版会漏掉某些合法文件名,操作员看到空扫描而错过实际发现。 Linux 上,两条读取路径现在能打开 Git 实际记录的文件。普通名对照保持一致,带回车的诊断可按 UTF-8 读取。 验证范围是本地 Linux Git、文件系统与两个实际命令;Windows 文件系统及已安装分发未单独验证。

改动思路 / Approach

Git 负责提供路径身份,现有扫描器和 ratchet 负责判定,现有 JSON 输出负责把结果交给调用方。 将无损文件路径和显示字段的职责留在各自已有边界,复用现有 JSON 出口。两种 Git probe 可继续共享相同的超时与清理策略。 两个入口按 NUL 切分,再按系统文件名规则还原路径。CR 不再改名,POSIX 的不可解码字节也能用于实际文件读取。 扫描枚举

具体改动 / Changes

关键代码讲解

ratchet 将还原后的路径用于模块发现,合约扫描将其用于诊断定位。两者随后把代理码点交给 ensure_ascii=False。已是 UTF-8 的标准输出保留原错误策略,在复现环境把它写回原始字节。 ratchet 文件身份 合约扫描文件身份 ratchet JSON 出口

按 docs/reference/public-boundary-read-cost.md 在 eba8a7f 的验收依据,Git-owned coverage 已验证。Explicit check reporting 仍需让机器调用者读取发现。 JSON 输出契约 已接受的验收依据

对主干的风险 / Risks

在同一 Linux 容器中,普通违规文件在两版都返回可解析的失败报告。含 CR 文件基线扫描数为 0,候选为 1 且能解析。不可解码文件在候选进入实际扫描和 ratchet 发现,两个失败报告均无法严格解码。 文件查找继续使用无损字节身份。将路径和派生标识投影到现有报告时做可区分的转义,使 JSON 始终可按 UTF-8 解码并定位对应条目。 在真实 Linux Git fixture 上覆盖普通名、含 CR 和含不可解码字节的名称。让违规命中进入两个实际 JSON 入口,分别断言退出码、严格解码、发现内容和位置。

语义与 CI 对齐

166 项定向测试通过,包含 Git 预算与真实 fsmonitor 子进程清理。Standard premerge 六项直接检查、18 项选中检查均通过。 Linux 实际 JSON 负例仍失败。 托管 CI 未查询。

我的整体评价 / Verdict

请求修改。保留字节路径修复,补齐两个现有结构化出口的报告边界。 此改动的用户入口为命令行结构化报告。已直接检查其 stdout 字节和严格解析。 验证范围是本地 Linux Git、文件系统与两个实际命令;Windows 文件系统及已安装分发未单独验证。

English verdict: REQUEST_CHANGES. On a208bc6, a tracked Linux pathname with an undecodable byte reaches both scanners but makes their failing JSON reports invalid UTF-8.

for record in completed.stdout.split(b"\0"):
if not record:
continue
path = repository_root / os.fsdecode(record)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Linux 文件名进入发现后会破坏 JSON 输出。 在 Linux Git 索引中加入一个名称含不可解码字节的文件,让该文件生成一条检查发现,再请求 JSON 输出。 两个命令均已返回失败退出码,但 stdout 混入原始字节,严格 UTF-8 解码在 JSON 解析前失败,机器调用者无法读取违规位置。 文件查找继续使用无损字节身份。将路径和派生标识投影到现有报告时做可区分的转义,使 JSON 始终可按 UTF-8 解码并定位对应条目。 在真实 Linux Git fixture 上覆盖普通名、含 CR 和含不可解码字节的名称。让违规命中进入两个实际 JSON 入口,分别断言退出码、严格解码、发现内容和位置。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants