Skip to content

Bound memory during native Docker artifact collection - #8488

Merged
huangruiteng merged 1 commit into
mainfrom
codex/spooled-artifact-collection
Nov 10, 2026
Merged

huangruiteng merged 1 commit into
mainfrom
codex/spooled-artifact-collection

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Terminal collection currently joins each native Docker archive into one bytes object before extraction. Collect the same archive through a context-managed temporary file, then use the existing tar data filter. The SDK iterator and spool close on success, interrupted reads, extraction errors and disk-write failures.

The collected paths, optional research directory, session-only credential boundary, error observations, retained-source handling and native stop/removal/capacity-release rules stay with their existing owners. The unrelated native bytes-return API is unchanged. This remains a Docker provider IO change, with no new capability, option or frontend behavior.

Validation on Python 3.12/Linux:

  • 341 runtime and artifact-collection tests passed against the repository's configured default native source and compatibility patch. Coverage includes modes/content, relative and unsafe links, HTTP and injected ENOSPC errors, interruption, spool/iterator closure, explicit retention and failed native cleanup/capacity release.
  • Three fresh processes per version collected a pre-existing 64 MiB synthetic tree through the real Docker backend. Host Python high-water RSS median decreased from 174.0 MiB to 46.6 MiB; bytes, modes and SHA-256 matched.
  • Real default/unconfigured collection, bytes API compatibility, symlink/hardlink fidelity, unsafe-link refusal, HTTP failures, injected temporary-file ENOSPC and explicit retention checks passed. All disposable containers were removed after their expected lifecycle assertions.
  • The final committed sources also passed the complete owned Docker upload/collection path. Ruff and git diff --check passed.

Tested head: 4cfc91f, based on main 4617995. The native-provider default pin is unchanged. The comparison baseline uses identical collector/runtime bytes from main before this IO patch.

The fixture used one network-disabled container at a time with a 512 MiB container limit; reported memory belongs to its host Python process. Disk-full errors were injected into real temporary-file writes rather than filling shared storage. Spooling adds transient archive disk IO/space under the existing TMPDIR policy. Full-scale resource use and an end-to-end latency improvement are not established. No model or grading workload was launched.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; reasoning_effort=max.

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

使用 native Docker runtime 收集终态材料的调用者,会在材料较大时承担整份 archive 的 Python 内存成本。
原实现先把 Docker GET archive 合并成 bytes,再解包;新实现将同一流写入临时文件,随后按原 data filter 解包。
独立真实 Docker 夹具验证了内容、mode 和 links 保真,并区分出修改前后的 archive 级 Python 分配。
本 PR 不修改评分、任务输入、运行预算、恢复权限或非 Docker provider,也不承诺完整尺寸的总资源或时延改善。

改动思路 / Approach

该修改复用既有 Docker provider IO owner;临时文件只承接 tar 生命周期,不成为新的策略、状态或权限 owner。

当前 PR 边界是文件传输的内存表示及异常资源释放;既有调用、筛选和终态清理规则继续由原 owner 负责。

对比不修改、仅关闭 BytesIO、改 public bytes API 和临时文件方案后,当前方案在现有 owner 内解决了可复现问题;不会增加用户确认轮次。资源策略有意由 Python heap 转为 TMPDIR IO,磁盘不足仍交给原异常/清理 owner。

具体改动 / Changes

关键代码讲解

  • _extract_artifact_archive 获取真实 SDK iterator 后,在 seekable TemporaryFile 中写完 tar,rewind 并使用原 filter="data" 解包。文件的 with 与 iterator 的 finally 分别负责外部资源生命周期,tar 对象的成员引用不会延长打开文件的寿命。
  • cleanup_container 仍只遍历原来的五个路径;research_sidecar 为真才增加第六个路径。每项失败仍记录异常类;外层 finally 的保留/移除以及容器存在时禁止释放容量保持原 owner,不由新 helper 作判断。

规范依据是 benchmark/README.md,spec_revision 46179959769eaecc7b04debd498963c55c07db1c。L-NATIVE 保留原生任务、runner、验证及评分 authority;L-OWNER 把 provider IO 留在 adapter;L-PRIVATE 保持 session-only 收集和公开/私有边界。三项均有代码和真实 default/research 夹具对应。

独立复跑 341 项 Linux3.12.3 runtime/collection tests 使用正式默认 native pin 476b34d17e759a7dbcf9ec36a12ff42b757b462a 与 nonfinite-checkpoint-v2 patch(patch SHA256 02d21096a2e55f625d3c8beee2335728caf08aa6c75a5a27b1ab412bc46323bf);没有把 raw upstream native main 当默认依赖,也没有静默更改默认 pin 或夹带 finite-run 能力。

独立 32MiB、每版本一次、真实 Docker public-entry 比较如下。这里是 IO 调用期间 tracemalloc 记录的 Python 分配峰值,不是 RSS;base/head 的文件 SHA、mode 和 links 都通过。

指标 base exact head
traced Python peak 67,407,009 B 248,043 B
独立 archive-sized heap oracle FAIL(预期历史缺陷) PASS

测量从 fixture/seed 准备及 gc 之后、public IO 方法调用之前开始,返回/异常后立即停止;不含后续读取整个结果文件的校验,也不覆盖 C 分配、OS page cache 或进程总资源。Docker SDK 7.2.0 的正常 GET 使用默认 chunk_size=2MiB、encode_stream=False;fixture 的 wrapper 原样转发真实 GET chunks,没有切小或替换正常数据。当次实际 chunk 长度分布未记录;urllib3 的 chunked 读取可返回小于上限的块,因此这个峰值不是普适 chunk/内存上界。HTTP 故障病例才单独中断真实流。作者的 64MiB/n3 RSS 结果是另一项证据,本次没有替换或复测该结论。

对主干的风险 / Risks

最强反例是临时盘或下载中断导致材料不完整,同时错误跳过 native 清理或提前释放容量。独立真实 Docker 覆盖 temp 创建和写入 ENOSPC、HTTP 中断、unsafe absolute link,以及保留与原 native 移除失败分支:所有 head 临时文件/取得的 streams 均关闭;移除失败时实际容器仍在,synthetic release callback 不调用。

配对 default-off 检查在同一容器内放入 research 和两个 synthetic auth 文件,base/head 都只产生五项收集且不收集 auth;enabled 后恰为六项。另在 backend 建立后增加 research 文件,base/head 都收集到该文件。保留回调证明的是既有保留/最终 teardown 规则,并非真实 failed-worker resume 资格。CI 尚有 pending/skipped 诊断,独立本地证据覆盖本 PR 的改动不变量;merge readiness 仍单独判断。

早期 reviewer 夹具曾有非 root 目标/校验权限假设错误,以及错误的移除故障注入 seam;这些失败均保留并单独归因为夹具问题,未当作 PR 通过或产品回归。修正后才执行上述有效 base/head 与异常对照,所有自有容器及 scratch 已 teardown。

我的整体评价 / Verdict

未发现阻断问题。该增量在真实入口避免了 archive 级 Python heap 分配,且保持原 artifact、隐私与生命周期 owner;一个私有 IO helper 的维护成本与问题相称。

同一32MiB合成材料在真实入口避免了 archive 级 Python bytes 分配,同时保留内容及失败路径;完整尺寸与并发总资源仍未资格化。

调用者沿用原入口、目标路径和返回/错误约定,无新增配置选择或确认轮次;TMPDIR 磁盘空间成为明确的资源代价。

检查的改动仅在 Docker IO、终态收集和合成测试中,没有渲染界面或交互布局变化。 unverified_scope: none。语义评估复用既有 contract,未新增状态分类、通用义务或权限 vocabulary;只改变 IO 的内部资源承载。完整尺寸/并发总资源、临时盘 headroom 和端到端时延仍需运行方按原边界资格化,不据合成测量授予运行或合并权限。

English verdict: APPROVE — head 4cfc91f; no blocking finding. Linux focused suites and bounded real Docker base/head, failure and lifecycle cases passed; full-scale resource use and merge readiness remain separate.

@huangruiteng
huangruiteng merged commit a8ffad3 into main Nov 10, 2026
31 of 39 checks passed
@huangruiteng
huangruiteng deleted the codex/spooled-artifact-collection branch November 10, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants