Skip to content

Pull optimization: reduce per-file path lookups and buffer allocations in unpackLayer - #3759

Draft
Amy (amyxu-08) wants to merge 1 commit into
agent-substrate:mainfrom
amyxu-08:remove-redundant-walks
Draft

Amy (amyxu-08) wants to merge 1 commit into
agent-substrate:mainfrom
amyxu-08:remove-redundant-walks

Conversation

@amyxu-08

Copy link
Copy Markdown

Part of the optimizations outlined in Investigating Substrate Image Pull Speed

Unpacking a large image layer on a cache miss goes through redundant path lookups through *os.Root and uses per-file buffer allocations. For each regular file, unpackLayer walked the path six times (MkdirAll, Lstat in replaceExisting, OpenFile, Lchown, Chmod, and Chtimes) and allocated a fresh 32 KiB buffer inside io.Copy when (*os.File).ReadFrom fell back to genericReadFrom.

  • Apply owner and mode on the open *os.File before closing it via tarutil.Fchown and setOpenFileMode.
  • Track directories created during the unpack in existingDirs so root.MkdirAll runs once per parent directory, and clear existingDirs in replaceExisting.
  • Create regular files with O_CREATE|O_WRONLY|O_EXCL and call replaceExisting only on os.ErrExist.
  • Wrap outFile in writerOnly and pass a single 32 KiB copyBuf to io.CopyBuffer.

AI use: AI-assisted, I reviewed and tested it (go test -race -count=3 -shuffle=on ./internal/imagecache andhack/run-e2e.sh against dev cluster).

  • Tests pass
  • Docs updated (none applicable)

Change-Id: I593a1b6714fc0bc18e1ca282b427f29bf41f4cee
@amyxu-08 Amy (amyxu-08) changed the title reduce per-file path lookups and buffer allocations in unpackLayer Pull optimization: reduce per-file path lookups and buffer allocations in unpackLayer Nov 10, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant