Skip to content

perf: stage replayed files in a single git.add call - #463

Open
SammyTourani wants to merge 1 commit into
cloudflare:mainfrom
SammyTourani:fix/issue-312
Open

SammyTourani wants to merge 1 commit into
cloudflare:mainfrom
SammyTourani:fix/issue-312

Conversation

@SammyTourani

Copy link
Copy Markdown

Fixes #312

In the replay loop of GitCloneService.buildRepository (worker/agents/git/git-clone-service.ts), all working-tree files are now staged with one git.add({ fs, dir: '/', filepath: allFiles }) call per replayed commit, instead of one call per file. git.add accepts an array and processes it under a single index acquisition, so the index is read and written once per commit instead of once per file.

The output does not change:

  • The replay writes every file for a commit before staging, so .gitignore rules see the same working tree whatever the add order.
  • Index entries are sorted when the index is written.
  • The resulting commit OIDs are identical to before. This matters because the GitHub export force-pushes, and a changed history would rewrite repositories that were already exported.

I also added worker/agents/git/git-clone-service.test.ts, a focused test for the replay. buildRepository had no test coverage. The test checks:

  • the template base commit
  • commit order, parents and author metadata
  • the tree contents of each replayed commit

This roughly halves the CPU time of the step that exceeds the Durable Object limit. It does not remove the limit for very large histories.

Refs #312, #226

Verification

Command: bunx vitest run --no-file-parallelism
Result:

 Test Files  41 passed (41)
      Tests  475 passed | 1 skipped (476)
   Duration  69.39s
  • New test before and after. The new test passes on both the original and the patched code. It fails when staging is broken: I tried filepath: [] and staging only the agent's files, and both fail with the expected file-list mismatch.

  • Same output. I ran the original and patched buildRepository side by side on synthetic agent histories covering:

    • files ignored by the template's .gitignore
    • .gitignore edits partway through the history (both unignoring and newly ignoring files)
    • deletions
    • duplicate file contents
    • deeply nested directories
    • no template

    Both versions produced identical commit and tree OIDs for every commit in every scenario.

  • CPU time. I measured buildRepository process CPU time under Node 26 (V8). The synthetic app had 80 template files and 40 agent files, with 1 to 5 files changed per commit. Values are the median of 3 runs, and HEAD OIDs were identical in all runs:

    Agent commits Before After
    50 3.6 s 1.7 s
    100 6.7 s 3.3 s
    200 14.6 s 7.0 s
  • Type check and lint. bun run typecheck (tsc -b) passes with no errors. The worker config excludes *.test.ts, so I also type-checked the new test under the same compiler settings: no errors in it. eslint is clean on the changed source file.

GitCloneService.buildRepository replays every agent commit on top of the
template base and staged the working tree with one git.add call per file.
Each call re-reads and re-parses the index, reloads the config, and
re-serializes and hashes the whole index, so every replayed commit cost
O(files^2) index work. This replay runs inside the agent Durable Object
during GitHub export and in the Worker during git clone.

Pass all paths to a single git.add call per replayed commit. The staged
set, ignore handling and resulting commits are unchanged (same commit
OIDs); only the per-file index round-trips are removed.

Add a focused test for the replay.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

export to github and clone: fails due to durable timeouts

1 participant