Skip to content

fix: survive flaky GitHub responses and keep stats files small - #111

Merged
jamesconwaydev merged 9 commits into
mainfrom
fix-review-wave2
Nov 6, 2026
Merged

jamesconwaydev merged 9 commits into
mainfrom
fix-review-wave2

Conversation

@jamesconwaydev

@jamesconwaydev jamesconwaydev commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

One 502 used to throw away a whole rot run. One 409 on the stats branch could lose the record of an issue we'd just opened, and the next hour would open it again. And two stats files were heading for the 1 MB line where the contents API stops returning them. This PR fixes all three, plus a few smaller items from the final review.

Retries (I6). gh.py now retries transient failures: three attempts, exponential backoff with jitter, and an injectable sleep.

  • GETs and GraphQL queries retry on 500, 502, 503, 504 and on network errors.
  • PUT, PATCH and DELETE retry on 502, 503 or 504. Contents PUTs are sha-guarded, so a repeat can't double-apply.
  • POSTs and GraphQL mutations create things, and a 502 can arrive after the item already exists. They retry only when the connection never opened (refused, DNS).
  • A 4xx never retries, and the rate-limit rules haven't changed.

409s on the stats branch (I7). Every stats write in sync_likes, on_merge and rot now goes through update_json. On a 409 it re-reads the file, re-applies the change and tries again, up to 3 attempts. If someone else changed likes.json or meta.json in the meantime, their version stands until the next hour. flags.json lays this run's new and changed records over the stored copy, so issue numbers that are already stored survive, and so do the new ones. discussions.json merges the same way, and a mapping that's already stored still wins.

Files over 1 MB (I10).

  • read_json falls back to the git blobs API when the contents API answers with encoding: none.
  • The trust cache drops records past its 7-day TTL and is stored as compact JSON.
  • Only the last 30 snapshots are kept. Older ones are deleted after the new one is written, which also keeps the directory well under the 1,000-file listing limit (M1).

Carried over from the last wave.

  • sync_likes exits cleanly without writing anything while a rename is half done. That means a listed entry has no thread, and an unlisted mapped key was ranked in the last likes.json. The skip stops once likes.json is 3 hours old: after that it logs an error and syncs anyway.
  • The site tests get a future-dated fixture row that covers the Trending age gate.
  • There's a comment on how renames.json keys its records.

CI (M3). Every job now has timeout-minutes. Most get 10. Submit apply and report hide get 35, because they wait up to 15 minutes for a merge. Likes gets 30, on-merge 15, ClusterFuzzLite 20 (its runs take up to 8 minutes), and the weekly health check 120. When a submit, report or health-check job is cancelled or times out, it closes its armed PR so auto-merge can't land it behind the next writer, and submit and reports park the issue under needs-review. Once the merge wait ends, each of those jobs mints a fresh App token with the same scopes, so finish, the comments, the park step and the cancel-close never run on a token that expired during a long run.

Part of the v0.1 launch.

GET retries on 500/502/503/504 and network errors; writes retry only
on 502/503/504 or a connection that never opened, so a request that
may have reached GitHub is not sent twice. Three attempts in all.
The contents API returns an empty body with encoding none above 1 MB,
so read_json falls back to the git blobs API. write_json can emit
compact JSON, and delete_file removes a stats file, retrying once on
a 409.
Every stats-branch write in sync_likes, on_merge and rot now goes
through update_json, which re-reads and re-applies on a 409. flags.json
merges this run's records into the stored copy, so an opened issue's
number is never lost and never opened twice.

The trust cache drops records past the 7-day TTL and is stored
compact. Only the last 30 snapshots are kept, which also keeps the
directory under the 1,000-entry listing limit.

sync_likes skips a run, writing nothing, while a rename is half done:
a listed entry has no thread mapping and the mapping holds a key that
is no longer listed.
The new fixture row climbs by 9 with no seed flag, so only the 7-day
age gate keeps its data-trend mark off. Counts and orderings that the
extra row shifts are updated.
A job stuck in a rate-limit sleep or a network stall held its
concurrency group for 6 hours. Most jobs get 10 minutes; the jobs that
wait up to 15 minutes for a merge get 25, likes 30, on-merge 15,
ClusterFuzzLite 20, and the weekly health check 120.
A 502 can arrive after GitHub has already created the issue, thread,
comment or dispatch. Status retries now cover GET, PUT, PATCH, DELETE
and GraphQL queries only; POSTs and mutations retry only when the
connection never opened.
sync_likes skips only when an unlisted mapped key was ranked in the
previous likes.json, and only while likes.json is under 3 hours old;
past that it logs an error and syncs. A removed entry can no longer
freeze the board.

After a 409, likes.json and meta.json are written only if the file is
still what this run read. update_json makes 3 attempts, and
delete_file treats a 404 as already gone.
A cancel or timeout releases list-writes while auto-merge is still
armed, so the PR could land behind the next writer. Submit, reports
and the health check now close it on cancelled(), and submit and
reports park the issue as they do on failure. Submit apply and report
hide get 35 minutes.
App tokens last an hour, and rot's job may run two. Submit, reports
and the health check now mint tokens with the same scopes once the
merge wait ends, and every later step (finish, comments, park,
cancel-close) uses them, falling back to the first token.
@jamesconwaydev
jamesconwaydev merged commit 7df8824 into main Nov 6, 2026
8 checks passed
@jamesconwaydev
jamesconwaydev deleted the fix-review-wave2 branch November 6, 2026 05:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant