Repository navigation
fix: survive flaky GitHub responses and keep stats files small - #111
Merged
Merged
Conversation
GET retries on 500/502/503/504 and network errors; writes retry only on 502/503/504 or a connection that never opened, so a request that may have reached GitHub is not sent twice. Three attempts in all.
The contents API returns an empty body with encoding none above 1 MB, so read_json falls back to the git blobs API. write_json can emit compact JSON, and delete_file removes a stats file, retrying once on a 409.
Every stats-branch write in sync_likes, on_merge and rot now goes through update_json, which re-reads and re-applies on a 409. flags.json merges this run's records into the stored copy, so an opened issue's number is never lost and never opened twice. The trust cache drops records past the 7-day TTL and is stored compact. Only the last 30 snapshots are kept, which also keeps the directory under the 1,000-entry listing limit. sync_likes skips a run, writing nothing, while a rename is half done: a listed entry has no thread mapping and the mapping holds a key that is no longer listed.
The new fixture row climbs by 9 with no seed flag, so only the 7-day age gate keeps its data-trend mark off. Counts and orderings that the extra row shifts are updated.
A job stuck in a rate-limit sleep or a network stall held its concurrency group for 6 hours. Most jobs get 10 minutes; the jobs that wait up to 15 minutes for a merge get 25, likes 30, on-merge 15, ClusterFuzzLite 20, and the weekly health check 120.
A 502 can arrive after GitHub has already created the issue, thread, comment or dispatch. Status retries now cover GET, PUT, PATCH, DELETE and GraphQL queries only; POSTs and mutations retry only when the connection never opened.
sync_likes skips only when an unlisted mapped key was ranked in the previous likes.json, and only while likes.json is under 3 hours old; past that it logs an error and syncs. A removed entry can no longer freeze the board. After a 409, likes.json and meta.json are written only if the file is still what this run read. update_json makes 3 attempts, and delete_file treats a 404 as already gone.
A cancel or timeout releases list-writes while auto-merge is still armed, so the PR could land behind the next writer. Submit, reports and the health check now close it on cancelled(), and submit and reports park the issue as they do on failure. Submit apply and report hide get 35 minutes.
App tokens last an hour, and rot's job may run two. Submit, reports and the health check now mint tokens with the same scopes once the merge wait ends, and every later step (finish, comments, park, cancel-close) uses them, falling back to the first token.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One 502 used to throw away a whole rot run. One 409 on the stats branch could lose the record of an issue we'd just opened, and the next hour would open it again. And two stats files were heading for the 1 MB line where the contents API stops returning them. This PR fixes all three, plus a few smaller items from the final review.
Retries (I6).
gh.pynow retries transient failures: three attempts, exponential backoff with jitter, and an injectable sleep.409s on the stats branch (I7). Every stats write in
sync_likes,on_mergeandrotnow goes throughupdate_json. On a 409 it re-reads the file, re-applies the change and tries again, up to 3 attempts. If someone else changedlikes.jsonormeta.jsonin the meantime, their version stands until the next hour.flags.jsonlays this run's new and changed records over the stored copy, so issue numbers that are already stored survive, and so do the new ones.discussions.jsonmerges the same way, and a mapping that's already stored still wins.Files over 1 MB (I10).
read_jsonfalls back to the git blobs API when the contents API answers withencoding: none.Carried over from the last wave.
sync_likesexits cleanly without writing anything while a rename is half done. That means a listed entry has no thread, and an unlisted mapped key was ranked in the lastlikes.json. The skip stops oncelikes.jsonis 3 hours old: after that it logs an error and syncs anyway.renames.jsonkeys its records.CI (M3). Every job now has
timeout-minutes. Most get 10. Submit apply and report hide get 35, because they wait up to 15 minutes for a merge. Likes gets 30, on-merge 15, ClusterFuzzLite 20 (its runs take up to 8 minutes), and the weekly health check 120. When a submit, report or health-check job is cancelled or times out, it closes its armed PR so auto-merge can't land it behind the next writer, and submit and reports park the issue underneeds-review. Once the merge wait ends, each of those jobs mints a fresh App token with the same scopes, sofinish, the comments, the park step and the cancel-close never run on a token that expired during a long run.Part of the v0.1 launch.