Repository navigation
fix: read large files and stop one stuck item from blocking the bots - #112
Merged
Merged
Conversation
The contents API returns encoding "none" for a file over 1 MB. A big LICENSE raised Undecided, and a big README passed on its size alone. Both now come through the git blobs API and are capped at the usual scan limits (100 KB per license file, README_SCAN for the README). Undecided is left for content that genuinely can't be decoded.
decide now exits 3 on Undecided and writes undecided.json. A follow-up step counts the run in a hidden marker inside the bot's own comment, so a submitter can't forge it. On the third undecided run the issue is labelled needs-review and gets the error comment once, and the sweeper moves on to the next dropped submission.
Full runs now count consecutive undecided weeks per entry in undecided.json on the stats branch. At 3 weeks the run opens one needs-review issue, deduplicated by title among the bot's own open issues. A one-repo run or a retry in the same week doesn't count, and a decided week resets the count. A failure here is logged and never stops the check itself.
Each dead report queued a one-repo health check in the shared list-writes slot. One-repo runs are now named "Health check: owner/name", and a dead report skips the dispatch when that entry was checked in the last 24 hours. Past 5 one-repo checks in an hour, new dead reports are parked as needs-review instead. The sweeper's exact "Health check" match for full runs is unchanged.
A document counts as a read only if it opens with "query" or "{"
after any leading # comment lines. Anything else, including a
fragment-first document, is treated as a mutation and not retried
after it may have reached GitHub.
Only Undecided was bounded, so a submission or report whose decide kept failing with an API error was re-dispatched on every sweep. Now every run that ends without a decision writes undecided.json and is counted, and the third parks the issue as needs-review with the error comment. A failed rot dispatch from a dead report counts too. A rate limit whose reset is still ahead writes nothing, so waiting for it spends no attempt. The counter moves to scripts/devstream/attempts.py so submit and reports share it.
The plan step now only records undecided weeks and lists the entries due a review in plan.json. A separate step, with its own token scoped to issues: write and nothing else, opens the needs-review issues. The main rot token is back to its original scopes, and a failure in the review step never stops the check.
The blobs API returns the full file, so a 100 MB license cost about 600 MB of memory on every run. The size from the contents response is now checked first, and nothing over 5 MB is fetched. Such a license file counts as no license text, which fails closed; rot's two-sighting guard still applies. A README that size is Undecided, so the attempt and review bounds take over.
Parking only happened when the count was exactly 3, so after a maintainer removed needs-review to retry, the item could fail forever. needs-review is now re-added on every failure from the third on, and the error comment is still posted only once, at 3. At the limit the counter comment no longer promises another try.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes I1, I2 and I8 from the final review, plus the GraphQL read check in
gh.py.Files over 1 MB (I1, I2). The contents API returns
encoding: nonefor a file over 1 MB. A big LICENSE raisedUndecided, and a big README passed on its size alone. Both now come through the git blobs API and are capped at the existing scan limits: 100 KB per license file,README_SCANfor the README. So a 2 MB BUSL license gets rejected, and a 2 MB README is actually measured. Nothing over 5 MB is downloaded at all. A license file that big counts as no license text, which fails closed, and a README that big isUndecided.Undecidednow means content that really can't be decoded.A stuck issue no longer holds the sweeper. Any submission or report decide run that ends without a decision now writes
undecided.jsonand exits 3. That coversUndecided, API errors, timeouts, and a failed rot dispatch from a dead report. The next step mints an issues-write token and counts the run in a hidden marker inside the bot's own comment. Only bot-authored comments count, so the author can't forge the number. On the third failed run, the issue getsneeds-reviewand one error comment, and the sweeper moves on. If a maintainer's retry fails again, the label goes straight back on without another comment. If a rate limit's reset is still in the future, the run writes nothing, so waiting for it doesn't use up an attempt.An undecidable entry no longer escapes the health check. Full runs keep a count of consecutive undecided weeks per entry in
undecided.jsonon the stats branch. At 3 weeks, the run opens oneneeds-reviewissue, deduplicated by title among the bot's own open issues. One-repo runs and same-week retries don't add to the count, and a decided week resets it. The plan step only records the count. A separate step opens the issues, using its own token with justissues: write, so the main rot token keeps its original scopes.Dead reports (I8). One-repo health checks are now named
Health check: owner/name. A dead report skips the dispatch when that entry was checked in the last 24 hours. Once 3 one-repo checks have run in the past hour, new dead reports are parked asneeds-reviewinstead. The reports decide token gainsactions: readto list runs. The sweeper's exactHealth checkmatch for full runs still works as before.GraphQL reads. A document counts as a read only if it opens with
queryor{once any leading#comment lines are stripped. Anything else counts as a mutation and isn't retried.Part of the v0.1 launch.