Skip to content

fix: read large files and stop one stuck item from blocking the bots - #112

Merged
jamesconwaydev merged 10 commits into
mainfrom
fix-review-wave3
Nov 6, 2026
Merged

jamesconwaydev merged 10 commits into
mainfrom
fix-review-wave3

Conversation

@jamesconwaydev

@jamesconwaydev jamesconwaydev commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes I1, I2 and I8 from the final review, plus the GraphQL read check in gh.py.

Files over 1 MB (I1, I2). The contents API returns encoding: none for a file over 1 MB. A big LICENSE raised Undecided, and a big README passed on its size alone. Both now come through the git blobs API and are capped at the existing scan limits: 100 KB per license file, README_SCAN for the README. So a 2 MB BUSL license gets rejected, and a 2 MB README is actually measured. Nothing over 5 MB is downloaded at all. A license file that big counts as no license text, which fails closed, and a README that big is Undecided. Undecided now means content that really can't be decoded.

A stuck issue no longer holds the sweeper. Any submission or report decide run that ends without a decision now writes undecided.json and exits 3. That covers Undecided, API errors, timeouts, and a failed rot dispatch from a dead report. The next step mints an issues-write token and counts the run in a hidden marker inside the bot's own comment. Only bot-authored comments count, so the author can't forge the number. On the third failed run, the issue gets needs-review and one error comment, and the sweeper moves on. If a maintainer's retry fails again, the label goes straight back on without another comment. If a rate limit's reset is still in the future, the run writes nothing, so waiting for it doesn't use up an attempt.

An undecidable entry no longer escapes the health check. Full runs keep a count of consecutive undecided weeks per entry in undecided.json on the stats branch. At 3 weeks, the run opens one needs-review issue, deduplicated by title among the bot's own open issues. One-repo runs and same-week retries don't add to the count, and a decided week resets it. The plan step only records the count. A separate step opens the issues, using its own token with just issues: write, so the main rot token keeps its original scopes.

Dead reports (I8). One-repo health checks are now named Health check: owner/name. A dead report skips the dispatch when that entry was checked in the last 24 hours. Once 3 one-repo checks have run in the past hour, new dead reports are parked as needs-review instead. The reports decide token gains actions: read to list runs. The sweeper's exact Health check match for full runs still works as before.

GraphQL reads. A document counts as a read only if it opens with query or { once any leading # comment lines are stripped. Anything else counts as a mutation and isn't retried.

Part of the v0.1 launch.

The contents API returns encoding "none" for a file over 1 MB. A big
LICENSE raised Undecided, and a big README passed on its size alone.
Both now come through the git blobs API and are capped at the usual
scan limits (100 KB per license file, README_SCAN for the README).
Undecided is left for content that genuinely can't be decoded.
decide now exits 3 on Undecided and writes undecided.json. A follow-up
step counts the run in a hidden marker inside the bot's own comment,
so a submitter can't forge it. On the third undecided run the issue
is labelled needs-review and gets the error comment once, and the
sweeper moves on to the next dropped submission.
Full runs now count consecutive undecided weeks per entry in
undecided.json on the stats branch. At 3 weeks the run opens one
needs-review issue, deduplicated by title among the bot's own open
issues. A one-repo run or a retry in the same week doesn't count, and
a decided week resets the count. A failure here is logged and never
stops the check itself.
Each dead report queued a one-repo health check in the shared
list-writes slot. One-repo runs are now named "Health check:
owner/name", and a dead report skips the dispatch when that entry was
checked in the last 24 hours. Past 5 one-repo checks in an hour, new
dead reports are parked as needs-review instead. The sweeper's exact
"Health check" match for full runs is unchanged.
A document counts as a read only if it opens with "query" or "{"
after any leading # comment lines. Anything else, including a
fragment-first document, is treated as a mutation and not retried
after it may have reached GitHub.
Only Undecided was bounded, so a submission or report whose decide
kept failing with an API error was re-dispatched on every sweep. Now
every run that ends without a decision writes undecided.json and is
counted, and the third parks the issue as needs-review with the error
comment. A failed rot dispatch from a dead report counts too. A rate
limit whose reset is still ahead writes nothing, so waiting for it
spends no attempt. The counter moves to scripts/devstream/attempts.py
so submit and reports share it.
The plan step now only records undecided weeks and lists the entries
due a review in plan.json. A separate step, with its own token scoped
to issues: write and nothing else, opens the needs-review issues. The
main rot token is back to its original scopes, and a failure in the
review step never stops the check.
The blobs API returns the full file, so a 100 MB license cost about
600 MB of memory on every run. The size from the contents response is
now checked first, and nothing over 5 MB is fetched. Such a license
file counts as no license text, which fails closed; rot's two-sighting
guard still applies. A README that size is Undecided, so the attempt
and review bounds take over.
Parking only happened when the count was exactly 3, so after a
maintainer removed needs-review to retry, the item could fail forever.
needs-review is now re-added on every failure from the third on, and
the error comment is still posted only once, at 3. At the limit the
counter comment no longer promises another try.
@jamesconwaydev
jamesconwaydev merged commit 2526079 into main Nov 6, 2026
8 checks passed
@jamesconwaydev
jamesconwaydev deleted the fix-review-wave3 branch November 6, 2026 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant