Repository navigation
perf(meta): lower a repo list filter once per query on PostgreSQL - #785
Conversation
Change-Id: I5e7433bf15297431dc114deb5bfa612fbe2034cc Signed-off-by: Thomas Kosiewski <tk@coder.com>
Change-Id: I97e7a315892d46babb307f4727baf7529285f881 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Change-Id: I2f2387113dadc6e2dfd9138ea7840b91dc232217 Signed-off-by: Thomas Kosiewski <tk@coder.com>
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Keep it up! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Why
On PostgreSQL,
GET /repos?q=and?prefix=cost about rows × filter length.url ILIKE plowers the patternpagain for every row it tests. In a multibyte database,Generic_Text_IC_like(like.cin PostgreSQL 17 and 18) callslower()on the pattern and on the url, and then matches the two asLIKEdoes. So oneorg:readrequest with a longqorprefixcan hold a PostgreSQL backend for seconds or minutes (numbers below). SQLite and S3-WAL already lower the filter once per query (#783).What changes
ListReposfilters withlower(url) LIKE (SELECT lower($2::text))instead ofurl ILIKE $2. That is the matchILIKEcomputes for each row, so every answer stays the same. The scalar subquery is an InitPlan: PostgreSQL lowers the pattern once per query, in custom and generic plans alike. The per-row match is bounded by the url's length, not by the filter's.pgListReposQuery, so a test can plan the statement thatListReposruns.internal/meta/README.mdstates the invariant. The SQLite comment that citedurl ILIKE ?names the new form.Why not answer an empty page above a url length bound
CreateRepoandCreateRepoWithStatewriteurl(PostgreSQL, SQLite and S3-WAL), and the API takes it from the token'srepoclaim without a length check.(org_id, url)btree index limits the compressed index entry, not the url. The test PostgreSQL stored a 100,005-byte url of one repeated letter and refused a 4,005-byte varied one. So no length bounds every url, and a shortcut above some length would change answers.Verification
TestPGListReposLowersFilterOnceplans theListReposstatement withEXPLAIN (GENERIC_PLAN), the kind of plan that pgx's statement cache can run. It requires an InitPlan, no read of the pattern parameter anywhere in the plan, and a filter oflower(url) ~~(LIKE) with no~~*(ILIKE). ILIKE lowers its pattern again for each row even when an InitPlan computes that pattern. On main'surl ILIKE $2it fails: the scan's filter reads$2for every row.TestPGListReposMatchesILIKEruns 592 filters over 23 urls and compares each page withurl ILIKEon the same rows: the same urls in the same order. The urls fold outside ASCII (ß and ẞ, İ and ı, Σ σ ς, Dž, ffi, the Kelvin and Ångström signs, a combining accent) and hold%,_and\. The filters are each url rune in every case Go knows for it, alone and afteracme/, and each start of each url.TestListReposLongFilters(aqand aprefixof 200 KiB list nothing, and 30,000 percent signs match only their url) passes unchanged on PostgreSQL, SQLite and in-memory S3-WAL.url ILIKE $2,lower(url) LIKE lower($2::text)without the subquery,url ILIKE (SELECT lower($2::text))andlower(url) ILIKE (SELECT lower($2::text))fail onlyTestPGListReposLowersFilterOnce. A url that is not lowered fails it,TestPGListReposMatchesILIKEand the existingListRepostests. A pattern that is not lowered andESCAPE '#'failTestPGListReposMatchesILIKEand the existingListRepostests.en_US.utf8, median of 3 runs. The host load average (1 min) read 6.10 at the start and 93.94 at the end.ILIKE)qof 49,990 bytesqof 200 KiBqof 1 MiBprefixof 200 KiBprefixof 1 MiB-race: build, vet (with and without-tags=openstorage_viewcheck), golangci-lint (0 issues), the doc-link check, the script unit tests and both--fingerprints-onlychecks pass.internal/metaandinternal/apipass with-tags=openstorage_viewcheckon PostgreSQL and SQLite.Size
205 changed lines: 51 of production Go, 6 of a SQLite comment, 147 of tests and 1 of docs.
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:xhigh