Skip to content

perf(api): drop the extra COUNT query from OffsetPaginator - #9948

Open
shivsin25 wants to merge 1 commit into
makeplane:previewfrom
shivsin25:perf/paginator-redundant-count
Open

shivsin25 wants to merge 1 commit into
makeplane:previewfrom
shivsin25:perf/paginator-redundant-count

Conversation

@shivsin25

@shivsin25 shivsin25 commented Oct 5, 2026 •

Copy link
Copy Markdown

Description

OffsetPaginator.get_result() runs two COUNT queries for every page:

total_count = self.total_count_queryset.count() if self.total_count_queryset else queryset.count()
...
next_cursor = Cursor(limit, page + 1, False, page_results.count() > limit)

The second one exists only to answer "is there a next page?". It counts the rows in the current page window (queryset[offset : offset + limit + 1]) and checks whether there are more than limit of them.

But total_count already answers that. If the total row count runs past the end of the current window, there's a next page:

has_next = total_count > offset + limit

So the page_results queryset and its COUNT can go away entirely. One less query per page.

This is worth doing because OffsetPaginator is the default paginator in BasePaginator.paginate(), which every list endpoint in plane/api, plane/app and plane/space goes through.

Why the two are equivalent

page_results is queryset[offset : offset + limit + 1], so its count is min(limit + 1, total_count - offset). Therefore:

page_results.count() > limit
  ⟺ min(limit + 1, total_count - offset) > limit
  ⟺ total_count - offset > limit
  ⟺ total_count > offset + limit

Exact, including at the boundaries — an exactly-full final page still correctly reports no next page. Those cases are covered by parametrised tests below.

When total_count_queryset is passed in, it's a pre-annotation copy of the same filtered queryset (copy.deepcopy(issue_queryset) in issue/base.py, view/base.py, module/issue.py and others) — same rows, just cheaper to count — so the equivalence holds there too.

One thing I deliberately did not do

The tempting version of this fix is to materialise the page and use len():

results = list(queryset[offset:stop])
has_next = len(results) > limit

That breaks on_results. issue_on_results in plane/utils/grouper.py does list(issues.values(*required_fields)) on whatever it's handed, so results has to stay an unevaluated queryset. This change leaves it completely alone, and there's a test pinning that behaviour so it doesn't regress later.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • Feature (non-breaking change which adds functionality)
  • Improvement (change that would cause existing functionality to not work as expected)
  • Code refactoring
  • Performance improvements
  • Documentation update

Screenshots and Media (if applicable)

N/A — no user-facing change. Same response payload, one less query per page.

Test Scenarios

Added to the existing plane/tests/unit/utils/test_paginator.py. No database needed — they use a small fake queryset that records count() calls and whether the result set was evaluated.

  • test_next_page_flag_matches_row_count — parametrised over 8 cases including the boundaries: empty result set, exactly one full page, one row past the boundary, exact multiples, and the final partial page.
  • test_page_costs_a_single_count_query — asserts count() is called exactly once per page.
  • test_total_count_queryset_is_counted_instead_of_the_page_queryset — when total_count_queryset is supplied, that one is counted and the page queryset is not.
  • test_results_are_handed_back_unevaluated — guards the on_results contract described above.

The 5 existing GHSA-wwgj-929g-42cm regression tests in the same file still pass.

docker compose -f docker-compose-test.yml run --rm api-tests \
  pytest plane/tests/unit/utils/test_paginator.py -vv
platform linux -- Python 3.12.5, pytest-9.0.3, pluggy-1.6.0 -- /usr/local/bin/python
cachedir: .pytest_cache
django: version: 5.2.15, settings: plane.settings.test (from env)
rootdir: /code
configfile: pytest.ini
plugins: mock-3.11.1, xdist-3.3.1, cov-4.1.0, django-4.12.0, anyio-4.15.1, Faker-25.0.0
collected 16 items

plane/tests/unit/utils/test_paginator.py::TestPaginateGroupByValidation::test_invalid_group_by_raises_parse_error PASSED
plane/tests/unit/utils/test_paginator.py::TestPaginateGroupByValidation::test_invalid_sub_group_by_raises_parse_error PASSED
plane/tests/unit/utils/test_paginator.py::TestPaginateGroupByValidation::test_unrecognised_field_never_reaches_paginator_constructor PASSED
plane/tests/unit/utils/test_paginator.py::TestPaginateGroupByValidation::test_valid_group_by_and_sub_group_by_pass_through PASSED
plane/tests/unit/utils/test_paginator.py::TestPaginateGroupByValidation::test_no_group_by_is_unaffected PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[25-0-True] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[25-1-True] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[25-2-False] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[0-0-False] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[10-0-False] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[11-0-True] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[20-1-False] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_next_page_flag_matches_row_count[21-1-True] PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_page_costs_a_single_count_query PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_total_count_queryset_is_counted_instead_of_the_page_queryset PASSED
plane/tests/unit/utils/test_paginator.py::TestOffsetPaginatorNextPage::test_results_are_handed_back_unevaluated PASSED

=============================== 16 passed ===============================

ruff check and ruff format --check are clean on both files.

References

No existing issue — spotted while reading the paginator. Happy to open one first if that's the preferred route for perf changes.

@coderabbitai

coderabbitai Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 193aee2f-cf8d-42d0-bf12-7c3f3b08e3d5
📥 Commits

Reviewing files that changed from the base of the PR and between 7466675 and 0b3afea.

📒 Files selected for processing (2)
  • apps/api/plane/tests/unit/utils/test_paginator.py
  • apps/api/plane/utils/paginator.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

OffsetPaginator now determines whether a next page exists by comparing the total count with the offset and limit. Unit tests cover next-page flags, count usage, and lazy result handling.

Changes

Offset pagination

Layer / File(s) Summary
Count-based next-page flag
apps/api/plane/utils/paginator.py, apps/api/plane/tests/unit/utils/test_paginator.py
OffsetPaginator derives the next cursor's has_results flag from the total count, offset, and limit. Tests cover empty and boundary-sized results, count-query selection, and unevaluated results.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Refactor

Merge Risk: ⚪ Minimal · up to 0b3af

The pagination change is ready to merge after normal checks; no actionable regression was established.

Security Architecture Review

Security architecture risk: ⚪ Minimal · up to 0b3af

The change removes a redundant page-window count without changing which rows are returned, access controls, or response fields. No introduced security concern was identified. Equivalent next-page behavior depends on consistent counts and page contents.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed behavior affects next-page metadata for users of this offset paginator. It does not itself broaden the supplied queryset, introduce another data source, or grant additional authority.

Trust Boundaries and Controls

  • observed — Request-supplied page size and cursor continue through the existing parsing and validation path. Offset bounds and grouped-field allowlist checks are unchanged by the inspected production diff.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly and concisely describes the main change: removing the extra COUNT query from OffsetPaginator.
Description check ✅ Passed The description is complete. It explains the change and its rationale, identifies the performance improvement, lists test scenarios and results, and addresses each template section.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

OffsetPaginator ran two COUNT queries per page: one for total_count, and
a second one over the page window purely to work out whether a next page
exists. total_count already answers that, so the second query is redundant.

Every list endpoint in plane/api, plane/app and plane/space goes through
this paginator, so this is one less query per list request across the board.

results is deliberately left as an unevaluated queryset -- on_results
callbacks call queryset methods on it (issue_on_results does .values()).

Adds unit tests covering the next-page flag at the page boundaries, the
query count, and that results is handed back unevaluated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant