perf(users): avoid full-table sorts in list queries by dr-hoseyn · Pull Request #774 · PasarGuard/panel

dr-hoseyn · 2026-08-14T14:50:30Z

Summary

remove page ordering and wide user columns from the /api/users total-count query
add a direction-matched id tie-breaker for deterministic offset pagination
add (created_at, id) for the dashboard's default user-list order
build/drop the PostgreSQL index concurrently to keep user writes available during upgrades
add cross-dialect SQL and index regression tests

Closes #773

Type of change

Checklist

I tested the change locally or explained why it cannot be tested.
I added or updated tests for behavior changes.
I updated documentation, translations, or examples if needed. (Not applicable: API schema is unchanged.)
I checked database migrations when models or schema changed.
I did not include secrets, tokens, private keys, or unrelated changes.

Testing

uv run ruff check . — passed
targeted Ruff format check — passed
uv run pytest -q tests/test_user_list_query_scaling.py — 4 passed
targeted endpoint/count tests — 6 passed
uv run pytest -q tests/api/test_user.py — 82 passed
uv run alembic check on SQLite — no new upgrade operations
empty SQLite migration chain to head — passed
one-revision downgrade/upgrade and index introspection — passed
PostgreSQL offline SQL — CREATE INDEX CONCURRENTLY / DROP INDEX CONCURRENTLY
MySQL migration compilation — passed

Synthetic SQLite benchmark with 500,000 users and a 50-row page:

Operation	Before	After
Total count	203.143 ms median	0.224 ms median
Default page	44.950 ms median	0.052 ms median

Before the change both plans used a full scan and temporary B-tree sort. The optimized page uses idx_users_created_at_id; the count no longer sorts. The benchmark is directional, not a production guarantee.

Screenshots

Not applicable (backend/database-only change).

Notes for reviewers

Response fields, filters, and pagination parameters are unchanged.
Owner/admin joins remain in the count statement; regression coverage verifies this.
PostgreSQL concurrent DDL is isolated with Alembic's autocommit_block().
This PR and perf(groups): add reverse group membership indexes #772 both independently target the current dev migration head. If one lands first, the other migration should be rebased onto the new head before merge to avoid parallel Alembic heads.

Summary by CodeRabbit

Performance
- Improved user list loading and counting, especially for filtered results.
- Added a database index to support faster, more consistent user sorting.
Bug Fixes
- User lists now maintain stable ordering when multiple users share the same creation time.
- User counts preserve existing filters and owner-related conditions accurately.

dr-hoseyn · 2026-08-14T14:50:49Z

@coderabbitai review

coderabbitai · 2026-08-14T14:50:51Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: 85e9ee48-562e-4418-938a-88368851342e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ebc329ec-9c3f-4c8d-9c17-d974f3d21ad7

📥 Commits

Reviewing files that changed from the base of the PR and between 3331421 and 3b4e1cd.

📒 Files selected for processing (4)

app/db/crud/user.py
app/db/migrations/versions/b7e2c4d91f60_add_user_list_sort_index.py
app/db/models.py
tests/test_user_list_query_scaling.py

Walkthrough

User listing now uses deterministic sorting and narrow filtered count queries. The database schema adds a (created_at, id) index, with dialect-specific migration handling. Tests cover query construction, filters, sorting, and index metadata.

Changes

User list query scaling

Layer / File(s)	Summary
Deterministic sorting and filtered counts `app/db/crud/user.py`, `tests/test_user_list_query_scaling.py`	`get_users` now uses centralized sort clauses with an ID tie-breaker matching the final direction. Count queries remove ordering, count `User.id`, preserve joins and filters, and exclude wide user columns.
Composite sort index and migration `app/db/models.py`, `app/db/migrations/versions/...`, `tests/test_user_list_query_scaling.py`	The `User` model defines the `(created_at, id)` index. The migration creates and removes it, using concurrent PostgreSQL operations and standard operations for other dialects. Tests verify the index name and column order.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: immohammad20000, m03ed

Merge Risk: ⚪ Minimal · up to 3b4e1

This change optimizes user-list counting and pagination while preserving the API contract; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the primary performance change for user list queries.
Linked Issues check	✅ Passed	The changes implement the count rewrite, deterministic sorting, composite index, concurrent PostgreSQL DDL, and tests required by [`#773`].
Out of Scope Changes check	✅ Passed	All code changes directly support the user-list query performance and compatibility objectives in [`#773`].
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{A rabbit sorted users in line,
With dates and IDs aligned just fine.
Counts grew light, orders stayed true,
An index joined the query crew.
“Hop!” said the rabbit, “Less work to do!”}

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai · 2026-08-14T14:51:05Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…rrent head The migration branched off an older revision, and a merge migration joined it to dev's head (b4c7e8f1a2d3). That works alone, but once another migration PR lands the same way, dev has two Alembic heads and `alembic upgrade head` fails with "Multiple head revisions". Point the migration at dev's head and drop the merge migration. Also drop an extra blank line after the imports that ruff flags (I001).

Resolve app/db/crud/user.py after the HWID count change: keep this PR's _build_user_sort_clauses, which already routes every option through _build_user_sort_clause (including the hwid_count subquery) and adds an id tie-break for every sort, and its _build_user_count_stmt.

…the user list sort index The previous migration PR in the series (PasarGuard#774) has landed, so point this migration at its revision (b7e2c4d91f60) to keep a single Alembic head.

* perf(statistics): reduce historical query load * test: remove added PR tests * docs: remove statistics performance notes * fix(migrations): chain statistics index after latest dev migration * fix(migrations): chain the subscription update index migration onto the current head The migration branched off an older revision, and a merge migration joined it to dev's head (b4c7e8f1a2d3). That works alone, but once another migration PR lands the same way, dev has two Alembic heads and `alembic upgrade head` fails with "Multiple head revisions". Point the migration at dev's head and drop the merge migration. Also drop an extra blank line after the imports that ruff flags (I001). * fix(migrations): chain the subscription update index migration after the user list sort index The previous migration PR in the series (#774) has landed, so point this migration at its revision (b7e2c4d91f60) to keep a single Alembic head. --------- Co-authored-by: T3ST3ST3R0N <T3ST3ST3R0N@gmail.com>

… node and core sort order The other migration PRs in the series (PasarGuard#774, PasarGuard#865, PasarGuard#875) have landed, so point this migration at the latest revision (d73f8a2c4e91) to keep a single Alembic head.

* perf(groups): scale membership summaries * test: remove added PR tests * test: remove PR-specific changes to existing tests * fix(migrations): chain the group membership index migration onto the current head The migration branched off an older revision, and a merge migration joined it to dev's head (b4c7e8f1a2d3). That works alone, but once another migration PR lands the same way, dev has two Alembic heads and `alembic upgrade head` fails with "Multiple head revisions". Point the migration at dev's head and drop the merge migration. Also drop an extra blank line after the imports that ruff flags (I001). * refactor(groups): keep only the reverse membership indexes The member count rewrite overlaps #967, which counts members in SQL in load_group_attrs without loading them and keeps a fallback when the count was not loaded. Leave Group.total_users and get_group as on dev so this PR adds only the reverse membership indexes, which those group-centric counts and bulk membership deletes use. * fix(migrations): chain the group membership index migration after the node and core sort order The other migration PRs in the series (#774, #865, #875) have landed, so point this migration at the latest revision (d73f8a2c4e91) to keep a single Alembic head. --------- Co-authored-by: T3ST3ST3R0N <T3ST3ST3R0N@gmail.com>

perf(users): scale sorted list queries

922a1ac

dr-hoseyn added 2 commits September 6, 2026 14:44

test: remove added PR tests

e61935a

Merge upstream dev and rebase user sort migration

181187c

dr-hoseyn force-pushed the perf/user-list-count-and-sort branch from fd99ff4 to 181187c Compare October 8, 2026 22:29

dr-hoseyn and others added 5 commits October 9, 2026 02:17

Merge dev and unify migration heads for user list CI

5de2672

Merge dev and preserve scalable user listing with HWID sorting

1a4c3ec

Integrate concurrent PR update and retain latest dev

177c66a

T3ST3ST3R0N merged commit 61bf853 into PasarGuard:dev Oct 10, 2026
9 checks passed

T3ST3ST3R0N mentioned this pull request Oct 10, 2026

Avoid full-table sorts in paginated user lists #773

Closed

Navigation Menu

Repository navigation

perf(users): avoid full-table sorts in list queries - #774

perf(users): avoid full-table sorts in list queries#774
T3ST3ST3R0N merged 8 commits into
PasarGuard:devfrom
dr-hoseyn:perf/user-list-count-and-sort

dr-hoseyn commented Aug 14, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

dr-hoseyn commented Aug 14, 2026

Uh oh!

coderabbitai Bot commented Aug 14, 2026 •

edited

Loading

Review skipped

Uh oh!

coderabbitai Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dr-hoseyn commented Aug 14, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of change

Checklist

Testing

Screenshots

Notes for reviewers

Summary by CodeRabbit

Uh oh!

dr-hoseyn commented Aug 14, 2026

Uh oh!

coderabbitai Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Uh oh!

coderabbitai Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dr-hoseyn commented Aug 14, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 14, 2026 •

edited

Loading

coderabbitai Bot commented Aug 14, 2026 •

edited

Loading