perf(statistics): reduce query load on large installs by dr-hoseyn · Pull Request #865 · PasarGuard/panel

dr-hoseyn · 2026-09-04T13:09:22Z

Summary

reduce /api/users/sub_update/chart from two history aggregations to one by deriving pie totals from the period rows
remove unnecessary users joins from owner-wide subscription history and unscoped online-count queries
add a created_at index for owner-wide user_subscription_updates time ranges
default subscription history to one month and move aggregate user-stat polling from 2s to 30s while keeping live resource polling at 2s
document the query-load safeguards, migration caveat, and directional benchmark

Closes #858

Type of change

Checklist

I tested the change locally or explained why it cannot be tested.
I added or updated tests for behavior changes.
I updated documentation, translations, or examples if needed.
I checked database migrations when models or schema changed.
I did not include secrets, tokens, private keys, or unrelated changes.

Testing

.venv/Scripts/python.exe -m pytest -q tests/test_statistics_query_load.py tests/test_system_user_metrics.py tests/api/test_usage_functions_timezone.py — 32 passed
targeted Ruff lint and format checks — passed
npm run build in dashboard — passed
fresh SQLite alembic upgrade head, downgrade to 7c4bd5128e62, and upgrade again — passed; the new index appeared, disappeared, and reappeared as expected
MySQL DDL compilation — CREATE INDEX idx_user_subscription_updates_created_at ON user_subscription_updates (created_at)

Directional in-memory SQLite benchmark, 400,000 rows per history table, median of seven warmed runs:

Query shape	Before	After	Change
Unscoped online count	406.16 ms	325.05 ms	-20.0%
Subscription chart	896.68 ms	502.13 ms	-44.0%

These numbers validate direction and query shape; they are not a production MySQL latency guarantee. The reporter's offered EXPLAIN ANALYZE output remains the best follow-up validation on the affected installation.

The three standalone tests/api/test_user.py permission cases were also attempted, but the current API harness did not seed the testadmin login and failed in the existing access_token fixture before reaching the endpoint. The operation path and response equivalence are covered by the new database-backed test.

Screenshots

Not applicable. The UI changes only adjust polling and the initial time range; layout is unchanged.

Notes for reviewers

I checked related prior and in-flight work before implementing this:

perf(system): collapse dashboard user metrics into one query #791 is already merged and reduces /api/system/users to one SQL statement; this PR keeps it and reduces how often the full aggregate runs on the Statistics page.
perf(frontend): integrate page visibility API for smart polling #604 slows polling only when the document is hidden. This issue reproduces while the tab is visible, so the changes are complementary.
feat(cleanup): add automatic retention policies #801 adds retention policies, which limit future table growth but do not remove the current duplicate scan or joins.
perf(db): bound multi-worker connection pools #789 bounds connection pools and fix(dashboard): cancel obsolete API requests at the HTTP transport #852 forwards request cancellation; neither changes these query shapes.
perf(usage): coalesce PostgreSQL history writes #787 targets PostgreSQL write amplification and intentionally leaves MySQL unchanged.

The new index must scan the existing subscription-update table. Large MySQL/MariaDB installations should apply this migration during a maintenance window because DDL locking depends on server version and table configuration. All-time history remains available as an explicit chart selection.

Summary by CodeRabbit

Performance
- Improved Statistics page query performance for subscription history and user metrics.
- Added faster time-based processing for subscription update records.
- Reduced unnecessary database lookups in common statistics views.
Usability
- Subscription charts now default to displaying the past month.
- Statistics refresh less frequently to reduce system load.
Documentation
- Added guidance on Statistics page performance behavior and verification.
Tests
- Added coverage for subscription chart totals and query behavior.

coderabbitai · 2026-09-04T13:09:28Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI
Review profile: CHILL
Plan: Advanced
Run ID: fae83d2a-3dcd-494d-814e-b99cdb10e264

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 041d3b93-cb43-41bd-9b42-9696bfa5906c

📥 Commits

Reviewing files that changed from the base of the PR and between 630b33d and caec921.

📒 Files selected for processing (8)

app/db/crud/user.py
app/db/migrations/versions/86f7c2a14d3e_add_subscription_update_created_index.py
app/db/models.py
app/operation/user.py
dashboard/src/components/charts/user-sub-update-pie-chart.tsx
dashboard/src/pages/_dashboard.statistics.tsx
docs/statistics-performance.md
tests/test_statistics_query_load.py

Included review availability: Your plan provides up to 8 included reviews per hour; 6 remain after this review.

Walkthrough

The Statistics page now reduces database joins and chart queries, adds an index for subscription history, limits the initial history range, slows polling, and adds SQLite coverage and performance documentation.

Changes

Statistics performance changes

Layer / File(s)	Summary
Query paths and subscription index `app/db/crud/user.py`, `app/db/models.py`, `app/db/migrations/versions/...`	Subscription history queries avoid the `User` join unless required. Online user counts also avoid that join without admin filtering. Subscription history gains a `created_at` index with upgrade and downgrade support.
Chart aggregation and dashboard timing `app/operation/user.py`, `dashboard/src/components/charts/user-sub-update-pie-chart.tsx`, `dashboard/src/pages/_dashboard.statistics.tsx`	Chart agent totals are derived from period statistics. The chart defaults to one month. System user statistics polling changes to a 30-second interval with a 15-second stale time.
Performance validation and documentation `tests/test_statistics_query_load.py`, `docs/statistics-performance.md`	Tests verify query counts, chart totals, and conditional joins. Documentation records safeguards, migration notes, verification criteria, and benchmark results.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: x0sina

Merge Risk: ⚪ Minimal · up to caec9

This reduces Statistics-page query volume and polling frequency while retaining scoped query behavior and explicit all-time history selection. Current evidence indicates the change is ready to merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 7 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary change: reducing Statistics page query load for large installations.
Linked Issues check	✅ Passed	The changes address the requirements in [`#858`] by reducing duplicate aggregations and unnecessary users joins, adding the subscription-history created_at index, changing the default history range, and…
Out of Scope Changes check	✅ Passed	The code, migration, dashboard updates, tests, and performance documentation all support the linked issue objective of reducing Statistics page resource usage. No unrelated changes are evident.

Full details: Docstring Coverage

Explanation

Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 7 files. (1 skipped: 1 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

❌ Error creating Unit Test PR.

Create PR with unit tests
Commit unit tests in branch codex/fix-statistics-query-load

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.}

_{Comment @coderabbitai help to get the list of available commands.}

dr-hoseyn · 2026-09-04T13:13:42Z

@coderabbitai review

coderabbitai · 2026-09-04T13:13:53Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

ImMohammad20000 · 2026-09-07T16:42:15Z

remove docs

dr-hoseyn · 2026-09-08T22:34:44Z

Done

coderabbitai · 2026-09-10T08:14:17Z

Note

Unit test generation is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it.

Generating unit tests... This may take up to 20 minutes.

coderabbitai · 2026-09-10T08:14:49Z

Failed to create Coding Agent finishing-touch task.

…he current head The migration branched off an older revision, and a merge migration joined it to dev's head (b4c7e8f1a2d3). That works alone, but once another migration PR lands the same way, dev has two Alembic heads and `alembic upgrade head` fails with "Multiple head revisions". Point the migration at dev's head and drop the merge migration. Also drop an extra blank line after the imports that ruff flags (I001).

…the user list sort index The previous migration PR in the series (PasarGuard#774) has landed, so point this migration at its revision (b7e2c4d91f60) to keep a single Alembic head.

…he subscription update index The previous migration PR in the series (PasarGuard#865) has landed, so point this migration at its revision (86f7c2a14d3e) to keep a single Alembic head.

* feat(nodes): persist node and core config order * fix(db): batch reorder updates for sqlite limits * fix(migrations): merge parallel heads * fix(dashboard): address reorder review findings * fix(nodes): limit reorder locks and payload size * docs: cover reorder migration and operation paths * fix(ordering): serialize sqlite writes * fix(ci): align reorder limits and finalmask test * fix(ordering): bound reorders to supported scope * fix(migrations): chain the node and core sort order migration onto the current head The migration branched off an older revision, and a merge migration joined it to dev's head (b4c7e8f1a2d3). That works alone, but once another migration PR lands the same way, dev has two Alembic heads and `alembic upgrade head` fails with "Multiple head revisions". Point the migration at dev's head and drop the merge migration. Both merge migrations (b7f4e6d9a1c2 and c12e9a875f04) only joined heads and carried no schema changes. * fix(migrations): chain the node and core sort order migration after the subscription update index The previous migration PR in the series (#865) has landed, so point this migration at its revision (86f7c2a14d3e) to keep a single Alembic head. --------- Co-authored-by: T3ST3ST3R0N <T3ST3ST3R0N@gmail.com>

… node and core sort order The other migration PRs in the series (PasarGuard#774, PasarGuard#865, PasarGuard#875) have landed, so point this migration at the latest revision (d73f8a2c4e91) to keep a single Alembic head.

* perf(groups): scale membership summaries * test: remove added PR tests * test: remove PR-specific changes to existing tests * fix(migrations): chain the group membership index migration onto the current head The migration branched off an older revision, and a merge migration joined it to dev's head (b4c7e8f1a2d3). That works alone, but once another migration PR lands the same way, dev has two Alembic heads and `alembic upgrade head` fails with "Multiple head revisions". Point the migration at dev's head and drop the merge migration. Also drop an extra blank line after the imports that ruff flags (I001). * refactor(groups): keep only the reverse membership indexes The member count rewrite overlaps #967, which counts members in SQL in load_group_attrs without loading them and keeps a fallback when the count was not loaded. Leave Group.total_users and get_group as on dev so this PR adds only the reverse membership indexes, which those group-centric counts and bulk membership deletes use. * fix(migrations): chain the group membership index migration after the node and core sort order The other migration PRs in the series (#774, #865, #875) have landed, so point this migration at the latest revision (d73f8a2c4e91) to keep a single Alembic head. --------- Co-authored-by: T3ST3ST3R0N <T3ST3ST3R0N@gmail.com>

perf(statistics): reduce historical query load

6492ff9

dr-hoseyn mentioned this pull request Sep 4, 2026

[Bug]: Opening the Statistics page saturates all CPU cores on large installs #858

Closed

2 tasks done

test: remove added PR tests

7c2d07a

docs: remove statistics performance notes

089b4f4

fix(migrations): chain statistics index after latest dev migration

e16fa51

dr-hoseyn force-pushed the codex/fix-statistics-query-load branch from f89d62d to e16fa51 Compare October 8, 2026 22:29

Merge dev and unify statistics migration heads for CI

6f954c0

mark-avd mentioned this pull request Oct 10, 2026

feat(statistics): rank users by traffic with a per-node split #991

Open

7 of 11 tasks

T3ST3ST3R0N added 2 commits October 10, 2026 18:57

fix(migrations): chain the subscription update index migration after …

7449332

…the user list sort index The previous migration PR in the series (PasarGuard#774) has landed, so point this migration at its revision (b7e2c4d91f60) to keep a single Alembic head.

T3ST3ST3R0N merged commit b984fa3 into PasarGuard:dev Oct 10, 2026
10 checks passed

Navigation Menu

Repository navigation

perf(statistics): reduce query load on large installs - #865

perf(statistics): reduce query load on large installs#865
T3ST3ST3R0N merged 7 commits into
PasarGuard:devfrom
dr-hoseyn:codex/fix-statistics-query-load

dr-hoseyn commented Sep 4, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading

Review skipped

❌ Failed checks (1 warning)

Uh oh!

dr-hoseyn commented Sep 4, 2026

Uh oh!

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading

Uh oh!

ImMohammad20000 commented Sep 7, 2026

Uh oh!

dr-hoseyn commented Sep 8, 2026

Uh oh!

coderabbitai Bot commented Sep 10, 2026

Uh oh!

coderabbitai Bot commented Sep 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dr-hoseyn commented Sep 4, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of change

Checklist

Testing

Screenshots

Notes for reviewers

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

dr-hoseyn commented Sep 4, 2026

Uh oh!

coderabbitai Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ImMohammad20000 commented Sep 7, 2026

Uh oh!

dr-hoseyn commented Sep 8, 2026

Uh oh!

coderabbitai Bot commented Sep 10, 2026

Uh oh!

coderabbitai Bot commented Sep 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dr-hoseyn commented Sep 4, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading

coderabbitai Bot commented Sep 4, 2026 •

edited

Loading