perf(bulk): batch user reloads and aggregate reset usage

[dr-hoseyn]

Summary

Bulk expiry, data-limit and group changes loaded admin/role, next plan, reset history and groups separately for each returned user. Bulk proxy updates already batched those relationships, but still materialized every reset-history row. For 100 affected users with 1,000 reset records each, this loaded 100,000 history objects after a single bulk operation.

Reuse the existing user-select builder to join scalar relationships, batch-load admin roles/groups and calculate lifetime usage with an indexed SQL aggregate. Apply the same reload to all five bulk operations. Group removal and proxy updates capture IDs before mutation rather than loading full user objects twice.

The reload uses populate_existing=True, which also fixes stale group collections: with expire_on_commit=False, a user whose groups were already loaded could retain the old membership after a bulk add/remove. The returned object now reflects the committed membership.

Before / after

Local benchmark: 100 users, 1,000 reset records per user, one shared admin/role and one group. Complete CRUD call, including target selection, mutation, commit and reload:

Operation	Median before → after	SQL statements before → after	Peak Python allocations before → after
Data limit	1,370.881 → 23.604 ms	306 → 6	115.061 → 0.777 MiB
Expiration	1,264.995 → 29.114 ms	306 → 6	115.061 → 0.774 MiB
Add groups	1,416.775 → 25.759 ms	306 → 6	115.095 → 0.800 MiB
Remove groups	1,385.899 → 24.804 ms	305 → 6	115.060 → 0.746 MiB
Proxy settings	1,958.861 → 23.752 ms	6 → 5	152.659 → 0.759 MiB

Reset-history ORM objects: 100,000 → 0 in every case. The benchmark asserts equal affected counts and complete serialized UserResponse values for both implementations, including lifetime usage, updated fields and group membership.

Method: Python 3.14.6 on Windows, SQLite in memory, baseline 234ab68c. One warmup per version, five timing samples per version with alternating execution order, fresh sessions and equivalent fixture state for each sample. SQL/object counts and tracemalloc peaks are measured in separate untimed runs. Fixture setup/reset, response validation and node dispatch are excluded. Peak allocations are not process RSS or database memory. These are local CRUD measurements, not production PostgreSQL/MySQL or end-to-end API latency claims.

Reproduce from the repository root:

python scripts/benchmark_bulk_user_loading.py --baseline 234ab68c --users 100 --logs 1000 --runs 5

The script uses its own in-memory database. A separate --users 2 --logs 0 --runs 1 smoke run also verified equal responses with no history.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor / cleanup
• Documentation
• Tests / CI

Testing

46 passed, including 20 new regression cases and existing node-sync, WireGuard and admin tests:

$env:DATABASE_URL = 'sqlite+aiosqlite:///:memory:'
.venv/Scripts/python.exe -m pytest tests/test_bulk_user_loading.py tests/test_node_sync.py tests/test_node_manager_sync.py tests/test_wireguard_alloc_unit.py tests/test_review_admins_unit.py -q

The database override isolates the existing admin test suite from a local test.db referencing a migration absent from this branch. The new bulk tests always create their own in-memory database.

Regression coverage includes all five operations, full response validation, absence of history-object loads, positive/negative adjustments, unchanged and empty scopes, cached membership refresh, and query growth across SQLAlchemy's 500-parent select-in batch boundary. Before the fix, the initial regressions reproduced history loads, stale memberships, and 192 SQL statements for 62 returned users; the latter now uses 6.

ruff check --no-fix, ruff format --check on all three changed files, and git diff --check passed.

Checklist

• I tested the change locally or explained why it cannot be tested.
• I added or updated tests for behavior changes.
• I updated documentation, translations, or examples if needed. Reproduction instructions are included in the benchmark.
• I checked database migrations when models or schema changed. No model/schema changes.
• I did not include secrets, tokens, private keys, or unrelated changes.

Screenshots

Not applicable; no UI changes.

Notes for reviewers

• Independent of perf(users): avoid materializing reset history in lists #850/perf(nodes): avoid materializing reset history in lists #851/fix(dashboard): cancel obsolete API requests at the HTTP transport #852/perf(nats): avoid repeated serialization when sizing user batches #853 and perf(groups): return updates before large user sync #827; based directly on dev. Only app/db/crud/bulk.py changes application behavior.
• Existing filtering, status transitions, affected counts and sync dispatch remain in place. Admin roles and next plans remain available, and proxy results preserve captured ID order. Direct usage_logs access is no longer eagerly supported by these bulk results; current callers only use them for allocation/node sync. Lifetime values remain available through the aggregate.
• SQL still scans the matching indexed reset rows for the sum; this removes transfer/materialization of history, not the database's aggregation work. Relationship queries grow by select-in batches for larger populations, not one fixed query count for every size. Small populations with no history can have slightly more query bookkeeping/allocation overhead.
• No dependencies, migrations or changes to retention. Reset-history rows remain intact.

Summary by CodeRabbit

• Improvements
  • Bulk user-management operations now more reliably reflect updated group memberships, expiration status, data limits, and proxy settings.
  • Results remain accurate when processing large user selections and users with previously loaded information.
  • No-op and empty selections are handled cleanly, and only newly affected users are reported for certain status changes.
• Performance
  • Bulk operations are more efficient when processing many users.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: ba05c6e0-ffa8-4202-90f2-05e855aeb202

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 1f847f5a-c4c5-4046-85c7-7300f24604fc

📥 Commits

Reviewing files that changed from the base of the PR and between 234ab68 and 2bd3db6.

📒 Files selected for processing (3)

• app/db/crud/bulk.py
• scripts/benchmark_bulk_user_loading.py
• tests/test_bulk_user_loading.py

Included review availability: Your plan provides up to 8 included reviews per hour; 5 remain after this review.

---

Walkthrough

Bulk CRUD operations now capture affected user IDs and reload users through a shared batched query. New tests cover response data, relationship refreshes, query counts, state changes, and empty scopes. A benchmark compares the current implementation with a baseline.

Changes

Bulk user reload optimization

Layer / File(s)	Summary
Shared bulk reload flow app/db/crud/bulk.py	Bulk operations use _bulk_user_reload_stmt() to reload affected users with batched relationships, lifetime traffic, and populate_existing=True. Group removal and proxy updates capture only user IDs before mutation.
Bulk reload behavior tests tests/test_bulk_user_loading.py	Tests verify response fields, group refreshes, bounded query counts, negative adjustments, no-op scopes, empty scopes, and avoidance of UserUsageResetLogs history loading.
Baseline benchmark harness scripts/benchmark_bulk_user_loading.py	The script compares baseline and current implementations across bulk actions using timing, SQL counts, load-event counts, allocation data, and response equality.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: ⚪ Minimal · up to 2bd3d

Bulk user operations now reload affected users in batches and calculate lifetime usage without materializing reset-history rows. The covered behavior preserves returned data and state transitions, with no current merge-blocking risk identified.

Poem

I twitch my nose at queries neat,
Batched reloads make hops complete.
The user rows return in line,
With fresh group state and traffic fine.
Tests and benchmarks softly cheer.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 22 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main changes: batching bulk user reloads and aggregating reset usage for performance.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[dr-hoseyn]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.