perf(group): make group list, sync, bulk operations and delete scale to 50k+ users

[ChownYourLife]

Summary

Adding an inbound to a group with many users (PUT /api/group/{id}) is very slow, fails outright on PostgreSQL above ~33k members, and listing groups loads far more data than it needs. Bulk requests that name many users (or touch every user) hit the same bind limit and the same per-user inbound queries. Seven root causes:

1. Group summaries hydrate every member just to count them. Group.total_users was len(self.users) (app/db/models.py). get_group used selectinload(Group.users) (app/db/crud/group.py), and load_group_attrs awaited group.awaitable_attrs.users for get_group_by_id (used by get/modify/delete), create_group, modify_group and the bulk enable/disable path. Every one of those loaded all User rows for a number.

   • total_users is now a SQL COUNT: with_expression(Group._total_users_query, Group.total_users) in get_group / get_group_by_id, and a single COUNT in load_group_attrs. The total_users hybrid uses the loaded value and falls back to len(users), so API responses are unchanged. Same query_expression mechanism as User._reseted_usage_query.

2. Syncing a group's users to nodes issued one inbound query per user. GroupOperation.modify_group (app/operation/group.py) re-syncs every member. get_users only did selectinload(User.groups) (app/db/crud/user.py), so Group.inbounds was never loaded and User.inbounds() fell back to one SELECT DISTINCT inbounds.tag ... per user. The sync path also loaded usage_logs for every user, which it does not use.

   • New get_users(load_group_inbounds=False) nests selectinload(Group.inbounds) under User.groups. The group operation paths that feed sync_users (modify, remove, bulk remove by id, bulk enable/disable) now pass load_group_inbounds=True, load_usage_logs=False.

3. One giant IN (...) over all member ids / usernames breaks PostgreSQL at ~33k+ users. asyncpg rejects a statement with more than 32,767 bind parameters (InterfaceError: the number of query arguments cannot exceed 32767). Two places build such a list from a group's members, and both fail on origin/dev at 50k users:

   • get_users_accessible_tags (app/db/crud/wireguard.py, called by sync_users_allocations from modify_group): ... WHERE users_groups_association.user_id IN (<every member id>). This runs after the group change has been committed, so the request errors and the nodes are never synced.
   • get_users(UserListQuery(username=...)) in remove_group and bulk_remove_groups_by_id, same failure after the group is already deleted.
   • get_users_accessible_tags now issues one query per 10k ids, and a new GroupOperation._get_users_for_sync loads users 10k usernames per query; it replaces the two username call sites. MariaDB has no such limit and was unaffected.

4. bulk_add_groups / bulk_remove_groups hit the same giant IN (...) and the same missing inbound preload. add_groups_to_users / remove_groups_from_users (app/db/crud/bulk.py) selected user_id IN (<every target id>) (twice or three times per call), which asyncpg rejects at 50k users on PostgreSQL. The users they reload for the node sync came from _bulk_user_reload_stmt() without their groups' inbounds, so User.inbounds() ran one query per affected user (MariaDB, 50k users: 20.9 s and ~50k statements for a bulk add).

   • The affected users are now found with a correlated SQL predicate (a membership COUNT for add, EXISTS for remove) instead of fetching all ids and then the existing pairs. Missing (user, group) pairs are written with one INSERT ... SELECT ... NOT EXISTS per chunk of users instead of building and executemany-inserting rows in Python. Insert / delete / reload statements take at most 5k ids (the group ids are bound next to them, so every statement stays at or below 10k bind parameters). Users are reloaded with load_group_inbounds=True and no usage logs (_reload_group_users; _build_user_select_stmt gets load_group_inbounds).

5. remove_group deleted association rows one by one and loaded every member first. db.delete(group) makes SQLAlchemy load Group.users and delete users_groups_association rows with an executemany of one DELETE per member (MariaDB, 50k users: ~4 s of ~7.7 s; PostgreSQL ~0.13 s). GroupOperation.remove_group and bulk_remove_groups_by_id also called get_users per group (full User rows) just to collect usernames.

   • crud.remove_group now calls remove_groups, which already deletes the association rows (users, templates, inbounds) with one statement each and then the group; the Group object stays usable for logging / notifications. get_group_usernames(db, group_ids) returns the distinct member usernames with one SELECT DISTINCT per 10k group ids, and both remove paths use it.
   • On origin/dev, remove_group also dispatched wrong inbounds. It loaded the members before the delete with get_users(db, query=UserListQuery(group_ids=[db_group.id])), and those User objects, with their User.groups (still containing the removed group), stayed in the session's identity map. After the delete, the get_users(UserListQuery(username=...)) reload returned those same identity-map objects with the stale User.groups, so members whose only group was the removed one were still sent the removed group's inbound (measured on TimescaleDB, 20k users: origin/dev dispatched 10,000 users with in-b and 10,000 with in-a, the deleted group's inbound; this PR dispatches 10,000 with in-b and 10,000 with no inbounds). Usernames are now collected with get_group_usernames (no User rows) and the users are reloaded fresh after the delete with their remaining groups' inbounds preloaded, so the sync sees the post-delete memberships.

6. A bulk request naming more than ~32k users fails on PostgreSQL. Explicit user ids are set[int] request fields with no size cap (BulkGroup.users, BulkUserFilter.users which BulkUser / BulkUsersProxy inherit, and BulkUsersSelection.ids for bulk disable / enable / status / reset usage / revoke / set owner / apply template / remove). Each went into one User.id IN (<every id>): _create_group_filter and _create_final_filter (app/db/crud/bulk.py, OR-combined with the other filter conditions and reused by the count, id-select and UPDATE statements), and get_users(ids=...), get_users_by_ids and remove_users (app/db/crud/user.py). asyncpg rejects the statement above 32,767 binds (InterfaceError, 40k ids at 50k users on origin/dev).

   • New _explicit_ids(column, ids) (app/db/crud/user.py, imported by bulk.py) builds column.in_(bindparam(None, ids, expanding=True, literal_execute=True)). SQLAlchemy writes the integers into the SQL text at execution time, so the statement binds no id parameters on PostgreSQL, MariaDB and SQLite, keeps the OR semantics and needs no dialect branch. It is one line per call site.
   • Why not the alternatives: chunking cannot work for a filter OR-combined with other conditions without resolving the matching ids first and rewriting every statement per chunk (otherwise the other conditions are re-evaluated per chunk, and relative updates such as the expire / datalimit amount could be applied twice to a user matching both parts). = ANY(:array) is a single bind but PostgreSQL-only, so it needs a dialect branch plus a second path for MariaDB / SQLite. A temp table needs DDL and an insert per request, again per dialect.
   • Trade-off: the statement text grows with the list (about 7 bytes per id; 40k ids were run on PostgreSQL and MariaDB, see the tables below). SQLite's default 1,000,000-byte SQL limit is reached at roughly 140k ids; SQLite is not a production backend here.

7. Other bulk operations reloaded users without their groups' inbounds, and bound every affected id.

   • update_users_expire / update_users_datalimit / update_users_proxy_settings (app/db/crud/bulk.py) reloaded the affected users with _bulk_user_reload_stmt().where(User.id.in_(<all ids>)) and no inbound preload, so the node sync ran one User.inbounds() query per user (MariaDB, 50k users: 40,085 statements for one expire) and the IN failed on PostgreSQL at 50k. They now use _reload_group_users (5k ids per statement, inbounds preloaded, no usage logs).
   • reset_all_users_data_usage selected every user id and bound them in one UPDATE and three DELETEs (asyncpg InterfaceError at 50k). It now updates with the admin filter (or none) and deletes with a User.id subquery, so no ids are bound.
   • AdminOperation disable-all / activate-all loaded all of the admin's users with usage logs and no inbounds before syncing (activate-all, 50k: 25,157 statements). Both skip usage logs; activate-all also preloads group inbounds (disable-all leaves no active user, so inbounds are unused there).
   • UserOperation._load_users_by_ids, used only to reload users for a node sync after bulk status / disable / enable / apply template, now calls get_users_by_ids(load_group_inbounds=True).

Files

• app/db/crud/bulk.py: add_groups_to_users / remove_groups_from_users rewritten as above; _chunked, _reload_group_users (also used by the expire / datalimit / proxy-settings reloads); _create_group_filter / _create_final_filter use _explicit_ids; reset_all_users_data_usage binds no ids.
• app/db/crud/group.py: remove_group delegates to remove_groups; get_group_usernames; SQL COUNT for total_users.
• app/db/crud/user.py: get_users(load_group_inbounds=...), _build_user_select_stmt(load_group_inbounds=...); _explicit_ids used by get_users(ids=...), get_users_by_ids(load_group_inbounds=...) and remove_users.
• app/db/crud/wireguard.py: chunked get_users_accessible_tags.
• app/db/models.py: Group._total_users_query / total_users.
• app/operation/group.py: preload group inbounds for syncs, _get_users_for_sync, username query on both remove paths.
• app/operation/admin.py: disable-all / activate-all skip usage logs; activate-all preloads group inbounds.
• app/operation/user.py: _load_users_by_ids preloads group inbounds for the node sync.
• tests/test_group_user_loading.py: regression tests.

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor / cleanup (performance)
• Documentation
• Tests / CI

Checklist

• I tested the change locally or explained why it cannot be tested.
• I added or updated tests for behavior changes.
• I updated documentation, translations, or examples if needed. (Not needed: no API or behavior change.)
• I checked database migrations when models or schema changed. (query_expression adds no column; make check-migrations is clean on PostgreSQL, MariaDB and TimescaleDB.)
• I did not include secrets, tokens, private keys, or unrelated changes.

Testing

Run against PostgreSQL (postgres:latest, 18.6) and MariaDB (mariadb:10.6, same images as .github/workflows/test-database-migrations.yml):

# PostgreSQL
export TEST_FROM=github TESTING=1 SQLALCHEMY_DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:55432/<db>
# MariaDB
export TEST_FROM=github TESTING=1 SQLALCHEMY_DATABASE_URL=mysql+asyncmy://testuser:testpassword@localhost:53306/<db> \
  NATS_ENABLED=1 NATS_URL=nats://localhost:54222 UVICORN_WORKERS=2 ROLE=all-in-one

make run-migration && make check-migrations && make test           # fresh database per run
uv run pytest tests/test_group_user_loading.py tests/api/test_group.py -q   # 35 passed on SQLite (also part of the full suites below)
uv run pytest tests/test_group_user_loading.py tests/api/test_group.py tests/api -k "group or bulk" -q   # 86 passed on SQLite
uv run ruff check <changed files> && uv run ruff format --check <changed files>   # clean

Full suite (make test) on this branch, fresh Alembic-migrated database each, make check-migrations clean on both:

Database	Result
MariaDB 10.6	749 passed, 34 skipped, 0 failed
PostgreSQL 18.6	733 passed, 34 skipped, 16 failed
TimescaleDB (timescale/timescaledb:latest-pg17, PG 17.11, extension 2.30.2)	733 passed, 34 skipped, 16 failed (origin/dev: 712 passed, 34 skipped, 16 failed, same 16)

The 16 PostgreSQL failures are pre-existing and unrelated: they fail identically on a clean origin/dev checkout (test_admin_usage_returns_stats_for_admin, 4x test_subscription_reads_see_updates_after_request_snapshot, and 11 tests in tests/api/test_usage_functions_timezone.py).

TimescaleDB was run like the test-timescaledb CI job: timescale/timescaledb:latest-pg17 (POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres POSTGRES_HOST_AUTH_METHOD=trust), TEST_FROM=github TESTING=1 SQLALCHEMY_DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:<port>/<db> NATS_ENABLED=1 NATS_URL=nats://localhost:<port> UVICORN_WORKERS=2, nats:2.10-alpine -js, fresh database per run, make run-migration && make check-migrations && ROLE=all-in-one make test, on this branch and on origin/dev. Migrations and check-migrations pass on both. The failing set is identical on both (the same 16 pre-existing failures listed above), so the PR adds no failure on TimescaleDB. uv run pytest tests/test_group_user_loading.py tests/api/test_group.py -q on the branch against that database: 35 passed. The app and its migrations create no hypertables (timescaledb_information.hypertables is empty after make run-migration), so on TimescaleDB the touched tables are ordinary PostgreSQL tables. As an extra check, node_user_usages was converted to a hypertable by hand (PK (id, created_at), 1-day chunks, 30 chunks, 100k rows): reset_all_users_data_usage(clean_chart_data=True) at 50k users completed in 1.46 s / 7 statements and the DELETE ... WHERE user_id IN (SELECT id FROM users) ran in 23 ms (ModifyHypertable over the 30 chunks).

ruff check . reports 4 errors (app/routers/passkey.py x2, tests/api/test_user.py, tests/test_review_users_unit.py). They are identical on a clean origin/dev checkout and are not touched by this PR.

New tests (tests/test_group_user_loading.py, crud level, in-memory SQLite) assert that get_group / get_group_by_id / load_group_attrs return correct total_users without loading Group.users or touching the users table, that get_users(load_group_inbounds=True) leaves nothing for User.inbounds() to query, and that get_users_accessible_tags / _get_users_for_sync never send more than 10,000 bind parameters per statement for a 25,000-id / 25,000-username list. They also cover the bulk and delete paths: add_groups_to_users / remove_groups_from_users on 25,000 users (at most 10,000 bind parameters per statement, correct memberships and affected users, group inbounds preloaded so no per-user inbound query), remove_group (no executemany association DELETE, members never loaded, template / inbound associations cleaned, group id and name still readable), get_group_usernames (distinct, chunked), and GroupOperation.remove_group / bulk_remove_groups_by_id syncing users with fresh groups and no per-user inbound queries.

For the bulk user paths: add_groups_to_users with 40,000 explicit ids OR-combined with has_group_ids, and update_users_expire / update_users_datalimit / update_users_proxy_settings with 40,000 explicit ids, never send more than 10,000 bind parameters per statement and return the right users with their inbounds preloaded (no per-user inbound query); get_users(ids=...) and get_users_by_ids with 40,000 ids; remove_users on 12,000 users; reset_all_users_data_usage (scoped to one admin and for everyone, 12,000 users, no id binds); UserOperation._load_users_by_ids with no per-user inbound query; and AdminOperation activate-all / disable-all syncing users without usage logs and, for activate-all, without per-user inbound queries. Each new test was run with its fix reverted and fails.

Before / after

Measured on real PostgreSQL 18.6 and MariaDB 10.6 (Docker, fresh Alembic-migrated database per run, users bulk-inserted). One big group with N users plus a second group sharing half its members, node dispatch stubbed, calling GroupOperation.modify_group, get_all_groups, get_group_by_id and GroupOperation.remove_group. "Before" is origin/dev; "After" is this PR (the remove_group rows are re-measured with the final code). Cells are wall time / SQL statements. Timings are single runs and include a cold first call, so treat small values as order-of-magnitude.

PostgreSQL

Scenario	Users	Before	After
Add inbound (modify_group)	5,000	1.22 s, 2,542 queries	0.45 s, 32 queries
Add inbound (modify_group)	20,000	4.46 s, 10,132 queries	0.98 s, 92 queries
Add inbound (modify_group)	50,000	ERROR after 2.9 s, 312 queries (InterfaceError: the number of query arguments cannot exceed 32767)	2.47 s, 216 queries
Group list	5,000	0.049 s, 4 queries	0.021 s, 3 queries
Group list	20,000	0.223 s, 4 queries	0.038 s, 3 queries
Group list	50,000	0.592 s, 4 queries	0.083 s, 3 queries
Get group by id	5,000	0.065 s, 3 queries	0.002 s, 2 queries
Get group by id	20,000	0.152 s, 3 queries	0.004 s, 2 queries
Get group by id	50,000	0.412 s, 3 queries	0.009 s, 2 queries
remove_group	5,000	1.18 s, 2,571 queries	0.16 s, 31 queries
remove_group	20,000	5.80 s, 10,251 queries	0.61 s, 94 queries
remove_group	50,000	ERROR after 3.1 s, 309 queries (same asyncpg 32767 error, in get_users(username=...))	1.61 s, 223 queries

MariaDB

Scenario	Users	Before	After
Add inbound (modify_group)	5,000	1.70 s, 2,542 queries	0.40 s, 32 queries
Add inbound (modify_group)	20,000	6.20 s, 10,132 queries	0.94 s, 92 queries
Add inbound (modify_group)	50,000	14.55 s, 25,312 queries	2.77 s, 216 queries
Group list	5,000	0.044 s, 4 queries	0.011 s, 3 queries
Group list	20,000	0.181 s, 4 queries	0.009 s, 3 queries
Group list	50,000	0.578 s, 4 queries	0.014 s, 3 queries
Get group by id	5,000	0.029 s, 3 queries	0.002 s, 2 queries
Get group by id	20,000	0.153 s, 3 queries	0.003 s, 2 queries
Get group by id	50,000	0.404 s, 3 queries	0.005 s, 2 queries
remove_group	5,000	2.24 s, 2,571 queries	0.16 s, 31 queries
remove_group	20,000	7.70 s, 10,251 queries	0.54 s, 94 queries
remove_group	50,000	18.69 s, 25,611 queries	1.42 s, 223 queries

Bulk operations (GroupOperation.bulk_add_groups / bulk_remove_groups, same fresh databases, node dispatch stubbed). Group g0 has N users, g1 half of them, g2 none. "Before" is the parent commit of the bulk change (294f5f6); bulk.py is untouched by the earlier commits of this PR, so it equals origin/dev for these paths. Cells are wall time / SQL statements. ERR = InterfaceError: the number of query arguments cannot exceed 32767 (asyncpg).

Scenario	DB	5k before	5k after	20k before	20k after	50k before	50k after
bulk_add_groups +g1,+g2 (all N users affected)	PostgreSQL	1.51 s / 5,018	0.26 s / 19	6.98 s / 20,050	0.90 s / 59	ERR	2.51 s / 140
bulk_add_groups +g1 (N/2 affected)	PostgreSQL	0.87 s / 2,513	0.19 s / 14	2.94 s / 10,028	0.41 s / 32	ERR	1.17 s / 73
bulk_remove_groups -g1 (N/2 affected)	PostgreSQL	1.25 s / 2,513	0.08 s / 14	3.66 s / 10,028	0.35 s / 32	ERR	0.83 s / 73
bulk_add_groups +g1,+g2	MariaDB	1.62 s / 5,018	0.20 s / 19	6.60 s / 20,050	0.81 s / 59	20.9 s / 50,116	1.95 s / 140
bulk_add_groups +g1	MariaDB	0.94 s / 2,513	0.09 s / 14	3.38 s / 10,028	0.33 s / 32	8.39 s / 25,061	0.88 s / 73
bulk_remove_groups -g1	MariaDB	0.83 s / 2,513	0.08 s / 14	3.19 s / 10,028	0.29 s / 32	11.3 s / 25,060	0.76 s / 73

After the change no statement binds more than 10,000 parameters. For the bulk operations, where both versions completed, the users dispatched to nodes and group memberships are identical (compared per user including inbound tags). For remove_group, the same N users are dispatched, each with the inbound tags of its remaining groups.

Where both versions completed, total_users (N and N/2), the modify response (inbound_tags, member count) and the number of users dispatched to nodes (5k / 20k / 50k) were identical before and after.

TimescaleDB (timescale/timescaledb:latest-pg17, PG 17.11, fresh Alembic-migrated database per run cloned from one seeded template, 50,000 users: g0 all, g1 25,000, g2 none; node dispatch stubbed; via GroupOperation / UserOperation; wall time / SQL statements; "Before" is origin/dev, "After" is this PR). ERROR is InterfaceError: the number of query arguments cannot exceed 32767 (asyncpg).

Scenario (50k users)	Before	After
Add inbound (modify_group)	ERROR after 2.60 s / 315 (nothing dispatched; the group change is already committed)	1.28 s / 219, 50,000 dispatched
Group list (get_all_groups)	0.51 s / 7	0.04 s / 6 (same total_users: 50,000 / 25,000 / 0)
remove_group (g0, 50k members)	ERROR after 2.60 s / 312 (group already deleted)	1.37 s / 226, 50,000 dispatched
bulk_add_groups +g2 (all 50k affected)	ERROR after 0.12 s / 7	1.97 s / 143, 50,000 dispatched
bulk_remove_groups -g1 (25k affected)	ERROR after 0.12 s / 7	0.60 s / 76, 25,000 dispatched
Bulk expire -2,700,000 s (about -31 d) on g0 (45k status changes)	ERROR after 0.43 s / 7	1.15 s / 114, 45,000 dispatched
Reset usage, all users, clean_chart_data=True	ERROR after 0.11 s / 5 (nothing changed)	0.40 s / 7 (used traffic, usage logs, node usages and next plans all reset)
bulk_add_groups with 40,000 explicit user ids	ERROR after 0.11 s / 7	1.62 s / 117, 40,000 dispatched

The "After" dispatched inbound tags match the expected memberships in every row (for example modify_group: 25,000 users with in-a,in-b,in-c and 25,000 with in-a,in-c). To compare results where origin/dev also completes, the same eight operations were run on a 20,000-user database (15,000 explicit ids): the final database state and the dispatched users with their tags are identical for every operation except remove_group, where origin/dev dispatches the removed group's inbound (see root cause 5). 20k wall time / statements, before → after: modify_group 0.86 s / 135 → 0.49 s / 96, group list 0.21 s / 7 → 0.03 s / 6, remove_group 1.27 s / 254 → 0.49 s / 97, bulk_add_groups 0.87 s / 52 → 0.78 s / 62, bulk_remove_groups 0.33 s / 31 → 0.28 s / 35, expire 0.48 s / 43 → 0.47 s / 50, reset usage 0.29 s / 8 → 0.17 s / 7, explicit ids 0.68 s / 42 → 0.64 s / 50.

Bulk user operations (explicit ids, filters, activate / disable all, reset usage)

Same fresh Alembic-migrated databases, node dispatch stubbed, called through UserOperation.bulk_modify_expire / bulk_modify_datalimit / bulk_modify_proxy_settings, AdminOperation disable-all / activate-all, UserOperation.reset_users_data_usage. "Before" is 88941e81, "After" is this PR. Users are active / expired / limited / disabled, in two admins, g0 for all and g1 for half. Scenarios run in order on one database per size and state-changing ones leave their effect for the next. Cells are wall time / SQL statements; ERR is the asyncpg InterfaceError. "explicit ids" use every user at 5k / 20k and 40,000 ids at 50k. After the change no statement binds more than 10,000 parameters (explicit ids are rendered into the SQL, 0 id binds; the largest remaining statements are the 5k-id chunks that also bind group ids). Timings are single runs, with the PostgreSQL and MariaDB runs executing concurrently on one machine (the PostgreSQL 50k "after" column is an isolated re-run), so statement counts are exact and wall times indicative.

PostgreSQL (before → after)

Scenario	5k	20k	50k
expire -3d, filter admins	0.33 s / 12 → 0.32 s / 13	0.87 s / 33 → 0.72 s / 40	ERR → 1.79 s / 94
expire +5d, filter status	1.68 s / 4,013 → 0.37 s / 14	7.2 s / 16,037 → 0.90 s / 47	ERR → 2.03 s / 107
datalimit +5GB, filter admins	0.41 s / 506 → 0.26 s / 7	3.21 s / 2,009 → 0.46 s / 10	4.98 s / 5,015 → 0.71 s / 16
proxy settings, filter admin	1.10 s / 2,509 → 0.30 s / 10	12.35 s / 10,024 → 0.66 s / 28	13.46 s / 25,054 → 1.36 s / 67
activate all disabled (one admin)	1.21 s / 2,522 → 0.30 s / 18	14.23 s / 10,067 → 0.68 s / 48	14.23 s / 25,157 → 1.41 s / 108
disable all active (one admin)	0.30 s / 21 → 0.30 s / 16	1.01 s / 66 → 0.67 s / 46	1.65 s / 156 → 1.42 s / 106
reset usage, all users	0.27 s / 4 → 0.31 s / 3	0.51 s / 4 → 0.32 s / 3	ERR → 0.42 s / 3
bulk_add_groups, explicit ids	0.49 s / 20 → 0.40 s / 20	1.31 s / 63 → 1.43 s / 63	ERR → 2.25 s / 121
bulk_add_groups, explicit ids OR admin filter	0.34 s / 15 → 0.32 s / 15	0.81 s / 34 → 0.77 s / 34	ERR → 1.56 s / 78
bulk_remove_groups, explicit ids	0.61 s / 20 → 0.61 s / 20	1.23 s / 63 → 1.33 s / 63	ERR → 2.22 s / 121
expire +1d, explicit ids	0.27 s / 6 → 0.26 s / 7	1.41 s / 9 → 0.48 s / 10	ERR → 0.63 s / 14
expire -4d, explicit ids OR status filter	0.28 s / 6 → 0.26 s / 7	0.67 s / 9 → 0.41 s / 10	ERR → 0.71 s / 14
datalimit +1GB, explicit ids	0.26 s / 3 → 0.30 s / 3	0.45 s / 3 → 0.38 s / 3	ERR → 0.48 s / 3
proxy settings, explicit ids	1.53 s / 3,514 → 0.42 s / 15	8.80 s / 14,044 → 1.06 s / 54	ERR → 1.95 s / 106

MariaDB (before → after)

Scenario	5k	20k	50k
expire -3d, filter admins	0.32 s / 12 → 0.33 s / 13	1.22 s / 33 → 0.75 s / 40	1.84 s / 75 → 1.81 s / 94
expire +5d, filter status	1.73 s / 4,013 → 0.35 s / 14	20.88 s / 16,037 → 0.79 s / 47	22.73 s / 40,085 → 1.69 s / 107
datalimit +5GB, filter admins	0.40 s / 506 → 0.23 s / 7	5.26 s / 2,009 → 0.37 s / 10	3.26 s / 5,015 → 0.53 s / 16
proxy settings, filter admin	1.23 s / 2,509 → 0.27 s / 10	5.24 s / 10,024 → 0.59 s / 28	13.07 s / 25,054 → 1.17 s / 67
activate all disabled (one admin)	1.14 s / 2,522 → 0.27 s / 18	6.51 s / 10,067 → 0.58 s / 48	8.71 s / 25,157 → 1.15 s / 108
disable all active (one admin)	0.31 s / 21 → 0.28 s / 16	0.61 s / 66 → 0.62 s / 46	1.48 s / 156 → 1.14 s / 106
reset usage, all users	0.24 s / 4 → 0.21 s / 3	0.30 s / 4 → 0.23 s / 3	0.45 s / 4 → 0.25 s / 3
bulk_add_groups, explicit ids	0.47 s / 20 → 0.43 s / 20	1.30 s / 63 → 1.64 s / 63	1.88 s / 121 → 3.15 s / 121 (see rerun below)
bulk_add_groups, explicit ids OR admin filter	0.34 s / 15 → 0.34 s / 15	1.25 s / 34 → 0.86 s / 34	1.44 s / 78 → 2.27 s / 78 (see rerun below)
bulk_remove_groups, explicit ids	0.42 s / 20 → 0.43 s / 20	1.52 s / 63 → 1.22 s / 63	1.85 s / 121 → 2.22 s / 121 (see rerun below)
expire +1d, explicit ids	0.25 s / 6 → 0.27 s / 7	0.40 s / 9 → 0.37 s / 10	0.49 s / 13 → 0.51 s / 14
expire -4d, explicit ids OR status filter	0.25 s / 6 → 0.27 s / 7	0.38 s / 9 → 0.38 s / 10	0.50 s / 13 → 0.60 s / 14
datalimit +1GB, explicit ids	0.23 s / 3 → 0.23 s / 3	0.30 s / 3 → 0.31 s / 3	0.42 s / 3 → 0.35 s / 3
proxy settings, explicit ids	1.57 s / 3,514 → 0.37 s / 15	7.78 s / 14,044 → 1.20 s / 54	10.22 s / 28,084 → 2.26 s / 106

Where both versions completed (every row at 5k and 20k on both databases, and every row on MariaDB 50k), the number of affected users returned, the users dispatched to nodes with their inbound tags, and the resulting database state (status, expire, data limit, used traffic, shadowsocks method, group memberships, usage log / next plan counts) are identical. The PostgreSQL 50k "before" runs stop at the first error, so their later state diverges; the PostgreSQL 50k "after" results were instead compared to the MariaDB 50k "before" results and are identical in all 16 scenarios.

Does the literal IN slow MariaDB down? No. The 1.88 s → 3.15 s / 1.85 s → 2.22 s cells above were measured while PostgreSQL was benchmarked on the same machine. Re-run alone on MariaDB (50k users, bulk_add_groups then bulk_remove_groups with 40,000 explicit ids, 3 runs each, seconds; "before" is 88941e81):

Operation	Before min / median	After min / median
add (round 1)	1.82 / 1.84	1.82 / 1.83
remove (round 1)	1.66 / 1.66	1.65 / 1.65
add (round 2, order swapped)	2.04 / 2.07	1.82 / 1.86
remove (round 2, order swapped)	1.89 / 1.93	1.57 / 1.59

By-id bulk endpoints (UserOperation.bulk_disable_users / bulk_enable_users, 40,000 ids of 50,000 users)

Wall time / SQL statements / largest bind count in one statement; every dispatch below is 40,000 users.

Database	Operation	Before	After
PostgreSQL	disable	ERR (40,001 binds)	8.2 s / 407 / 500
PostgreSQL	enable	ERR (40,001 binds)	8.5 s / 405 / 500
MariaDB	disable	12.9 s / 406 / 40,001	13.5 s / 407 / 500
MariaDB	enable	23.1 s / 40,404 / 40,001	11.1 s / 405 / 500

The users dispatched to nodes with their inbound tags and the final status counts are identical (PostgreSQL "after" equals MariaDB "before"). The per-user ORM loops of these endpoints are unchanged (see Notes).

Is the COUNT fast without a groups_id index? Yes, at this scale; no index or migration is added here.

• PostgreSQL: the primary key is (user_id, groups_id), so the count does a Seq Scan on users_groups_association (75k association rows at 50k users). EXPLAIN ANALYZE execution time was 5.6 ms for the correlated list-style count and 3.3 ms for a single-group count. It grows linearly with the table, so perf(groups): add reverse group membership indexes #772's (groups_id, user_id) index would help PostgreSQL, but it isn't needed to justify this PR.
• MariaDB: InnoDB already creates fk_users_groups_association_groups_id_groups (groups_id); EXPLAIN shows ref ... Using index, and the whole group list takes about 14 ms at 50k.
• with_expression and the nested selectinload produced correct counts and 3 / 216 statements on MariaDB.

Screenshots

Not applicable.

Notes for reviewers

• Refs Scale group summaries for large user counts #771. This PR covers its "populate total_users with a SQL count without loading Group.users" item.
• Overlap with perf(groups): add reverse group membership indexes #772: it also replaces the total_users hybrid on the model with a query_expression and adds the reverse association indexes. Both touch the same lines in models.py / crud/group.py, so one will need a rebase. This PR keeps the len(users) fallback in the hybrid and adds the count in load_group_attrs, which perf(groups): add reverse group membership indexes #772's with_expression approach is compatible with. Without perf(groups): add reverse group membership indexes #772's (groups_id, user_id) index the COUNT still scans users_groups_association on PostgreSQL (3-6 ms at 75k rows, see above); MariaDB uses its FK index.
• load_group_attrs now takes db first and load_total_users replaces load_users; get_group_by_id no longer takes load_users. All callers are updated (crud/group.py, operation/group.py).
• Node sync after a removal reloads users fresh. origin/dev's GroupOperation.remove_group loaded the members before deleting the group with get_users(db, query=UserListQuery(group_ids=[db_group.id])); the get_users(UserListQuery(username=...)) reload after the delete returned those same session-cached User objects, whose User.groups still included the removed group, so it sent the removed group's inbound to nodes for members who had no other group providing it (TimescaleDB, 20k users: origin/dev dispatched 10,000 users with in-b and 10,000 with in-a; this PR dispatches 10,000 with in-b and 10,000 with none). This is a bug on origin/dev that this PR fixes: usernames come from get_group_usernames and users are reloaded after the delete (_get_users_for_sync, _reload_group_users, get_users_by_ids) with their remaining groups' inbounds preloaded.
• Out of scope, deliberately not changed here:
  • Moving the user/node sync to a background task so the request returns early, and skipping it on name-only edits (perf(groups): return updates before large user sync #827).
  • The by-id bulk endpoints (bulk_disable_users, bulk_enable_users, bulk_reset_user_data_usage, bulk_revoke_user_sub, bulk_set_owner, bulk_apply_template_to_users, bulk_remove_users) still run per-user ORM loops, in particular refresh_and_load_user per user in bulk_reset_user_data_usage, bulk_revoke_user_sub and bulk_set_owner, and bulk_reset_user_data_usage / bulk_revoke_user_sub still sync users that were refreshed one by one, so their groups' inbounds are not preloaded. Only their explicit-id IN (bind limit) and the reload used for the sync after status / disable / enable / apply template were changed.
  • update_users_status (scheduled status jobs) and get_users_simple(ids=...) (list API query parameter) still bind their ids in one IN; they are not reachable from a bulk request body. admins / group_ids / has_group_ids filters are small sets and are still bound.
  • create_user_template / modify_user_template still load group members (perf(templates): avoid loading group members and batch list relations #944).

Summary by CodeRabbit

• Improvements
  • Group user totals are available without loading every group member, improving responsiveness when viewing groups with many users.
  • Bulk user and group changes now process large selections in batches, helping avoid request limits during membership updates, removals, and data-usage resets.
  • User and group updates continue to synchronize affected users with their group access settings.
• Bug Fixes
  • Group totals and membership updates now reflect the full affected user scope, including for large groups.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 39affe6e-72f7-42ca-bb45-3c233f0ca87c

---

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 49626df0-6d64-4eb0-ab2e-3a92955fb918

---

---

📥 Commits

Reviewing files that changed from the base of the PR and between fe7c5f9 and 530ceb1.

---

---

📒 Files selected for processing (9)

• app/db/crud/bulk.py
• app/db/crud/group.py
• app/db/crud/user.py
• app/db/crud/wireguard.py
• app/db/models.py
• app/operation/admin.py
• app/operation/group.py
• app/operation/user.py
• tests/test_group_user_loading.py

---

---

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

---

---

Walkthrough

The change updates group and user loading to compute group totals in SQL and optionally preload group inbounds. Bulk ID, membership, reset, and accessible-tag queries process large inputs in batches. Group and admin operations use the updated loaders when synchronizing affected users. Tests cover loading, batching, and bulk operations.

Changes

Group and User Loading

Layer / File(s)	Summary
Group counts and user loading app/db/models.py, app/db/crud/group.py, app/db/crud/user.py, tests/test_group_user_loading.py	Group queries compute member counts in SQL without loading member rows. User queries can preload group inbounds. Tests cover counts and eager loading.
Batched IDs and bulk operations app/db/crud/bulk.py, app/db/crud/group.py, app/db/crud/user.py, app/db/crud/wireguard.py, tests/test_group_user_loading.py	Explicit ID filters and bulk operations process large input sets in batches or through server-side queries. Updated user results include group inbounds.
Operation-level user synchronization app/operation/admin.py, app/operation/group.py, app/operation/user.py, tests/test_group_user_loading.py	Group and admin operations load affected users with group inbounds and exclude usage logs. Group removal uses batched username-based loading.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Suggested reviewers: m03ed

Merge Risk: ⚪ Minimal · up to 530ce

The reviewed group and bulk-operation paths have no established issue requiring a fix before merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 40.79% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 76 functions across 9 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the primary performance changes to group listing, synchronization, bulk operations, and deletion for large user counts.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

---

---

---

✨ Finishing Touches 💡 1

⚔️ Resolve merge conflicts 💡

• Resolve merge conflict in branch perf/group-inbounds-and-listing

---

---

🧪 Generate unit tests (beta)

• Create a new PR

---

---

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{A rabbit checks the counts in rows,
Then batches IDs where each one goes.
Group inbounds arrive in view,
The sync hops through its users too.
Fresh queries leave neat tracks behind.}

_{Comment @coderabbitai help to get the list of available commands.}