Skip to content

fix(node): let lifecycle operations preempt user sync - #903

Open
dr-hoseyn wants to merge 4 commits into
PasarGuard:devfrom
dr-hoseyn:codex/fix-node-operation-sync-lock
Open

dr-hoseyn wants to merge 4 commits into
PasarGuard:devfrom
dr-hoseyn:codex/fix-node-operation-sync-lock

Conversation

@dr-hoseyn

@dr-hoseyn dr-hoseyn commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Node edits, disables, and removals could wait for many minutes when a bulk or full user sync held the per-node lock. A failed chunked sync made this especially visible because the bridge fallback retried users individually while lifecycle requests remained queued.

  • Track bulk and full user-sync tasks per node.
  • Let real configuration changes and removals cancel stale sync work before taking the lifecycle lock.
  • Preserve unchanged health-check reconnects without interrupting an active sync.
  • Release the sync lock between bounded batches and resolve the current managed node for every batch, preventing work from continuing on a replaced instance.
  • Keep each per-node lock identity stable for the manager lifetime so concurrent waiters cannot bypass one another through a replacement lock.

The PR changes only app/node/__init__.py; it adds no documentation, test files, migrations, dependencies, or unrelated changes.

Type of change

  • Bug fix
  • New feature
  • Breaking change
  • Refactor / cleanup
  • Documentation
  • Tests / CI

Checklist

  • I tested the change locally or explained why it cannot be tested.
  • I added or updated tests for behavior changes. (No test files requested for this hotfix; existing suites plus isolated concurrency regressions were run.)
  • I updated documentation, translations, or examples if needed. (Not needed; no public behavior or configuration change.)
  • I checked database migrations when models or schema changed. (No model or schema changes.)
  • I did not include secrets, tokens, private keys, or unrelated changes.

Testing

  • python -m pytest tests/test_node_manager.py tests/test_node_manager_sync.py -q — 11 passed.
  • python -m ruff check --no-fix app/node/__init__.py — passed.
  • python -m ruff format --check app/node/__init__.py — passed.
  • git diff --check — passed.
  • Isolated async regression: a legacy per-user fallback blocked indefinitely, then concurrent removal cancelled it and completed in 0.0001s.
  • Isolated async regression: an unchanged reconnect preserved the in-flight sync, while a real config update cancelled it, replaced the node, and completed in 0.0003s.

Notes for reviewers

Cancellation is limited to user-sync tasks for the affected node and only happens for an actual configuration change or lifecycle removal. Automated reconnects with an unchanged connection signature and metadata continue to reuse the live node without disrupting synchronization.

Summary by CodeRabbit

  • Bug Fixes
    • Improved node lifecycle handling during user synchronization.
    • Synchronization is canceled when a node is removed or its connection settings change, and checks node availability between batches.
    • Unchanged reconnects no longer interrupt active synchronization; metadata updates wait for active synchronization to finish.
    • When synchronization is preempted by a node change, the operation returns a retry response without marking the node as errored.

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 747bbd5e-6f3d-4730-bc0e-289a3047e157

📥 Commits

Reviewing files that changed from the base of the PR and between 70903fa and 56f1364.


📒 Files selected for processing (3)
  • app/node/__init__.py
  • app/operation/node.py
  • tests/test_node_sync_preemption.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 2 remain after this review.



Walkthrough

NodeManager tracks user-sync tasks and coordinates them with node updates and removal. Bulk sync can resolve the current node between batches. Full-sync lifecycle cancellation raises UserSyncPreemptedError, which the local operation maps to HTTP 409 without changing node status.

Changes

Node sync lifecycle

Layer / File(s) Summary
Task tracking and node lifecycle
app/node/__init__.py
NodeManager tracks sync tasks and cancels them before connection changes or removal. Unchanged connection and metadata values reuse the live node. Metadata-only updates wait for the per-node lock, which remains registered after removal.
User-sync execution
app/node/__init__.py
Bulk sync can resolve the current node and release the lock between batches. Full sync runs in a tracked task and distinguishes lifecycle preemption from caller cancellation. Sync cancellation is logged separately from other failures.
Operation response and lifecycle tests
app/operation/node.py, tests/test_node_sync_preemption.py
The local sync operation maps UserSyncPreemptedError to HTTP 409 without updating node status. Tests cover removal, connection changes, metadata edits, unchanged reconnects, and caller cancellation.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant NodeOperation
  participant NodeManager
  participant UserSyncTask
  NodeOperation->>NodeManager: start user sync
  NodeManager->>UserSyncTask: track sync task
  NodeManager->>UserSyncTask: cancel on connection change or removal
  UserSyncTask-->>NodeManager: lifecycle cancellation
  NodeManager-->>NodeOperation: raise UserSyncPreemptedError
  NodeOperation-->>NodeOperation: return HTTP 409 without node status update
Loading

Suggested reviewers: m03ed

Merge Risk: 🔵 Low · up to 56f13

A sync started during a node change or removal can briefly defeat preemption. The PR is mergeable with awareness of this bounded concurrency risk.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 12.12% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 33 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: allowing node lifecycle operations to preempt in-flight user synchronization.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.

  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the syncs in flight,
Node changes pause the batches’ light.
Locks remain while tasks unwind,
New nodes serve the work assigned.
A retry waits beyond the bend,
Then users sync from start to end.

Comment @coderabbitai help to get the list of available commands.

@dr-hoseyn

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

…on-sync-lock

# Conflicts:
#	app/node/__init__.py
@dr-hoseyn
dr-hoseyn force-pushed the codex/fix-node-operation-sync-lock branch from 3aef26a to 47ddab7 Compare October 8, 2026 22:29
Cancel in-flight user syncs from update_node only when the connection
signature changed (and from remove_node as before). A name or usage
coefficient edit now waits for the per-node lock instead: bulk deliveries
release it between batches and continue on the replacement object, so
pending user deltas are not dropped. Once metadata-only edits stop
restarting the node, nothing else would re-send cancelled deltas.

Run sync_full in its own tracked task. Preemption then cancels only the
sync, never the caller (in single-process mode that is the admin's
POST /node/{id}/sync request), and the caller gets UserSyncPreemptedError,
which the node operation maps to a 409 without marking the node as
errored. Cancelling the caller still cancels the sync.

Log deliveries cancelled by a lifecycle change at debug level in
_update_users instead of dropping the CancelledError results silently.
- removal cancels a stuck per-user fallback delivery
- an unchanged health-check reconnect keeps a running full or bulk sync,
  while a connection change cancels it
- a name or usage coefficient edit waits for the batch in flight and the
  remaining batches reach the replacement object
- a metadata edit waits for a running full sync
- preempting sync_full raises UserSyncPreemptedError in the caller and
  cancelling the caller cancels the sync
- the sync endpoint maps preemption to 409 without an error status
@T3ST3ST3R0N

Copy link
Copy Markdown
Collaborator

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants