Skip to content

feat: Apply RETRY to FDv2 synchronizer orchestration - #252

Open
beekld wants to merge 21 commits into
mainfrom
bklimt/SDK-3402/retry-parking
Open

beekld wants to merge 21 commits into
mainfrom
bklimt/SDK-3402/retry-parking

Conversation

@beekld

@beekld beekld commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Applies the RETRY behavior to FDv2 synchronizer orchestration in the server SDK. A synchronizer that fails is parked and retried later, instead of being blocked for good.

Parking

  • A synchronizer that fails unexpectedly is parked: a terminal error, an exception from next(), a failure to build, or shutting down on its own. The park delay starts at 5 minutes, doubles with each failure up to 1 hour, and has jitter. After a park, the next synchronizer starts after a failover delay of 0.5-1 seconds.
  • A synchronizer's park count resets when it meets its reset threshold. For polling, that's two successful polls in a row. For streaming, it's staying connected for 60 seconds.
  • When every synchronizer is parked, the data system waits for the earliest unpark time. Synchronization stops only when the data source is closed.
  • When the fallback condition is met but no other synchronizer is available, the current synchronizer keeps running.

Directed fallback

  • The fallback to FDv1 now ends after a TTL, which defaults to 1 hour with jitter. The data system then restarts the FDv2 synchronizers.
  • A directive received during the fallback extends the TTL.
  • A directive with no FDv1 fallback synchronizer configured no longer halts the data system. It reports INTERRUPTED and restarts the FDv2 synchronizers when the TTL ends.

Conditions removed

  • The Conditions class and FDv2DataSourceConditions are removed. They weren't compatible with the new parking and FDv1 fallback timers, so the synchronizer loop is simplified to check all of its timers each time it wakes.

Not included

  • Honoring Retry-After.
  • Reading the TTL from the response. Every directive uses the 1-hour default.

Note

Overview
Replaces the old condition/timer model (FDv2DataSourceConditions, FDv2DataSource.Conditions) with a single synchronizer loop that wakes on results or deadlines (fallback, recovery, unpark, FDv1 TTL). Failed synchronizers are parked with exponential backoff (5 min → 1 h, jitter) instead of being blocked for the process lifetime; terminal errors and build failures park the source rather than permanently blocking it.

Adds AsyncLatch so close() and timed waits can race CompletableFuture.anyOf without leaking completion nodes on long-lived streams.

FDv1 fallback is time-bounded (default 1 h, jitter): SourceManager tracks end time, blocks FDv2 sources during fallback, and returns when TTL expires. Server directives extend TTL or switch to FDv1 without halting the whole data system when no FDv1 synchronizer exists (reports INTERRUPTED and waits for TTL).

Reset threshold is propagated via FDv2SourceResult.resetThresholdMet — polling (2 successes), streaming (60 s healthy), FDv1 adapter (2 VALID) — and resets park counts. HTTP/transport classification (FailureClass) turns unexpected errors into terminal results in polling, streaming, and the FDv1 synchronizer adapter.

close() releases the latch, shuts down sources, and joins the run thread (5 s cap) before reporting OFF.

Reviewed by Cursor Bugbot for commit 94f7478. Bugbot is set up for automated code reviews on this repo. Configure here.

@beekld
beekld force-pushed the bklimt/SDK-3402/retry-parking branch from 0cf207f to 8aa8c77 Compare November 6, 2026 00:49
@beekld
beekld marked this pull request as ready for review November 6, 2026 00:54
@beekld
beekld requested a review from a team as a code owner November 6, 2026 00:55
@beekld
beekld added this pull request to stack #254 November 6, 2026 03:27
@beekld
beekld force-pushed the bklimt/SDK-3402/retry-parking branch from f78d65f to 0dafa42 Compare November 6, 2026 17:37

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 0dafa42. Configure here.

}
break;
case VALID:
consecutiveSuccessfulResults++;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you can conclude success from consecutive valid status.

StreamProcessor calls updateStatus(State.VALID, null) at the end of handleMessage for every message event, including the default: branch that logs "Unexpected event found in stream" (StreamProcessor.java:327-331). Two unknown event types in a row (the forward-compatibility case: any new server event name) produce two VALIDs with zero data delivered.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this code kind of assumes that the FDv1 fallback is a polling synchronizer. Making this work for any kind of synchronizer would require adding a way to signal the reset from any kind of FDv1 DataSource, and I don't think that's worth the effort, given FDv1 isn't the future anyway.

switch (res.getResultType()) {
case CHANGE_SET:
consecutiveSuccessfulPolls++;
if (consecutiveSuccessfulPolls == RESET_THRESHOLD_SUCCESSFUL_POLLS) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if (consecutiveSuccessfulPolls == RESET_THRESHOLD_SUCCESSFUL_POLLS) {
if (consecutiveSuccessfulPolls >= RESET_THRESHOLD_SUCCESSFUL_POLLS) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to keep it as ==, so the reset only gets set once, per the spec. I don't think it makes much difference right now, but it would be more fragile to have the reset signal firing on every result.

@beekld
beekld force-pushed the bklimt/SDK-3402/retry-parking branch from 0dafa42 to a700bff Compare November 10, 2026 21:42
@beekld
beekld requested a review from tanderson-ld November 11, 2026 01:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants