Add periodic EOB completion check to Coordinator for bootstrap datastreams#1015
Closed
mittalprince wants to merge 2 commits into
Closed
Add periodic EOB completion check to Coordinator for bootstrap datastreams#1015mittalprince wants to merge 2 commits into
mittalprince wants to merge 2 commits into
Conversation
added 2 commits
May 20, 2026 14:56
Cover previously untested edge cases: - No tasks assigned yet -> bootstrap not started, skip - Task with null status -> not treated as COMPLETE - updateDatastream returns false -> no exception, retries next cycle - Multiple bootstrap groups -> only the complete one gets flagged
…reams - Add IS_EOB_COMPLETE constant to DatastreamMetadataConstants - Add periodicEobCheck scheduled task on the leader that stamps system.isEOBComplete=true on bootstrap datastream groups once all assigned tasks reach COMPLETE status - Add eobCheckPeriodMs and eobCheckConnectorNames config to CoordinatorConfig
mittalprince
changed the title
| * {@link com.linkedin.datastream.server.DatastreamTaskStatus.Code#COMPLETE} status, indicating | ||
| * that End-of-Bootstrap (EOB) events have been successfully produced to all destination partitions. | ||
| */ | ||
| public static final String IS_EOB_COMPLETE = "system.isEOBComplete"; |
Collaborator
There was a problem hiding this comment.
should it be renamed as system.isBootstrapComplete?
| Datastream representative = group.getDatastreams().get(0); | ||
|
|
||
| // Bootstrap datastreams are identified by the "bst-" prefix in their name. | ||
| if (!representative.getName().contains("bst-")) { |
Collaborator
There was a problem hiding this comment.
this check can return false positives.
It would be better to check this via datastream's connector name.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
periodicEobCheckscheduled task inCoordinatorthat runs on the leader and checks whether all tasks for bootstrap datastreams have reachedCOMPLETEstatusmaybeMarkEOBCompleteForBootstrapDatastreamswhich stampssystem.isEOBComplete=trueon all datastreams in the group (including deduped streams) when all tasks are completeIS_EOB_COMPLETEconstant toDatastreamMetadataConstantseobCheckPeriodMsandeobCheckConnectorNamesconfig toCoordinatorConfigwith defaults (1 min period, all connectors)Testing Done
TestCoordinator