Repository navigation
feat(schedule): keep an always-on stream running with a keepalive DAG - #1018
Merged
Merged
Conversation
A forge Spark stream that is not trigger.availableNow and declares execution.retries gets a DAG of its own: one task that runs the reconcile-only fluid apply for that stream (KEEPALIVE_APPLY_ARGS, the one place it is named), every 5 minutes or on the build's execution.orchestration.airflow.dagConfig.schedule, with max_active_runs=1, no catch-up and no Airflow retries, so the build's own retries bound the restarts. It declares no outlets: a stream updates its tables all the time. A contract whose only builds are such streams now needs a DAG.
The keepalive DAG's task runs fluid apply <contract> [--env] --mode reconcile --build-id <stream> --yes. That runs no tofu apply: it starts the stream again on its checkpoint within the restart budget the build's execution.retries sets, and fails the run when the stream drifted from the contract or the budget is spent. The arguments stay in KEEPALIVE_APPLY_ARGS, joined one space apart whatever they hold, and a test parses them with fluid's own parser and its --build-id check, so the DAG cannot run a mode or flag that fluid apply refuses.
A keepalive DAG reads only the contract's builds: a stream is in no build graph, since nothing waits for it. So _keepalive_dags() sits with _keeps_alive() and _keepalive_schedule() rather than in _BuildGraph, which keeps to the dependencies between the builds a DAG runs.
…nothing Four places counted a keepalive DAG as more than it is: - A dataset trigger on an expose that only a kept-alive stream writes was no longer refused, though a keepalive declares no outlets and the trigger would never fire. It is refused again, and the message says why. - fluid generate schedule --scheduler prefect or dagster wrote a placeholder flow for a contract whose only builds are kept-alive streams, and fluid scaffold-composer wrote a daily plan-and-apply DAG for one. Neither renders a keepalive. Both write nothing again, and name the Airflow keepalive DAG. - A stream that trigger.type: continuous or a Flink job leaves to another platform got a keepalive DAG. It no longer does, as _classify already has it.
The keepalive DAG's text said a run fails when the stream drifted or the budget is spent. fluid apply --mode reconcile also fails when nothing is deployed or the stream's cluster ended, and on EMR Serverless, EMR on EKS and Glue the service's own retries are the restarts. It leaves a run cancelled on purpose stopped until fluid apply starts it. The DAG text, the KEEPALIVE_APPLY_ARGS comment and the CHANGELOG entry now say so.
The autouse fixture in test_scheduler_framework.py emptied the shared scheduler registry and left it empty. The built-in schedulers register only when fluid_build.schedulers is first imported, so a later test in the same worker that looked up prefect or dagster by name found no scheduler. CI runs the suite under xdist with pytest-randomly, so such a test failed only on some orders. Each test now gets an empty registry of its own through monkeypatch, and the shared one is back in place after it.
…edule-stream-keepalive
…live fails on The keepalive DAG text and its CHANGELOG entry now also name two cases a reconcile exits 1 on. The job config the last apply deployed cannot be read: it is read on every run, while the stream is healthy too. The worker's FLUID_SPARK_* settings or forge release differ from the CI apply's: a policy gate refuses the build, or the compiled config reads as drift.
The 0.33.0 cut turned the old Unreleased section into 0.33.0, and this entry moved with it. The keepalive DAG is not in 0.33.0, so it is listed under Unreleased, after the entry for Spark stream batch bounds.
…edule-stream-keepalive
… DAG The reconcile paragraph names a keepalive as a scheduler task every few minutes. fluid generate schedule now renders one, as an Airflow DAG, for a stream that declares execution.retries.
The build-graph tests import scaffold_composer, SimpleNamespace and fluid_apply at module level instead of again inside each test, and the registry fixture patches _SCHEDULERS by its dotted path, so the module needs no second import of the registry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
fluid generate schedulenow gives an always-on forge Spark stream its own small DAG when the build declaresexecution.retries:fluid_keepalive, runsfluid apply <contract> [--env X] --mode reconcile --build-id <stream> --yes.execution.orchestration.airflow.dagConfig.schedule.@onceis refused, because a keepalive has to recur.max_active_runs=1,catchup=False, retries 0 and a 30-minute task timeout. The restart budget is the build's ownexecution.retries(maxAttemptsrestarts per rolling hour), which--mode reconcileapplies, so Airflow adds no retries on top of it. Each wait inside a reconcile is bounded byFLUID_SPARK_WAIT_SECONDS(900 s by default).--mode reconcile(#1013) runs notofu apply:runtime: local; EMR Serverless and EMR on EKS retry a failed run themselves.FLUID_SPARK_*settings or forge release differ from the CI apply's. The DAG run then fails, and Airflow's own alerting picks it up.fluid applystarts it, so the keepalive never undoes an operator's cancel.Which streams get a keepalive DAG:
properties.stream, nottrigger.availableNow, run by Airflow (the default engine), that declaresexecution.retries.execution.retriesis the opt-in.--mode reconcilewould restart a stream without it too (5 restarts per rolling hour by default). But a DAG that runsfluid applyevery 5 minutes is something a contract should ask for, not something every streaming contract gets on its nextgenerate schedule.What does not change:
execution.retriesstill gets no DAG.trigger.type: continuousconnector is still left to its platform.A contract whose only builds are such streams now gets this DAG, so
fluid generate scheduleno longer answers "No build needs a DAG" for it.The seam
KEEPALIVE_APPLY_ARGS, and are joined one space apart, whatever they hold.fluid apply's--build-idcheck, so the DAG cannot run a mode or flag thatfluid applyrefuses.fluid applytask (cd "$FLUID_PROJECT_DIR", the env allowlist,env -i) but for that one line, and it is rendered by the samerender_build_task().FLUID_*variables pass through as for everyfluid applytask, so the reconcile sees the operator settings the CI apply has (FLUID_SPARK_TARGET_ALLOWLISTand the rest)._keepalive_dags()reads the builds alone, because a stream is in no build graph. It sits beside_keeps_alive()and_keepalive_schedule().A test fix on the way
tests/test_scheduler_framework.pyemptied the shared scheduler registry around each test and left it empty. The built-in schedulers register only whenfluid_build.schedulersis first imported, so a later test in the same worker that namedprefectordagsterfound no scheduler. Under xdist with pytest-randomly, that fails on some test orders only. This PR's--scheduler prefect|dagstertest hit it. Each test there now gets an empty registry of its own throughmonkeypatch, and the shared one is left as it was.Same output as before for other contracts
A contract with no such stream renders byte for byte as before: every existing golden file is unchanged. The new
kept_alive_streamgolden pins the keepalive DAG.Borrowed, not built
maxFailedAttemptsPerHour) and restarts it from its checkpoint: https://docs.aws.amazon.com/emr/latest/EMR-Serverless-UserGuide/jobs-resiliency.htmlrestartPolicy: OnFailurewithonFailureRetries, which is a count, not a window.catchupexplicitly, and documentsmax_active_runsas a cap on concurrent runs: https://www.astronomer.io/docs/learn/airflow-dag-parametersdagConfig.scheduleoverrides it.fluid apply.Validation
ruff check fluid_build/ tests/, the dead-code ruff (copilot, forge_datamodel),black --check(24.10.0, 2281 files), the license headers, the product guardrail (0 failing), and detect-secrets 1.5.0 on the changed files.lint-imports: 6 contracts kept.generate schedule,scaffold-composer, continuous triggers, Airflow 3 compatibility, the authoring lint, schedule-sync, the apply modes, the reconcile budget and the startup budget. Two failures were already there before this change:test_a_secret_ref_resolves_on_the_worker, whose stub imports another checkout's package.test_fluid_apply_accepts_the_keepalive_argvpasses with--mode reconcileon main. Before feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013 it fails withinvalid choice: 'reconcile', which is the failure it exists to catch._BuildGraph, which brought the class back to its earlier size.test_fluid_apply_airflow_live.pypasses on both, re-run on this head. It runs aDagBagparse of every golden contract, the kept-alive one included.airflow dags test platform.events__aws__stream-rawsucceeds on both. Its task ranfluid apply <contract> --env aws --mode reconcile --build-id stream-raw --yes, recorded by a stubfluidpassed in throughFLUID_BIN.fluid validateandfluid generate schedule --env awson a stream-plus-daily-batch contract wrote two DAGs:stream_raw_dag.py: the keepalive, with*/5 * * * *,max_active_runs=1,catchup=Falseand retries 0;daily_curated_dag.py.Reviews
/security-review: no HIGH or MEDIUM findings.dagConfig.schedulecron is validated and reaches the source only as a string literal. The script is built from module constants. Ids are validated twice. The argv is fixed in code.scaffold-composerwrote a daily plan-and-apply DAG for a contract whose only builds are kept-alive streams. Fixed.--scheduler prefectordagsterwrote a placeholder flow for such a contract. It now writes nothing and names the Airflow keepalive.scaffold-composerhad the same problem (the delivery review's P2).trigger.type: continuous) got a keepalive DAG. It no longer does.--scheduler airflow,orchestration.engine: prefectand an unknown scheduler).KEEPALIVE_APPLY_ARGScomment and the CHANGELOG entry now name every case a reconcile run fails on, from feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013's final behaviour (an unreadable job config and the worker's settings included). The registry fixture fix is test-only.Documentation
Pages:
docs/cli/generate.md(generate schedule): the keepalive DAG for a stream withexecution.retries. Cover its cadence and thedagConfig.scheduleoverride, the single run with no retries and no outlets, and the fact that a streams-with-retries contract no longer gets "No build needs a DAG".docs/advanced/airflow.md: keeping an always-on stream running from Airflow. Cover stopping a stream on purpose (pause the keepalive DAG, then cancel the run; to resume, unpause and runfluid apply), and that the worker needs the sameFLUID_SPARK_*settings and forge release as the CI apply: a missing allowlist refuses the build, and a setting that changes the compiled config reads as drift. The worker's role needs the product'sfluid-<product>-spark-callerpolicy: every start, a restart included, reads the stream's job config and runner (s3:GetObject), and on EMR on EC2 it also callsDescribeClusterand reads the step role.docs/advanced/spark-iceberg-aws.md: a stream'sexecution.retriesas its restart budget, with the keepalive DAG that applies it.docs/upgrading.md("Not in a release yet"): one line. A contract whose stream declaresexecution.retriesgets one more DAG file on its nextfluid generate schedule.Follow-ups (not in this PR)
--mode reconcilehas a default budget. It stays opt-in here, for the reason under "Which streams get a keepalive DAG".