Skip to content

feat(schedule): keep an always-on stream running with a keepalive DAG - #1018

Merged
fas89 merged 14 commits into
mainfrom
feat/schedule-stream-keepalive
Nov 19, 2026
Merged

fas89 merged 14 commits into
mainfrom
feat/schedule-stream-keepalive

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

fluid generate schedule now gives an always-on forge Spark stream its own small DAG when the build declares execution.retries:

  • One task, fluid_keepalive, runs fluid apply <contract> [--env X] --mode reconcile --build-id <stream> --yes.
  • It runs every 5 minutes, or on the build's execution.orchestration.airflow.dagConfig.schedule.
    • That cron is validated like any DAG cron.
    • @once is refused, because a keepalive has to recur.
  • The DAG sets max_active_runs=1, catchup=False, retries 0 and a 30-minute task timeout. The restart budget is the build's own execution.retries (maxAttempts restarts per rolling hour), which --mode reconcile applies, so Airflow adds no retries on top of it. Each wait inside a reconcile is bounded by FLUID_SPARK_WAIT_SECONDS (900 s by default).
  • It has no outlets. A stream updates its tables all the time, and an asset event every 5 minutes would start every DAG scheduled on them.

--mode reconcile (#1013) runs no tofu apply:

  • When the stream died (a Spot interruption, a failed driver), it starts again on its checkpoint, within that budget. forge restarts it on EMR on EC2, Glue and runtime: local; EMR Serverless and EMR on EKS retry a failed run themselves.
  • It exits non-zero when the stream cannot be kept running: it drifted from the contract, nothing is deployed, the job config the last apply deployed cannot be read, its cluster ended, or the restarts are spent. It can also fail when the worker's FLUID_SPARK_* settings or forge release differ from the CI apply's. The DAG run then fails, and Airflow's own alerting picks it up.
  • A spent budget is not final where forge does the restarting (EMR on EC2, Glue, local). Once the oldest restart is an hour old, the next run restarts the stream, so a burst of Spot interruptions heals without an operator. Every run before then fails.
  • A run cancelled on purpose stays stopped until fluid apply starts it, so the keepalive never undoes an operator's cancel.

Which streams get a keepalive DAG:

  • a Spark build with properties.stream, not trigger.availableNow, run by Airflow (the default engine), that declares execution.retries.
  • Declaring execution.retries is the opt-in. --mode reconcile would restart a stream without it too (5 restarts per rolling hour by default). But a DAG that runs fluid apply every 5 minutes is something a contract should ask for, not something every streaming contract gets on its next generate schedule.

What does not change:

  • A stream without execution.retries still gets no DAG.
  • A trigger.type: continuous connector is still left to its platform.
  • A batch that depends on the stream still does not wait for it.

A contract whose only builds are such streams now gets this DAG, so fluid generate schedule no longer answers "No build needs a DAG" for it.

The seam

  • The arguments are named in one place, KEEPALIVE_APPLY_ARGS, and are joined one space apart, whatever they hold.
  • A test parses them with fluid's own parser and runs fluid apply's --build-id check, so the DAG cannot run a mode or flag that fluid apply refuses.
  • The task runs the same static script as every other fluid apply task (cd "$FLUID_PROJECT_DIR", the env allowlist, env -i) but for that one line, and it is rendered by the same render_build_task().
  • FLUID_* variables pass through as for every fluid apply task, so the reconcile sees the operator settings the CI apply has (FLUID_SPARK_TARGET_ALLOWLIST and the rest).
  • _keepalive_dags() reads the builds alone, because a stream is in no build graph. It sits beside _keeps_alive() and _keepalive_schedule().

A test fix on the way

tests/test_scheduler_framework.py emptied the shared scheduler registry around each test and left it empty. The built-in schedulers register only when fluid_build.schedulers is first imported, so a later test in the same worker that named prefect or dagster found no scheduler. Under xdist with pytest-randomly, that fails on some test orders only. This PR's --scheduler prefect|dagster test hit it. Each test there now gets an empty registry of its own through monkeypatch, and the shared one is left as it was.

Same output as before for other contracts

A contract with no such stream renders byte for byte as before: every existing golden file is unchanged. The new kept_alive_stream golden pins the keepalive DAG.

Borrowed, not built

Validation

  • Gates on the branch:
    • Clean: ruff check fluid_build/ tests/, the dead-code ruff (copilot, forge_datamodel), black --check (24.10.0, 2281 files), the license headers, the product guardrail (0 failing), and detect-secrets 1.5.0 on the changed files.
    • lint-imports: 6 contracts kept.
    • The import-cycle, license-header and startup-budget tests pass (in the run below).
  • Touched tests: 1183 passed, across the schedulers, the scheduler registry, generate schedule, scaffold-composer, continuous triggers, Airflow 3 compatibility, the authoring lint, schedule-sync, the apply modes, the reconcile budget and the startup budget. Two failures were already there before this change: test_a_secret_ref_resolves_on_the_worker, whose stub imports another checkout's package.
  • The CLI-parser pin test_fluid_apply_accepts_the_keepalive_argv passes with --mode reconcile on main. Before feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013 it fails with invalid choice: 'reconcile', which is the failure it exists to catch.
  • A trial merge with feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013's head d63e1ab on main 07ac9da, before feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013 merged: every gate above clean, and 1315 passed across the touched suites and feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013's own Spark runner, apply-modes and AWS Spark IaC tests.
  • Full suite: this PR's CI, on every Python version CI runs.
  • Quality delta, run against a no-op control on the same base:
  • Real Airflow 2.11.2 and 3.4.0:
    • The gated test_fluid_apply_airflow_live.py passes on both, re-run on this head. It runs a DagBag parse of every golden contract, the kept-alive one included.
    • airflow dags test platform.events__aws__stream-raw succeeds on both. Its task ran fluid apply <contract> --env aws --mode reconcile --build-id stream-raw --yes, recorded by a stub fluid passed in through FLUID_BIN.
  • Real CLI: fluid validate and fluid generate schedule --env aws on a stream-plus-daily-batch contract wrote two DAGs:
    • stream_raw_dag.py: the keepalive, with */5 * * * *, max_active_runs=1, catchup=False and retries 0;
    • daily_curated_dag.py.

Reviews

  • /security-review: no HIGH or MEDIUM findings.
    • Checked: the dagConfig.schedule cron is validated and reaches the source only as a string literal. The script is built from module constants. Ids are validated twice. The argv is fixed in code.
    • Re-run on the fix: no HIGH or MEDIUM findings (the new message text goes only to the CLI error, and the new checks only narrow what is emitted).
  • Delivery review: CONCERNS, one P2. scaffold-composer wrote a daily plan-and-apply DAG for a contract whose only builds are kept-alive streams. Fixed.
  • Defect scan: 1 High and 3 Medium. All four were confirmed, because the regression tests fail on the pre-fix commit, and all four are fixed:
    • A dataset trigger on an expose that only a kept-alive stream writes passed silently, and would never fire. It is refused again.
    • --scheduler prefect or dagster wrote a placeholder flow for such a contract. It now writes nothing and names the Airflow keepalive.
    • scaffold-composer had the same problem (the delivery review's P2).
    • A stream on another platform (trigger.type: continuous) got a keepalive DAG. It no longer does.
    • Re-check: an auditor re-ran each trigger on the fix commit: all four FIXED, and no new Critical, High or Medium defect (it probed mixed writers, an explicit --scheduler airflow, orchestration.engine: prefect and an unknown scheduler).
  • Rounds: 1.
  • After the gate, text only: the DAG text, the KEEPALIVE_APPLY_ARGS comment and the CHANGELOG entry now name every case a reconcile run fails on, from feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013's final behaviour (an unreadable job config and the worker's settings included). The registry fixture fix is test-only.

Documentation

  • Docs TODO: the forge_docs pages below need this change. Their docs PR follows it.

Pages:

  • docs/cli/generate.md (generate schedule): the keepalive DAG for a stream with execution.retries. Cover its cadence and the dagConfig.schedule override, the single run with no retries and no outlets, and the fact that a streams-with-retries contract no longer gets "No build needs a DAG".
  • docs/advanced/airflow.md: keeping an always-on stream running from Airflow. Cover stopping a stream on purpose (pause the keepalive DAG, then cancel the run; to resume, unpause and run fluid apply), and that the worker needs the same FLUID_SPARK_* settings and forge release as the CI apply: a missing allowlist refuses the build, and a setting that changes the compiled config reads as drift. The worker's role needs the product's fluid-<product>-spark-caller policy: every start, a restart included, reads the stream's job config and runner (s3:GetObject), and on EMR on EC2 it also calls DescribeCluster and reads the step role.
  • docs/advanced/spark-iceberg-aws.md: a stream's execution.retries as its restart budget, with the keepalive DAG that applies it.
  • docs/upgrading.md ("Not in a release yet"): one line. A contract whose stream declares execution.retries gets one more DAG file on its next fluid generate schedule.

Follow-ups (not in this PR)

  • A keepalive for every always-on stream, now that --mode reconcile has a default budget. It stays opt-in here, for the reason under "Which streams get a keepalive DAG".
  • When a kept-alive stream and a build that runs outside every scheduler (a Flink job, for example) both write an expose, a dataset trigger on it is refused, as it should be. The message, though, gives the generic reason ("which no DAG runs") instead of naming the keepalive.

fas89 added 14 commits November 18, 2026 23:38
A forge Spark stream that is not trigger.availableNow and declares
execution.retries gets a DAG of its own: one task that runs the
reconcile-only fluid apply for that stream (KEEPALIVE_APPLY_ARGS, the one
place it is named), every 5 minutes or on the build's
execution.orchestration.airflow.dagConfig.schedule, with max_active_runs=1,
no catch-up and no Airflow retries, so the build's own retries bound the
restarts. It declares no outlets: a stream updates its tables all the time.
A contract whose only builds are such streams now needs a DAG.
The keepalive DAG's task runs fluid apply <contract> [--env] --mode
reconcile --build-id <stream> --yes. That runs no tofu apply: it starts the
stream again on its checkpoint within the restart budget the build's
execution.retries sets, and fails the run when the stream drifted from the
contract or the budget is spent. The arguments stay in KEEPALIVE_APPLY_ARGS,
joined one space apart whatever they hold, and a test parses them with
fluid's own parser and its --build-id check, so the DAG cannot run a mode or
flag that fluid apply refuses.
A keepalive DAG reads only the contract's builds: a stream is in no build graph, since nothing waits for it. So _keepalive_dags() sits with _keeps_alive() and _keepalive_schedule() rather than in _BuildGraph, which keeps to the dependencies between the builds a DAG runs.
…nothing

Four places counted a keepalive DAG as more than it is:

- A dataset trigger on an expose that only a kept-alive stream writes was no
  longer refused, though a keepalive declares no outlets and the trigger would
  never fire. It is refused again, and the message says why.
- fluid generate schedule --scheduler prefect or dagster wrote a placeholder
  flow for a contract whose only builds are kept-alive streams, and fluid
  scaffold-composer wrote a daily plan-and-apply DAG for one. Neither renders a
  keepalive. Both write nothing again, and name the Airflow keepalive DAG.
- A stream that trigger.type: continuous or a Flink job leaves to another
  platform got a keepalive DAG. It no longer does, as _classify already has it.
The keepalive DAG's text said a run fails when the stream drifted or the
budget is spent. fluid apply --mode reconcile also fails when nothing is
deployed or the stream's cluster ended, and on EMR Serverless, EMR on EKS
and Glue the service's own retries are the restarts. It leaves a run
cancelled on purpose stopped until fluid apply starts it. The DAG text, the
KEEPALIVE_APPLY_ARGS comment and the CHANGELOG entry now say so.
The autouse fixture in test_scheduler_framework.py emptied the shared
scheduler registry and left it empty. The built-in schedulers register only
when fluid_build.schedulers is first imported, so a later test in the same
worker that looked up prefect or dagster by name found no scheduler. CI runs
the suite under xdist with pytest-randomly, so such a test failed only on some
orders. Each test now gets an empty registry of its own through monkeypatch,
and the shared one is back in place after it.
…live fails on

The keepalive DAG text and its CHANGELOG entry now also name two cases a
reconcile exits 1 on. The job config the last apply deployed cannot be read:
it is read on every run, while the stream is healthy too. The worker's
FLUID_SPARK_* settings or forge release differ from the CI apply's: a policy
gate refuses the build, or the compiled config reads as drift.
The 0.33.0 cut turned the old Unreleased section into 0.33.0, and this
entry moved with it. The keepalive DAG is not in 0.33.0, so it is listed
under Unreleased, after the entry for Spark stream batch bounds.
… DAG

The reconcile paragraph names a keepalive as a scheduler task every few minutes. fluid generate schedule now renders one, as an Airflow DAG, for a stream that declares execution.retries.
The build-graph tests import scaffold_composer, SimpleNamespace and fluid_apply at module level instead of again inside each test, and the registry fixture patches _SCHEDULERS by its dotted path, so the module needs no second import of the registry.
@github-actions github-actions Bot added cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes labels Nov 19, 2026
@fas89
fas89 merged commit cb95dab into main Nov 19, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli Changes to the CLI surface or implementation tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant