Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ Every contract is delivered through an 11-stage pipeline. Each stage is a hard g
| `replace` | auto-snapshot → `CREATE OR REPLACE TABLE` | — | **dropped**; backup retained |
| `replace-and-build` | same as `replace` | `dbt run --full-refresh` | **dropped**; rebuilt |

`--mode reconcile [--build-id <stream>]` runs no `tofu`, no DDL and no build, and refuses a `plan.json`: it keeps the contract's always-on Spark streams running with the config the last apply deployed (the job config it uploaded, and the config hash its runs carry). A stream that died starts again on its checkpoint within `execution.retries.maxAttempts` restarts per rolling hour (5 by default, at most 10); drift, a spent budget or nothing deployed starts nothing and exits 1; a run cancelled through the runtime is left stopped. A keepalive (a scheduler task every few minutes, with `--build-id <stream>`) keeps a stream up. A plain `fluid apply` restarts a dead stream under the same budget. EMR Serverless and EMR on EKS retry failed runs themselves, so forge never restarts a FAILED run there; Glue's job is emitted with MaxRetries 0, so forge restarts Glue streams as it does EMR on EC2 and local ones. Code: `build_runners/spark/_reconcile.py`, `cli/_spark_apply.py::reconcile_only`.
`--mode reconcile [--build-id <stream>]` runs no `tofu`, no DDL and no build, and refuses a `plan.json`: it keeps the contract's always-on Spark streams running with the config the last apply deployed (the job config it uploaded, and the config hash its runs carry). A stream that died starts again on its checkpoint within `execution.retries.maxAttempts` restarts per rolling hour (5 by default, at most 10); drift, a spent budget or nothing deployed starts nothing and exits 1; a run cancelled through the runtime is left stopped. A keepalive (a scheduler task every few minutes, with `--build-id <stream>`) keeps a stream up; `fluid generate schedule` renders one as an Airflow DAG for a stream that declares `execution.retries`. A plain `fluid apply` restarts a dead stream under the same budget. EMR Serverless and EMR on EKS retry failed runs themselves, so forge never restarts a FAILED run there; Glue's job is emitted with MaxRetries 0, so forge restarts Glue streams as it does EMR on EC2 and local ones. Code: `build_runners/spark/_reconcile.py`, `cli/_spark_apply.py::reconcile_only`.

`--allow-data-loss` is required for `replace*` when `FLUID_ENV != dev` OR target has rows. `--no-verify-plan-binding` is an emergency DR waiver for plan-binding checks (logged at WARNING level so audit trails catch it).

Expand Down Expand Up @@ -156,7 +156,7 @@ forge-cli/
| Provider plugins (planning) | `fluid_build/providers/{local,gcp,aws,snowflake,odps,odcs}/` — `plan()` emits abstract actions |
| **IaC emitter (apply)** | `fluid_build/iac/` — modular OpenTofu plugin registry. `cli/_apply_opentofu_engine.py` compiles the contract to `main.tf.json` and shells `tofu` for cloud providers. `cli/generate_iac.py` is `fluid generate iac` for review-only emit. One plugin per cloud at `iac/providers/{aws,gcp,snowflake}.py` (dbt-adapter pattern). |
| Path-B schedulers | `fluid_build/providers/aws/plan/schedule.py` (EventBridge/MWAA/Lambda/Step-Functions via stage 6 `SchedulePlanner`) |
| Airflow DAGs | `fluid_build/schedulers/airflow/` is the one renderer. `render_dag()` (`__init__.py`, operators in `_operators.py`) renders a contract's `orchestration` section for gcp/aws/snowflake; `fluid_apply.py` renders each build as one `fluid apply --mode amend-and-build --build-id` task, a forge Spark build included (its build phase starts the run and waits; a stream that is not `availableNow` gets no DAG). Builds that `builds[].dependencies` link share one DAG (`scheduled_dags()`: the graph and the schedule; `render_build_task()`: the one function that writes a build's node), on the one trigger they declare: a cron, `trigger.datasets` (`dataset`), or both (`schedule_and_dataset`). Every task declares its build's exposes as Airflow asset `outlets`, named as Airflow's providers name the storage (`_assets.py`). `fluid generate schedule`, `fluid export --engine airflow` and `fluid generate-airflow` (a deprecated alias of the first) all come through it. Task variables go through `codegen_utils.unique_task_identifiers`. |
| Airflow DAGs | `fluid_build/schedulers/airflow/` is the one renderer. `render_dag()` (`__init__.py`, operators in `_operators.py`) renders a contract's `orchestration` section for gcp/aws/snowflake; `fluid_apply.py` renders each build as one `fluid apply --mode amend-and-build --build-id` task, a forge Spark build included (its build phase starts the run and waits; a stream that is not `availableNow` gets no DAG). Builds that `builds[].dependencies` link share one DAG (`scheduled_dags()`: the graph and the schedule; `render_build_task()`: the one function that writes a build's node), on the one trigger they declare: a cron, `trigger.datasets` (`dataset`), or both (`schedule_and_dataset`). Every task declares its build's exposes as Airflow asset `outlets`, named as Airflow's providers name the storage (`_assets.py`). A Spark stream that declares `execution.retries` gets a keepalive DAG whose task runs `fluid apply --mode reconcile --build-id <stream>` every 5 minutes (`KEEPALIVE_APPLY_ARGS`). `fluid generate schedule`, `fluid export --engine airflow` and `fluid generate-airflow` (a deprecated alias of the first) all come through it. Task variables go through `codegen_utils.unique_task_identifiers`. |
| Contract schemas | `fluid_build/schemas/*.json` (0.7.2 is latest; `SchemaManager.latest_bundled_version()` is the dynamic lookup) |
| Policy engine | `fluid_build/policy/` (compiler, agent_policy, sovereignty, guardrails) |
| Plan-binding helpers | `fluid_build/forge/core/plan_digest.py` — `inject_digests`, `verify_plan_binding`, `PlanBindingError` |
Expand Down
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`fluid.dead-letter.raw-key` header: the Kafka sink is at least once, and a batch sent again
after a failure carries the same keys, so a consumer drops repeats by key.

- **A keepalive DAG for an always-on stream that declares `execution.retries`.** `fluid
generate schedule` gives each forge Spark stream that is not `trigger.availableNow` and
declares `execution.retries` a small DAG whose one task runs
`fluid apply <contract> --mode reconcile --build-id <stream> --yes`, so a stream a Spot
interruption or a failed driver stopped starts again on its checkpoint. It runs every 5
minutes, or on the build's `execution.orchestration.airflow.dagConfig.schedule`, one run at a
time, with no catch-up and no Airflow retries: the build's `execution.retries` is the restart
budget, and a run fails when the stream cannot be kept running (it drifted from the contract,
nothing is deployed, its job config cannot be read, its cluster ended, or the restarts are
spent). The worker needs the `FLUID_SPARK_*` settings and the forge release the CI apply has,
or a run can fail too. A run cancelled on purpose stays stopped. The task declares no
outlets, so a dataset trigger on what only the stream writes is refused, as before. A contract
whose only builds are such streams now gets this DAG on Airflow instead of "No build needs a
DAG"; on Prefect or Dagster, and in `fluid scaffold-composer`, it still has nothing to
schedule, and the message names the Airflow keepalive.

### Fixed

- **`fluid apply` runs a dbt build whatever its trigger, finds the dbt that has its adapter, and
Expand Down
12 changes: 10 additions & 2 deletions fluid_build/cli/generate_schedule.py
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,11 @@ def run(args: Any, logger: logging.Logger) -> int:
# stream fluid apply keeps running, or a build whose trigger is
# continuous) has nothing to schedule, which is not an error: unless it
# hand-declares orchestration tasks, or names a scheduler that does not
# exist.
if fluid_apply.runs_only_outside_schedulers(contract) and not (
# exist. Only the Airflow DAGs keep a stream running (its keepalive DAG),
# so on another scheduler a stream that declares execution.retries has
# nothing to schedule either.
keepalive = (scheduler_name or fluid_apply.DEFAULT_ENGINE) == fluid_apply.DEFAULT_ENGINE
if fluid_apply.runs_only_outside_schedulers(contract, keepalive=keepalive) and not (
fluid_apply.has_explicit_tasks(contract)
):
from fluid_build.schedulers import has_scheduler
Expand All @@ -183,6 +186,11 @@ def run(args: Any, logger: logging.Logger) -> int:
"continuously (trigger.type: continuous), so there is nothing to "
"schedule. No files written."
)
if not fluid_apply.runs_only_outside_schedulers(contract):
cprint(
"A stream that declares execution.retries gets a keepalive DAG on "
"Airflow: fluid generate schedule --scheduler airflow."
)
return 0

if not scheduler_name:
Expand Down
20 changes: 15 additions & 5 deletions fluid_build/cli/scaffold_composer.py
Original file line number Diff line number Diff line change
Expand Up @@ -202,16 +202,23 @@ def _wrote_dag(dag_path: str, dag_id: str) -> bool:
)


def _report_nothing_to_schedule(dag_path: str, dag_id: str) -> None:
def _report_nothing_to_schedule(dag_path: str, dag_id: str, *, keepalive: bool = False) -> None:
"""Say why no DAG is written, as ``fluid generate schedule`` does, and name
a DAG this command wrote for the product before: left in place, it keeps
scheduling the product wherever it was deployed. A file of the same name
that it did not write is someone else's, and is not mentioned."""
that it did not write is someone else's, and is not mentioned. With
``keepalive``, a stream declares ``execution.retries``: say which command
writes the DAG that keeps it running."""
cprint(
"Every build runs outside forge-cli (a Flink job, for example), as a stream that fluid "
"apply keeps running, or continuously (trigger.type: continuous), so there is nothing "
"to schedule. No DAG written."
)
if keepalive:
cprint(
"A stream that declares execution.retries gets a keepalive DAG from fluid generate "
"schedule, which restarts it when it stops."
)
if _wrote_dag(dag_path, dag_id):
# Printed as written: soft_wrap leaves wrapping to the terminal, so the
# path is never broken by an inserted newline and copies whole, and no
Expand Down Expand Up @@ -241,15 +248,18 @@ def run(args, logger: logging.Logger) -> int:
# apply keeps running, so a contract with no other build has nothing to
# schedule and gets no DAG. The test is `fluid generate schedule`'s,
# hand-declared orchestration tasks included, so the two commands agree
# on such a contract.
# on such a contract. A stream's keepalive DAG is generate schedule's
# alone: this command's validate, plan and apply DAG cannot keep a
# stream running, so here a stream with execution.retries is outside too.
from fluid_build.schedulers.airflow.fluid_apply import (
has_explicit_tasks,
runs_only_outside_schedulers,
)

if runs_only_outside_schedulers(c) and not has_explicit_tasks(c):
if runs_only_outside_schedulers(c, keepalive=False) and not has_explicit_tasks(c):
dag_path = os.path.join(os.path.abspath(args.out_dir), f"{dag_id}.py")
_report_nothing_to_schedule(dag_path, dag_id)
kept_alive = not runs_only_outside_schedulers(c)
_report_nothing_to_schedule(dag_path, dag_id, keepalive=kept_alive)
return 0
cron = _validate_cron(_extract_cron(c))
# SECURITY (shell + Python-literal injection): the contract path is
Expand Down
Loading
Loading