Skip to content

docs(spark): fluid generate schedule writes a keepalive DAG for an always-on stream, and a refused start leaves the old run going (forge-cli #1018, #1019) - #275

Merged
fas89 merged 1 commit into
mainfrom
claude/docs-spark-keepalive
Nov 19, 2026
Merged

fas89 merged 1 commit into
mainfrom
claude/docs-spark-keepalive

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Documents two forge-cli pull requests that are in 0.34.0: forge-cli #1018, a keepalive DAG for an always-on Spark stream, and forge-cli #1019, a refused start that cancels nothing and a scheduler policy that decrypts through S3.

  • advanced/airflow.md: the new section "Keeping a stream running". It says which streams get a keepalive DAG, what the DAG is (the file, the DAG id, the fluid_keepalive task and its command), its schedule and limits, why it has no outlets, what a run does and when it fails, how to stop it, and what the worker needs. It has five real outputs: what fluid generate schedule prints, the DAG's docstring, the message on Prefect and Dagster, the message of fluid scaffold-composer, and the refusal of @once. Two existing sentences change: a stream that declares execution.retries is the exception to "a streaming build gets no DAG".
  • A stream that declares execution.retries counts as a scheduled build, so stage 3 (fluid generate artifacts) writes its DAG when the contract sets no engine. cli/generate-artifacts.md, cli/schedule-sync.md, walkthrough/11-stage-pipeline.md, walkthrough/airflow-declarative.md and advanced/production-troubleshooting.md said that it writes none in that case. cli/scaffold-composer.md notes the second line of the command's message.
  • cli/generate.md, cli/apply.md and "The restart budget" in advanced/spark-iceberg-aws.md each point to the new section.
  • advanced/spark-iceberg-aws.md, "A run starts only on the config this apply compiled": the read-back comes before the cancel of a running job of another config, so a refused start leaves that job running. A second real output line shows it. On EMR on EC2 the check that the policy is attached is part of the start, after that cancel. In "The caller's permissions", the scheduler's policy holds kms:Decrypt through S3 only, in an S3Kms statement, and the sentences that said it holds none are rewritten.
  • upgrading.md: one bullet for the keepalive DAG, and the two statements that #1019 made stale are rewritten.
  • scripts/forge-cli-releases.json maps #1018 and #1019 to 0.34.0 (66 pull requests mapped, reviewed through 0.34.0).

Related CLI PRs

Areas Updated

  • Getting Started
  • CLI Reference
  • Provider Docs
  • Walkthroughs / Examples
  • Contributing / Community Docs
  • Other: Airflow, upgrading, troubleshooting

Decisions

  • Both pull requests merged before forge-cli cut 0.34.0 (127fa5bf on main), and the changelog's 0.34.0 entry describes them, so the markers read *(since 0.34.0, ...)*. The pin stays at 0.33.0.
  • #1019 has no bullet of its own in upgrading.md. It fixes code that #1014 and #1013 had introduced and that no release contained, so 0.33.0 never had a start that reads the job config back, an AccessDenied on a bucket that a KMS key encrypts, or a stream left stopped by a refused start. forge-cli's changelog tells it as a fix, and these pages describe 0.34.0 against 0.33.0. So the two statements that it made stale ("the stream stays stopped", "holds no kms:Decrypt") are now statements of the behavior, and the two groups they sit in cite #1019 as well.
  • The scheduler's policy decrypts through S3 only, so the pages say that a bucket's KMS key needs no IAM addition for the scheduler's role, and that the key's own policy still has to allow the role unless it leaves that to the account's IAM policies.
  • The outputs of the keepalive section are from the real fluid generate schedule and fluid scaffold-composer on forge-cli main at the cut, on the example stream with execution: {retries: {maxAttempts: 5}} added. No Airflow ran here: forge-cli's own tests load the DAG in a real Airflow 2 and Airflow 3 when one is named.
  • The two refusal lines are from forge-cli's own post-apply reconcile on an EMR on EC2 stream, with AWS answered by botocore's Stubber (one with no job running, one with a running job of another config).

Checklist

  • python scripts/check_cli_docs.py against the 0.33.0 CLI and forge-cli main's changelog, check_providers.py and gen_contract_reference.py --check pass
  • Site build, built-link check, offline link check and the linked anchors pass locally

Notes

  • A reviewer checked 62 sentences against forge-cli main at the cut and the 0.33.0 package, with about 48 runs and reads, and found nine problems. All are fixed: stage 3 also writes the keepalive DAG (five pages said it does not, and the reviewer did not name schedule-sync.md, which said the same); on EMR on EC2 the policy check comes after the cancel; the list of outcomes that fail a run was incomplete; "needs nothing added" for a KMS key left out the key's own policy; and the message of fluid scaffold-composer names fluid generate schedule, not Airflow. I reran the commands behind each fix; the fixes did not get a second review.
  • Not covered: a real Airflow loading the DAG, and the behavior of the S3Kms statement on a real key (only the emitted policy was checked). The 0.33.0 result of stage 3 is read from the code, because that module would not import in the 0.33.0 environment.
  • Docs main (docs: every forge-cli change in 0.34.0 reads since 0.34.0, and the releases map is refreshed #274, the 0.34.0 marker conversion) is the base.

…ways-on stream, and a refused start leaves the old run going (forge-cli #1018, #1019)

Documents forge-cli #1018 and #1019, both in 0.34.0.

- advanced/airflow.md: the section "Keeping a stream running": who gets a
  keepalive DAG, what it is, its schedule and limits, what a run does and
  what the worker needs, with real outputs.
- Stage 3, schedule-sync, scaffold-composer and the walkthroughs: a Spark
  stream that declares execution.retries counts as a scheduled build.
- advanced/spark-iceberg-aws.md: the read-back comes before the cancel of a
  running job of another config, so a refused start leaves that job running;
  the scheduler's policy holds kms:Decrypt through S3 in an S3Kms statement.
- upgrading.md: a bullet for the keepalive DAG, and two statements rewritten.
- scripts/forge-cli-releases.json maps #1018 and #1019 to 0.34.0.
@fas89
fas89 merged commit 58a2aab into main Nov 19, 2026
9 checks passed

This branch was previously deployed

1 inactive deployment
github-pages — cae90972 Deployed Nov 19, 2026 by fas89 via deploy #433
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant