Repository navigation
docs(spark): fluid generate schedule writes a keepalive DAG for an always-on stream, and a refused start leaves the old run going (forge-cli #1018, #1019) - #275
Merged
Conversation
…ways-on stream, and a refused start leaves the old run going (forge-cli #1018, #1019) Documents forge-cli #1018 and #1019, both in 0.34.0. - advanced/airflow.md: the section "Keeping a stream running": who gets a keepalive DAG, what it is, its schedule and limits, what a run does and what the worker needs, with real outputs. - Stage 3, schedule-sync, scaffold-composer and the walkthroughs: a Spark stream that declares execution.retries counts as a scheduled build. - advanced/spark-iceberg-aws.md: the read-back comes before the cancel of a running job of another config, so a refused start leaves that job running; the scheduler's policy holds kms:Decrypt through S3 in an S3Kms statement. - upgrading.md: a bullet for the keepalive DAG, and two statements rewritten. - scripts/forge-cli-releases.json maps #1018 and #1019 to 0.34.0.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents two forge-cli pull requests that are in 0.34.0: forge-cli #1018, a keepalive DAG for an always-on Spark stream, and forge-cli #1019, a refused start that cancels nothing and a scheduler policy that decrypts through S3.
advanced/airflow.md: the new section "Keeping a stream running". It says which streams get a keepalive DAG, what the DAG is (the file, the DAG id, thefluid_keepalivetask and its command), its schedule and limits, why it has no outlets, what a run does and when it fails, how to stop it, and what the worker needs. It has five real outputs: whatfluid generate scheduleprints, the DAG's docstring, the message on Prefect and Dagster, the message offluid scaffold-composer, and the refusal of@once. Two existing sentences change: a stream that declaresexecution.retriesis the exception to "a streaming build gets no DAG".execution.retriescounts as a scheduled build, so stage 3 (fluid generate artifacts) writes its DAG when the contract sets no engine.cli/generate-artifacts.md,cli/schedule-sync.md,walkthrough/11-stage-pipeline.md,walkthrough/airflow-declarative.mdandadvanced/production-troubleshooting.mdsaid that it writes none in that case.cli/scaffold-composer.mdnotes the second line of the command's message.cli/generate.md,cli/apply.mdand "The restart budget" inadvanced/spark-iceberg-aws.mdeach point to the new section.advanced/spark-iceberg-aws.md, "A run starts only on the config this apply compiled": the read-back comes before the cancel of a running job of another config, so a refused start leaves that job running. A second real output line shows it. On EMR on EC2 the check that the policy is attached is part of the start, after that cancel. In "The caller's permissions", the scheduler's policy holdskms:Decryptthrough S3 only, in anS3Kmsstatement, and the sentences that said it holds none are rewritten.upgrading.md: one bullet for the keepalive DAG, and the two statements that #1019 made stale are rewritten.scripts/forge-cli-releases.jsonmaps #1018 and #1019 to 0.34.0 (66 pull requests mapped, reviewed through 0.34.0).Related CLI PRs
Areas Updated
Decisions
127fa5bfonmain), and the changelog's 0.34.0 entry describes them, so the markers read*(since 0.34.0, ...)*. The pin stays at 0.33.0.upgrading.md. It fixes code that #1014 and #1013 had introduced and that no release contained, so 0.33.0 never had a start that reads the job config back, anAccessDeniedon a bucket that a KMS key encrypts, or a stream left stopped by a refused start. forge-cli's changelog tells it as a fix, and these pages describe 0.34.0 against 0.33.0. So the two statements that it made stale ("the stream stays stopped", "holds nokms:Decrypt") are now statements of the behavior, and the two groups they sit in cite #1019 as well.fluid generate scheduleandfluid scaffold-composeron forge-climainat the cut, on the example stream withexecution: {retries: {maxAttempts: 5}}added. No Airflow ran here: forge-cli's own tests load the DAG in a real Airflow 2 and Airflow 3 when one is named.Checklist
python scripts/check_cli_docs.pyagainst the 0.33.0 CLI and forge-climain's changelog,check_providers.pyandgen_contract_reference.py --checkpassNotes
mainat the cut and the 0.33.0 package, with about 48 runs and reads, and found nine problems. All are fixed: stage 3 also writes the keepalive DAG (five pages said it does not, and the reviewer did not nameschedule-sync.md, which said the same); on EMR on EC2 the policy check comes after the cancel; the list of outcomes that fail a run was incomplete; "needs nothing added" for a KMS key left out the key's own policy; and the message offluid scaffold-composernamesfluid generate schedule, not Airflow. I reran the commands behind each fix; the fixes did not get a second review.S3Kmsstatement on a real key (only the emitted policy was checked). The 0.33.0 result of stage 3 is read from the code, because that module would not import in the 0.33.0 environment.main(docs: every forge-cli change in 0.34.0 reads since 0.34.0, and the releases map is refreshed #274, the 0.34.0 marker conversion) is the base.