Skip to content

docs(spark): a dead stream restarts within a budget, fluid apply --mode reconcile keeps streams running, and a destroy waits for a starting application (forge-cli #1013, #1017) - #272

Merged
fas89 merged 2 commits into
mainfrom
claude/docs-spark-aws-wave18
Nov 19, 2026
Merged

fas89 merged 2 commits into
mainfrom
claude/docs-spark-aws-wave18

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Documents forge-cli #1013 (95b58b31) and #1017 (b73a60b2), on main after the 0.33.0 cut and in no release.

  • #1013. A Spark stream that died is started again on its checkpoint within a restart budget, on a plain fluid apply too, and fluid apply --mode reconcile keeps the always-on streams running and deploys nothing.
  • #1017. A destroy of a removed build waits for an EMR Serverless application that is still being created or started.

What changed:

  • cli/apply.md: a row of the mode table, the --build-id sentence, and a new section "Reconcile mode" (what it reads, which streams, what it never does, --dry-run and the prompt, the operator's settings, the table of what it finds and does for each stream, and five real outputs).
  • advanced/spark-iceberg-aws.md: a new section "The restart budget" (the rule, what is a death and what is a cancel, who restarts, how it is counted, the spent budget, the config hash, two applies at once, and outputs for EMR on EC2, EMR Serverless, EMR on EKS and a local build); the reconcile table row, the Spot checklist bullet, the dead-letter text and two older sentences that said a failed stream is not started again; the scheduler policy (it now covers streams, one statement for each runtime, no kms:Decrypt); and, for #1017, the destroy's step 3 and the third line of its refusal.
  • advanced/airflow.md: one sentence: a scheduler task keeps a stream running with --mode reconcile.
  • upgrading.md: four bullets under "Not in a release yet".

Related CLI PRs

Areas Updated

  • Getting Started
  • CLI Reference
  • Provider Docs
  • Walkthroughs / Examples
  • Contributing / Community Docs
  • Other: advanced guides, upgrading

Decisions

  • The markers read *On forge-cli main, not yet in a release ([forge-cli #NNNN](...)).*: both pull requests merged after the 0.33.0 cut. The pin is 0.33.0 (docs: forge-cli 0.33.0 (pin, contract reference, upgrade notes) #271), and the gates ran against that release.
  • One pull request for the two, as they merged within the hour and both touch "Starting and reconciling runs" and the end of upgrading.md.
  • The page documents --mode reconcile as a command that a keepalive runs. The generator of a keepalive DAG is not on forge-cli main, so nothing says that forge writes one.
  • The outputs are real. The local ones are the real fluid apply on a stream of runtime: local with a stand-in spark-submit, killed from outside with kill -9, on main and, for the comparison, on 0.33.0's code. The EMR on EC2 ones are forge-cli's reconcile step with AWS answered by botocore's test stubs: no cluster was touched. The EMR Serverless and EMR on EKS texts are the strings of the same code for a run that ended FAILED. The scheduler policy is the emitter's output on 0.33.0's code and on main.
  • A stand-in does not show Spark's own behaviour, so the page says where it is a stand-in.

Checklist

  • python scripts/check_cli_docs.py (with forge-cli's live CHANGELOG.md), check_providers.py and gen_contract_reference.py --check pass
  • Site build, built-link check, offline link check and the linked anchors pass locally

Notes

  • Two reviewers checked about 118 sentences against the code (with about 80 runs: a real local stream killed and reconciled, forge-cli's reconcile step with AWS stubbed, the scheduler policy on 0.33.0's code and on main, and the destroy step). Twelve findings are fixed. The real ones: a restart whose new run ends at once exits 1; the scheduler policy has two more statements (StepRoleEmrEc2, FindApplication); a local run that was killed on its own was already failed in 0.33.0, and only a supervisor that got a signal read as cancelled; the options table of --mode lacked reconcile; the local table and the reconcile table still read as the current rule for a failed stream; and the advice to pause a keepalive before cancelling contradicted the rule it followed, so it is gone.
  • Docs main is the base (docs(spark): a Spark job is granted its own table where Lake Formation governs new tables (forge-cli #1010) #270 is merged).
  • The code of #1013 and #1017 on main equals the code the text was written and reviewed against (the pull requests' heads 428e8920 and b064a2be) for every code file.

…de reconcile keeps streams running, and a destroy waits for a starting application (forge-cli #1013, #1017)

Reconcile mode, the restart budget and its outputs, the scheduler policy for streams, and the destroy of an EMR Serverless application that is still being created or started.
@fas89
fas89 merged commit 4c0516e into main Nov 19, 2026
9 checks passed

This branch was previously deployed

1 inactive deployment
github-pages — 1fed7386 Deployed Nov 19, 2026 by fas89 via deploy #427
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant