Repository navigation
docs(spark): a dead stream restarts within a budget, fluid apply --mode reconcile keeps streams running, and a destroy waits for a starting application (forge-cli #1013, #1017) - #272
Merged
Conversation
…de reconcile keeps streams running, and a destroy waits for a starting application (forge-cli #1013, #1017) Reconcile mode, the restart budget and its outputs, the scheduler policy for streams, and the destroy of an EMR Serverless application that is still being created or started.
4 of 8 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents forge-cli #1013 (
95b58b31) and #1017 (b73a60b2), onmainafter the 0.33.0 cut and in no release.fluid applytoo, andfluid apply --mode reconcilekeeps the always-on streams running and deploys nothing.What changed:
cli/apply.md: a row of the mode table, the--build-idsentence, and a new section "Reconcile mode" (what it reads, which streams, what it never does,--dry-runand the prompt, the operator's settings, the table of what it finds and does for each stream, and five real outputs).advanced/spark-iceberg-aws.md: a new section "The restart budget" (the rule, what is a death and what is a cancel, who restarts, how it is counted, the spent budget, the config hash, two applies at once, and outputs for EMR on EC2, EMR Serverless, EMR on EKS and a local build); the reconcile table row, the Spot checklist bullet, the dead-letter text and two older sentences that said a failed stream is not started again; the scheduler policy (it now covers streams, one statement for each runtime, nokms:Decrypt); and, for #1017, the destroy's step 3 and the third line of its refusal.advanced/airflow.md: one sentence: a scheduler task keeps a stream running with--mode reconcile.upgrading.md: four bullets under "Not in a release yet".Related CLI PRs
Areas Updated
Decisions
*On forge-climain, not yet in a release ([forge-cli #NNNN](...)).*: both pull requests merged after the 0.33.0 cut. The pin is 0.33.0 (docs: forge-cli 0.33.0 (pin, contract reference, upgrade notes) #271), and the gates ran against that release.upgrading.md.--mode reconcileas a command that a keepalive runs. The generator of a keepalive DAG is not on forge-climain, so nothing says that forge writes one.fluid applyon a stream ofruntime: localwith a stand-inspark-submit, killed from outside withkill -9, onmainand, for the comparison, on 0.33.0's code. The EMR on EC2 ones are forge-cli's reconcile step with AWS answered by botocore's test stubs: no cluster was touched. The EMR Serverless and EMR on EKS texts are the strings of the same code for a run that endedFAILED. The scheduler policy is the emitter's output on 0.33.0's code and onmain.Checklist
python scripts/check_cli_docs.py(with forge-cli's liveCHANGELOG.md),check_providers.pyandgen_contract_reference.py --checkpassNotes
main, and the destroy step). Twelve findings are fixed. The real ones: a restart whose new run ends at once exits1; the scheduler policy has two more statements (StepRoleEmrEc2,FindApplication); a local run that was killed on its own was already failed in 0.33.0, and only a supervisor that got a signal read as cancelled; the options table of--modelackedreconcile; the local table and the reconcile table still read as the current rule for a failed stream; and the advice to pause a keepalive before cancelling contradicted the rule it followed, so it is gone.mainis the base (docs(spark): a Spark job is granted its own table where Lake Formation governs new tables (forge-cli #1010) #270 is merged).mainequals the code the text was written and reviewed against (the pull requests' heads428e8920andb064a2be) for every code file.