Repository navigation
docs(spark): a run starts only on the job config fluid apply compiled (forge-cli #1014) - #268
Merged
Merged
Conversation
… (forge-cli #1014) The two steps, the three refusals, the digest, and the permissions of the identity that runs fluid apply.
…ark-aws-wave15 # Conflicts: # docs/upgrading.md
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents forge-cli #1014, on
main(abf57a48) after the 0.33.0 cut and in no release: a Spark run starts only on the job config thatfluid applycompiled.advanced/spark-iceberg-aws.md: a new section, "A run starts only on the config this apply compiled": the two steps (fluid applyreads the job config and the runner back from S3 before it starts a run on AWS; every runtime passes--fluid-job-sha256and the runner refuses another config), the three refusals of the start, the runner's refusal, the digest and what is left out of it, what the check does not cover, and the identity's permissions. A paragraph after the argv block of the local run, and one in "The caller's permissions".upgrading.md: one bullet under "Not in a release yet".Related CLI PRs
Areas Updated
Decisions
*On forge-climain, not yet in a release ([forge-cli #1014](...)).*: the pull request merged after the 0.33.0 cut. The pin stays at 0.32.0.fluid applyruns aftertofu apply) with botocore's test stubs answering AWS, the reads of S3 included, for the three refusals of the start; forge's runner run as a script, with a stand-in forpyspark, for its refusal; and a localfluid applywith a stand-inspark-submitfor the argv of a local run. They were run again onmainafter the merge and match, but for the config hash and the runner's path, which the runner of #1015 changed.fluid applycancels a running job of another config before it starts the new one, and the check runs inside the start, so a mismatch ends the apply after the cancel. The page says so and tells the reader to runfluid applyagain, which uploads both objects and starts the run.fluid-<product>-spark-callerholdss3:GetObjecton both objects (checked in the generated policy on 0.33.0's code and onmain) and nokms:Decrypt. forge-cli's changelog says that a scheduler's role already reads both objects, which holds fors3:GetObject; the page and the bullet name the missingkms:Decryptfor a bucket that encrypts them with a KMS key.Checklist
python scripts/check_cli_docs.py(with forge-cli's liveCHANGELOG.md),check_providers.pyandgen_contract_reference.py --checkpassNotes
kms:Decryptit lacks (twice).mainequals the code the text was written and reviewed against (the pull request's headd291327f) for every file but the changelog and the runner, whose #1014 part is the same.