Skip to content

docs(spark): a run starts only on the job config fluid apply compiled (forge-cli #1014) - #268

Merged
fas89 merged 2 commits into
mainfrom
claude/docs-spark-aws-wave15
Nov 19, 2026
Merged

fas89 merged 2 commits into
mainfrom
claude/docs-spark-aws-wave15

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Documents forge-cli #1014, on main (abf57a48) after the 0.33.0 cut and in no release: a Spark run starts only on the job config that fluid apply compiled.

  • advanced/spark-iceberg-aws.md: a new section, "A run starts only on the config this apply compiled": the two steps (fluid apply reads the job config and the runner back from S3 before it starts a run on AWS; every runtime passes --fluid-job-sha256 and the runner refuses another config), the three refusals of the start, the runner's refusal, the digest and what is left out of it, what the check does not cover, and the identity's permissions. A paragraph after the argv block of the local run, and one in "The caller's permissions".
  • upgrading.md: one bullet under "Not in a release yet".

Related CLI PRs

Areas Updated

  • Getting Started
  • CLI Reference
  • Provider Docs
  • Walkthroughs / Examples
  • Contributing / Community Docs
  • Other: advanced guide, upgrading

Decisions

  • The markers read *On forge-cli main, not yet in a release ([forge-cli #1014](...)).*: the pull request merged after the 0.33.0 cut. The pin stays at 0.32.0.
  • The outputs are real, without an AWS account or a Spark: forge-cli's reconcile step (the code fluid apply runs after tofu apply) with botocore's test stubs answering AWS, the reads of S3 included, for the three refusals of the start; forge's runner run as a script, with a stand-in for pyspark, for its refusal; and a local fluid apply with a stand-in spark-submit for the argv of a local run. They were run again on main after the merge and match, but for the config hash and the runner's path, which the runner of #1015 changed.
  • A refused start leaves a stream stopped. The reviewer found that fluid apply cancels a running job of another config before it starts the new one, and the check runs inside the start, so a mismatch ends the apply after the cancel. The page says so and tells the reader to run fluid apply again, which uploads both objects and starts the run.
  • The scheduler's policy fluid-<product>-spark-caller holds s3:GetObject on both objects (checked in the generated policy on 0.33.0's code and on main) and no kms:Decrypt. forge-cli's changelog says that a scheduler's role already reads both objects, which holds for s3:GetObject; the page and the bullet name the missing kms:Decrypt for a bucket that encrypts them with a KMS key.
  • The runner changed, so the page says each running Spark stream restarts once on its next apply, as the pull request's changelog does. The runner of #1015 changed it as well, so a release that holds both restarts a stream once.

Checklist

  • python scripts/check_cli_docs.py (with forge-cli's live CHANGELOG.md), check_providers.py and gen_contract_reference.py --check pass
  • Site build, built-link check, offline link check and the linked anchors pass locally

Notes

  • A reviewer checked 46 sentences against the code, with 26 runs. Its four findings are fixed: the refusal does not leave a stream running (above), the word in the parentheses of the unreadable-object message is the name of the error when AWS did not answer, and the scheduler's policy and the kms:Decrypt it lacks (twice).
  • The code of #1014 on main equals the code the text was written and reviewed against (the pull request's head d291327f) for every file but the changelog and the runner, whose #1014 part is the same.

… (forge-cli #1014)

The two steps, the three refusals, the digest, and the permissions of the identity that runs fluid apply.
…ark-aws-wave15

# Conflicts:
#	docs/upgrading.md

This branch was previously deployed

1 inactive deployment
github-pages — 9c622662 Deployed Nov 19, 2026 by fas89 via deploy #419
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant