Skip to content

fix(spark): a scheduled start decrypts its artifacts, and a refused start leaves the old run going - #1019

Merged
fas89 merged 2 commits into
mainfrom
claude/spark-caller-kms-d88a8f
Nov 19, 2026
Merged

fas89 merged 2 commits into
mainfrom
claude/spark-caller-kms-d88a8f

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Description

Two defects in the start-only-on-the-compiled-config gate (#1014).

  • A scheduled start failed AccessDenied with an SSE-KMS artifact bucket. A start reads
    job.json and the runner back from S3 before it runs (_adapters.checked_run), as the
    scheduler's identity for a scheduled build. The scheduler's policy
    (aws_identity.caller_document) granted s3:GetObject on them and no kms:Decrypt, so
    with a customer-managed key as the bucket's default encryption every scheduled start was
    refused. Before feat(spark): start a run only on the job config fluid apply compiled #1014 the scheduler never read S3. The policy now carries an S3Kms
    statement like the job role's: kms:Decrypt on any key, only through S3 in the provider's
    region (kms:ViaService) and only for those objects, or their bucket under an S3 Bucket Key
    (kms:EncryptionContext:aws:s3:arn). The largest one-build scheduler policy (an isolated
    EMR Serverless build) is 1,766 characters of IAM's 6,144, and the nine-build case feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013 pins
    is 2,879.
  • A refused start left a stream stopped. fluid apply cancelled a stream's run of another
    config, waited for it to stop, and only then started the new one, whose start ran the check.
    A deployed job.json another apply had changed was refused with the old run already gone,
    so the stream stayed down until the next apply. When a run of another config is to be cancelled,
    the reconcile now reads the two artifacts back first (Adapter.checked(), read once: the
    start uses the run it returned). A refusal is the build's refused outcome naming the run
    left running, nothing is cancelled or started, and fluid apply exits 1. A start with
    nothing to cancel keeps its order, so on EMR on EC2 the policy check and the ended-cluster
    check still share one DescribeCluster.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

Documentation

  • Docs PR linked
  • No docs needed
  • Docs TODO (forge_docs)
    • Operator permissions page: the scheduler's kms:Decrypt through S3 for an SSE-KMS artifact
      bucket.
    • Spark on AWS page: a start the deployed artifacts refuse leaves the old run going.

Testing

  • tests/iac/test_iac_aws_spark_infra.py: the scheduler's S3Kms statement (kms:Decrypt
    only, kms:ViaService on S3 in the provider's region, the encryption context naming the
    bucket and the two artifacts and nothing else); the policy stays under IAM's 6,144
    characters on every runtime; the nine-build policy's statement list gains S3Kms, whose
    context names exactly the artifacts.
  • tests/build_runners/spark/test_reconcile.py: a deployed job.json that is not the one this
    apply compiled, with an active run of the old config, cancels nothing and starts nothing,
    the outcome is refused naming the run left running, and reconcile_after_apply returns 1.
  • tests/build_runners/spark/test_adapters.py: the check before a cancel and the start read
    the two artifacts once.
  • Gates: ruff, the isolated dead-code gate, black 24.10.0, license headers, lint-imports, the
    product guardrail, detect-secrets on the changed files, mypy strict hotspots.

A start reads the job config and the runner back from S3 before it runs
(checked_run), as the scheduler's identity for a scheduled build. Its policy
granted s3:GetObject on them and no kms:Decrypt, so with an SSE-KMS artifact
bucket every scheduled start failed with AccessDenied.

The caller policy now carries an S3Kms statement like the job role's:
kms:Decrypt on any key, only through S3 in the provider's region
(kms:ViaService) and only for those objects, or their bucket under an S3
Bucket Key (kms:EncryptionContext:aws:s3:arn). The largest scheduler policy,
an isolated EMR Serverless build, is 1,766 characters of IAM's 6,144.
…d run

fluid apply cancelled a stream's run of another config, waited for it to stop,
and only then started the new one, whose start reads the deployed job config
and runner back and refuses them when another apply has changed them. A
refused start so left the stream stopped until the next apply.

When a run of another config is to be cancelled, the reconcile now reads the
two artifacts back first (Adapter.checked, read once: the start then uses the
run it returned). A refusal is the build's refused outcome naming the run left
running, nothing is cancelled or started, and fluid apply exits 1. A start
with nothing to cancel keeps its order (on EMR on EC2 the policy check and the
ended-cluster check still share one DescribeCluster).
@github-actions github-actions Bot added the tests Test coverage or test infrastructure changes label Nov 19, 2026
@fas89
fas89 merged commit 73ea0e0 into main Nov 19, 2026
61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests Test coverage or test infrastructure changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant