Repository navigation
fix(spark): a scheduled start decrypts its artifacts, and a refused start leaves the old run going - #1019
Merged
Conversation
A start reads the job config and the runner back from S3 before it runs (checked_run), as the scheduler's identity for a scheduled build. Its policy granted s3:GetObject on them and no kms:Decrypt, so with an SSE-KMS artifact bucket every scheduled start failed with AccessDenied. The caller policy now carries an S3Kms statement like the job role's: kms:Decrypt on any key, only through S3 in the provider's region (kms:ViaService) and only for those objects, or their bucket under an S3 Bucket Key (kms:EncryptionContext:aws:s3:arn). The largest scheduler policy, an isolated EMR Serverless build, is 1,766 characters of IAM's 6,144.
…d run fluid apply cancelled a stream's run of another config, waited for it to stop, and only then started the new one, whose start reads the deployed job config and runner back and refuses them when another apply has changed them. A refused start so left the stream stopped until the next apply. When a run of another config is to be cancelled, the reconcile now reads the two artifacts back first (Adapter.checked, read once: the start then uses the run it returned). A refusal is the build's refused outcome naming the run left running, nothing is cancelled or started, and fluid apply exits 1. A start with nothing to cancel keeps its order (on EMR on EC2 the policy check and the ended-cluster check still share one DescribeCluster).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two defects in the start-only-on-the-compiled-config gate (#1014).
job.jsonand the runner back from S3 before it runs (_adapters.checked_run), as thescheduler's identity for a scheduled build. The scheduler's policy
(
aws_identity.caller_document) granteds3:GetObjecton them and nokms:Decrypt, sowith a customer-managed key as the bucket's default encryption every scheduled start was
refused. Before feat(spark): start a run only on the job config fluid apply compiled #1014 the scheduler never read S3. The policy now carries an
S3Kmsstatement like the job role's:
kms:Decrypton any key, only through S3 in the provider'sregion (
kms:ViaService) and only for those objects, or their bucket under an S3 Bucket Key(
kms:EncryptionContext:aws:s3:arn). The largest one-build scheduler policy (an isolatedEMR Serverless build) is 1,766 characters of IAM's 6,144, and the nine-build case feat(spark): a dead stream restarts within a budget; fluid apply --mode reconcile #1013 pins
is 2,879.
fluid applycancelled a stream's run of anotherconfig, waited for it to stop, and only then started the new one, whose start ran the check.
A deployed
job.jsonanother apply had changed was refused with the old run already gone,so the stream stayed down until the next apply. When a run of another config is to be cancelled,
the reconcile now reads the two artifacts back first (
Adapter.checked(), read once: thestart uses the run it returned). A refusal is the build's
refusedoutcome naming the runleft running, nothing is cancelled or started, and
fluid applyexits 1. A start withnothing to cancel keeps its order, so on EMR on EC2 the policy check and the ended-cluster
check still share one DescribeCluster.
Type of Change
Documentation
kms:Decryptthrough S3 for an SSE-KMS artifactbucket.
Testing
tests/iac/test_iac_aws_spark_infra.py: the scheduler'sS3Kmsstatement (kms:Decryptonly,
kms:ViaServiceon S3 in the provider's region, the encryption context naming thebucket and the two artifacts and nothing else); the policy stays under IAM's 6,144
characters on every runtime; the nine-build policy's statement list gains
S3Kms, whosecontext names exactly the artifacts.
tests/build_runners/spark/test_reconcile.py: a deployedjob.jsonthat is not the one thisapply compiled, with an active run of the old config, cancels nothing and starts nothing,
the outcome is
refusednaming the run left running, andreconcile_after_applyreturns 1.tests/build_runners/spark/test_adapters.py: the check before a cancel and the start readthe two artifacts once.
product guardrail, detect-secrets on the changed files, mypy strict hotspots.