Repository navigation
docs(spark): a stream bounds its batches, a topic dead letter is keyed by its source, and the runner pins failOnDataLoss (forge-cli #1015) - #267
Merged
Conversation
…d by its source, and the runner pins failOnDataLoss (forge-cli #1015) Bounding a batch, the topic's key and header, and the Spot checklist item. The runner changed, so each running stream restarts once.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents forge-cli #1015, on
main(931274a3) after the 0.33.0 cut and in no release: a Spark stream can bound what one micro-batch reads, the runner pinsfailOnDataLoss=trueon every Kafka source, and a dead letter sent to a Kafka topic is keyed by where it came from.advanced/spark-iceberg-aws.md: themaxOffsetsPerTriggerrow of "Stream settings" and a new section, "Bounding a batch" (the accepted range, whatfluid validateprints for each refused value, the options the runner gives Spark's Kafka source, the restart on a change, the refusals of 0.32.0 and of managed Flink); "The topic" (the new key<topic>-<partition>@<offset>, the headerfluid.dead-letter.raw-key, repeats after a failure, and what a compacted topic does with them); "Masked builds"; and item 5 of the Spot checklist (failOnDataLoss).upgrading.md: four bullets under "Not in a release yet".Related CLI PRs
Areas Updated
Decisions
*On forge-climain, not yet in a release ([forge-cli #1015](...)).*: the pull request merged after the 0.33.0 cut. The pin stays at 0.32.0, so the generated contract reference does not list the new key until the pin moves.fluid validateof the example contract with each value (5000and the maximum pass;0,true,"5",5.0and one above the maximum are refused), the Spark runner's own option function on the generatedjob.json, the pinned 0.32.0 CLI, and the config hash of an unchanged contract before and after the pull request. No Kafka or Spark ran here, so the split of the limit across partitions is worded as forge-cli's schema words it, and the key and the headers of a topic dead letter are read from the runner's code and its tests.fluid.dead-letter.raw-keyis null for a masked build and that its dead letters have no value. Kafka treats a record with a key and a null value as a delete marker on a compacted topic, so the page tells the reader to give a masked build's dead letters a topic that deletes by retention. That consequence is mine, from the code and Kafka's documentation, and not from the pull request.89e6fabc00be7408before,620697bad7baf2abafter), because the runner's sha256 is part of it.Checklist
python scripts/check_cli_docs.py(with forge-cli's liveCHANGELOG.md),check_providers.pyandgen_contract_reference.py --checkpassNotes
groupIdPrefix, the validate block dropped itsValidation completedline, the Spark error text is part of the message and not its start, and a masked build is not the only source of a dead letter with no value.mainequals the code the text was written and reviewed against (the pull request's head76a4095b) for every file but the changelog.main(docs(spark): removing a Spark build on AWS is not a data loss (forge-cli #1007) #266) is the base.