Skip to content

docs(spark): a stream bounds its batches, a topic dead letter is keyed by its source, and the runner pins failOnDataLoss (forge-cli #1015) - #267

Merged
fas89 merged 1 commit into
mainfrom
claude/docs-spark-aws-wave14
Nov 19, 2026
Merged

fas89 merged 1 commit into
mainfrom
claude/docs-spark-aws-wave14

Conversation

@fas89

@fas89 fas89 commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Documents forge-cli #1015, on main (931274a3) after the 0.33.0 cut and in no release: a Spark stream can bound what one micro-batch reads, the runner pins failOnDataLoss=true on every Kafka source, and a dead letter sent to a Kafka topic is keyed by where it came from.

  • advanced/spark-iceberg-aws.md: the maxOffsetsPerTrigger row of "Stream settings" and a new section, "Bounding a batch" (the accepted range, what fluid validate prints for each refused value, the options the runner gives Spark's Kafka source, the restart on a change, the refusals of 0.32.0 and of managed Flink); "The topic" (the new key <topic>-<partition>@<offset>, the header fluid.dead-letter.raw-key, repeats after a failure, and what a compacted topic does with them); "Masked builds"; and item 5 of the Spot checklist (failOnDataLoss).
  • upgrading.md: four bullets under "Not in a release yet".

Related CLI PRs

Areas Updated

  • Getting Started
  • CLI Reference
  • Provider Docs
  • Walkthroughs / Examples
  • Contributing / Community Docs
  • Other: advanced guide, upgrading

Decisions

  • The markers read *On forge-cli main, not yet in a release ([forge-cli #1015](...)).*: the pull request merged after the 0.33.0 cut. The pin stays at 0.32.0, so the generated contract reference does not list the new key until the pin moves.
  • The outputs are real: fluid validate of the example contract with each value (5000 and the maximum pass; 0, true, "5", 5.0 and one above the maximum are refused), the Spark runner's own option function on the generated job.json, the pinned 0.32.0 CLI, and the config hash of an unchanged contract before and after the pull request. No Kafka or Spark ran here, so the split of the limit across partitions is worded as forge-cli's schema words it, and the key and the headers of a topic dead letter are read from the runner's code and its tests.
  • The text says that fluid.dead-letter.raw-key is null for a masked build and that its dead letters have no value. Kafka treats a record with a key and a null value as a delete marker on a compacted topic, so the page tells the reader to give a masked build's dead letters a topic that deletes by retention. That consequence is mine, from the code and Kafka's documentation, and not from the pull request.
  • The runner changed, so the page says each running Spark stream restarts once on its next apply. The pull request's changelog says only that a stream without the limit keeps its job config; the hash of an unchanged contract is another one (checked: 89e6fabc00be7408 before, 620697bad7baf2ab after), because the runner's sha256 is part of it.

Checklist

  • python scripts/check_cli_docs.py (with forge-cli's live CHANGELOG.md), check_providers.py and gen_contract_reference.py --check pass
  • Site build, built-link check, offline link check and the linked anchors pass locally

Notes

  • A reviewer checked about 62 sentences against the code, with about 32 runs and fetches (forge-cli's code, the pinned CLI, Spark 3.5.1's Kafka source and Kafka's design text). Its four findings are fixed: the options output left out groupIdPrefix, the validate block dropped its Validation completed line, the Spark error text is part of the message and not its start, and a masked build is not the only source of a dead letter with no value.
  • The code of #1015 on main equals the code the text was written and reviewed against (the pull request's head 76a4095b) for every file but the changelog.
  • Docs main (docs(spark): removing a Spark build on AWS is not a data loss (forge-cli #1007) #266) is the base.

…d by its source, and the runner pins failOnDataLoss (forge-cli #1015)

Bounding a batch, the topic's key and header, and the Spot checklist item. The runner changed, so each running stream restarts once.
@fas89
fas89 merged commit f836eb6 into main Nov 19, 2026
9 checks passed

This branch was previously deployed

1 inactive deployment
github-pages — 767cfce9 Deployed Nov 19, 2026 by fas89 via deploy #417
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant