Repository navigation
fix: a Kafka outage no longer leaves finished pods behind or fails logins - #432
Merged
Merged
Conversation
…gins, and the unread publish_failed flag goes The flag was set on every failed publish and nothing read it. Events sent after their work is done now go through produce_or_log, which logs a Kafka error instead of raising. So the result processor no longer throws past the pod monitor's cleanup, and login, register and settings changes succeed while Kafka is down.
HardMax71
enabled auto-merge (squash)
October 21, 2026 22:48
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
|
Codecov Report❌ Patch coverage is
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
Fixes #298
A failed Kafka publish set
publish_failedon the stored event, and nothing ever read it. Since #428 the result is written before anything is published, so the execution no longer staysqueued. But with Kafka down when a pod finishes, the result processor's error still escaped to the pod monitor. The monitor then skipped deleting the pod and freeing the slot. Login and register answered 500 too, although nothing consumes those events.Changes
The flag and
mark_publish_failedgo. Prod has 0 flagged events among the 39 stored since 2026-10-12. Events sent after their work is done go through a newproduce_or_log: it still raises a MongoDB error, since the stored event is the record, but only logs a Kafka one. That covers auth and settings events, pod progress,pod_created, and the result processor's terminal event andresult_stored. A failed result write still publishesresult_failed. A publish failure after the write doesn't, since the result was stored. Executions, cancels and replays keep raising; their callers handle it.Type of Change
Test Plan
cd backend && uv run pytest tests/unit)cd frontend && npm run test)docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')cd frontend && npm run test:e2e)./deploy.sh openapi, then./deploy.sh types), never edited by handValidation: 542 unit tests, ruff, mypy strict, the orphan check and
mkdocs build --strictpass. The new outage test fails on main withKafkaErrorand passes here. Backend e2e is left to CI.Checklist
pre-commit run --all-filesand all checks passSummary by cubic
Fixes #298. A Kafka outage no longer leaves finished pods behind or fails logins, and the unread
publish_failedflag goes.When Kafka was down and a pod finished, the result processor's publish error escaped to the pod monitor, which then skipped deleting the pod and freeing its slot. Login and register answered 500 too, although nothing consumes those events. The
publish_failedflag set on every failed publish was never read by anything.produce_or_log, which stores an event likeproducebut only logs a Kafka error; a MongoDB error still raises, since the stored event is the record.pod_created, and the result processor's terminal event andresult_storedthrough it.result_failed; a publish failure after the write doesn't, since the result was stored.mark_publish_failedand thepublish_failedfields. Prod has 0 flagged events among the 39 stored since 2026-10-12.Written for commit ed07194. Summary will update on new commits.