Skip to content

fix: a Kafka outage no longer leaves finished pods behind or fails logins - #432

Merged
HardMax71 merged 1 commit into
mainfrom
max/298-publish-failures
Oct 21, 2026
Merged

HardMax71 merged 1 commit into
mainfrom
max/298-publish-failures

Conversation

@HardMax71

@HardMax71 HardMax71 commented Oct 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes #298

A failed Kafka publish set publish_failed on the stored event, and nothing ever read it. Since #428 the result is written before anything is published, so the execution no longer stays queued. But with Kafka down when a pod finishes, the result processor's error still escaped to the pod monitor. The monitor then skipped deleting the pod and freeing the slot. Login and register answered 500 too, although nothing consumes those events.

Changes

The flag and mark_publish_failed go. Prod has 0 flagged events among the 39 stored since 2026-10-12. Events sent after their work is done go through a new produce_or_log: it still raises a MongoDB error, since the stored event is the record, but only logs a Kafka one. That covers auth and settings events, pod progress, pod_created, and the result processor's terminal event and result_stored. A failed result write still publishes result_failed. A publish failure after the write doesn't, since the result was stored. Executions, cancels and replays keep raising; their callers handle it.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)
  • CI/CD or tooling changes

Test Plan

  • Backend unit tests pass (cd backend && uv run pytest tests/unit)
  • Frontend unit tests pass (cd frontend && npm run test)
  • Backend E2E tests pass (docker compose exec -T backend sh -c 'uv sync --group test --no-dev --frozen --no-install-project --no-build && pytest tests/e2e')
  • Frontend E2E tests pass (cd frontend && npm run test:e2e)
  • API spec and client regenerated if routes or schemas changed (./deploy.sh openapi, then ./deploy.sh types), never edited by hand
  • Manual testing performed

Validation: 542 unit tests, ruff, mypy strict, the orphan check and mkdocs build --strict pass. The new outage test fails on main with KafkaError and passes here. Backend e2e is left to CI.

Checklist

  • My code follows the project's style guidelines
  • I have run pre-commit run --all-files and all checks pass
  • I have added tests that prove my fix/feature works
  • I have updated documentation if needed
  • My changes don't introduce new warnings

Summary by cubic

Fixes #298. A Kafka outage no longer leaves finished pods behind or fails logins, and the unread publish_failed flag goes.

When Kafka was down and a pod finished, the result processor's publish error escaped to the pod monitor, which then skipped deleting the pod and freeing its slot. Login and register answered 500 too, although nothing consumes those events. The publish_failed flag set on every failed publish was never read by anything.

  • Adds produce_or_log, which stores an event like produce but only logs a Kafka error; a MongoDB error still raises, since the stored event is the record.
  • Routes auth and settings events, pod progress, pod_created, and the result processor's terminal event and result_stored through it.
  • A failed result write still publishes result_failed; a publish failure after the write doesn't, since the result was stored.
  • Executions, cancels, and replays keep raising; their callers handle the error.
  • Removes mark_publish_failed and the publish_failed fields. Prod has 0 flagged events among the 39 stored since 2026-10-12.

Written for commit ed07194. Summary will update on new commits.

View guided diff Turn on auto-fix

…gins, and the unread publish_failed flag goes

The flag was set on every failed publish and nothing read it. Events sent after their work is done now go through produce_or_log, which logs a Kafka error instead of raising. So the result processor no longer throws past the pod monitor's cleanup, and login, register and settings changes succeed while Kafka is down.
@github-actions github-actions Bot added documentation Improvements or additions to documentation area: api FastAPI routes, auth, rate limiting, settings area: execution Queue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes area: events Kafka consumers and middlewares, producer, idempotency, DLQ, replay labels Oct 21, 2026
@HardMax71
HardMax71 enabled auto-merge (squash) October 21, 2026 22:48
@coderabbitai

coderabbitai Bot commented Oct 21, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5ed3f165-1408-4c36-a244-a34353bbbdc0
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

@codecov-commenter

codecov-commenter commented Oct 21, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.93939% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
backend/app/services/k8s_worker/worker.py 0.00% 1 Missing ⚠️
backend/app/services/pod_monitor/monitor.py 50.00% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Flag Coverage Δ
backend-e2e 82.83% <72.72%> (-0.11%) ⬇️
backend-unit 75.93% <84.84%> (+0.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
backend/app/core/providers.py 97.02% <100.00%> (ø)
backend/app/db/repositories/event_repository.py 73.62% <100.00%> (+2.79%) ⬆️
backend/app/events/core/producer.py 100.00% <100.00%> (ø)
backend/app/services/auth_service.py 95.49% <100.00%> (ø)
backend/app/services/result_processor/processor.py 92.77% <100.00%> (+6.52%) ⬆️
backend/app/services/user_settings_service.py 97.61% <100.00%> (ø)
backend/app/services/k8s_worker/worker.py 40.60% <0.00%> (ø)
backend/app/services/pod_monitor/monitor.py 94.07% <50.00%> (ø)
Components Coverage Δ
Backend 93.24% <93.93%> (+0.13%) ⬆️
Frontend 92.11% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HardMax71
HardMax71 merged commit 544981b into main Oct 21, 2026
22 checks passed
@HardMax71
HardMax71 deleted the max/298-publish-failures branch October 21, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: api FastAPI routes, auth, rate limiting, settings area: events Kafka consumers and middlewares, producer, idempotency, DLQ, replay area: execution Queue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: events flagged publish_failed are never re-sent

2 participants