[sentry-junior]

Work in progress. This draft scopes a fix and is not ready for review.

The worker child enforces processing_deadline_duration by raising ProcessingDeadlineExceeded (a BaseException) from a SIGALRM handler. That can interrupt library cleanup at any point. In Sentry it lands inside Django's Atomic.__exit__ after in_atomic_block is cleared but before COMMIT. psycopg2 then refuses set_autocommit(True) with set_session cannot be used inside a transaction, which replaces the deadline. The Django connection is left in manual transaction management with a transaction that never commits. The child keeps the connection and runs later tasks on it, so they fail with on_commit() cannot be used in manual transaction management or "You can't execute queries until the end of the 'atomic' block", or they lose their writes, until the child reaches its max task count.

Examples in Sentry:

• https://sentry.sentry.io/issues/7678768611/ shows the deadline → set_session chain.
• Afterwards one child failed every task until it was recycled about 40 minutes later: https://sentry.sentry.io/issues/7825042819/ (8k+ events).
• https://sentry.sentry.io/issues/7821208638/ is the on_commit symptom.

What this PR does:

• Masked deadlines are handled as deadlines. If the exception that reaches the worker has a ProcessingDeadlineExceeded anywhere in its __cause__/__context__ chain, it goes through the deadline path: same report, same processing_deadline_exceeded metric (with a new masked tag), and retry decided against the deadline. Today it falls into except Exception, skips the deadline metric, and may retry silently.
• New optional TaskCompletionHook (TaskbrokerApp(completion_hooks=[...])). after_task(activation, status, deadline_exceeded) runs after every task, success or failure, once timeout_alarm has exited. A hook can repair what the deadline broke without the deadline interrupting the repair. Hook errors are logged and counted, and never stop the child.

The client stays Django-agnostic. The Sentry side would register a hook that checks each initialized connection (Django autocommit, in_atomic_block, driver autocommit, driver transaction_status == IDLE) and closes and resets any unhealthy one. That check issues no SQL. It has to run after every task, not only when deadline_exceeded is set, because a deadline can also be swallowed outright: sentry_sdk's capture_internal_exceptions() catches BaseException.

Open questions before this is ready:

• Should the child exit after a deadline instead of, or as well as, the hook? handle_alarm's docstring already says it should. Recycling covers state a hook can't know about (client sessions, locks, contextvars), but on busy pools it would mean a process restart for every deadline. It could be a per-pool or per-task option.
• Is after_task(activation, status, deadline_exceeded) the right shape, or should hooks get a result object so fields can be added later without breaking them?
• handle_task_exception was moved out of the loop body unchanged so the deadline handling could be shared. The diff is larger than the behavior change.

via Rudra Patel.

--

View Session [Sentry]