Skip to content

[Bug]: execution queue slots leak and executions stay queued forever #297

Closed

Description

@HardMax71

Bug Description

Several saga failure paths leave the execution queued with an SSE stream that never closes, and two of them also keep its Redis queue slot forever. Once max_concurrent_executions slots (10 by default) have leaked, nothing runs at all.

A failed saga step goes to compensation and marks the saga FAILED. It publishes nothing and never calls release(). _execute_saga swallows the error, so the retry branch in try_schedule_from_queue doesn't run either:

else:
self.logger.error("Saga step failed", step=step.name, saga_id=instance.saga_id)
if self.config.enable_compensation:
await self._compensate_saga(instance.saga_id, context)
else:
await self._fail_saga(instance.saga_id, "Step failed without compensation")
return
self.logger.info("Saga steps done, waiting for execution completion event", saga_id=instance.saga_id)
except SagaConcurrencyError:
self.logger.info("Saga modified concurrently, stopping execution", saga_id=instance.saga_id)
except Exception as e:
self.logger.error("Error executing saga", saga_id=instance.saga_id, exc_info=True)
if self.config.enable_compensation:
await self._compensate_saga(instance.saga_id, context)
else:
await self._fail_saga(instance.saga_id, str(e))

An easy trigger today: admin settings accept a max timeout up to 3600, but ValidateExecutionStep raises above 300. That mismatch is getting fixed separately, but any other step failure leaks the slot the same way:

if len(event.script) > 1024 * 1024:
raise ValueError("Script size exceeds limit")
if event.timeout_seconds is not None and event.timeout_seconds > 300:
raise ValueError("Timeout exceeds maximum allowed")

check_timeouts writes TIMEOUT and nothing else:

async def check_timeouts(self) -> None:
"""Check for timed-out sagas and mark them. Single invocation."""
cutoff_time = datetime.now(UTC) - timedelta(seconds=self.config.timeout_seconds)
timed_out = await self._repo.find_timed_out_sagas(cutoff_time)
for instance in timed_out:
self.logger.warning("Saga timed out", saga_id=instance.saga_id)
await self._repo.save_saga(
instance.saga_id,
state=SagaState.TIMEOUT,
error_message=f"Saga timed out after {self.config.timeout_seconds} seconds",
completed_at=datetime.now(UTC),
)

After three failed saga starts the orchestrator releases the slot but publishes no ExecutionFailedEvent (L131-L152).

The queue.active and queue.depth counters drift as well. release() records a release even when SREM removed nothing, and the three-retry path calls it twice. remove() and the lost-event branch in try_schedule never decrement depth:

if event_json is None:
self._logger.error(
"Event data expired/missing for scheduled execution — execution lost",
execution_id=execution_id,
)
await self._redis.srem(_ACTIVE_KEY, execution_id)
self._metrics.record_event_data_lost()
return None
event_str = event_json if isinstance(event_json, str) else event_json.decode()
event = ExecutionRequestedEvent.model_validate_json(event_str)
wait_seconds = time.time() - enqueue_score
self._metrics.record_wait_time(wait_seconds, str(priority))
self._metrics.record_schedule()
self._logger.info(
"Scheduled execution from queue", execution_id=execution_id, wait_seconds=round(wait_seconds, 3)
)
return execution_id, event
async def increment_retry_count(self, execution_id: str) -> int:
"""Atomically increment and return the retry count for an execution."""
key = _retry_key(execution_id)
count: int = await self._redis.incr(key)
await self._redis.expire(key, _EVENT_TTL)
return count
async def release(self, execution_id: str) -> None:
pipe = self._redis.pipeline(transaction=True)
pipe.srem(_ACTIVE_KEY, execution_id)
pipe.delete(_event_key(execution_id))
await pipe.execute()
self._metrics.record_release()
self._logger.debug("Released execution from active set", execution_id=execution_id)
async def update_priority(self, execution_id: str, new_priority: QueuePriority) -> bool:
script = await self._get_update_priority_script()
result = await script(
keys=[_pending_key(new_priority), *_PENDING_KEYS],
args=[execution_id],
)
if not result:
return False
self._logger.info(
"Updated execution priority",
execution_id=execution_id,
new_priority=new_priority,
)
return True
async def remove(self, execution_id: str) -> bool:
pipe = self._redis.pipeline(transaction=True)
for key in _PENDING_KEYS:
pipe.zrem(key, execution_id)
pipe.delete(_event_key(execution_id))
pipe.delete(_retry_key(execution_id))
pipe.srem(_ACTIVE_KEY, execution_id)
results = await pipe.execute()
removed = any(results[: len(_PENDING_KEYS)]) or bool(results[-1])
if removed:
self._logger.info("Removed execution from queue", execution_id=execution_id)
return removed

#287 leaks allocations on these same paths, and #289 hides the drift since the orchestrator's metrics aren't exported.

Steps to Reproduce

Traced in the code, not reproduced end to end yet.

  1. As admin, set the max timeout to 600 in Admin > Settings.
  2. Run 10 scripts. Each saga fails at validate_execution and every execution stays queued.
  3. Set the timeout back to 30 and run one more. It stays queued too, and SCARD exec_queue:active returns 10.

Expected Behavior

Every path that ends a saga releases the slot and publishes ExecutionFailedEvent (or a timeout), so the result processor writes a terminal status and SSE closes. The queue metrics move only when Redis actually changes, or get read from SCARD and ZCARD.

Component

Workers (Saga, DLQ, etc.)

Activity

  1. added
    bugSomething isn't working
    area: executionQueue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes
    on Oct 10, 2026
  2. HardMax71 commented on Oct 21, 2026

    @HardMax71
    OwnerAuthor

    Fixed by #428: the saga paths are gone, a pod that cannot be created fails its execution and frees both slots, and the queue numbers come from ZCARD and ZCOUNT. One rare case is left: an execution queued for more than 24 hours loses its event data in Redis and stays queued.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: executionQueue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimesbugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions