Repository navigation
Description
Bug Description
Several saga failure paths leave the execution queued with an SSE stream that never closes, and two of them also keep its Redis queue slot forever. Once max_concurrent_executions slots (10 by default) have leaked, nothing runs at all.
A failed saga step goes to compensation and marks the saga FAILED. It publishes nothing and never calls release(). _execute_saga swallows the error, so the retry branch in try_schedule_from_queue doesn't run either:
Integr8sCode/backend/app/services/saga/saga_orchestrator.py
Lines 240 to 260 in 8cd314a
| else: | |
| self.logger.error("Saga step failed", step=step.name, saga_id=instance.saga_id) | |
| if self.config.enable_compensation: | |
| await self._compensate_saga(instance.saga_id, context) | |
| else: | |
| await self._fail_saga(instance.saga_id, "Step failed without compensation") | |
| return | |
| self.logger.info("Saga steps done, waiting for execution completion event", saga_id=instance.saga_id) | |
| except SagaConcurrencyError: | |
| self.logger.info("Saga modified concurrently, stopping execution", saga_id=instance.saga_id) | |
| except Exception as e: | |
| self.logger.error("Error executing saga", saga_id=instance.saga_id, exc_info=True) | |
| if self.config.enable_compensation: | |
| await self._compensate_saga(instance.saga_id, context) | |
| else: | |
| await self._fail_saga(instance.saga_id, str(e)) |
An easy trigger today: admin settings accept a max timeout up to 3600, but ValidateExecutionStep raises above 300. That mismatch is getting fixed separately, but any other step failure leaks the slot the same way:
Integr8sCode/backend/app/services/saga/execution_saga.py
Lines 30 to 34 in 8cd314a
| if len(event.script) > 1024 * 1024: | |
| raise ValueError("Script size exceeds limit") | |
| if event.timeout_seconds is not None and event.timeout_seconds > 300: | |
| raise ValueError("Timeout exceeds maximum allowed") |
check_timeouts writes TIMEOUT and nothing else:
Integr8sCode/backend/app/services/saga/saga_orchestrator.py
Lines 326 to 337 in 8cd314a
| async def check_timeouts(self) -> None: | |
| """Check for timed-out sagas and mark them. Single invocation.""" | |
| cutoff_time = datetime.now(UTC) - timedelta(seconds=self.config.timeout_seconds) | |
| timed_out = await self._repo.find_timed_out_sagas(cutoff_time) | |
| for instance in timed_out: | |
| self.logger.warning("Saga timed out", saga_id=instance.saga_id) | |
| await self._repo.save_saga( | |
| instance.saga_id, | |
| state=SagaState.TIMEOUT, | |
| error_message=f"Saga timed out after {self.config.timeout_seconds} seconds", | |
| completed_at=datetime.now(UTC), | |
| ) |
After three failed saga starts the orchestrator releases the slot but publishes no ExecutionFailedEvent (L131-L152).
The queue.active and queue.depth counters drift as well. release() records a release even when SREM removed nothing, and the three-retry path calls it twice. remove() and the lost-event branch in try_schedule never decrement depth:
Integr8sCode/backend/app/services/execution_queue.py
Lines 137 to 198 in 8cd314a
| if event_json is None: | |
| self._logger.error( | |
| "Event data expired/missing for scheduled execution — execution lost", | |
| execution_id=execution_id, | |
| ) | |
| await self._redis.srem(_ACTIVE_KEY, execution_id) | |
| self._metrics.record_event_data_lost() | |
| return None | |
| event_str = event_json if isinstance(event_json, str) else event_json.decode() | |
| event = ExecutionRequestedEvent.model_validate_json(event_str) | |
| wait_seconds = time.time() - enqueue_score | |
| self._metrics.record_wait_time(wait_seconds, str(priority)) | |
| self._metrics.record_schedule() | |
| self._logger.info( | |
| "Scheduled execution from queue", execution_id=execution_id, wait_seconds=round(wait_seconds, 3) | |
| ) | |
| return execution_id, event | |
| async def increment_retry_count(self, execution_id: str) -> int: | |
| """Atomically increment and return the retry count for an execution.""" | |
| key = _retry_key(execution_id) | |
| count: int = await self._redis.incr(key) | |
| await self._redis.expire(key, _EVENT_TTL) | |
| return count | |
| async def release(self, execution_id: str) -> None: | |
| pipe = self._redis.pipeline(transaction=True) | |
| pipe.srem(_ACTIVE_KEY, execution_id) | |
| pipe.delete(_event_key(execution_id)) | |
| await pipe.execute() | |
| self._metrics.record_release() | |
| self._logger.debug("Released execution from active set", execution_id=execution_id) | |
| async def update_priority(self, execution_id: str, new_priority: QueuePriority) -> bool: | |
| script = await self._get_update_priority_script() | |
| result = await script( | |
| keys=[_pending_key(new_priority), *_PENDING_KEYS], | |
| args=[execution_id], | |
| ) | |
| if not result: | |
| return False | |
| self._logger.info( | |
| "Updated execution priority", | |
| execution_id=execution_id, | |
| new_priority=new_priority, | |
| ) | |
| return True | |
| async def remove(self, execution_id: str) -> bool: | |
| pipe = self._redis.pipeline(transaction=True) | |
| for key in _PENDING_KEYS: | |
| pipe.zrem(key, execution_id) | |
| pipe.delete(_event_key(execution_id)) | |
| pipe.delete(_retry_key(execution_id)) | |
| pipe.srem(_ACTIVE_KEY, execution_id) | |
| results = await pipe.execute() | |
| removed = any(results[: len(_PENDING_KEYS)]) or bool(results[-1]) | |
| if removed: | |
| self._logger.info("Removed execution from queue", execution_id=execution_id) | |
| return removed |
#287 leaks allocations on these same paths, and #289 hides the drift since the orchestrator's metrics aren't exported.
Steps to Reproduce
Traced in the code, not reproduced end to end yet.
- As admin, set the max timeout to 600 in Admin > Settings.
- Run 10 scripts. Each saga fails at validate_execution and every execution stays
queued. - Set the timeout back to 30 and run one more. It stays
queuedtoo, andSCARD exec_queue:activereturns 10.
Expected Behavior
Every path that ends a saga releases the slot and publishes ExecutionFailedEvent (or a timeout), so the result processor writes a terminal status and SSE closes. The queue metrics move only when Redis actually changes, or get read from SCARD and ZCARD.
Component
Workers (Saga, DLQ, etc.)
Activity
HardMax71 commentedon Oct 21, 2026
Fixed by #428: the saga paths are gone, a pod that cannot be created fails its execution and frees both slots, and the queue numbers come from ZCARD and ZCOUNT. One rare case is left: an execution queued for more than 24 hours loses its event data in Redis and stays
queued.