Repository navigation
Description
Bug Description
Cancelling an execution doesn't stop it. The pod keeps running, the execution never becomes cancelled, and the SSE stream stays open.
cancel_execution only publishes an ExecutionCancelledEvent:
Integr8sCode/backend/app/services/execution_service.py
Lines 227 to 235 in 8cd314a
| metadata = self._create_event_metadata(user_id=user_id) | |
| event = ExecutionCancelledEvent( | |
| execution_id=execution_id, | |
| aggregate_id=execution_id, | |
| cancelled_by=user_id, | |
| metadata=metadata, | |
| ) | |
| await self.producer.produce(event_to_produce=event, key=execution_id) |
The saga orchestrator removes it from the queue, marks the saga CANCELLED and releases the slot. It never compensates, so no DeletePodCommand goes out:
Integr8sCode/backend/app/services/saga/saga_orchestrator.py
Lines 88 to 117 in 8cd314a
| async def handle_execution_cancelled(self, event: DomainEvent) -> None: | |
| """Handle EXECUTION_CANCELLED — remove from queue and mark saga as cancelled.""" | |
| if not isinstance(event, ExecutionCancelledEvent): | |
| raise TypeError(f"Expected ExecutionCancelledEvent, got {type(event).__name__}") | |
| await self._queue.remove(event.execution_id) | |
| await self._resolve_completion(event.execution_id, SagaState.CANCELLED, f"cancelled by {event.cancelled_by}") | |
| async def _resolve_completion(self, execution_id: str, state: SagaState, error_message: str | None = None) -> None: | |
| """Look up the active saga for an execution and transition it to a terminal state. | |
| Always releases the queue slot and attempts to schedule the next pending | |
| execution, even when no saga is found or the saga is already terminal. | |
| ``release()`` is idempotent (Redis SREM on a missing member is a no-op). | |
| """ | |
| saga = await self._repo.get_saga_by_execution_and_name(execution_id, _SAGA_NAME) | |
| if not saga: | |
| self.logger.debug("No execution_saga found for execution", execution_id=execution_id) | |
| elif saga.state.is_terminal: | |
| self.logger.debug("Saga already in terminal state", saga_id=saga.saga_id, state=saga.state) | |
| else: | |
| self.logger.info("Marking saga terminal state", saga_id=saga.saga_id, state=state) | |
| await self._repo.save_saga( | |
| saga.saga_id, | |
| state=state, | |
| error_message=error_message, | |
| completed_at=datetime.now(UTC), | |
| ) | |
| await self._queue.release(execution_id) | |
| await self.try_schedule_from_queue() |
The saga API's cancel_saga is the one that compensates (L373-L387). The comment in topics.py says cancel deletes the pod through that compensation, which this path never reaches:
Integr8sCode/backend/app/infrastructure/kafka/topics.py
Lines 53 to 56 in 8cd314a
| # Events that signal the executor pod has reached a terminal state. | |
| # The pod monitor uses this to decide when to delete the pod. | |
| # Note: EXECUTION_CANCELLED is excluded — cancellation triggers pod deletion | |
| # via the saga's DeletePodCommandEvent compensation, not the monitor. |
No result-processor subscriber handles execution_cancelled, so Mongo keeps queued. If the pod was already running, it finishes later and the result processor writes whatever it ended with, after its slot went to the next execution. So more pods can run than max_concurrent_executions allows.
EXECUTION_CANCELLED is also missing from the set that closes the SSE stream:
Integr8sCode/backend/app/infrastructure/kafka/topics.py
Lines 68 to 75 in 8cd314a
| EXECUTION_PIPELINE_TERMINAL_EVENT_TYPES: frozenset[EventType] = frozenset( | |
| { | |
| EventType.RESULT_STORED, | |
| EventType.RESULT_FAILED, | |
| EventType.EXECUTION_FAILED, | |
| EventType.EXECUTION_TIMEOUT, | |
| } | |
| ) |
Cancel skips the allocation release too, which is the leak from #287.
There's no cancel button in the frontend, so only API callers hit this.
Steps to Reproduce
Found by reading the code, not reproduced end to end yet.
- Start a script that sleeps a few seconds.
POST /api/v1/executions/{id}/cancelwhile it runs.- The pod keeps running and
GET /api/v1/executions/{id}/resultsaysqueued, thencompleted. A second cancel returnscancellation_requestedagain instead ofalready_cancelled.
Expected Behavior
Cancel deletes the pod, releases the allocation and writes cancelled to the execution, and the cancelled event closes the SSE stream. handle_execution_cancelled could reuse the cancel_saga compensation path, plus a result-processor handler for execution_cancelled that writes the terminal status.
Component
Workers (Saga, DLQ, etc.)
Activity
HardMax71 commentedon Oct 21, 2026
Fixed by #428: cancel stores
cancelled, deletes the pod and frees the slot, andresult_storedcloses the SSE stream.