Skip to content

[Bug]: cancelling an execution doesn't stop it #296

Closed

Description

@HardMax71

Bug Description

Cancelling an execution doesn't stop it. The pod keeps running, the execution never becomes cancelled, and the SSE stream stays open.

cancel_execution only publishes an ExecutionCancelledEvent:

metadata = self._create_event_metadata(user_id=user_id)
event = ExecutionCancelledEvent(
execution_id=execution_id,
aggregate_id=execution_id,
cancelled_by=user_id,
metadata=metadata,
)
await self.producer.produce(event_to_produce=event, key=execution_id)

The saga orchestrator removes it from the queue, marks the saga CANCELLED and releases the slot. It never compensates, so no DeletePodCommand goes out:

async def handle_execution_cancelled(self, event: DomainEvent) -> None:
"""Handle EXECUTION_CANCELLED — remove from queue and mark saga as cancelled."""
if not isinstance(event, ExecutionCancelledEvent):
raise TypeError(f"Expected ExecutionCancelledEvent, got {type(event).__name__}")
await self._queue.remove(event.execution_id)
await self._resolve_completion(event.execution_id, SagaState.CANCELLED, f"cancelled by {event.cancelled_by}")
async def _resolve_completion(self, execution_id: str, state: SagaState, error_message: str | None = None) -> None:
"""Look up the active saga for an execution and transition it to a terminal state.
Always releases the queue slot and attempts to schedule the next pending
execution, even when no saga is found or the saga is already terminal.
``release()`` is idempotent (Redis SREM on a missing member is a no-op).
"""
saga = await self._repo.get_saga_by_execution_and_name(execution_id, _SAGA_NAME)
if not saga:
self.logger.debug("No execution_saga found for execution", execution_id=execution_id)
elif saga.state.is_terminal:
self.logger.debug("Saga already in terminal state", saga_id=saga.saga_id, state=saga.state)
else:
self.logger.info("Marking saga terminal state", saga_id=saga.saga_id, state=state)
await self._repo.save_saga(
saga.saga_id,
state=state,
error_message=error_message,
completed_at=datetime.now(UTC),
)
await self._queue.release(execution_id)
await self.try_schedule_from_queue()

The saga API's cancel_saga is the one that compensates (L373-L387). The comment in topics.py says cancel deletes the pod through that compensation, which this path never reaches:

# Events that signal the executor pod has reached a terminal state.
# The pod monitor uses this to decide when to delete the pod.
# Note: EXECUTION_CANCELLED is excluded — cancellation triggers pod deletion
# via the saga's DeletePodCommandEvent compensation, not the monitor.

No result-processor subscriber handles execution_cancelled, so Mongo keeps queued. If the pod was already running, it finishes later and the result processor writes whatever it ended with, after its slot went to the next execution. So more pods can run than max_concurrent_executions allows.

EXECUTION_CANCELLED is also missing from the set that closes the SSE stream:

EXECUTION_PIPELINE_TERMINAL_EVENT_TYPES: frozenset[EventType] = frozenset(
{
EventType.RESULT_STORED,
EventType.RESULT_FAILED,
EventType.EXECUTION_FAILED,
EventType.EXECUTION_TIMEOUT,
}
)

Cancel skips the allocation release too, which is the leak from #287.

There's no cancel button in the frontend, so only API callers hit this.

Steps to Reproduce

Found by reading the code, not reproduced end to end yet.

  1. Start a script that sleeps a few seconds.
  2. POST /api/v1/executions/{id}/cancel while it runs.
  3. The pod keeps running and GET /api/v1/executions/{id}/result says queued, then completed. A second cancel returns cancellation_requested again instead of already_cancelled.

Expected Behavior

Cancel deletes the pod, releases the allocation and writes cancelled to the execution, and the cancelled event closes the SSE stream. handle_execution_cancelled could reuse the cancel_saga compensation path, plus a result-processor handler for execution_cancelled that writes the terminal status.

Component

Workers (Saga, DLQ, etc.)

Activity

  1. added
    bugSomething isn't working
    area: executionQueue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimes
    on Oct 10, 2026
  2. HardMax71 commented on Oct 21, 2026

    @HardMax71
    OwnerAuthor

    Fixed by #428: cancel stores cancelled, deletes the pod and frees the slot, and result_stored closes the SSE stream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: executionQueue, scheduler, k8s worker, pod monitor, result processor, executor pods, runtimesbugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions