[shashankram]

Description

Requires #3635 to merge first. This branch builds on that PR; its separate follow-up commit is d30cb0bd.

Canceling message delivery during actor restoration can leave the actor RESUMING without a turn, holding worker capacity. The existing Session lifecycle worker now retries resume, then suspends the actor so a later message can resume normally.

• Retain the same cleanup claim through resume, suspension, and snapshot recording, preventing new turns from racing cleanup.
• Queue bounded attempts with exponential backoff capped at 15 seconds, freeing workers during delays. Database retries do not repeat completed runtime operations.
• Preserve uncertain claims, coordinate crash recovery, and allow explicit force deletion when cleanup cannot safely finish.

Interrupted-resume cleanup flow

sequenceDiagram
    participant D as Session store
    participant W as Lifecycle worker
    participant S as Substrate
    W->>D: Claim expired dispatch with no active turn
    W->>S: Read actor identity and state
    alt Actor is RESUMING
        W->>S: Resume actor
        S-->>W: Confirm RUNNING
    end
    W->>S: Suspend actor and save durable snapshot
    S-->>W: Confirm suspension and snapshot
    W->>D: Record snapshot and clear dispatch and claim
    Note over W,S: Retryable failures requeue the same owner with backoff

Loading

Testing

• Added coverage for interrupted resume, lost replies, retained ownership, competing lifecycle work, worker fairness, and shutdown.
• Added an E2E regression that cancels during RESUMING, verifies suspension with no tasks or worker assignment, then sends a successful subsequent message.

Change Type

/kind fix

Changelog

Actors left resuming after canceled message delivery now recover and suspend automatically.