Adds retry support to the Amazon.Lambda.DurableExecution#2363
Draft
GarrettBeatty wants to merge 5 commits into
Draft
Adds retry support to the Amazon.Lambda.DurableExecution#2363GarrettBeatty wants to merge 5 commits into
GarrettBeatty wants to merge 5 commits into
Conversation
stack-info: PR: #2361, branch: GarrettBeatty/stack/3
GarrettBeatty
added a commit
that referenced
this pull request
May 12, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
711bf82 to
4f05fa9
Compare
This was referenced May 12, 2026
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 12, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
4f05fa9 to
54d18f9
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 12, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
54d18f9 to
599445f
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 12, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
599445f to
e7a85e4
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 12, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
e7a85e4 to
8f23ebb
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 13, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
8f23ebb to
e39e68e
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 13, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
e39e68e to
52055d3
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
81b9144 to
531cbbe
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 13, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
531cbbe to
31ea7e8
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
31ea7e8 to
ef44439
Compare
GarrettBeatty
added a commit
that referenced
this pull request
May 13, 2026
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
ef44439 to
6bc97f2
Compare
GarrettBeatty
added a commit
that referenced
this pull request
May 13, 2026
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 13, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
6bc97f2 to
85eae3e
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
GarrettBeatty
changed the base branch from
GarrettBeatty/stack/2
to
feature/durablefunction
GarrettBeatty
added a commit
that referenced
this pull request
May 14, 2026
GarrettBeatty
force-pushed
the
GarrettBeatty/stack/3
branch
from
85eae3e to
0a32c0d
Compare
GarrettBeatty
changed the base branch from
feature/durablefunction
to
GarrettBeatty/stack/2
There was a problem hiding this comment.
Pull request overview
Builds on PR #2360 to add retry support to the Amazon.Lambda.DurableExecution SDK. Failed steps can now be retried with configurable backoff and jitter via service-mediated retries (the SDK checkpoints a RETRY operation and suspends the Lambda so the user is not billed during backoff). Adds at-most-once semantics for non-idempotent steps via a synchronously-flushed START checkpoint that allows crash detection on replay.
Changes:
- New public retry API:
IRetryStrategy,RetryDecision,RetryStrategyfactories (Default/Transient/None/Exponential/FromDelegate),JitterStrategy,StepSemantics, andStepConfig.RetryStrategy/StepConfig.Semantics. StepOperationaddsPENDING(retry-timer) andSTARTED(AtMostOnce crash-recovery) replay arms, aHandleStepFailureAsyncdecision tree, and START-checkpoint emission (sync for AtMostOnce, fire-and-forget for AtLeastOnce).- 21 new unit tests plus integration-test updates asserting
StepStartedevents and richer history logging.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
Config/IRetryStrategy.cs |
New strategy interface + RetryDecision struct |
Config/RetryStrategy.cs |
ExponentialRetryStrategy, DelegateRetryStrategy, JitterStrategy, StepSemantics, factories |
Config/StepConfig.cs |
Adds RetryStrategy and Semantics properties |
Internal/StepOperation.cs |
PENDING/STARTED replay arms, retry decision tree, START-checkpoint emission |
Internal/TerminationManager.cs |
Adds RetryScheduled termination reason |
Internal/CheckpointBatcher.cs |
Doc-only update describing fire-and-forget semantics |
Tests/RetryStrategyTests.cs |
14 unit tests for exponential math/jitter/filters/delegate |
Tests/DurableContextTests.cs |
6 retry/AtMostOnce/Pending replay tests |
Tests/DurableFunctionTests.cs |
Updated to assert START + SUCCEED + WAIT-START flat sequence |
IntegrationTests/*.cs |
Add StepStarted-event assertions; richer history dump in DurableFunctionDeployment |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Implements the minimum viable slice of the Amazon.Lambda.DurableExecution SDK: a workflow can run StepAsync and WaitAsync against a real Lambda, with replay-aware checkpointing wired through to the AWS service. Public API surface introduced: - DurableFunction.WrapAsync — entry point that handles the durable execution envelope (input hydration, output construction, status mapping) - IDurableContext.StepAsync / WaitAsync (4 Step overloads, 1 Wait) - StepConfig with serializer hook (retry deferred to follow-up PR) - ICheckpointSerializer interface - [DurableExecution] attribute (recognized by future source generator) - DurableExecutionException base + StepException Internals: - DurableExecutionHandler — Task.WhenAny race between user code and the suspension signal, returning Succeeded/Failed/Pending - ExecutionState — replay-aware operation lookup and pending checkpoint buffer - OperationIdGenerator — deterministic, replay-stable IDs - TerminationManager — TaskCompletionSource-based suspension trigger - LambdaDurableServiceClient — wraps AWSSDK.Lambda's checkpoint and state APIs Tests: - 86 unit tests covering enums, exceptions, models, configs, ID generation, termination, execution state, the handler race, the context (Step + Wait paths), and the WrapAsync entry point - 8 end-to-end integration tests deploying real Lambdas via Docker on the provided.al2023 runtime: StepWaitStep, MultipleSteps, WaitOnly, LongerWait, ReplayDeterminism, RetrySucceeds, RetryExhausts, StepFails Out of scope (follow-up PRs): - IRetryStrategy, ExponentialRetryStrategy, retry decision factories - DefaultJsonCheckpointSerializer - DurableLogger replay-suppression (currently returns NullLogger) - Callbacks, InvokeAsync, ParallelAsync, MapAsync, RunInChildContextAsync, WaitForConditionAsync — interface intentionally does not declare them - Annotations source-generator integration - DurableTestRunner / Amazon.Lambda.DurableExecution.Testing package - dotnet new lambda.DurableFunction blueprint stack-info: PR: #2360, branch: GarrettBeatty/stack/2 remove update update update update
GarrettBeatty
commented
May 14, 2026
| var history = await deployment.WaitForHistoryAsync( | ||
| arn!, | ||
| h => (h.Events?.Count(e => e.StepSucceededDetails != null) ?? 0) >= 2 | ||
| h => (h.Events?.Count(e => e.EventType == EventType.StepStarted) ?? 0) >= 2 |
Contributor
Author
There was a problem hiding this comment.
now that we are emitting START steps (which are needed for retries) we are asserting them in the IT tests
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| COPY bin/publish/ ${LAMBDA_TASK_ROOT} | ||
|
|
||
| ENTRYPOINT ["/var/task/bootstrap"] |
GarrettBeatty
commented
May 14, 2026
| /// Replay semantics — example: <c>await ctx.StepAsync(ChargeCard, "charge")</c> | ||
| /// Replay branches — example: <c>await ctx.StepAsync(ChargeCard, "charge")</c> | ||
| /// <list type="bullet"> | ||
| /// <item>Fresh: no prior state → run func → emit SUCCEED → return result.</item> |
Contributor
Author
There was a problem hiding this comment.
in previous PR only SUCCEEDED or FAILED mattered. But now for replays, we need to keep track of how many times the function was executed, which is done via the number of STARTED steps.
Match the Python / Java / JavaScript reference SDKs' replay-mode model: the workflow is "replaying" iff it has not yet revisited every checkpointed completed user-replayable operation. A single global flag flipped on the first fresh op (the prior model) misclassified workflow- body code that runs before the first step and would not generalize to Map/Parallel/Callback later. ExecutionState changes: - Replace `Mode`/`ExecutionMode`/`EnterExecutionMode()` with `IsReplaying` + `TrackReplay(operationId)`. - Initial replay decision: any non-EXECUTION op present means we're replaying. The service always sends an EXECUTION-type op carrying the input payload — that's bookkeeping, not user history, so it does not count toward replay (matches Python execution.py:258, Java ExecutionManager:81, JS execution-context.ts:62). - TrackReplay flips IsReplaying false once every checkpointed terminal- status non-EXECUTION op has been visited. Terminal set matches Python's: SUCCEEDED, FAILED, CANCELLED, STOPPED. Operation changes: - DurableOperation.ExecuteAsync calls TrackReplay(OperationId) at the top, so every operation participates in visit accounting without each subclass needing to remember. - StepOperation/WaitOperation drop their manual EnterExecutionMode calls. Tests: - ExecutionStateTests rewritten around IsReplaying/TrackReplay, including pinning regressions: only-EXECUTION-op ⇒ NotReplaying, all-visited ⇒ flips out of replay, PENDING ops do not block transition, idempotency. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PRs:
#2216
What
Adds retry support to the
Amazon.Lambda.DurableExecutionSDK on top of the foundation in #2360. After this PR a step that throws can be retried with configurable backoff and jitter; durable executions resume after the retry timer elapses without billing Lambda compute during the wait.Public API introduced:
IRetryStrategyRetryDecisionIRetryStrategy.ShouldRetry—ShouldRetryflag plusDelay.RetryStrategyDefault,Transient,None,Exponential(...),FromDelegate(...).JitterStrategyNone/Half/Fullfor exponential backoff.StepSemanticsAtLeastOncePerRetry(default) /AtMostOncePerRetry.StepConfig.RetryStrategy,StepConfig.SemanticsWhy
Real workflows fail. A step that calls a flaky downstream service or hits a transient throttle needs to retry without restarting the whole workflow. Durable execution makes service-mediated retries possible: the SDK checkpoints a
RETRYoperation with aNextAttemptDelaySeconds, suspends the Lambda, and the service re-invokes us when the timer fires. The user's compute isn't billed during the wait.AtMostOncePerRetrysemantics handle non-idempotent steps (e.g. charging a card): aSTARTcheckpoint is durably persisted before user code runs, so a Lambda crash mid-execution can be detected on replay and routed through the retry strategy rather than re-executing.How
Retry control flow. When a step throws,
StepOperation.HandleStepFailureAsyncconsults the configuredIRetryStrategy.ShouldRetry(ex, attemptNumber). If the decision says retry, the SDK enqueues aRETRYcheckpoint carryingNextAttemptDelaySeconds, then suspends viaTerminationManager.SuspendAndAwaitsoRunAsyncreturnsPendingto the service. On the next invocation,StepOperation.ReplayAsyncseesStatus == PENDINGand either re-suspends (timer not yet elapsed) or re-executes (timer fired) with the carried-forward attempt counter.At-most-once semantics. For non-idempotent steps,
Semantics = AtMostOncePerRetrywrites aSTARTcheckpoint and blocks until the batcher flushes it before user code runs. If Lambda crashes between user code and theSUCCEEDflush, replay seesSTARTEDwith no terminal record and routes throughHandleStepFailureAsyncas a failed attempt instead of re-executing — the side effect runs at most once per attempt.Retry strategy contract.
IRetryStrategy.ShouldRetry(Exception, int attemptNumber)returns aRetryDecision.ExponentialRetryStrategysupports configurable max attempts, initial/max delay, backoff rate, jitter (None/Half/Full), and exception filtering by type or message regex. Built-in factories:RetryStrategy.Default(6 attempts, 5s/60s, 2× backoff, full jitter),Transient(3 attempts, 1s/5s, half jitter),None.RetryStrategy.FromDelegate(...)for arbitrary policies.Key files:
Config/IRetryStrategy.cs— strategy interface +RetryDecisionvalue typeConfig/RetryStrategy.cs— built-in strategies,ExponentialRetryStrategy,JitterStrategy,StepSemantics,DelegateRetryStrategyConfig/StepConfig.cs— addsRetryStrategyandSemanticspropertiesInternal/StepOperation.cs— addsPENDING(retry timer) andSTARTED(AtMostOnce crash recovery) replay arms;HandleStepFailureAsyncdecision treeInternal/TerminationManager.cs— addsRetryScheduledreasonTesting
21 new unit tests in
Amazon.Lambda.DurableExecution.Tests(130 total, up from 109 in #2360):RetryStrategyTests(14 tests) — exponential backoff math, jitter strategies, max-attempt exhaustion, exception-type and message-pattern filtering, delegate strategiesDurableContextTestsretry block (6 tests) —FailsWithRetryStrategy_CheckpointsRetryAndSuspends,FailsNoRetryStrategy_CheckpointsFail,RetryExhausted_CheckpointsFail,PendingWithFutureTimestamp_Suspends,PendingWithPastTimestamp_ReExecutes,AtMostOnce_FlushesStartBeforeExecution,AtMostOnce_StartedReplay_TriggersRetryHandlerIntegration tests (
Amazon.Lambda.DurableExecution.IntegrationTests) —RetrySucceedsandRetryExhaustsend-to-end against the real durable-execution service.Out of scope (follow-up PRs)
MapAsync/ParallelAsync/RunInChildContextAsync/WaitForConditionAsyncCallbackAsync,InvokeAsyncDefaultJsonCheckpointSerializerDurableLoggerreplay-suppression (currentlyNullLogger)[DurableExecution]attributeDurableTestRunner/Amazon.Lambda.DurableExecution.Testingpackagedotnet new lambda.DurableFunctionblueprint