Skip to content

Allow configuring HTTP client timeout for DynamoDB backend#31797

Open
Kyslik wants to merge 2 commits intohashicorp:mainfrom
Kyslik:fix/26580
Open

Allow configuring HTTP client timeout for DynamoDB backend#31797
Kyslik wants to merge 2 commits intohashicorp:mainfrom
Kyslik:fix/26580

Conversation

@Kyslik
Copy link
Copy Markdown

@Kyslik Kyslik commented Feb 22, 2026

Description

This PR allows operators to configure the HTTP client timeout used by the DynamoDB backend.
By default, the timeout remains zero, which preserves existing behavior. In Go's net/http package, a timeout of zero means no timeout, so requests may block indefinitely.

As described in issue #26580, a deadlock scenario is possible when using the DynamoDB HA backend. If the active instance experiences an intermittent network drop, it may fail to release the lock while another instance acquires it, resulting in multiple nodes acting as active. Recovery typically requires a restart; however, if an intermediate proxy or firewall properly closes the stale connection, the instance can recover and transition back to standby. Configuring a non-zero HTTP client timeout mitigates this condition by preventing indefinitely blocked requests on these two calls

_, err := l.backend.client.UpdateItem(&dynamodb.UpdateItemInput{
resp, err := l.backend.client.GetItem(&dynamodb.GetItemInput{

The additional validation rejecting negative timeout values is defensive. Although time.ParseDuration permits negative durations, they are not meaningful in this context (related reading golang/go#39177).

A dedicated test was not added because this change is limited to configuration parsing and assignment of http.Client.Timeout, which relies entirely on Go's standard library (time.ParseDuration and net/http). The logic is straightforward, contains no custom timeout behavior, and does not alter request execution semantics beyond setting the standard client field, so additional unit coverage would provide limited value.


There is an alternative fix to make client sensibly timeout, using an UpdateItemWithContext and GetItemWithContext, however context is not in the Lock interface thus would require a lot more work.
🤔 , unless hardcoded context.WithTimeout can be used:

+	ctx, cancel := context.WithTimeout(context.Background(), "some sensible constant")
+	defer cancel()
...
+	_, err := l.backend.client.UpdateItemWithContext(ctx, &dynamodb.UpdateItemInput{
-	_, err := l.backend.client.UpdateItem(&dynamodb.UpdateItemInput{

I think the client timeout should be configurable even though UpdateItemWithContext may be favorable.

@vercel
Copy link
Copy Markdown

vercel Bot commented Feb 22, 2026

@Kyslik is attempting to deploy a commit to the HashiCorp Team on Vercel.

A member of the Team first needs to authorize it.

@hashicorp-cla-app
Copy link
Copy Markdown

hashicorp-cla-app Bot commented Feb 22, 2026

CLA assistant check
All committers have signed the CLA.

@Kyslik Kyslik marked this pull request as ready for review February 22, 2026 16:08
@Kyslik Kyslik requested a review from a team as a code owner February 22, 2026 16:08
@Kyslik Kyslik requested a review from mathew-amala February 22, 2026 16:08
@heatherezell heatherezell added storage/dynamodb bug Used to indicate a potential bug labels Feb 23, 2026
@Kyslik
Copy link
Copy Markdown
Author

Kyslik commented Mar 5, 2026

@mathew-amala could you please take a look 🤞🏻 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Used to indicate a potential bug storage/dynamodb

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants