From 8ff203523b475a91d66a58f4c819e1975780e2e5 Mon Sep 17 00:00:00 2001 From: starfleeth <128422269+starfleeth@users.noreply.github.com> Date: Fri, 9 Oct 2026 09:27:14 -0700 Subject: [PATCH] Remove source tags from references --- references/aws-lambda/constraints.md | 111 ++++++++++----------- references/aws-lambda/diagnostics.md | 91 ++++++++--------- references/aws-lambda/iam.md | 44 ++++---- references/aws-lambda/observability.md | 18 ++-- references/aws-lambda/sdk-dotnet.md | 2 - references/aws-lambda/sdk-go.md | 60 +++++------ references/aws-lambda/sdk-java.md | 24 ++--- references/aws-lambda/sdk-python.md | 65 ++++++------ references/aws-lambda/sdk-typescript.md | 71 ++++++------- references/aws-lambda/self-hosted.md | 24 ++--- references/aws-lambda/setup.md | 70 ++++++------- references/aws-lambda/versioning.md | 15 +-- references/concepts.md | 8 +- references/gcp-cloud-run/constraints.md | 11 +- references/gcp-cloud-run/diagnostics.md | 9 +- references/gcp-cloud-run/iam.md | 10 +- references/gcp-cloud-run/sdk-dotnet.md | 1 - references/gcp-cloud-run/sdk-go.md | 1 - references/gcp-cloud-run/sdk-java.md | 1 - references/gcp-cloud-run/sdk-python.md | 1 - references/gcp-cloud-run/sdk-typescript.md | 1 - references/gcp-cloud-run/self-hosted.md | 13 ++- references/gcp-cloud-run/setup.md | 5 +- references/gcp-cloud-run/versioning.md | 6 +- references/wci.md | 15 +-- 25 files changed, 292 insertions(+), 385 deletions(-) diff --git a/references/aws-lambda/constraints.md b/references/aws-lambda/constraints.md index 1accd65..45ccacd 100644 --- a/references/aws-lambda/constraints.md +++ b/references/aws-lambda/constraints.md @@ -1,117 +1,115 @@ # AWS Lambda — Execution-model constraints - This reference describes the invocation-based AWS Lambda model. For concepts shared across compute providers, see [Serverless Workers — Concepts](../concepts.md). ## Worker model -A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. -There is no always-on infrastructure to provision or scale. Temporal invokes the Worker when Tasks arrive on a Task Queue, and the Worker shuts down when the work is done. +A Serverless Worker is a Temporal Worker that runs on serverless compute instead of a long-lived process. +There is no always-on infrastructure to provision or scale. Temporal invokes the Worker when Tasks arrive on a Task Queue, and the Worker shuts down when the work is done. -A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. The difference is in the lifecycle: instead of the Worker starting and polling continuously, Temporal invokes the Serverless Worker on demand, the Worker starts, processes available Tasks, and then shuts down. +A Serverless Worker uses the same Temporal SDKs as a traditional long-lived Worker. It registers Workflows and Activities the same way. The difference is in the lifecycle: instead of the Worker starting and polling continuously, Temporal invokes the Serverless Worker on demand, the Worker starts, processes available Tasks, and then shuts down. -Serverless Workers require Worker Versioning. Each Serverless Worker must be associated with a Worker Deployment Version that has a compute provider configured. +Serverless Workers require Worker Versioning. Each Serverless Worker must be associated with a Worker Deployment Version that has a compute provider configured. -Each Workflow must have an `AutoUpgrade` or `Pinned` versioning behavior, set per-Workflow or as a Worker-level default. +Each Workflow must have an `AutoUpgrade` or `Pinned` versioning behavior, set per-Workflow or as a Worker-level default. ## How Serverless invocation works -With long-lived Workers, the Worker process starts, connects to Temporal, and polls a Task Queue for work. Temporal does not need to know anything about the Worker's infrastructure. +With long-lived Workers, the Worker process starts, connects to Temporal, and polls a Task Queue for work. Temporal does not need to know anything about the Worker's infrastructure. -With Serverless Workers, Temporal starts the Worker. +With Serverless Workers, Temporal starts the Worker. ### Invocation flow -The invocation flow works as follows: +The invocation flow works as follows: -1. A Task is submitted (for example, `StartWorkflow` or `ScheduleActivity`). -2. The Matching Service attempts to route the Task directly to an available Worker (a sync match). -3. If a Worker is available, the Task is routed to that Worker. -4. If no Worker is available (sync match fails), the Matching Service pushes a signal to the WCI, and the WCI invokes the configured compute provider. -5. The Serverless Worker starts, creates a Temporal Client, and begins polling the Task Queue. -6. The Worker processes available Tasks until it exits (see Worker lifecycle). +1. A Task is submitted (for example, `StartWorkflow` or `ScheduleActivity`). +2. The Matching Service attempts to route the Task directly to an available Worker (a sync match). +3. If a Worker is available, the Task is routed to that Worker. +4. If no Worker is available (sync match fails), the Matching Service pushes a signal to the WCI, and the WCI invokes the configured compute provider. +5. The Serverless Worker starts, creates a Temporal Client, and begins polling the Task Queue. +6. The Worker processes available Tasks until it exits (see Worker lifecycle). -Each invocation is independent. The Worker creates a fresh client connection on every invocation. There is no connection reuse or shared state across invocations. +Each invocation is independent. The Worker creates a fresh client connection on every invocation. There is no connection reuse or shared state across invocations. ## Autoscaling -The shared [WCI inputs](../wci.md#inputs) cause Lambda function invocations when Tasks need Workers. Each Worker exits when its invocation finishes, allowing the Lambda fleet to scale to zero. +The shared [WCI inputs](../wci.md#inputs) cause Lambda function invocations when Tasks need Workers. Each Worker exits when its invocation finishes, allowing the Lambda fleet to scale to zero. ## Scaling with long-lived Workers -Serverless Workers can share a Task Queue with long-lived Workers. Because Serverless Workers are only invoked on sync match failure, Serverless Workers only pick up Tasks that no long-lived Worker was available to handle. In practice, the Serverless Workers act as spillover capacity for the long-lived fleet. +Serverless Workers can share a Task Queue with long-lived Workers. Because Serverless Workers are only invoked on sync match failure, Serverless Workers only pick up Tasks that no long-lived Worker was available to handle. In practice, the Serverless Workers act as spillover capacity for the long-lived fleet. -**Warning:** If you configure Serverless and long-lived Workers on the same Task Queue, do not enable dynamic scaling on the long-lived Workers. The two groups cannot coordinate their scaling behavior. If both scale dynamically, the long-lived Workers may scale up to handle the same Tasks that Temporal is simultaneously invoking Serverless Workers for, leading to unnecessary invocations and unpredictable scaling. +**Warning:** If you configure Serverless and long-lived Workers on the same Task Queue, do not enable dynamic scaling on the long-lived Workers. The two groups cannot coordinate their scaling behavior. If both scale dynamically, the long-lived Workers may scale up to handle the same Tasks that Temporal is simultaneously invoking Serverless Workers for, leading to unnecessary invocations and unpredictable scaling. ## Worker lifecycle -A single Serverless Worker invocation has three phases: init, work, and shutdown. +A single Serverless Worker invocation has three phases: init, work, and shutdown. ### Init phase -The Worker initializes and establishes a client connection to Temporal. +The Worker initializes and establishes a client connection to Temporal. ### Work phase -The Worker polls the Task Queue and processes Tasks. +The Worker polls the Task Queue and processes Tasks. ### Shutdown phase -The Worker stops polling, waits for in-flight Tasks to finish, and runs any shutdown hooks (for example, OpenTelemetry telemetry flushes). Shutdown begins before the invocation deadline so the Worker can exit cleanly before the compute provider forcibly terminates the execution environment. +The Worker stops polling, waits for in-flight Tasks to finish, and runs any shutdown hooks (for example, OpenTelemetry telemetry flushes). Shutdown begins before the invocation deadline so the Worker can exit cleanly before the compute provider forcibly terminates the execution environment. ### Tuning for long-running Activities -If your Worker handles long-running Activities, set these three values together: +If your Worker handles long-running Activities, set these three values together: -- **Worker stop timeout > longest Activity runtime.** Gives in-flight Activities enough time to finish after polling stops. -- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** Ensures the drain and any shutdown hooks complete before the compute provider terminates the environment. -- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** Set on the compute provider to give each invocation enough total runtime. +- **Worker stop timeout > longest Activity runtime.** Gives in-flight Activities enough time to finish after polling stops. +- **Shutdown deadline buffer > Worker stop timeout + shutdown hook time.** Ensures the drain and any shutdown hooks complete before the compute provider terminates the environment. +- **Invocation deadline > longest Activity runtime + shutdown deadline buffer.** Set on the compute provider to give each invocation enough total runtime. -If your longest-running Activity runs longer than half the maximum invocation deadline, use Activity Heartbeats to record the state of the Activity execution so that the next retry can pick up where it left off. +If your longest-running Activity runs longer than half the maximum invocation deadline, use Activity Heartbeats to record the state of the Activity execution so that the next retry can pick up where it left off. -Example: if your longest Activity runtime is 5 minutes, and your shutdown hooks take 3 seconds, set the Worker stop timeout to more than 5 minutes, and the shutdown deadline buffer to more than 303 seconds (5 minutes + 3 seconds). Set your invocation deadline to at least 10 minutes and 3 seconds. +Example: if your longest Activity runtime is 5 minutes, and your shutdown hooks take 3 seconds, set the Worker stop timeout to more than 5 minutes, and the shutdown deadline buffer to more than 303 seconds (5 minutes + 3 seconds). Set your invocation deadline to at least 10 minutes and 3 seconds. -The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks. +The Worker stop timeout controls how long the Worker waits for in-flight Tasks to finish after it stops polling. The shutdown deadline buffer controls how much time before the invocation deadline the Worker stops polling for Tasks. -Raising only the shutdown deadline buffer makes the Worker stop polling earlier, but does not give in-flight Tasks any more time to complete. +Raising only the shutdown deadline buffer makes the Worker stop polling earlier, but does not give in-flight Tasks any more time to complete. -Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider might terminate the Worker before the full stop timeout completes. +Raising only the Worker stop timeout does not make the Worker stop polling earlier, which means the compute provider might terminate the Worker before the full stop timeout completes. ## Failure handling -Serverless Workers rely on Temporal's standard retry and timeout semantics to recover from failures. +Serverless Workers rely on Temporal's standard retry and timeout semantics to recover from failures. ### Worker crash -If a Worker invocation crashes (out of memory, unhandled exception, etc.): +If a Worker invocation crashes (out of memory, unhandled exception, etc.): -- The Activity Timeout fires after the configured duration. -- Temporal retries the Activity on a different Worker invocation. -- No manual intervention is required. +- The Activity Timeout fires after the configured duration. +- Temporal retries the Activity on a different Worker invocation. +- No manual intervention is required. ### Provider concurrency limit -If the compute provider's concurrency limit is reached (for example, AWS Lambda account concurrency): +If the compute provider's concurrency limit is reached (for example, AWS Lambda account concurrency): -- Further invocations from the WCI fail. -- Tasks remain in the Task Queue backlog. No data loss occurs. -- Processing slows until concurrency frees up. +- Further invocations from the WCI fail. +- Tasks remain in the Task Queue backlog. No data loss occurs. +- Processing slows until concurrency frees up. ### Resource exhaustion across Activity slots -By default, a single Worker invocation may run multiple Activity slots. A crash or resource exhaustion in one Activity can affect other Activities running in the same invocation. +By default, a single Worker invocation may run multiple Activity slots. A crash or resource exhaustion in one Activity can affect other Activities running in the same invocation. -To isolate Activities from each other: +To isolate Activities from each other: -- Split Workflow and Activity Workers into separate compute functions. -- Set Activity slots to 1 per invocation. +- Split Workflow and Activity Workers into separate compute functions. +- Set Activity slots to 1 per invocation. -With single-slot configuration, each Activity gets a dedicated execution environment. +With single-slot configuration, each Activity gets a dedicated execution environment. ## Constraints - | Constraint | Detail | |---|---| @@ -122,30 +120,28 @@ With single-slot configuration, each Activity gets a dedicated execution environ ## Worker Versioning with Serverless Workers -Serverless Workers require Worker Versioning, and the compute provider must invoke a stable, immutable build for each Worker Deployment Version. With AWS Lambda, this means aligning two versioning systems: +Serverless Workers require Worker Versioning, and the compute provider must invoke a stable, immutable build for each Worker Deployment Version. With AWS Lambda, this means aligning two versioning systems: -- **Temporal Worker Deployment Versions** — identified by deployment name and Build ID. Each Workflow runs against a specific Worker Deployment Version (Pinned) or moves between them on routing changes (Auto-Upgrade). -- **AWS Lambda function versions** — immutable numbered snapshots of your Lambda function code (`1`, `2`, `3`, ...). +- **Temporal Worker Deployment Versions** — identified by deployment name and Build ID. Each Workflow runs against a specific Worker Deployment Version (Pinned) or moves between them on routing changes (Auto-Upgrade). +- **AWS Lambda function versions** — immutable numbered snapshots of your Lambda function code (`1`, `2`, `3`, ...). -For production workloads, map each Worker Deployment Version to exactly one Lambda function version, and configure the compute provider with the qualified versioned ARN for that Lambda version (for example, `arn:aws:lambda:us-east-1:123:function:my-worker:5`). +For production workloads, map each Worker Deployment Version to exactly one Lambda function version, and configure the compute provider with the qualified versioned ARN for that Lambda version (for example, `arn:aws:lambda:us-east-1:123:function:my-worker:5`). -For development or non-critical workloads, you can use an unqualified ARN to iterate without publishing a new Lambda function version each time. +For development or non-critical workloads, you can use an unqualified ARN to iterate without publishing a new Lambda function version each time. -**Caution:** An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Without a versioned ARN, deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as Pinned. +**Caution:** An unqualified ARN (no version suffix) points at `$LATEST`, which changes on every redeploy. Without a versioned ARN, deploying replay-unsafe code causes non-determinism errors for in-flight Workflows, even for Workflows annotated as Pinned. -The choice of Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets Lambda. Both behaviors expect a versioned ARN that points at one immutable Lambda function version. +The choice of Pinned or Auto-Upgrade controls how Workflows move between Worker Deployment Versions in Temporal. It does not change how a Worker Deployment Version targets Lambda. Both behaviors expect a versioned ARN that points at one immutable Lambda function version. | Versioning Behavior | With versioned Lambda ARN | Without versioned Lambda ARN | |---|---|---| | **Pinned** | Existing Workflows stay on their original Lambda function version until they complete. | Existing Workflows stay on their original Worker Deployment Version, but the underlying Lambda code has already changed since `$LATEST` updated at redeploy. The new code must be replay-compatible. | | **Auto-Upgrade** | Existing Workflows move to the new Worker Deployment Version and its new Lambda function version at the next Workflow Task after you move the Current Version. | The Lambda redeploy already changed the code for all versions. Setting the Current Version only changes routing, not which code runs. | - See `versioning.md` for the step-by-step `aws lambda publish-version` workflow and `setup.md` (Step 4) for how to configure the compute provider with a versioned ARN. ## Why use Serverless Workers? - - **Reduce operational overhead.** No always-on infrastructure to manage and no autoscaling policies to tune. Temporal and the compute provider handle invocation and scaling. - **Get started faster.** Deploying a Worker is as simple as deploying a function. No Kubernetes, container orchestration, or scaling strategy required. @@ -154,7 +150,6 @@ See `versioning.md` for the step-by-step `aws lambda publish-version` workflow a ## When to use Serverless Workers - Good fit when: @@ -166,7 +161,6 @@ Good fit when: May not be ideal when: - - Activities are long-running and cannot be interrupted. AWS Lambda has a 15-minute execution limit. Activities that run longer and cannot be broken into smaller steps need a different hosting strategy or a provider with longer limits. - Workloads require sustained high throughput. Long-lived Workers on dedicated compute may be more cost-effective and performant. @@ -174,7 +168,6 @@ May not be ideal when: ## How Serverless Workers compare to long-lived Workers - | | Long-lived Worker | Serverless Worker | |---|---|---| diff --git a/references/aws-lambda/diagnostics.md b/references/aws-lambda/diagnostics.md index ddf163d..a935959 100644 --- a/references/aws-lambda/diagnostics.md +++ b/references/aws-lambda/diagnostics.md @@ -1,56 +1,52 @@ # AWS Lambda — Diagnostics & troubleshooting - ## Invocation flow (when working correctly) -When a Serverless Worker invocation works correctly, the following sequence happens: +When a Serverless Worker invocation works correctly, the following sequence happens: -1. You deploy the Worker function on Lambda. -2. You configure a Worker Deployment Version with a compute provider. This starts a Worker Controller Instance (WCI) Workflow and a validation invocation of the Lambda function. -3. The Lambda polls the Temporal Service successfully, binding the Task Queue configured on the Worker to the Worker Deployment Version. -4. The WCI continuously monitors the associated Task Queue on a schedule. The Matching Service also notifies the WCI Workflow of sync match failures immediately as they happen. -5. A Task arrives on the Task Queue and the WCI detects the backlog. -6. The WCI invokes the Lambda function. -7. The Lambda function starts, the Worker connects to Temporal and polls the Task Queue. -8. The Worker processes Tasks and shuts down gracefully. +1. You deploy the Worker function on Lambda. +2. You configure a Worker Deployment Version with a compute provider. This starts a Worker Controller Instance (WCI) Workflow and a validation invocation of the Lambda function. +3. The Lambda polls the Temporal Service successfully, binding the Task Queue configured on the Worker to the Worker Deployment Version. +4. The WCI continuously monitors the associated Task Queue on a schedule. The Matching Service also notifies the WCI Workflow of sync match failures immediately as they happen. +5. A Task arrives on the Task Queue and the WCI detects the backlog. +6. The WCI invokes the Lambda function. +7. The Lambda function starts, the Worker connects to Temporal and polls the Task Queue. +8. The Worker processes Tasks and shuts down gracefully. ## Diagnostic decision tree -Start by determining whether the Lambda function is being invoked at all, then narrow down from there. +Start by determining whether the Lambda function is being invoked at all, then narrow down from there. -Check the Lambda function's CloudWatch metrics or invocation logs. In the AWS Console, go to **Lambda > Functions > your function > Monitor**. Look for recent invocations in the **Invocations** graph. You can also check **CloudWatch > Log groups > /aws/lambda/your-function-name** for execution logs. +Check the Lambda function's CloudWatch metrics or invocation logs. In the AWS Console, go to **Lambda > Functions > your function > Monitor**. Look for recent invocations in the **Invocations** graph. You can also check **CloudWatch > Log groups > /aws/lambda/your-function-name** for execution logs. -- If there are no invocations: see [Lambda is not being invoked](#lambda-not-invoked). -- If the Lambda is being invoked but Workflows are not progressing: see [Lambda is invoked but Tasks are not completing](#lambda-invoked-not-completing). +- If there are no invocations: see [Lambda is not being invoked](#lambda-not-invoked). +- If the Lambda is being invoked but Workflows are not progressing: see [Lambda is invoked but Tasks are not completing](#lambda-invoked-not-completing). --- ## Lambda is not being invoked {#lambda-not-invoked} -Work through the following checks in order. +Work through the following checks in order. ### 1. Validate the connection to Lambda -Go to **Workers > Deployments > select your deployment**, open the **Actions** menu on the version, and click **Validate Connection**. +Go to **Workers > Deployments > select your deployment**, open the **Actions** menu on the version, and click **Validate Connection**. -A successful validation confirms that: +A successful validation confirms that: - The Worker Deployment Version has a compute provider configured. - Temporal can assume the invocation role. - The Lambda function can be invoked. -If validation fails: +If validation fails: - Verify the Lambda function ARN and invocation role ARN in the Worker Deployment Version configuration are correct. - Verify the invocation role was created using the CloudFormation template and that the External ID matches the value in the Worker Deployment Version configuration. **Not every validation failure is on your side.** If the error is Temporal failing to obtain its **own** base AWS credentials (for example, `no EC2 IMDS role found`) while assuming the `wci-lambda-invoke` role, that is a Temporal-Cloud-side credential problem occurring *before* Temporal ever reaches your invocation role or Lambda. Signature: it reproduces identically no matter what you change on the AWS side, and (for Cloud) is independent of your trust policy or External ID. Do not keep editing your AWS config — verify your artifacts once (Lambda `Active` with the correct runtime; trust policy lists all five `wci-lambda-invoke` principals plus the `sts:ExternalId` condition; invoke permissions scoped to the function ARN), then escalate to Temporal support. Expect infrastructure-side rough edges while the feature is in Public Preview. -If the Worker Deployment Version does not have a compute provider configured, no WCI Workflow exists and the Lambda is never automatically invoked. +If the Worker Deployment Version does not have a compute provider configured, no WCI Workflow exists and the Lambda is never automatically invoked. -**Common cause:** Manually invoking the Lambda function before creating the Worker Deployment Version in the UI or CLI. When the Lambda runs, the Worker connects to Temporal and polls the Task Queue. That polling registers the Worker Deployment Version and binds the Task Queue on the server, but the version has no compute provider. To fix, create or update the Worker Deployment Version with the compute provider flags. +**Common cause:** Manually invoking the Lambda function before creating the Worker Deployment Version in the UI or CLI. When the Lambda runs, the Worker connects to Temporal and polls the Task Queue. That polling registers the Worker Deployment Version and binds the Task Queue on the server, but the version has no compute provider. To fix, create or update the Worker Deployment Version with the compute provider flags. **Recovering a provider-less version via the CLI.** In practice there is no `update-provider` command, and `describe-version` does **not** show whether a compute provider is attached, so you cannot confirm the provider-less state by inspecting the version — you infer it because `create-version` reports the version already exists. Re-running `create-version` fails for the same reason. The reliable fix is to **delete the provider-less version and recreate it** with the compute-provider flags. This is clean as long as the version was never set as current. @@ -60,14 +56,13 @@ temporal worker deployment delete-version \ --build-id # then re-run: temporal worker deployment create-version --aws-lambda-function-arn ... (see setup.md, Step 4) ``` - -`delete-version` requires the version to not be Current, Ramping, or have active pollers; add `--skip-drainage` to ignore the draining restriction. +`delete-version` requires the version to not be Current, Ramping, or have active pollers; add `--skip-drainage` to ignore the draining restriction. ### 2. Check that the version is set as current -The Worker Deployment Version must be set as the current version for new Tasks to route to it. If you created the version through the CLI, you need to set it as current. +The Worker Deployment Version must be set as the current version for new Tasks to route to it. If you created the version through the CLI, you need to set it as current. -Verify the current version with: +Verify the current version with: ```bash temporal worker deployment describe --name @@ -77,9 +72,9 @@ temporal worker deployment describe --name ### 3. Check that the WCI is detecting Tasks -If the connection validates successfully but the Lambda is still not being invoked, the WCI may not be detecting Tasks on the Task Queue. +If the connection validates successfully but the Lambda is still not being invoked, the WCI may not be detecting Tasks on the Task Queue. -Check which Task Queues are bound to the Worker Deployment Version and whether there is a backlog: +Check which Task Queues are bound to the Worker Deployment Version and whether there is a backlog: ```bash temporal worker deployment describe-version \ @@ -88,19 +83,18 @@ temporal worker deployment describe-version \ --build-id \ --report-task-queue-stats ``` - -If no Task Queues are listed, the binding has not been established. The server binds a Task Queue to a Worker Deployment Version when a Worker with that deployment version successfully connects and polls the Task Queue. +If no Task Queues are listed, the binding has not been established. The server binds a Task Queue to a Worker Deployment Version when a Worker with that deployment version successfully connects and polls the Task Queue. **What a healthy version looks like.** The inverse of the check above is the single most useful positive signal in the whole setup: the Task Queue appearing here, for both workflow and activity task types, proves that Temporal assumed the invocation role, invoked the Lambda, the package loaded, the env vars were right, the Worker authenticated to the Namespace, and the timeout was long enough to reach a poll. Getting this output means every remaining failure is downstream (routing, versioning-behavior, or application code), so check it immediately after `create-version` and before touching anything else. `describe-version` still does **not** report whether a compute provider is attached, so it cannot rule that in or out. #### Failed first invocation -A common cause of missing Task Queue bindings is a failed first invocation. When you create a Worker Deployment Version, the WCI invokes the Lambda to validate the configuration. If that first invocation fails (for example, due to missing environment variables, incorrect TLS configuration, missing dependencies, or a Lambda timeout that is too short), the Worker never connects to Temporal and never polls. Without a successful poll, the Task Queue binding is never created. +A common cause of missing Task Queue bindings is a failed first invocation. When you create a Worker Deployment Version, the WCI invokes the Lambda to validate the configuration. If that first invocation fails (for example, due to missing environment variables, incorrect TLS configuration, missing dependencies, or a Lambda timeout that is too short), the Worker never connects to Temporal and never polls. Without a successful poll, the Task Queue binding is never created. -Pay particular attention to the Lambda timeout. AWS Lambda functions default to a 3-second timeout, which is often too short for the Worker to start, connect to Temporal, and register the Task Queue before AWS terminates the invocation. If the function times out during this initial invocation, the binding is never established and the Lambda is not invoked again. +Pay particular attention to the Lambda timeout. AWS Lambda functions default to a 3-second timeout, which is often too short for the Worker to start, connect to Temporal, and register the Task Queue before AWS terminates the invocation. If the function times out during this initial invocation, the binding is never established and the Lambda is not invoked again. -To diagnose: invoke the Lambda function manually from the AWS Console. The console displays the execution result and any errors directly, making it easier to identify configuration issues than searching through CloudWatch logs. Once the Lambda runs successfully and the Worker connects to Temporal, the Task Queue binding is established. +To diagnose: invoke the Lambda function manually from the AWS Console. The console displays the execution result and any errors directly, making it easier to identify configuration issues than searching through CloudWatch logs. Once the Lambda runs successfully and the Worker connects to Temporal, the Task Queue binding is established. **A manual invoke runs for nearly the whole timeout.** A serverless Worker keeps polling until its shutdown-deadline buffer, so a synchronous manual invoke runs for roughly the full Lambda timeout (e.g. ~590s of a 600s timeout), not a few seconds. From the AWS CLI (`aws lambda invoke`) this trips the default **60-second client read timeout** with a "Read timeout on endpoint" error — that is expected, **not** a Worker crash. Add `--cli-read-timeout 0` (or use `--invocation-type Event` for an async invoke), and judge health from the CloudWatch startup logs (Worker connected + polling), not from the CLI's exit. Remember this successful poll also auto-registers a provider-less version — see "Common cause" above. @@ -108,17 +102,17 @@ To diagnose: invoke the Lambda function manually from the AWS Console. The conso ## Lambda is invoked but Tasks are not completing {#lambda-invoked-not-completing} -If CloudWatch shows Lambda invocations but Workflows are not progressing, the problem is in the Worker's execution within the Lambda function. +If CloudWatch shows Lambda invocations but Workflows are not progressing, the problem is in the Worker's execution within the Lambda function. ### Check Lambda execution logs -Check CloudWatch logs for errors during Worker startup. In the AWS Console, go to **CloudWatch > Log groups > /aws/lambda/your-function-name** and look for recent error messages. +Check CloudWatch logs for errors during Worker startup. In the AWS Console, go to **CloudWatch > Log groups > /aws/lambda/your-function-name** and look for recent error messages. -Common errors include: +Common errors include: -- **Connection failures**: The Worker cannot reach the Temporal Service. Check that the `TEMPORAL_ADDRESS` and `TEMPORAL_API_KEY` environment variables (or `temporal.toml` config file) are correctly set on the Lambda function. For self-hosted deployments, verify network reachability. -- **TLS errors**: The TLS certificate or key is missing, expired, or does not match the Namespace. -- **Authentication errors**: The API key is invalid or does not have access to the Namespace. +- **Connection failures**: The Worker cannot reach the Temporal Service. Check that the `TEMPORAL_ADDRESS` and `TEMPORAL_API_KEY` environment variables (or `temporal.toml` config file) are correctly set on the Lambda function. For self-hosted deployments, verify network reachability. +- **TLS errors**: The TLS certificate or key is missing, expired, or does not match the Namespace. +- **Authentication errors**: The API key is invalid or does not have access to the Namespace. ### Language-specific signatures @@ -128,21 +122,21 @@ For the exact logging cause and fix, see the selected SDK reference. ### Check for Lambda timeout -If the Lambda function reaches its configured timeout before the Worker finishes processing, AWS terminates the invocation. +If the Lambda function reaches its configured timeout before the Worker finishes processing, AWS terminates the invocation. -The Worker begins graceful shutdown before the Lambda deadline. If Activities take longer than the available execution window, the Activities are abandoned mid-execution and retried on the next invocation. +The Worker begins graceful shutdown before the Lambda deadline. If Activities take longer than the available execution window, the Activities are abandoned mid-execution and retried on the next invocation. -For long-running Activities, increase the Lambda timeout and the Worker's shutdown buffer together. +For long-running Activities, increase the Lambda timeout and the Worker's shutdown buffer together. ### Check that the deployment name and build ID match -If CloudWatch shows rapid, repeated invocations with no Workflow progress, the deployment name or build ID in the Worker code may not match the Worker Deployment Version configuration. +If CloudWatch shows rapid, repeated invocations with no Workflow progress, the deployment name or build ID in the Worker code may not match the Worker Deployment Version configuration. -The deployment name and build ID in your Lambda function code must exactly match the values you used when creating the Worker Deployment Version. Compare the values in your code against the WCI Workflow ID (`temporal-sys-worker-controller-instance::`) and the output of `temporal worker deployment describe`. +The deployment name and build ID in your Lambda function code must exactly match the values you used when creating the Worker Deployment Version. Compare the values in your code against the WCI Workflow ID (`temporal-sys-worker-controller-instance::`) and the output of `temporal worker deployment describe`. -A mismatch causes an invocation loop: the WCI invokes the Lambda, the Worker starts and polls with a different deployment version than the WCI expects, the Task is not processed, and the WCI invokes the Lambda again. +A mismatch causes an invocation loop: the WCI invokes the Lambda, the Worker starts and polls with a different deployment version than the WCI expects, the Task is not processed, and the WCI invokes the Lambda again. -To fix the loop, update the deployment name and build ID in the Worker code to match the Worker Deployment Version, then redeploy the Lambda function. +To fix the loop, update the deployment name and build ID in the Worker code to match the Worker Deployment Version, then redeploy the Lambda function. --- @@ -150,7 +144,7 @@ To fix the loop, update the deployment name and build ID in the Worker code to m Use [Worker Controller Instance (WCI)](../wci.md#inspect-the-wci) to find the WCI, inspect its Activity results, and interpret its Running state before changing AWS resources. -Describe a Worker Deployment Version and check Task Queue stats: +Describe a Worker Deployment Version and check Task Queue stats: ```bash temporal worker deployment describe-version \ @@ -159,4 +153,3 @@ temporal worker deployment describe-version \ --build-id \ --report-task-queue-stats ``` - diff --git a/references/aws-lambda/iam.md b/references/aws-lambda/iam.md index 0e58d2a..c7266e9 100644 --- a/references/aws-lambda/iam.md +++ b/references/aws-lambda/iam.md @@ -1,9 +1,5 @@ # AWS Lambda — IAM & permissions - This file covers three distinct identities: the **operator** (whose credentials run the deploy commands), the **execution role** (grants the function permission to run), and the **Temporal invocation role** (grants Temporal permission to invoke the function). The deploy steps themselves are in `setup.md`; self-hosted server enablement is in `self-hosted.md`. @@ -96,19 +92,18 @@ If the organization wraps credentials in its own tool (`aws-vault`, `granted`/`a ## Execution role -The Lambda execution role grants the function permission to run. It is trusted by `lambda.amazonaws.com` and must have at least the `AWSLambdaBasicExecutionRole` managed policy attached (which includes the CloudWatch Logs permissions the Worker needs). This is separate from the Temporal invocation role below. Pass its ARN as `--role` when you create the function (`setup.md`, Step 2). +The Lambda execution role grants the function permission to run. It is trusted by `lambda.amazonaws.com` and must have at least the `AWSLambdaBasicExecutionRole` managed policy attached (which includes the CloudWatch Logs permissions the Worker needs). This is separate from the Temporal invocation role below. Pass its ARN as `--role` when you create the function (`setup.md`, Step 2). ## Step 3: Configure IAM for Temporal invocation ### Temporal Cloud -This section applies to Temporal Cloud. For self-hosted, see the self-hosted section below. +This section applies to Temporal Cloud. For self-hosted, see the self-hosted section below. -Temporal needs permission to invoke your Lambda function and check its status. The Temporal server assumes an IAM role in your AWS account with a handful of Lambda permissions scoped to your Worker functions. The trust policy on the role includes an External ID condition to prevent confused deputy attacks. +Temporal needs permission to invoke your Lambda function and check its status. The Temporal server assumes an IAM role in your AWS account with a handful of Lambda permissions scoped to your Worker functions. The trust policy on the role includes an External ID condition to prevent confused deputy attacks. #### CloudFormation template parameters - | Parameter | Description | |---|---| @@ -174,19 +169,19 @@ The `:*` form is also what makes future `publish-version` builds work without an #### Trust policy principals -The Cloud template trusts five Temporal Cloud AWS account IDs with the role `wci-lambda-invoke`: +The Cloud template trusts five Temporal Cloud AWS account IDs with the role `wci-lambda-invoke`: -- `arn:aws:iam::902542641901:role/wci-lambda-invoke` -- `arn:aws:iam::160190466495:role/wci-lambda-invoke` -- `arn:aws:iam::819232936619:role/wci-lambda-invoke` -- `arn:aws:iam::829909441867:role/wci-lambda-invoke` -- `arn:aws:iam::354116250941:role/wci-lambda-invoke` +- `arn:aws:iam::902542641901:role/wci-lambda-invoke` +- `arn:aws:iam::160190466495:role/wci-lambda-invoke` +- `arn:aws:iam::819232936619:role/wci-lambda-invoke` +- `arn:aws:iam::829909441867:role/wci-lambda-invoke` +- `arn:aws:iam::354116250941:role/wci-lambda-invoke` -The IAM policy grants `lambda:InvokeFunction` and `lambda:GetFunction` on the specified Lambda function ARNs. +The IAM policy grants `lambda:InvokeFunction` and `lambda:GetFunction` on the specified Lambda function ARNs. #### Deploy the CloudFormation stack -This skill ships the complete, ready-to-deploy template at `assets/temporal-cloud-serverless-worker-role.yaml` (transcribed verbatim from the docs). Copy it into your working directory, or point `--template-body` at the skill's copy — no need to author it by hand. +This skill ships the complete, ready-to-deploy template at `assets/temporal-cloud-serverless-worker-role.yaml` (transcribed verbatim from the docs). Copy it into your working directory, or point `--template-body` at the skill's copy — no need to author it by hand. ```bash aws cloudformation create-stack \ @@ -199,7 +194,6 @@ aws cloudformation create-stack \ --capabilities CAPABILITY_NAMED_IAM \ --region ``` - Notes on the parameters above: @@ -210,24 +204,23 @@ Notes on the parameters above: EXTERNAL_ID=$(openssl rand -hex 16) ``` -Retrieve the IAM role ARN from the stack outputs: +Retrieve the IAM role ARN from the stack outputs: ```bash aws cloudformation describe-stacks --stack-name --query 'Stacks[0].Outputs[?OutputKey==`RoleARN`].OutputValue' --output text --region ``` - ### Self-hosted invocation role For self-hosted server enablement (dynamic config, WCI, and the server's own AWS credentials), see `self-hosted.md`. The invocation role itself is below. -Self-hosted Serverless Workers require Temporal Service v1.31.0 or later. +Self-hosted Serverless Workers require Temporal Service v1.31.0 or later. #### Create the Lambda invocation role (self-hosted) -Temporal invokes Lambda functions by assuming an IAM role in your AWS account. This role needs `lambda:GetFunction` and `lambda:InvokeFunction` permission on your Worker Lambda functions, and a trust policy that allows the Temporal server's identity to assume it. +Temporal invokes Lambda functions by assuming an IAM role in your AWS account. This role needs `lambda:GetFunction` and `lambda:InvokeFunction` permission on your Worker Lambda functions, and a trust policy that allows the Temporal server's identity to assume it. -This skill ships the complete self-hosted template at `assets/temporal-self-hosted-serverless-worker-role.yaml` (verbatim from the docs). Copy it locally or point `--template-body` at the skill's copy. +This skill ships the complete self-hosted template at `assets/temporal-self-hosted-serverless-worker-role.yaml` (verbatim from the docs). Copy it locally or point `--template-body` at the skill's copy. ```bash aws cloudformation create-stack \ @@ -240,7 +233,6 @@ aws cloudformation create-stack \ --capabilities CAPABILITY_NAMED_IAM \ --region ``` - | Parameter | Description | |---|---| @@ -248,9 +240,8 @@ aws cloudformation create-stack \ | `AssumeRoleExternalId` | A unique string to prevent confused deputy attacks. Use the same value when creating the Worker Deployment Version. | | `LambdaFunctionARNs` | Comma-separated list of Lambda function ARNs that Temporal may invoke. To allow any published version, use a wildcard suffix (for example, `arn:aws:lambda:...:function:my-temporal-worker:*`). | | `RoleName` | Base name for the created IAM role. Defaults to `Temporal-Serverless-Worker`. Provide a new role name if creating more than one stack. | - -Retrieve the role ARN: +Retrieve the role ARN: ```bash aws cloudformation describe-stacks \ @@ -259,6 +250,5 @@ aws cloudformation describe-stacks \ --output text \ --region ``` - -**Key distinction:** The Lambda execution role (trusted by `lambda.amazonaws.com`) is separate from the Temporal invocation role (trusted by Temporal's `wci-lambda-invoke` principals for Cloud, or the Temporal Service's own IAM identity for self-hosted). The execution role grants the function permission to run. The invocation role grants Temporal permission to invoke the function. +**Key distinction:** The Lambda execution role (trusted by `lambda.amazonaws.com`) is separate from the Temporal invocation role (trusted by Temporal's `wci-lambda-invoke` principals for Cloud, or the Temporal Service's own IAM identity for self-hosted). The execution role grants the function permission to run. The invocation role grants Temporal permission to invoke the function. diff --git a/references/aws-lambda/observability.md b/references/aws-lambda/observability.md index 0492608..4d2155f 100644 --- a/references/aws-lambda/observability.md +++ b/references/aws-lambda/observability.md @@ -2,7 +2,7 @@ ## Overview -Each SDK provides an OpenTelemetry integration package with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. When enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. +Each SDK provides an OpenTelemetry integration package with defaults configured for the AWS Distro for OpenTelemetry (ADOT) Lambda layer. When enabled, the Worker emits SDK metrics and distributed traces for Workflow and Activity executions. The ADOT Lambda layer collects this telemetry and can forward traces to AWS X-Ray and metrics to Amazon CloudWatch. Load the selected SDK reference's **Observability** section: @@ -18,9 +18,9 @@ The remaining steps in this file are shared across SDKs. ## Custom Collector configuration required -The default ADOT Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. +The default ADOT Collector configuration does not route OpenTelemetry Protocol (OTLP) data to the traces pipeline. You must provide a custom Collector configuration that wires the OTLP receiver to both the traces and metrics pipelines. -Example `otel-collector-config.yaml` (bundle in your Lambda deployment package): +Example `otel-collector-config.yaml` (bundle in your Lambda deployment package): For the Collector configuration environment variable, see the selected SDK reference. @@ -59,7 +59,6 @@ service: metrics: address: localhost:8888 ``` - ## Enable X-Ray active tracing @@ -68,16 +67,15 @@ aws lambda update-function-configuration \ --function-name \ --tracing-config Mode=Active ``` - ## Required IAM permissions -The Lambda execution role must have permissions to write to X-Ray and CloudWatch: +The Lambda execution role must have permissions to write to X-Ray and CloudWatch: -- `xray:PutTraceSegments` -- `xray:PutTelemetryRecords` -- `cloudwatch:PutMetricData` +- `xray:PutTraceSegments` +- `xray:PutTelemetryRecords` +- `cloudwatch:PutMetricData` -Without these permissions, the Collector fails silently and no telemetry appears. +Without these permissions, the Collector fails silently and no telemetry appears. For language-specific IAM notes, see the selected SDK reference. diff --git a/references/aws-lambda/sdk-dotnet.md b/references/aws-lambda/sdk-dotnet.md index 400d28d..86d48f1 100644 --- a/references/aws-lambda/sdk-dotnet.md +++ b/references/aws-lambda/sdk-dotnet.md @@ -1,6 +1,5 @@ # .NET SDK on AWS Lambda -Sources: [`Temporalio.Extensions.Aws.Lambda` 1.18.0](https://www.nuget.org/packages/Temporalio.Extensions.Aws.Lambda/1.18.0), [Lambda extension source](https://github.com/temporalio/sdk-dotnet/tree/1.18.0/src/Temporalio.Extensions.Aws.Lambda), [OpenTelemetry extension source](https://github.com/temporalio/sdk-dotnet/tree/1.18.0/src/Temporalio.Extensions.Aws.Lambda.OpenTelemetry), and the [maintained Lambda Worker sample](https://github.com/temporalio/samples-dotnet/tree/6aba4fb9ea08177e303352ec9a4c61e303cefb0e/src/LambdaWorker). Use this reference for .NET SDK-specific package, entry-point, Worker configuration, tuned defaults, observability, and diagnostic details. For shared AWS Lambda execution constraints, deployment, observability infrastructure, and diagnostic flow, see `constraints.md`, `setup.md`, `observability.md`, and `diagnostics.md`. @@ -85,7 +84,6 @@ Registrations go through `config.WorkerOptions`, an ordinary `TemporalWorkerOpti ## Lambda-tuned defaults - | Setting | Lambda default | |---|---| diff --git a/references/aws-lambda/sdk-go.md b/references/aws-lambda/sdk-go.md index 0bfa64c..14d1837 100644 --- a/references/aws-lambda/sdk-go.md +++ b/references/aws-lambda/sdk-go.md @@ -1,16 +1,15 @@ # Go SDK on AWS Lambda - Use this reference for Go SDK-specific package, entry-point, Worker configuration, tuned defaults, and observability details. For shared AWS Lambda execution constraints, deployment, and observability infrastructure, see `constraints.md`, `setup.md`, and `observability.md`. ## Package -Import: `lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"` +Import: `lambdaworker "go.temporal.io/sdk/contrib/aws/lambdaworker"` Install: `go get go.temporal.io/sdk/contrib/aws/lambdaworker` — **this is a separate Go module** from `go.temporal.io/sdk`, versioned independently (`v0.1.1` at the time of writing). Having the main SDK in `go.mod` does not make it importable; add it explicitly, then `go mod tidy`. Verify the installed surface with `go doc go.temporal.io/sdk/contrib/aws/lambdaworker` before generating code — the API is Public Preview and drifts. -- Go: [Go Lambda Worker sample](https://github.com/temporalio/samples-go/tree/main/lambda-worker) +- Go: [Go Lambda Worker sample](https://github.com/temporalio/samples-go/tree/main/lambda-worker) List the exported API of the installed module version before generating code: @@ -23,11 +22,11 @@ go doc go.temporal.io/sdk/contrib/aws/lambdaworker.Options ## Entry point -`lambdaworker.RunWorker` — starts a Lambda-based Worker. Pass a `WorkerDeploymentVersion` and a callback that registers Workflows and Activities. +`lambdaworker.RunWorker` — starts a Lambda-based Worker. Pass a `WorkerDeploymentVersion` and a callback that registers Workflows and Activities. ## Configure callback -The `Options` callback gives access to the same registration methods as a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`. +The `Options` callback gives access to the same registration methods as a traditional Worker: `RegisterWorkflow`, `RegisterWorkflowWithOptions`, `RegisterActivity`, `RegisterActivityWithOptions`, and `RegisterNexusService`. Go, Python and TypeScript invoke it per invocation. @@ -35,14 +34,14 @@ In Go it is a direct field on the options object (`opts.TaskQueue`). ## Versioning behavior -Set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`. -Or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. +Set per-Workflow at registration time with `workflow.VersioningBehaviorPinned` or `workflow.VersioningBehaviorAutoUpgrade`. +Or set a Worker-level default with `DefaultVersioningBehavior` in `DeploymentOptions`. **Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). ## Handler example -Use the Go SDK's `lambdaworker` package. +Use the Go SDK's `lambdaworker` package. ```go package main @@ -69,11 +68,9 @@ func main() { }) } ``` - ## Lambda-tuned defaults - | Setting | Lambda default | |---|---| @@ -91,41 +88,39 @@ func main() { Note: Go sticky cache size is 100, while Python and TypeScript are 30. These values come from each SDK's own docs and are not interchangeable. -These are the same `worker.Options` available to any Temporal Worker, just with lower values for Lambda's constrained environment. Except for `ShutdownDeadlineBuffer`, which is specific to the `lambdaworker` package. +These are the same `worker.Options` available to any Temporal Worker, just with lower values for Lambda's constrained environment. Except for `ShutdownDeadlineBuffer`, which is specific to the `lambdaworker` package. -`DisableEagerActivities` is always true and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. +`DisableEagerActivities` is always true and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. -`ShutdownDeadlineBuffer` controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `WorkerStopTimeout` + 2 seconds. +`ShutdownDeadlineBuffer` controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `WorkerStopTimeout` + 2 seconds. -If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. +If your Worker handles long-running Activities, increase `WorkerStopTimeout`, `ShutdownDeadlineBuffer`, and the Lambda invocation deadline (`--timeout`) together. ## Connection configuration -The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). +The `lambdaworker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). -TOML config file resolution order: +TOML config file resolution order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. -The file is optional. If absent, only environment variables are used. +The file is optional. If absent, only environment variables are used. ## Build and package -Cross-compile for Lambda's Linux runtime: +Cross-compile for Lambda's Linux runtime: ```bash GOOS=linux GOARCH=amd64 go build -tags lambda.norpc -o bootstrap ./worker ``` - -Package the binary into a zip file: +Package the binary into a zip file: ```bash zip function.zip bootstrap ``` - **Add `CGO_ENABLED=0`, and match the architecture you deploy.** The `provided.al2023` runtime expects a self-contained binary; building with cgo enabled links against host libraries that may not resolve inside the runtime. Set `CGO_ENABLED=0` for a statically linked binary, and keep `GOARCH` consistent with the function's `--architectures` (`amd64` ↔ `x86_64`, `arm64` ↔ `arm64`). Also adjust the trailing package path to your layout — `.` when `main` is in the repo root, `./worker` when it is in a `worker/` subdirectory. A reusable script: @@ -157,10 +152,9 @@ aws lambda create-function \ --memory-size 256 \ --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' ``` - -- `--runtime`: `provided.al2023` for custom Go binaries. -- `--handler`: `bootstrap` when using the `provided.al2023` custom runtime. +- `--runtime`: `provided.al2023` for custom Go binaries. +- `--handler`: `bootstrap` when using the `provided.al2023` custom runtime. | Variable | Description | |---|---| @@ -168,13 +162,13 @@ aws lambda create-function \ ## Observability -Import: `otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel"` +Import: `otel "go.temporal.io/sdk/contrib/aws/lambdaworker/otel"` -- `otel.ApplyDefaults` — configures both metrics and tracing. -- `otel.ApplyMetrics` — configures metrics only. -- `otel.ApplyTracing` — configures tracing only. +- `otel.ApplyDefaults` — configures both metrics and tracing. +- `otel.ApplyMetrics` — configures metrics only. +- `otel.ApplyTracing` — configures tracing only. -Usage in the configure callback: +Usage in the configure callback: ```go if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != nil { @@ -182,11 +176,11 @@ if err := otel.ApplyDefaults(opts, &opts.ClientOptions, otel.Options{}); err != } ``` -By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. -Attach the ADOT Collector layer to your Lambda function. -Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary. +Attach the ADOT Collector layer to your Lambda function. +Go does not need a language-specific ADOT layer because the OTel SDK is compiled into the binary. -`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. diff --git a/references/aws-lambda/sdk-java.md b/references/aws-lambda/sdk-java.md index 7967e9a..dbf4e66 100644 --- a/references/aws-lambda/sdk-java.md +++ b/references/aws-lambda/sdk-java.md @@ -1,12 +1,11 @@ # Java SDK on AWS Lambda - Use this reference for Java SDK-specific package, entry-point, Worker configuration, tuned defaults, observability, and diagnostic details. For shared AWS Lambda execution constraints, deployment, observability infrastructure, and diagnostic flow, see `constraints.md`, `setup.md`, `observability.md`, and `diagnostics.md`. ## Package -Import: `io.temporal.aws.lambda.LambdaWorker`, `io.temporal.aws.lambda.LambdaWorkerOptions`, `io.temporal.common.WorkerDeploymentVersion` +Import: `io.temporal.aws.lambda.LambdaWorker`, `io.temporal.aws.lambda.LambdaWorkerOptions`, `io.temporal.common.WorkerDeploymentVersion` Install: `io.temporal:temporal-aws-lambda` — a **separate Maven artifact** from `io.temporal:temporal-sdk`, but published on the **same version line** (both 1.38.0). This is a third packaging pattern: unlike Go and TypeScript it is not independently versioned, and unlike Python it does not ship inside the main SDK. Use `io.temporal:temporal-bom` in `dependencyManagement` to keep them aligned. @@ -23,7 +22,7 @@ Install: `io.temporal:temporal-aws-lambda` — a **separate Maven artifact** fro `aws-lambda-java-core` (1.4.0) arrives transitively from `temporal-aws-lambda`; declare it explicitly if you compile against `RequestHandler`/`Context`. -- Java: [Java Lambda Worker sample](https://github.com/temporalio/samples-java/tree/main/lambda-worker) — three Gradle subprojects (`worker/` handler + greeting Workflow/Activity, `starter/` local client, `deploy/` IAM and deploy scripts plus a CloudFormation template) +- Java: [Java Lambda Worker sample](https://github.com/temporalio/samples-java/tree/main/lambda-worker) — three Gradle subprojects (`worker/` handler + greeting Workflow/Activity, `starter/` local client, `deploy/` IAM and deploy scripts plus a CloudFormation template) List the real public API of the resolved artifact before generating code: @@ -41,7 +40,7 @@ javap -cp 'io.temporal.aws.lambda.LambdaWorkerOptions$Builder' **`LambdaWorker.define(version, configure)`** — returns a `RequestHandler` that your handler class delegates to. There are four public overloads: `define` (2- and 3-arg) and `newHandler` (2- and 3-arg, taking a pre-built `LambdaWorkerOptions`). -Note that Java's entry point is not "run"-shaped like the other SDKs' (`RunWorker`, `run_worker`, `runWorker`) — confirm the method name against the version you install. +Note that Java's entry point is not "run"-shaped like the other SDKs' (`RunWorker`, `run_worker`, `runWorker`) — confirm the method name against the version you install. ## Configure callback — two phases, unlike the other SDKs @@ -82,7 +81,7 @@ public class MyWorkflowImpl implements MyWorkflow { ## Handler example -Use the `temporal-aws-lambda` module. The handler class implements `RequestHandler` and delegates to the handler returned by `LambdaWorker.define`. +Use the `temporal-aws-lambda` module. The handler class implements `RequestHandler` and delegates to the handler returned by `LambdaWorker.define`. ```java package com.example.temporal; @@ -117,7 +116,6 @@ The entry point is `define` (or `newHandler` for pre-built options) — not a "r ## Lambda-tuned defaults - | Setting | Lambda default | |---|---| @@ -170,7 +168,7 @@ Resolution order (`LambdaWorkerOptions.resolveConfigFilePath`): Build an uber-jar with all dependencies bundled. A JAR is a valid zip, so it uploads directly with no extra packaging step. -**Gradle** (what the official sample uses): `./gradlew shadowJar` → `build/libs/-all.jar`. +**Gradle** (what the official sample uses): `./gradlew shadowJar` → `build/libs/-all.jar`. **Maven**: `maven-shade-plugin`, bound to `package` → `target/.jar`. @@ -239,13 +237,12 @@ aws lambda create-function \ --environment file:///tmp/lambda-env.json ``` -- `--runtime`: `java17` (or another supported Java version). -- `--handler`: `fully.qualified.Class::method` — **a different format from every other SDK**, which use `module.function` / `module.export`. Point it at the method that delegates to the `LambdaWorker.define` handler. +- `--runtime`: `java17` (or another supported Java version). +- `--handler`: `fully.qualified.Class::method` — **a different format from every other SDK**, which use `module.function` / `module.export`. Point it at the method that delegates to the `LambdaWorker.define` handler. - `--zip-file`: the shaded jar directly; no separate zip step. Switch to `--code S3Bucket=…,S3Key=…` once the jar exceeds 50 MB, which happens early in Java (see packaging above). - **`HOME=/tmp` is not needed** — unlike the Go and TypeScript examples. Verified: the Java module never reads `HOME`, and a missing config file is non-fatal. → Connection configuration above. -- `--memory-size`: the docs recommend starting at `1024` because "Java Workers typically need more memory than other runtimes," then adjusting from CloudWatch. A measured hello-world used **240 MB of 1024** (`Max Memory Used` in the invocation's REPORT line), so `512` is usually ample for small Workers — and since Lambda bills GB-seconds, halving memory halves the bill. Start at 1024, read the metric, then cut. +- `--memory-size`: the docs recommend starting at `1024` because "Java Workers typically need more memory than other runtimes," then adjusting from CloudWatch. A measured hello-world used **240 MB of 1024** (`Max Memory Used` in the invocation's REPORT line), so `512` is usually ample for small Workers — and since Lambda bills GB-seconds, halving memory halves the bill. Start at 1024, read the metric, then cut. - **`--timeout` is a cost setting once it clears startup.** The per-SDK examples differ deliberately — 600 for Go, Python and TypeScript; 90 for Java — and both clear startup easily: a measured Java Worker bound its Task Queue **~10s** after `create-version`, JVM cold start included, against the 83s of polling a 90s deadline allows. Lambda's 3-second default is what fails this; 90 does not. @@ -259,11 +256,10 @@ Measured cold starts are ~1s for Python and Java alike (`Init Duration` in the R ## Observability -No extra dependency is required: `temporal-aws-lambda` already depends on `io.temporal:temporal-opentelemetry`, `io.opentelemetry:opentelemetry-api` (BOM 1.25.0), and `io.opentelemetry.contrib:opentelemetry-aws-xray`. The helper class ships inside the module. +No extra dependency is required: `temporal-aws-lambda` already depends on `io.temporal:temporal-opentelemetry`, `io.opentelemetry:opentelemetry-api` (BOM 1.25.0), and `io.opentelemetry.contrib:opentelemetry-aws-xray`. The helper class ships inside the module. Import: `io.temporal.aws.lambda.OtelLambdaWorkerConfigurationHelper` - - `configure(LambdaWorkerOptions.Builder)` — configures metrics and tracing with defaults. - `configure(LambdaWorkerOptions.Builder, Consumer)` — same, with customization. @@ -288,7 +284,7 @@ Defaults come from the constants `DEFAULT_OTLP_ENDPOINT` and `DEFAULT_SERVICE_NA **Flush before the deadline.** A Serverless Worker's invocation ends on a deadline rather than after a request, so telemetry buffered past that point is lost. Use `configureFlushHook` (or a report interval shorter than the invocation deadline) so metrics and spans are exported before shutdown. This matters more on Java's shorter recommended deadline (90s) than on the 600s used elsewhere. -Attach the ADOT Collector layer. Because the OpenTelemetry SDK arrives as an ordinary Maven dependency of `temporal-aws-lambda`, no language-specific auto-instrumentation layer is required for the Worker's own telemetry — the same situation as Go. +Attach the ADOT Collector layer. Because the OpenTelemetry SDK arrives as an ordinary Maven dependency of `temporal-aws-lambda`, no language-specific auto-instrumentation layer is required for the Worker's own telemetry — the same situation as Go. `OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` — the `_URI` form, as with Go and TypeScript. The official Java sample packages `otel-collector-config.template.yaml` into the artifact root as `otel-collector-config.yaml` during `shadowJar`; with Maven, add it under `src/main/resources`. diff --git a/references/aws-lambda/sdk-python.md b/references/aws-lambda/sdk-python.md index 6ff2bf6..dd15aa8 100644 --- a/references/aws-lambda/sdk-python.md +++ b/references/aws-lambda/sdk-python.md @@ -1,16 +1,15 @@ # Python SDK on AWS Lambda - Use this reference for Python SDK-specific package, entry-point, Worker configuration, tuned defaults, observability, and diagnostic details. For shared AWS Lambda execution constraints, deployment, observability infrastructure, and diagnostic flow, see `constraints.md`, `setup.md`, `observability.md`, and `diagnostics.md`. ## Package -Import: `from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker` +Import: `from temporalio.contrib.aws.lambda_worker import LambdaWorkerConfig, run_worker` Install: `pip install temporalio` — the contrib module ships inside the main package here (unlike Go and TypeScript, which need a separate dependency). Use `temporalio[lambda-worker-otel]` for OpenTelemetry support. -- Python: [Python Lambda Worker sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker) +- Python: [Python Lambda Worker sample](https://github.com/temporalio/samples-python/tree/main/lambda_worker) Read the installed API before generating code: @@ -24,11 +23,11 @@ python -c "import temporalio.contrib.aws.lambda_worker as m; help(m.LambdaWorker ## Entry point -`run_worker` — takes a `WorkerDeploymentVersion` and a configure callback, returns a Lambda handler. +`run_worker` — takes a `WorkerDeploymentVersion` and a configure callback, returns a Lambda handler. ## Configure callback -The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults. Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor. +The `configure` callback receives a `LambdaWorkerConfig` dataclass with fields pre-populated with Lambda-appropriate defaults. Set the Task Queue, Workflows, and Activities through `worker_config`, which accepts the same keyword arguments as the `Worker` constructor. Go, Python and TypeScript invoke it per invocation. @@ -36,14 +35,14 @@ Python and TypeScript pass Workflow and Activity collections into the worker con ## Versioning behavior -Set per-Workflow in the `@workflow.defn` decorator: `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`. -Or set a Worker-level default with `default_versioning_behavior` in the worker config. +Set per-Workflow in the `@workflow.defn` decorator: `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`. +Or set a Worker-level default with `default_versioning_behavior` in the worker config. **Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). ## Handler example -Use the Python SDK's `lambda_worker` contrib package. +Use the Python SDK's `lambda_worker` contrib package. ```python from temporalio.common import WorkerDeploymentVersion @@ -67,9 +66,8 @@ lambda_handler = run_worker( configure, ) ``` - -Versioning behavior: set per-Workflow in the `@workflow.defn` decorator with `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`, or set a Worker-level default with `default_versioning_behavior` in the worker config. +Versioning behavior: set per-Workflow in the `@workflow.defn` decorator with `VersioningBehavior.PINNED` or `VersioningBehavior.AUTO_UPGRADE`, or set a Worker-level default with `default_versioning_behavior` in the worker config. ```python from temporalio import workflow @@ -82,11 +80,9 @@ class MyWorkflow: async def run(self, input: str) -> str: ... ``` - ## Lambda-tuned defaults - | Setting | Lambda default | |---|---| @@ -102,44 +98,42 @@ class MyWorkflow: | `disable_eager_activity_execution` | Always `True` | | `shutdown_deadline_buffer` | 7 seconds | -`disable_eager_activity_execution` is always `True` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. +`disable_eager_activity_execution` is always `True` and cannot be overridden. Eager Activities require a persistent connection, which Lambda invocations don't maintain. -`shutdown_deadline_buffer` is specific to the `lambda_worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `graceful_shutdown_timeout` + 2 seconds. +`shutdown_deadline_buffer` is specific to the `lambda_worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `graceful_shutdown_timeout` + 2 seconds. -If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together. +If your Worker handles long-running Activities, increase `graceful_shutdown_timeout`, `shutdown_deadline_buffer`, and the Lambda invocation deadline (`--timeout`) together. ## Connection configuration -The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). +The `lambda_worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). -TOML config file resolution order: +TOML config file resolution order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. -The file is optional. If absent, only environment variables are used. +The file is optional. If absent, only environment variables are used. ## Build and package -Install dependencies into a local directory for packaging, using `--platform` for Linux-compatible binaries: +Install dependencies into a local directory for packaging, using `--platform` for Linux-compatible binaries: ```bash pip install --target ./package --platform manylinux2014_x86_64 --only-binary=:all: temporalio ``` - **Pin the download to the Lambda runtime's Python version and architecture, not your local interpreter's.** If they differ (e.g. local `3.14` vs the function's `python3.13`), add `--python-version 3.13` alongside `--only-binary=:all:` so pip fetches runtime-matching wheels, and keep `--platform` (`manylinux2014_x86_64` for `x86_64`, `manylinux2014_aarch64` for `arm64`) consistent with the function's `--architectures`. Mismatches surface as import errors only at invocation time, not at package time. -To include OpenTelemetry support, install `temporalio[lambda-worker-otel]` instead. +To include OpenTelemetry support, install `temporalio[lambda-worker-otel]` instead. -Package dependencies and application code: +Package dependencies and application code: ```bash cd package && zip -r ../function.zip . && cd .. zip function.zip lambda_function.py my_workflows.py my_activities.py ``` - ## Deploy the Lambda function @@ -154,10 +148,9 @@ aws lambda create-function \ --memory-size 256 \ --environment '{"Variables":{"TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' ``` - -- `--runtime`: `python3.13` (or another supported Python version). -- `--handler`: `lambda_function.lambda_handler` (entry point in `module.function` format, must point to the handler returned by `run_worker`). +- `--runtime`: `python3.13` (or another supported Python version). +- `--handler`: `lambda_function.lambda_handler` (entry point in `module.function` format, must point to the handler returned by `run_worker`). | SDK example | Memory | Full invocation | GB-seconds | |---|---|---|---| @@ -169,15 +162,15 @@ The `--environment` examples above pass `TEMPORAL_API_KEY` inline for brevity ## Observability -Import: `from temporalio.contrib.aws.lambda_worker.otel import apply_defaults` +Import: `from temporalio.contrib.aws.lambda_worker.otel import apply_defaults` -To install with OTel support: `pip install temporalio[lambda-worker-otel]` +To install with OTel support: `pip install temporalio[lambda-worker-otel]` -- `apply_defaults` — configures both metrics and tracing. -- `build_metrics_telemetry_config` — configures metrics only. -- `apply_tracing` — configures tracing only. +- `apply_defaults` — configures both metrics and tracing. +- `build_metrics_telemetry_config` — configures metrics only. +- `apply_tracing` — configures tracing only. -Usage in the configure callback: +Usage in the configure callback: ```python def configure(config: LambdaWorkerConfig) -> None: @@ -187,15 +180,15 @@ def configure(config: LambdaWorkerConfig) -> None: apply_defaults(config) ``` -By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. -Attach the ADOT Python Lambda layer to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch. +Attach the ADOT Python Lambda layer to your Lambda function. The layer includes both auto-instrumentation and an OpenTelemetry Collector that receives telemetry on `localhost:4317` and forwards traces to AWS X-Ray and metrics to Amazon CloudWatch. -`OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` +`OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/otel-collector-config.yaml` Note: Python uses `_FILE` while Go and TypeScript use `_URI`. -For Python, the `AWSXRayDaemonWriteAccess` managed policy can be attached instead. +For Python, the `AWSXRayDaemonWriteAccess` managed policy can be attached instead. For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. diff --git a/references/aws-lambda/sdk-typescript.md b/references/aws-lambda/sdk-typescript.md index 14f2326..f8636f1 100644 --- a/references/aws-lambda/sdk-typescript.md +++ b/references/aws-lambda/sdk-typescript.md @@ -1,16 +1,15 @@ # TypeScript SDK on AWS Lambda - Use this reference for TypeScript SDK-specific package, entry-point, Worker configuration, tuned defaults, and observability details. For shared AWS Lambda execution constraints, deployment, and observability infrastructure, see `constraints.md`, `setup.md`, and `observability.md`. ## Package -Import: `import { runWorker } from '@temporalio/lambda-worker'` +Import: `import { runWorker } from '@temporalio/lambda-worker'` Install: `npm install @temporalio/lambda-worker` — a separate npm package from `@temporalio/worker`, versioned independently. -- TypeScript: [TypeScript Lambda Worker sample](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker) +- TypeScript: [TypeScript Lambda Worker sample](https://github.com/temporalio/samples-typescript/tree/main/lambda-worker) Check the installed version, then read its type declarations: @@ -22,11 +21,11 @@ npm ls @temporalio/lambda-worker ## Entry point -`runWorker` — creates a Lambda handler that runs a Temporal Worker. Pass a deployment version and a configure callback. +`runWorker` — creates a Lambda handler that runs a Temporal Worker. Pass a deployment version and a configure callback. ## Configure callback -Set Worker options via `config.workerOptions`. For Workflow code, use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. +Set Worker options via `config.workerOptions`. For Workflow code, use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. Go, Python and TypeScript invoke it per invocation. @@ -34,7 +33,7 @@ Python and TypeScript pass Workflow and Activity collections into the worker con ## Pre-bundling Workflow code -Build the bundle as a separate build step: +Build the bundle as a separate build step: ```typescript import { bundleWorkflowCode } from '@temporalio/worker'; @@ -45,22 +44,21 @@ const { code } = await bundleWorkflowCode({ }); await writeFile('./workflow-bundle.js', code); ``` - -Then reference the bundle in your handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`. +Then reference the bundle in your handler with `workflowBundle: { codePath: require.resolve('./workflow-bundle.js') }`. ## Versioning behavior -Set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. -Values are `'PINNED'` or `'AUTO_UPGRADE'`. The default versioning behavior is `PINNED`. +Set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. +Values are `'PINNED'` or `'AUTO_UPGRADE'`. The default versioning behavior is `PINNED`. -Access via: `config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'` +Access via: `config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'` **Worker Versioning is always on.** The run-worker entry point enables it, so the only remaining decision is `Pinned` vs `AutoUpgrade` per Workflow (or a Worker-level default). ## Handler example -Use the `@temporalio/lambda-worker` package. +Use the `@temporalio/lambda-worker` package. ```typescript import { runWorker } from '@temporalio/lambda-worker'; @@ -75,15 +73,13 @@ export const handler = runWorker({ deploymentName: 'my-app', buildId: 'build-1' config.workerOptions.workerDeploymentOptions!.defaultVersioningBehavior = 'PINNED'; }); ``` - -Use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. +Use `workflowBundle` with pre-bundled code instead of `workflowsPath` to avoid webpack bundling overhead on Lambda cold starts. -Versioning behavior: set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. Values are `'AUTO_UPGRADE'` or `'PINNED'`. +Versioning behavior: set per-Workflow with `setWorkflowOptions` in the Workflow file, or set a default for all Workflows with `defaultVersioningBehavior` in the configure callback. Values are `'AUTO_UPGRADE'` or `'PINNED'`. ## Lambda-tuned defaults - | Setting | Lambda default | |---|---| @@ -98,41 +94,39 @@ Versioning behavior: set per-Workflow with `setWorkflowOptions` in the Workflow | `maxCachedWorkflows` | 30 | | `shutdownDeadlineBufferMs` | 7000 | -Eager Activities are not supported. Lambda invocations don't maintain persistent connections. +Eager Activities are not supported. Lambda invocations don't maintain persistent connections. -`shutdownDeadlineBufferMs` is specific to the `@temporalio/lambda-worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `shutdownGraceTime` (5s) + 2s. +`shutdownDeadlineBufferMs` is specific to the `@temporalio/lambda-worker` package. It controls how much time before the Lambda deadline the Worker begins its graceful shutdown. The default is `shutdownGraceTime` (5s) + 2s. -If your Worker handles long-running Activities, increase `shutdownGraceTime`, `shutdownDeadlineBufferMs`, and the Lambda invocation deadline (`--timeout`) together. +If your Worker handles long-running Activities, increase `shutdownGraceTime`, `shutdownDeadlineBufferMs`, and the Lambda invocation deadline (`--timeout`) together. ## Connection configuration -The `@temporalio/lambda-worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). +The `@temporalio/lambda-worker` package automatically loads Temporal client configuration from a TOML config file and environment variables (see the Environment Configuration docs, `/develop/environment-configuration`). -TOML config file resolution order: +TOML config file resolution order: 1. `TEMPORAL_CONFIG_FILE` environment variable, if set. 2. `temporal.toml` in `$LAMBDA_TASK_ROOT` (typically `/var/task`). 3. `temporal.toml` in the current working directory. -The file is optional. If absent, only environment variables are used. +The file is optional. If absent, only environment variables are used. ## Build and package -Build the Workflow bundle and compile the project: +Build the Workflow bundle and compile the project: ```bash npx ts-node src/scripts/build-workflow-bundle.ts npx tsc ``` - -Install production dependencies and package everything: +Install production dependencies and package everything: ```bash npm install --omit=dev zip -r function.zip lib/ node_modules/ workflow-bundle.js ``` - ## Deploy the Lambda function @@ -147,10 +141,9 @@ aws lambda create-function \ --memory-size 256 \ --environment '{"Variables":{"HOME":"/tmp","TEMPORAL_ADDRESS":":7233","TEMPORAL_NAMESPACE":"","TEMPORAL_API_KEY":""}}' ``` - -- `--runtime`: `nodejs22.x` (or another supported Node.js version, 20+). -- `--handler`: `lib/index.handler` (entry point in `module.export` format, must point to the handler exported by `runWorker`). +- `--runtime`: `nodejs22.x` (or another supported Node.js version, 20+). +- `--handler`: `lib/index.handler` (entry point in `module.export` format, must point to the handler exported by `runWorker`). | Variable | Description | |---|---| @@ -158,12 +151,12 @@ aws lambda create-function \ ## Observability -Import: `import { applyDefaults } from '@temporalio/lambda-worker/otel'` +Import: `import { applyDefaults } from '@temporalio/lambda-worker/otel'` -- `applyDefaults` — registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OTLP. -- `makeOtelPlugin` — returns a plugin for pre-bundling Workflow code that includes Workflow interceptor modules. +- `applyDefaults` — registers Temporal SDK interceptors for tracing and configures the Core SDK to export metrics via OTLP. +- `makeOtelPlugin` — returns a plugin for pre-bundling Workflow code that includes Workflow interceptor modules. -Usage in the configure callback: +Usage in the configure callback: ```typescript export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, (config) => { @@ -176,7 +169,7 @@ export const handler = runWorker({ deploymentName: 'sdk-demo', buildId: 'v1' }, }); ``` -When pre-bundling Workflow code, pass the plugin from `makeOtelPlugin()` so that Workflow interceptor modules are included in the bundle: +When pre-bundling Workflow code, pass the plugin from `makeOtelPlugin()` so that Workflow interceptor modules are included in the bundle: ```typescript import { bundleWorkflowCode } from '@temporalio/worker'; @@ -189,13 +182,13 @@ const { code } = await bundleWorkflowCode({ }); ``` -By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. +By default, telemetry is sent to `localhost:4317`, which is the ADOT Lambda layer's default collector endpoint. -Attach two ADOT Lambda layers: +Attach two ADOT Lambda layers: -1. The ADOT JavaScript layer for Node.js-side auto-instrumentation and trace export. -2. The ADOT Collector layer (`aws-otel-collector-amd64`) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch. +1. The ADOT JavaScript layer for Node.js-side auto-instrumentation and trace export. +2. The ADOT Collector layer (`aws-otel-collector-amd64`) to run the OTel Collector as a Lambda extension, receiving telemetry via OTLP on `localhost:4317` and forwarding traces to X-Ray and metrics to CloudWatch. -`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` +`OPENTELEMETRY_COLLECTOR_CONFIG_URI=/var/task/otel-collector-config.yaml` For the shared Collector configuration, X-Ray enablement, and execution-role permissions, see `observability.md`. diff --git a/references/aws-lambda/self-hosted.md b/references/aws-lambda/self-hosted.md index d6508e2..3ebde52 100644 --- a/references/aws-lambda/self-hosted.md +++ b/references/aws-lambda/self-hosted.md @@ -1,22 +1,19 @@ # AWS Lambda — Self-hosted Temporal Service - -Self-hosted Serverless Workers require Temporal Service v1.31.0 or later. +Self-hosted Serverless Workers require Temporal Service v1.31.0 or later. Complete this server enablement before following `setup.md`. For the Lambda invocation role Temporal assumes (the CloudFormation stack and trust policy), see `iam.md`. ## Network reachability -The Temporal Service frontend must be reachable from the Lambda execution environment. If the Temporal Service runs on a private network, you may need VPC access for Lambda, VPC peering, or a similar mechanism. +The Temporal Service frontend must be reachable from the Lambda execution environment. If the Temporal Service runs on a private network, you may need VPC access for Lambda, VPC peering, or a similar mechanism. ## Enable the Worker Controller Instance (WCI) -WCI is disabled by default and must be enabled through dynamic configuration. +WCI is disabled by default and must be enabled through dynamic configuration. -Add the following keys to your dynamic config file: +Add the following keys to your dynamic config file: ```yaml workercontroller.enabled: @@ -30,9 +27,8 @@ workercontroller.scaling_algorithms.enabled: - value: - no-sync ``` - -To enable WCI for specific Namespaces instead of globally, add a `constraints` section with the Namespace name under `workercontroller.enabled`: +To enable WCI for specific Namespaces instead of globally, add a `constraints` section with the Namespace name under `workercontroller.enabled`: ```yaml workercontroller.enabled: @@ -40,21 +36,19 @@ workercontroller.enabled: constraints: namespace: 'your-namespace' ``` - -The Temporal Service watches the dynamic config file for changes and applies updates without a restart. +The Temporal Service watches the dynamic config file for changes and applies updates without a restart. ## Configure AWS credentials -The Temporal Service needs AWS credentials to assume an IAM role that invokes Lambda functions. +The Temporal Service needs AWS credentials to assume an IAM role that invokes Lambda functions. -**On AWS infrastructure (EC2, ECS, EKS):** The server uses the attached instance role, task role, or pod role automatically. No additional credential configuration is needed. The attached role must have `sts:AssumeRole` permission for the Lambda invocation role. +**On AWS infrastructure (EC2, ECS, EKS):** The server uses the attached instance role, task role, or pod role automatically. No additional credential configuration is needed. The attached role must have `sts:AssumeRole` permission for the Lambda invocation role. -**Outside AWS:** Use IAM Roles Anywhere, or configure static AWS credentials in the server's environment (not recommended). These credentials must belong to an IAM user or role that has `sts:AssumeRole` permission for the Lambda invocation role. +**Outside AWS:** Use IAM Roles Anywhere, or configure static AWS credentials in the server's environment (not recommended). These credentials must belong to an IAM user or role that has `sts:AssumeRole` permission for the Lambda invocation role. ``` AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= AWS_REGION= ``` - diff --git a/references/aws-lambda/setup.md b/references/aws-lambda/setup.md index f732926..a29ffcd 100644 --- a/references/aws-lambda/setup.md +++ b/references/aws-lambda/setup.md @@ -1,29 +1,24 @@ # AWS Lambda — Setup (happy path) - This is the end-to-end golden path: connect, write the Worker, package and deploy, register a Worker Deployment Version, set it current, and verify. For the Lambda execution model and limits, see `constraints.md`. For the operator permissions and preflight, execution/invocation roles, and CloudFormation, see `iam.md`. For production build versioning (`publish-version`, qualified ARNs, rollback), see `versioning.md`. For self-hosted server enablement, see `self-hosted.md`. If it doesn't work, see `diagnostics.md`. ## Prerequisites -- A Temporal Cloud account with an AWS-hosted Namespace, or a self-hosted Temporal Service v1.31.0 or later. -- The Namespace's cloud provider must match the serverless compute provider. -- For self-hosted deployments, complete the self-hosted setup before following the deployment guide. -- Every Workflow must declare a versioning behavior, or the Worker must set a default versioning behavior. -- An AWS account with permissions to create and invoke Lambda functions and create IAM roles. For the exact operator actions and a preflight check, see `iam.md`. -- The AWS-specific steps require the `aws` CLI installed and configured with your AWS credentials. You may also use the AWS Console or the AWS SDKs. -- The Go SDK, Python SDK, TypeScript SDK, Java SDK, or .NET SDK, depending on your language. +- A Temporal Cloud account with an AWS-hosted Namespace, or a self-hosted Temporal Service v1.31.0 or later. +- The Namespace's cloud provider must match the serverless compute provider. +- For self-hosted deployments, complete the self-hosted setup before following the deployment guide. +- Every Workflow must declare a versioning behavior, or the Worker must set a default versioning behavior. +- An AWS account with permissions to create and invoke Lambda functions and create IAM roles. For the exact operator actions and a preflight check, see `iam.md`. +- The AWS-specific steps require the `aws` CLI installed and configured with your AWS credentials. You may also use the AWS Console or the AWS SDKs. +- The Go SDK, Python SDK, TypeScript SDK, Java SDK, or .NET SDK, depending on your language. - The `temporal` CLI, authenticated to the target Temporal Service — Steps 4–6 and the CLI troubleshooting paths use it. See "Temporal CLI and Cloud connection" below. The selected SDK reference links its maintained sample project. ## Temporal CLI and Cloud connection -Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it and authenticate it to the target Temporal Service before those steps, or commands default to `localhost:7233` and fail against Temporal Cloud. The serverless `worker deployment create-version` subcommand and its `--aws-lambda-*` flags also require a recent CLI build — see "Check the CLI version" in Step 4. +Steps 4–6 and the CLI troubleshooting paths use the `temporal` CLI. Install it and authenticate it to the target Temporal Service before those steps, or commands default to `localhost:7233` and fail against Temporal Cloud. The serverless `worker deployment create-version` subcommand and its `--aws-lambda-*` flags also require a recent CLI build — see "Check the CLI version" in Step 4. **Authenticate to Temporal Cloud (API key).** Export environment variables (the CLI and the serverless Worker packages both read these): @@ -40,7 +35,6 @@ temporal --profile prod config set --prop address --value ".." temporal --profile prod config set --prop api_key --value "" ``` - or configure an environment and pass `--env prod` (or set `TEMPORAL_ENV`): @@ -57,8 +51,8 @@ temporal env get --env prod # --env mechanism temporal config get --prop address # --profile mechanism ``` -- For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. -- Supplying an API key auto-enables TLS; no cert flags are needed for API-key auth. +- For Temporal Cloud the Namespace is the fully-qualified `.`, not the bare name. +- Supplying an API key auto-enables TLS; no cert flags are needed for API-key auth. - The `temporal ...` commands in Steps 4–6 assume this is configured. To create an API key, see `skill-temporal-ops`. **Temporal-side preflight.** Confirm the CLI can reach the Namespace before deploying — this is the Temporal side of the pre-deploy access check. It should list (empty is fine) without an auth or connection error: @@ -113,7 +107,7 @@ Do not proceed to Steps 4–6 on the assumption auth will work — re-run this c ## Step 1: Write Worker code -The Worker handles the per-invocation lifecycle: connecting to Temporal, polling for tasks, and gracefully shutting down before the invocation deadline. +The Worker handles the per-invocation lifecycle: connecting to Temporal, polling for tasks, and gracefully shutting down before the invocation deadline. Load the reference for the selected SDK alongside this shared deployment guide: @@ -183,7 +177,6 @@ aws lambda get-function --function-name my-temporal-worker \ ### Common parameters (all SDKs) - | Parameter | Description | |---|---| @@ -192,7 +185,7 @@ aws lambda get-function --function-name my-temporal-worker \ | `--timeout` | Invocation deadline in seconds. Maximum time each Lambda invocation can run before AWS terminates it. Set high enough for the Worker to start, process Tasks, and shut down gracefully. | | `--memory-size` | Memory in MB allocated to each invocation. | -**Caution:** AWS Lambda functions default to a 3-second timeout, which is too short for the Worker to start, connect to Temporal, and register the Task Queue. If the first invocation times out before the Worker polls, the Task Queue binding is never created and the Lambda is never invoked again. Always set `--timeout` high enough for the Worker to start, process Tasks, and shut down gracefully. +**Caution:** AWS Lambda functions default to a 3-second timeout, which is too short for the Worker to start, connect to Temporal, and register the Task Queue. If the first invocation times out before the Worker polls, the Task Queue binding is never created and the Lambda is never invoked again. Always set `--timeout` high enough for the Worker to start, process Tasks, and shut down gracefully. Set it from three constraints: @@ -204,7 +197,6 @@ Lambda bills **GB-seconds** — allocated memory × billed duration, however idl ### Environment variables - | Variable | Description | |---|---| @@ -215,9 +207,9 @@ Lambda bills **GB-seconds** — allocated memory × billed duration, however idl | `TEMPORAL_TLS_CLIENT_KEY_PATH` | Path to the TLS client key file for mTLS authentication. | | `TEMPORAL_API_KEY` | API key for API key authentication. Supplying it auto-enables TLS; mTLS cert paths are not needed. | -The serverless Worker packages read environment variables and configuration files automatically at startup. For the full list of supported environment variables, config file format, and profiles, see the Environment configuration docs (`/develop/environment-configuration`). +The serverless Worker packages read environment variables and configuration files automatically at startup. For the full list of supported environment variables, config file format, and profiles, see the Environment configuration docs (`/develop/environment-configuration`). -Sensitive values like TLS keys and API keys should be encrypted at rest. +Sensitive values like TLS keys and API keys should be encrypted at rest. For updating the function code and publishing immutable versions, see `versioning.md`. @@ -227,11 +219,10 @@ Step 3 (execution role, Temporal invocation role, and CloudFormation for Tempora ## Step 4: Create Worker Deployment Version -Create a Worker Deployment Version with a compute provider that points to your Lambda function. The compute configuration tells Temporal how to invoke your Worker: the provider type (`aws-lambda`), the Lambda function ARN, and the IAM role to assume. The deployment name and build ID must match the values in your Worker code. +Create a Worker Deployment Version with a compute provider that points to your Lambda function. The compute configuration tells Temporal how to invoke your Worker: the provider type (`aws-lambda`), the Lambda function ARN, and the IAM role to assume. The deployment name and build ID must match the values in your Worker code. ### Using Temporal UI - 1. In the Temporal UI, open your Namespace. 2. In the left pane, select **Workers**. @@ -243,11 +234,11 @@ Create a Worker Deployment Version with a compute provider that points to your L - **External ID**: the same value passed to the CloudFormation template. 6. Click **Save**. -When you create a version through the UI, the version is automatically set as current. +When you create a version through the UI, the version is automatically set as current. ### Using Temporal CLI -Use the CLI for manual setup, shell scripts, and CI/CD pipelines. When you create a version through the CLI, you must set the version as current as a separate step. +Use the CLI for manual setup, shell scripts, and CI/CD pipelines. When you create a version through the CLI, you must set the version as current as a separate step. **Check the CLI version before relying on these commands.** The `worker deployment create-version` subcommand and its `--aws-lambda-*` flags only exist in recent Temporal CLI builds, and a CLI old enough to lack them fails in a way that looks like a syntax mistake: @@ -258,16 +249,15 @@ temporal worker deployment create-version --help If the subcommand or the flags are missing, upgrade. Observed bounds: v1.5.0 (Homebrew) lacked `create-version` entirely; v1.8.0 (standalone) had the serverless flags. The exact minimum version is unconfirmed against the CLI changelog, so treat those as bounds rather than a threshold. A current standalone build can be installed alongside a package-managed one without disturbing it — worth doing rather than upgrading a CLI the user may depend on elsewhere. -First, create the Worker Deployment if it does not already exist: +First, create the Worker Deployment if it does not already exist: ```bash temporal worker deployment create \ --namespace \ --name my-app ``` - -Then create the version with the compute provider configuration: +Then create the version with the compute provider configuration: ```bash temporal worker deployment create-version \ @@ -278,7 +268,6 @@ temporal worker deployment create-version \ --aws-lambda-assume-role-arn \ --aws-lambda-assume-role-external-id ``` - | Flag | Description | |---|---| @@ -287,11 +276,10 @@ temporal worker deployment create-version \ | `--aws-lambda-function-arn` | Qualified versioned ARN of the Lambda function Temporal invokes for this version (for example, `function:my-worker:5`). An unqualified ARN is also accepted for development. | | `--aws-lambda-assume-role-arn` | IAM role Temporal assumes to invoke the function. This is the `RoleARN` output from the CloudFormation stack. This is not the Lambda execution role or your own IAM user/role. | | `--aws-lambda-assume-role-external-id` | External ID configured in the IAM role trust policy. | - ### Validate connection -Go to **Workers** > **Deployments** > select your deployment > open the **Actions** menu on the version and click **Validate Connection**. This checks that Temporal can assume the IAM role and invoke the function. +Go to **Workers** > **Deployments** > select your deployment > open the **Actions** menu on the version and click **Validate Connection**. This checks that Temporal can assume the IAM role and invoke the function. ### Checkpoint: confirm the validation invocation actually bound the Task Queue @@ -306,9 +294,9 @@ Task Queues listed = the invocation role, the Lambda package, the env vars, and ## Step 5: Set version as current -If you created the version through the Temporal UI, the version is already current — skip this step. +If you created the version through the Temporal UI, the version is already current — skip this step. -If you used the CLI, set the version as current. Without this step, tasks on the Task Queue will not route to the version, and Temporal will not invoke the Lambda function. +If you used the CLI, set the version as current. Without this step, tasks on the Task Queue will not route to the version, and Temporal will not invoke the Lambda function. ```bash temporal worker deployment set-current-version \ @@ -316,9 +304,8 @@ temporal worker deployment set-current-version \ --build-id build-1 \ --yes ``` - -**`set-current-version` asks for interactive confirmation.** Without `--yes` (`-y`) it prompts, and run non-interactively (scripts, CI, or an agent shell) it exits without applying the change — which reads as a silent no-op: the command appears to succeed but the version never becomes current. Pass `--yes` for any non-interactive use. `set-ramping-version` behaves the same way. Note that `delete-version` does *not* take `--yes` — its gating flag is `--skip-drainage`. +**`set-current-version` asks for interactive confirmation.** Without `--yes` (`-y`) it prompts, and run non-interactively (scripts, CI, or an agent shell) it exits without applying the change — which reads as a silent no-op: the command appears to succeed but the version never becomes current. Pass `--yes` for any non-interactive use. `set-ramping-version` behaves the same way. Note that `delete-version` does *not* take `--yes` — its gating flag is `--skip-drainage`. Confirm it took effect before moving on: @@ -328,7 +315,7 @@ temporal worker deployment describe --name my-app ## Step 6: Verify deployment -Start a Workflow on the same Task Queue to confirm that Temporal invokes your Lambda Worker. +Start a Workflow on the same Task Queue to confirm that Temporal invokes your Lambda Worker. ```bash temporal workflow start \ @@ -336,14 +323,13 @@ temporal workflow start \ --type MyWorkflow \ --input '"Hello, serverless!"' ``` - -Verify the invocation by checking: +Verify the invocation by checking: -- **Temporal UI:** The Workflow execution should show task completions in the event history. -- **AWS CloudWatch Logs:** The Lambda function's log group (`/aws/lambda/my-temporal-worker`) should show invocation logs with the Worker startup, task processing, and graceful shutdown. Requires the execution role to have CloudWatch Logs permissions (included in `AWSLambdaBasicExecutionRole`). +- **Temporal UI:** The Workflow execution should show task completions in the event history. +- **AWS CloudWatch Logs:** The Lambda function's log group (`/aws/lambda/my-temporal-worker`) should show invocation logs with the Worker startup, task processing, and graceful shutdown. Requires the execution role to have CloudWatch Logs permissions (included in `AWSLambdaBasicExecutionRole`). -If the Workflow does not progress or the Lambda is not invoked, see `diagnostics.md`. +If the Workflow does not progress or the Lambda is not invoked, see `diagnostics.md`. ## Teardown diff --git a/references/aws-lambda/versioning.md b/references/aws-lambda/versioning.md index 520171c..395d86a 100644 --- a/references/aws-lambda/versioning.md +++ b/references/aws-lambda/versioning.md @@ -1,8 +1,5 @@ # AWS Lambda — Versioning, updates & rollback - For the provider-specific model — Temporal Worker Deployment Versions vs. Lambda function versions, and Pinned vs. Auto-Upgrade behavior — see `constraints.md` ("Worker Versioning with Serverless Workers"). @@ -27,7 +24,6 @@ aws lambda update-function-code \ --function-name my-temporal-worker \ --zip-file fileb://function.zip ``` - **Wait for the update to settle before publishing.** `update-function-code` returns while `LastUpdateStatus` is still `InProgress`, and `publish-version` in that window either fails (`ResourceConflictException`) or, worse, snapshots the *previous* code. Block on the transition: @@ -37,24 +33,23 @@ aws lambda wait function-updated-v2 --function-name my-temporal-worker (`function-updated` on AWS CLI v1; see `setup.md` for the polling fallback.) -After updating, increment the Build ID in your Worker code and publish a new Lambda function version. +After updating, increment the Build ID in your Worker code and publish a new Lambda function version. ## Publish a Lambda function version -For production, create an immutable snapshot of your Lambda code after creating the function and after each `update-function-code`, and maintain a one-to-one mapping between each Lambda function version and each Temporal Worker Deployment Version Build ID. +For production, create an immutable snapshot of your Lambda code after creating the function and after each `update-function-code`, and maintain a one-to-one mapping between each Lambda function version and each Temporal Worker Deployment Version Build ID. ```bash aws lambda publish-version \ --function-name my-temporal-worker \ --description "Build ID build-5" ``` - -The command prints the `FunctionArn` for the new version, for example `arn:aws:lambda:us-east-1:123456789012:function:my-temporal-worker:5`. Use this qualified versioned ARN when you create the Worker Deployment Version. +The command prints the `FunctionArn` for the new version, for example `arn:aws:lambda:us-east-1:123456789012:function:my-temporal-worker:5`. Use this qualified versioned ARN when you create the Worker Deployment Version. -For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster. +For development or non-critical workloads, you can skip `publish-version` and use an unqualified ARN to iterate faster. -To roll back, revert the Temporal Current Version with `temporal worker deployment set-current-version`. The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again. +To roll back, revert the Temporal Current Version with `temporal worker deployment set-current-version`. The previous Worker Deployment Version still points at its original Lambda function version and is ready to receive traffic again. ```bash temporal worker deployment set-current-version \ diff --git a/references/concepts.md b/references/concepts.md index 0fb468f..db43836 100644 --- a/references/concepts.md +++ b/references/concepts.md @@ -1,6 +1,5 @@ # Serverless Workers — Concepts - ## Shared serverless concepts @@ -20,17 +19,16 @@ A Serverless Worker is a Temporal Worker whose compute lifecycle is managed by T ### Compute providers -A compute provider is the configuration that tells Temporal how to start or scale compute for a Worker Deployment Version. It is set on the Worker Deployment Version and specifies the provider type, the compute target, and the identity Temporal uses to act on that target. +A compute provider is the configuration that tells Temporal how to start or scale compute for a Worker Deployment Version. It is set on the Worker Deployment Version and specifies the provider type, the compute target, and the identity Temporal uses to act on that target. -For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function. +For example, an AWS Lambda compute provider includes the Lambda function ARN and the IAM role that Temporal assumes to invoke the function. A GCP Cloud Run compute provider names the Worker Pool and the invoker service account that Temporal impersonates to resize it. -Compute providers are only needed for Serverless Workers. Traditional long-lived Workers do not require a compute provider because the Worker process lifecycle is not managed by the Temporal server. +Compute providers are only needed for Serverless Workers. Traditional long-lived Workers do not require a compute provider because the Worker process lifecycle is not managed by the Temporal server. #### Supported providers - | Provider | Description | |---|---| diff --git a/references/gcp-cloud-run/constraints.md b/references/gcp-cloud-run/constraints.md index 2519eb9..baa21f3 100644 --- a/references/gcp-cloud-run/constraints.md +++ b/references/gcp-cloud-run/constraints.md @@ -4,7 +4,7 @@ Consequences of Cloud Run's execution model: **Temporal resizes a pool of long-l ## Worker lifetime is an instance, not an invocation -Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Some SDKs add optional conveniences, but none are required. +Each pool instance runs **standard long-lived Worker code**: it connects, registers Workflows and Activities, and polls the Task Queue for its whole lifetime. There is no handler, no per-Task lifecycle, and **no serverless Worker package**. Some SDKs add optional conveniences, but none are required. The WCI controls how many instances run; each instance manages its own polling and Task processing. For the shared WCI lifecycle, inputs, and inspection commands, see [Worker Controller Instance (WCI)](../wci.md). For Cloud Run-specific failure interpretation, see `diagnostics.md`. @@ -19,13 +19,12 @@ Cloud Run sends `SIGTERM` during scale-in and can send `SIGKILL` ten seconds lat ## What bounds an Activity instead: scale-in -**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. +**The WCI decides when to remove an instance from Task Queue activity, not from what any individual instance is doing.** It does not track how long an instance has been running or whether it is mid-Activity, so the instance Cloud Run stops may be one that is still executing work. Graceful shutdown lets short work drain but cannot guarantee an Activity will finish. **Use Activity Heartbeats and set a Workflow-side Heartbeat Timeout** so an interrupted attempt is detected and retried promptly. Heartbeat calls without a Heartbeat Timeout do not provide timely recovery; the retry may wait until Start-to-Close expires. Set Start-to-Close above the longest expected attempt and configure a Retry Policy appropriate for the operation. Record the next unprocessed item only after processing succeeds, then resume from that Heartbeat detail on retry. The selected SDK reference includes the complete Activity options. ## Autoscaling behavior - The shared [WCI inputs](../wci.md#inputs) feed Cloud Run's rate-based pool-sizing algorithm, which combines two mechanisms: @@ -36,7 +35,7 @@ It sizes to a **target utilization of 80% by default** rather than loading every **Scale-in is deliberately more conservative than scale-out:** it holds capacity while sync match failures are still occurring and applies a cooldown before reducing the pool. With no work, it can scale to zero; the next sync match failure or backlog scales it back up. -The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, target utilization `0.8`, and scale-down stabilization duration `90s`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The CLI couples these flags, so follow the canonical [compatibility and complete-group guidance](setup.md#step-6-register-the-worker-deployment-version). +The scaler defaults are **minimum `0`, maximum `30`, initial count `0`, target utilization `0.8`, and scale-down stabilization duration `90s`**. Configure them in the version's Scaling and Lifecycle settings or with the Temporal CLI. The CLI couples these flags, so follow the canonical [compatibility and complete-group guidance](setup.md#step-6-register-the-worker-deployment-version). The `90s` duration begins after the most recent sync-match failure and is only one gate in scale-in; it is not a promise that the pool reaches zero 90 seconds after registration or Workflow completion. To confirm scale-in, read the pool's requested instance count and the Worker's shutdown logs rather than waiting a fixed time. @@ -44,7 +43,7 @@ An initial count and minimum of zero do not suppress registration. The rate-base ## One Worker Pool per Worker Deployment Version -**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. +**The compute configuration names a project, region, and pool — not a revision.** Temporal runs whichever revision the pool serves at the time, which ties a pool to a single build. A new build needs a **new pool**, and the Build ID belongs in the pool name to keep that mapping visible. Keep an older version's pool in place while Pinned Workflows are still running on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. @@ -52,6 +51,6 @@ Do not deploy a new image into a pool used by a live Worker Deployment Version. ## Do not share a Task Queue with long-lived Workers -**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. +**The pool scales up to cover the Task Queue's full workload even when independently managed Workers are already handling all of it**, so you run and pay for duplicate capacity. The WCI sizes the pool from the rate of Tasks arriving on the version's Task Queues, and nothing in that measurement accounts for the long-lived Workers. Sync matching to a long-lived Worker suppresses the *immediate* scale-up, but the periodic re-sizing scales the pool up regardless. Fixing poller counts on the long-lived side does not help — use separate Task Queues. diff --git a/references/gcp-cloud-run/diagnostics.md b/references/gcp-cloud-run/diagnostics.md index d0d0b8e..c48a075 100644 --- a/references/gcp-cloud-run/diagnostics.md +++ b/references/gcp-cloud-run/diagnostics.md @@ -6,7 +6,6 @@ For the WCI lifecycle, inputs, Workflow ID pattern, and inspection commands, see ## Scaling flow (when working correctly) - 1. You deploy the Worker image to a Worker Pool at zero instances. 2. You create a Worker Deployment Version pointing at that pool. This starts a WCI Workflow. @@ -68,7 +67,7 @@ gcloud run worker-pools describe \ --region --project --format=yaml ``` -Three fields under `metadata.annotations`: +Three fields under `metadata.annotations`: | Field | What it tells you | |---|---| @@ -96,7 +95,7 @@ gcloud logging read \ Workers → Deployments → select deployment → Actions → **Validate Connection**. For Cloud Run this impersonates the invoker and reads the pool, confirming three things: the compute configuration names a pool that exists, Temporal can impersonate the invoker, and the invoker can read. -**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. +**It starts no instance and does not exercise `run.workerPools.update`.** Version registration is different: its Task Queue bootstrap does update the pool. An invoker with read but not update permission can therefore pass this manual validation, but version registration or a later resize fails. If validation succeeds but registration never changes `lastModifier`, **check the update permission**. On failure, check each part of the compute configuration against the pool: @@ -191,7 +190,7 @@ This returns the Worker's container output and the pool's resize audit entries ( **A scaled-to-zero pool emits no new logs.** Use `gcloud run worker-pools logs tail` only while an instance is running. An empty result may simply mean the pool has never started. -Common errors: +Common errors: - **Connection failures** — check `TEMPORAL_ADDRESS` and `TEMPORAL_NAMESPACE` on the pool. Self-hosted: verify network reachability from Cloud Run to the frontend. - **Missing secrets** — the instance cannot read the API key or TLS material. The **runner** service account needs `roles/secretmanager.secretAccessor` on the secret. That is the account in `spec.template.spec.serviceAccountName`, **not the invoker.** @@ -214,7 +213,7 @@ This is expected behavior, not a misconfiguration. Confirm the Worker handles `S ## Rule out a GCP-side cause -If every check passes, the cause may be in Cloud Run rather than your configuration: +If every check passes, the cause may be in Cloud Run rather than your configuration: - [Cloud Run known issues](https://cloud.google.com/run/docs/known-issues) — includes issues affecting how long pool operations take. - [Google Cloud Service Health](https://status.cloud.google.com/) — active incidents by product and region. diff --git a/references/gcp-cloud-run/iam.md b/references/gcp-cloud-run/iam.md index ff18d5d..434b40b 100644 --- a/references/gcp-cloud-run/iam.md +++ b/references/gcp-cloud-run/iam.md @@ -10,13 +10,13 @@ Cloud Run uses three identities: Temporal reaches the invoker through `roles/iam.serviceAccountTokenCreator`. The Terraform module described below creates the invoker and applies its grants. -**The two service accounts are not interchangeable.** The runner runs the pool and never scales it; the invoker scales the pool and never runs it. +**The two service accounts are not interchangeable.** The runner runs the pool and never scales it; the invoker scales the pool and never runs it. ## Runner service account The runtime identity the pool's instances use to reach other Google Cloud services. Set in `setup.md` Step 4 with `gcloud run worker-pools deploy --service-account`. It may be an account that already exists; a dedicated one is preferred. -**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: +**It needs no baseline role to run the Worker.** Cloud Run collects `stdout` and `stderr` into Cloud Logging through its own infrastructure, and the Cloud Run *service agent* — not the runner — pulls the container image. Grant only what your code actually reaches: - `roles/secretmanager.secretAccessor` on each secret you mount, including the Temporal API key. - `roles/logging.logWriter` **only if** the Worker writes through the Cloud Logging API rather than stdout/stderr. @@ -27,7 +27,7 @@ The Collector uses the runner's Application Default Credentials. If its configur ## Invoker service account -The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: +The identity Temporal Cloud impersonates to read and scale the pool. Two grants make it work: - Temporal's identity receives **`roles/iam.serviceAccountTokenCreator`** on the invoker, so it can impersonate it. - The invoker receives a project-level Cloud Run role with at least **`run.workerPools.get`** (read) and **`run.workerPools.update`** (scale). `roles/run.developer` includes both. @@ -36,13 +36,13 @@ The invoker also needs **`roles/iam.serviceAccountUser` on the runner service ac ### Validate Connection checks read access only -`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass, but version registration and scaling also require `run.workerPools.update`. Passing Validate Connection therefore does not prove that the invoker can scale the pool. See `diagnostics.md` for registration and scaling checks. +`run.workerPools.get` alone is enough for the UI's **Validate Connection** action to pass, but version registration and scaling also require `run.workerPools.update`. Passing Validate Connection therefore does not prove that the invoker can scale the pool. See `diagnostics.md` for registration and scaling checks. ## The Terraform module Temporal publishes [`serverless-workers/gcp/cloud-run`](https://github.com/temporalio/terraform-modules/tree/main/modules/serverless-workers/gcp/cloud-run), which creates the invoker service account and applies the grants. -Apply the module block below. Temporal Cloud's UI template is the source of the account-specific `impersonator_service_account_emails` only; see [Where `impersonator_service_account_emails` comes from](#where-impersonator_service_account_emails-comes-from). For a standalone root module, declare the Google provider constraint expected by the module and configure the project explicitly; reconcile any existing root constraint before running `terraform init`: +Apply the module block below. Temporal Cloud's UI template is the source of the account-specific `impersonator_service_account_emails` only; see [Where `impersonator_service_account_emails` comes from](#where-impersonator_service_account_emails-comes-from). For a standalone root module, declare the Google provider constraint expected by the module and configure the project explicitly; reconcile any existing root constraint before running `terraform init`: ```hcl terraform { diff --git a/references/gcp-cloud-run/sdk-dotnet.md b/references/gcp-cloud-run/sdk-dotnet.md index b2f6cc4..4cf2fa5 100644 --- a/references/gcp-cloud-run/sdk-dotnet.md +++ b/references/gcp-cloud-run/sdk-dotnet.md @@ -1,6 +1,5 @@ # .NET SDK on GCP Cloud Run - Use this reference for .NET-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. diff --git a/references/gcp-cloud-run/sdk-go.md b/references/gcp-cloud-run/sdk-go.md index 29fa0b7..7966a29 100644 --- a/references/gcp-cloud-run/sdk-go.md +++ b/references/gcp-cloud-run/sdk-go.md @@ -1,6 +1,5 @@ # Go SDK on GCP Cloud Run - Use this reference for Go-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. diff --git a/references/gcp-cloud-run/sdk-java.md b/references/gcp-cloud-run/sdk-java.md index d858d09..033eddc 100644 --- a/references/gcp-cloud-run/sdk-java.md +++ b/references/gcp-cloud-run/sdk-java.md @@ -1,6 +1,5 @@ # Java SDK on GCP Cloud Run - Use this reference for Java-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. diff --git a/references/gcp-cloud-run/sdk-python.md b/references/gcp-cloud-run/sdk-python.md index 5af5b49..8fc4275 100644 --- a/references/gcp-cloud-run/sdk-python.md +++ b/references/gcp-cloud-run/sdk-python.md @@ -1,6 +1,5 @@ # Python SDK on GCP Cloud Run - Use this reference for Python-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. diff --git a/references/gcp-cloud-run/sdk-typescript.md b/references/gcp-cloud-run/sdk-typescript.md index 9970ad6..e81284a 100644 --- a/references/gcp-cloud-run/sdk-typescript.md +++ b/references/gcp-cloud-run/sdk-typescript.md @@ -1,6 +1,5 @@ # TypeScript SDK on GCP Cloud Run - Use this reference for TypeScript-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. diff --git a/references/gcp-cloud-run/self-hosted.md b/references/gcp-cloud-run/self-hosted.md index 58df6e4..f925c4c 100644 --- a/references/gcp-cloud-run/self-hosted.md +++ b/references/gcp-cloud-run/self-hosted.md @@ -1,10 +1,10 @@ # GCP Cloud Run — self-hosted Temporal Service setup -Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. +Serverless Workers require **Temporal Service v1.31.0 or later**. Complete this page before following `setup.md`. ## 1. Cloud Run instances must reach the Temporal Service -The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). +The frontend must be reachable **from the Worker Pool instances**. If the Service runs on a private network, that likely means [Direct VPC egress](https://cloud.google.com/run/docs/configuring/vpc-direct-vpc) or a [Serverless VPC Access connector](https://cloud.google.com/run/docs/configuring/vpc-connectors). Note the direction: instances dial *out* to Temporal. Nothing needs to reach *into* Cloud Run — Temporal drives the pool through the Cloud Run admin API, not by connecting to your instances. @@ -25,7 +25,7 @@ workercontroller.scaling_algorithms.enabled: - rate-based ``` -**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. Pairing `no-sync` with `gcp-cloud-run` is rejected. +**Cloud Run requires the `rate-based` algorithm.** Because a pool is a set of long-lived instances, the WCI resizes it from arrival and backlog rates. Pairing `no-sync` with `gcp-cloud-run` is rejected. To enable per Namespace instead of globally: @@ -36,7 +36,7 @@ workercontroller.enabled: namespace: 'your-namespace' ``` -The Service watches the file and applies updates **without a restart**. +The Service watches the file and applies updates **without a restart**. Two optional global keys cover multi-hop impersonation: @@ -45,7 +45,6 @@ Two optional global keys cover multi-hop impersonation: | `workercontroller.compute_providers.gcp.intermediary_service_accounts` | Service accounts to impersonate in sequence before the invoker. Unset means the server impersonates the invoker directly. | | `workercontroller.compute_providers.gcp.first_delegate_as_base` | When `true`, the first entry is the identity the server's ambient credentials impersonate directly and the rest are passed as token-creator delegates. Defaults to `false`, passing the whole chain as delegates. | - ## 3. Give the Temporal Service a GCP identity @@ -53,10 +52,10 @@ The server impersonates the invoker service account, so it must first run as a G **On GCP (GCE, GKE):** the attached service account is used automatically through [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials). No extra credential configuration — you grant *that* account impersonation rights in step 4. -**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. +**Outside GCP:** use [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) and point `GOOGLE_APPLICATION_CREDENTIALS` at the credential configuration file it produces. That variable also accepts a service account key file, but **Google recommends against long-lived keys** — prefer federation, and never ask a user to paste a key. ## 4. Create the invoker service account The Service scales the pool as an **invoker** service account, which only reads and scales the pool. The identity instances *run as* is the separate **runner** service account set on the pool in `setup.md`. → `iam.md` for the full distinction. -Use the Terraform module in `iam.md`, setting `impersonator_service_account_emails` to the GCP identity used by the Temporal Service. That reference is authoritative for the invoker and runner grants; they are the same for Temporal Cloud and self-hosted deployments. Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version, then follow `setup.md` from Step 1. +Use the Terraform module in `iam.md`, setting `impersonator_service_account_emails` to the GCP identity used by the Temporal Service. That reference is authoritative for the invoker and runner grants; they are the same for Temporal Cloud and self-hosted deployments. Use the module's `invoker_email` output as `--gcp-cloud-run-service-account` when registering the Worker Deployment Version, then follow `setup.md` from Step 1. diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md index 36cab8a..02292d0 100644 --- a/references/gcp-cloud-run/setup.md +++ b/references/gcp-cloud-run/setup.md @@ -14,7 +14,6 @@ End-to-end: write a standard Worker, containerize it, push the image, create a W - A Temporal SDK supported by this skill: Go, Python, TypeScript, Java, or .NET. - A Temporal Cloud API key that can access the target Namespace. **The user creates it**, either in the Temporal Cloud UI under **Settings → API Keys** or by running `tcld apikey create --name --duration ` in their own terminal. Never run `tcld apikey create` from the agent's shell: it prints the new key. The command creates a key for the current user; add `--service-account-id ` to create it for an existing service account instead. Prefer a service-account-owned key for shared or long-lived Workers and a short-lived user key for a personal test. The key is needed twice, in the operator's Temporal CLI profile and in the Worker's Secret Manager runtime secret; the [API-key hand-off](#hand-off-the-temporal-api-key) sets both from one read. - The `temporal` CLI commands in Steps 6–8 authenticate through a named CLI profile that the user creates during the [API-key hand-off](#hand-off-the-temporal-api-key). An exported variable in the user's terminal does not reach the agent's shell, so do not rely on `TEMPORAL_API_KEY` there. Never append `--api-key ` or put the key in an inline assignment. On macOS the default profile file is `~/Library/Application Support/temporalio/temporal.toml`. The examples below pass `--profile ` explicitly; omit it only when using a different already-configured authentication mechanism, including self-hosted mTLS. @@ -146,7 +145,7 @@ Two things the Worker must do: 1. **Declare its Worker Deployment Version and enable versioning**, with a deployment name and build ID that exactly match the version you register in Step 6. 2. **Read its configuration from the environment** — address, Namespace, Task Queue, credentials — so one image can run against any Namespace. The pool supplies these via `--set-env-vars` and `--set-secrets`. -**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. +**The entrypoint must start the Worker process**, so an instance begins polling as soon as it starts. ## Step 2: Containerize the Worker @@ -300,7 +299,7 @@ Through the UI, the version is set current automatically; through the CLI it is ### Checkpoint: verify registration -Creating the Worker Deployment Version starts its WCI, which temporarily raises the pool to at least one instance so the Worker can bind its Task Queues. This occurs even when the minimum and initial instance counts are `0`. Wait for the expected Task Queue types before setting the version current. The separate UI **Validate Connection** action checks pool read access only and is not a substitute for this registration check. +Creating the Worker Deployment Version starts its WCI, which temporarily raises the pool to at least one instance so the Worker can bind its Task Queues. This occurs even when the minimum and initial instance counts are `0`. Wait for the expected Task Queue types before setting the version current. The separate UI **Validate Connection** action checks pool read access only and is not a substitute for this registration check. First run the machine-checkable [Task Queue binding check](diagnostics.md#start-here-did-the-expected-worker-bind). Do not continue until it finds the expected Task Queue and types. Then verify that Temporal wrote to the pool: diff --git a/references/gcp-cloud-run/versioning.md b/references/gcp-cloud-run/versioning.md index 28e2419..c3cc858 100644 --- a/references/gcp-cloud-run/versioning.md +++ b/references/gcp-cloud-run/versioning.md @@ -2,11 +2,11 @@ ## One Worker Pool per Build ID -**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool serves, so durable isolation requires a separate pool per build. Carry the Build ID in the pool name (`my-worker-pool-build-1`) to keep that mapping visible. A pool-level instance split can temporarily hold a revision, but the split remains mutable state outside Temporal. +**The compute configuration names a project, region, and Worker Pool — it does not name a [revision](https://cloud.google.com/run/docs/managing/revisions).** Temporal runs whichever revision the pool serves, so durable isolation requires a separate pool per build. Carry the Build ID in the pool name (`my-worker-pool-build-1`) to keep that mapping visible. A pool-level instance split can temporarily hold a revision, but the split remains mutable state outside Temporal. ## The hazard: redeploying into a live pool -A normal `gcloud run worker-pools deploy` creates a new revision and promotes it to every instance by default. The Temporal version still points to the same pool, so the code changes underneath it. Replay-unsafe changes can cause non-determinism errors for in-flight Workflows, including Pinned ones, because pinning selects a Temporal version rather than a Cloud Run revision. Use a new pool for durable isolation; `--no-promote` is only a same-pool guardrail. +A normal `gcloud run worker-pools deploy` creates a new revision and promotes it to every instance by default. The Temporal version still points to the same pool, so the code changes underneath it. Replay-unsafe changes can cause non-determinism errors for in-flight Workflows, including Pinned ones, because pinning selects a Temporal version rather than a Cloud Run revision. Use a new pool for durable isolation; `--no-promote` is only a same-pool guardrail. ## Guardrail and recovery when a pool is reused @@ -47,7 +47,7 @@ After emergency recovery, return to one pool per Build ID for the next release s 3. Register a new Worker Deployment Version pointing at the new pool, with a build ID matching the new Worker code. 4. Confirm registration bootstrapped the pool: the expected Task Queue types are bound, the deployed image digest matches the intended artifact, and the Worker startup log announces the new build ID. `lastModifier` showing the invoker proves only that Temporal wrote the pool's requested count. Treat the separate UI Validate Connection action as a read-only check. → `diagnostics.md`. 5. Set the new version current, or ramp to it. -6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. +6. **Leave the old pool in place** while Pinned Workflows still run on it. It can sit at zero instances; its WCI scales it back up when a Task arrives for that version. Only the invoker's permissions are shared across pools, so a new pool usually needs no IAM change — provided the invoker's `deploy_roles` are project-level, which is the module's default. Check `iam.md` if you scoped them to individual pools instead. diff --git a/references/wci.md b/references/wci.md index 993b510..e33f1aa 100644 --- a/references/wci.md +++ b/references/wci.md @@ -1,39 +1,35 @@ # Worker Controller Instance (WCI) - ## Role and lifecycle -The Worker Controller Instance (WCI) is a Temporal system Workflow that scales compute for a Worker Deployment Version from Task Queue conditions. One WCI runs for each Worker Deployment Version that has a compute provider configured, in the same Namespace as the Worker Deployment. +The Worker Controller Instance (WCI) is a Temporal system Workflow that scales compute for a Worker Deployment Version from Task Queue conditions. One WCI runs for each Worker Deployment Version that has a compute provider configured, in the same Namespace as the Worker Deployment. Temporal creates and manages the WCI automatically. Never create, start, or manage it yourself. ## Inputs -The WCI responds to sync match failures and Task Queue backlog or rate information. The selected provider's constraints and diagnostics references describe how those inputs become compute changes, along with the provider-specific lifecycle and failure signatures. +The WCI responds to sync match failures and Task Queue backlog or rate information. The selected provider's constraints and diagnostics references describe how those inputs become compute changes, along with the provider-specific lifecycle and failure signatures. ## Inspect the WCI Run these with the CLI connection the version was registered with: on Cloud Run, add `--profile ` (the profile the API-key hand-off writes); on AWS Lambda, use the connection configured in `aws-lambda/setup.md`. -List WCI Workflows in the Namespace: +List WCI Workflows in the Namespace: ```bash temporal workflow list \ --namespace \ --query 'TemporalNamespaceDivision = "TemporalWorkerControllerInstance" AND WorkflowId = "temporal-sys-worker-controller-instance::"' ``` - -WCI Workflow IDs follow this pattern: +WCI Workflow IDs follow this pattern: ```text temporal-sys-worker-controller-instance:: ``` -Inspect the WCI Workflow history for recent Activity results: +Inspect the WCI Workflow history for recent Activity results: ```bash temporal workflow show \ @@ -41,7 +37,6 @@ temporal workflow show \ --workflow-id 'temporal-sys-worker-controller-instance::' \ --detailed ``` - The default table shows event types but hides the payloads needed for diagnosis. Use `--detailed` as above or `-o json`. During registration, look for `ValidateSpec` and the `InvokeWorkersToRegisterTaskQueues` bootstrap. A result such as `worker_count: 1` means the WCI requested bootstrap capacity; it does not mean that an instance started or that the expected Worker registered and polled. Provider-specific scaling later appears through the provider action described in its diagnostics guide.