diff --git a/SKILL.md b/SKILL.md index 3ce5c78..fab9f54 100644 --- a/SKILL.md +++ b/SKILL.md @@ -101,7 +101,7 @@ Where the harness has a todo list, use it *in addition to* the printed checklist **A step is complete when its verification passed — not when its command exited zero.** Several commands in this workflow exit clean having done nothing: the traffic-shifting and key-revocation commands no-op when their confirmation prompt goes unanswered, and providers return from create and update calls while the resource is still settling. Check an item off against state you read back, not against an exit code. When a step's verification fails, say which step you are on and what it is blocked on rather than moving down the list. -1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, or .NET), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available. For Lambda, it must also be authenticated to the target at this point. For Cloud Run, check only that the `temporal` CLI is installed: its authenticated profile is created during the API-key hand-off after approval (`references/gcp-cloud-run/setup.md`). Each changes the specifics. Read `references/concepts.md` for shared concepts, `references//constraints.md` for the selected provider's lifecycle rules, and `references//setup.md` for compatibility and client-setup details. +1. **Scope the task.** Identify the SDK language (Go, Python, TypeScript, Java, or .NET), the deployment target (Temporal Cloud or self-hosted — self-hosted has its own server prerequisites), the compute provider, and whether this is a new setup, a configuration change, or troubleshooting. Confirm the deployment target is compatible with the chosen provider — see "A Namespace on the target cloud provider is required" under Provider-neutral principles. Ensure a Temporal client/CLI is available. For Lambda, it must also be authenticated to the target at this point. For Cloud Run, check only that the `temporal` CLI is installed: its authenticated profile is created by the operator hand-off after approval, unless the operator reuses a working one (`references/gcp-cloud-run/setup.md`). Each changes the specifics. Read `references/concepts.md` for shared concepts, `references//constraints.md` for the selected provider's lifecycle rules, and `references//setup.md` for compatibility and client-setup details. **Derive the compute provider from the Namespace's cloud provider.** An AWS-hosted Namespace uses AWS Lambda; a GCP-hosted Namespace uses GCP Cloud Run. State the derived provider with the Namespace choice, and if the request names a provider, check that it matches. When `tcld` cannot be used, the Namespace's region settles it: `aws-us-east-1` is AWS, `gcp-us-central1` is GCP. Ask the provider as a structured question only for a self-hosted Temporal Service, where either is possible; carry each option's status from the support table in its description, and note that Activity duration can decide it, because Lambda caps an invocation at 15 minutes and Cloud Run does not. @@ -164,7 +164,7 @@ Where the harness has a todo list, use it *in addition to* the printed checklist #### GCP Cloud Run - List the project and region; any APIs that still need enabling, and the Cloud Build submission; the Artifact Registry repository, if new, and the image; the Worker Pool; the runner service account and the dedicated invoker, with its Terraform state directory; the secret; the local CLI profile; the deployment name, build ID, and Task Queue; and logs. Say that the user creates the Temporal API key and enters it during the hand-off, and that they may also need to run `terraform apply` or the secret IAM grant in their own terminal if the agent's environment blocks them. Note that the Temporal deployment name is checked right after the hand-off, because the CLI profile does not exist before approval. + List the project and region; any APIs that still need enabling, and the Cloud Build submission; the Artifact Registry repository, if new, and the image; the Worker Pool; the runner service account and the dedicated invoker, with its Terraform state directory; the secret for the Worker's key; the operator's CLI profile, new or reused; the deployment name, build ID, and Task Queue; and logs. Say that the user creates the Temporal API keys, one for the Worker and one for the operator unless the operator reuses a working profile, and enters each in its own hand-off, and that they may also need to run `terraform apply` or the secret IAM grant in their own terminal if the agent's environment blocks them. Note that the Temporal deployment name is checked right after the hand-offs, because a new CLI profile does not exist before approval. ### AWS Lambda: steps 3–8 @@ -184,7 +184,7 @@ Where the harness has a todo list, use it *in addition to* the printed checklist ### GCP Cloud Run: steps 3–8 -Steps 3 and 5 need no API key: write and build the image, and apply Step 5's Terraform, while the user runs the key hand-off. Step 4 waits for the hand-off, so the checklist may mark Step 5 done first. Deliver the hand-off as `references/gcp-cloud-run/setup.md` describes: end the turn with the user's action first, and repeat it in full while the run is blocked on it. +Steps 3 and 5 need no API key: write and build the image, and apply Step 5's Terraform, while the user runs the key hand-offs. Step 4 waits for the Worker's key and Step 6 for the operator's profile, so the checklist may mark Step 5 done first. Deliver the hand-offs as `references/gcp-cloud-run/setup.md` describes: end the turn with the user's action first, and repeat it in full while the run is blocked on it. 3. **Author the Worker.** Write an ordinary long-lived Worker that starts polling when the container starts. There is no Cloud Run serverless Worker package or per-invocation handler. Set the required Worker Versioning behavior and make the deployment name and build ID match the version to be registered. → the selected `references/gcp-cloud-run/sdk-.md`. @@ -196,7 +196,7 @@ Steps 3 and 5 need no API key: write and build the image, and apply Step 5's Ter 7. **Verify.** Start a Workflow, confirm its history progresses, and confirm the Worker Pool logs show startup, polling, and Task execution. If it does not progress, start with the version's expected Task Queue bindings, then follow the WDV/WCI/provider decision table. → `references/gcp-cloud-run/diagnostics.md`. -8. **Hand back the inventory first; offer teardown as the closing note.** Include the project and region; the Artifact Registry repository, image tag, and digest; the Worker Pool; the runner and invoker service accounts, and whether this run created the invoker; the Terraform state directory; the secret and its versions; the deployment name, build ID, and Task Queue; the local CLI profile; and who created the Temporal API key. Follow the same inventory-before-teardown and approval rules as the Lambda path. → `references/gcp-cloud-run/setup.md`. +8. **Hand back the inventory first; offer teardown as the closing note.** Include the project and region; the Artifact Registry repository, image tag, and digest; the Worker Pool; the runner and invoker service accounts, and whether this run created the invoker; the Terraform state directory; the secret and its versions; the deployment name, build ID, and Task Queue; the local CLI profile; and, for each Temporal API key, who created it, who owns it, and whether it is the Worker's, the operator's, or both. Follow the same inventory-before-teardown and approval rules as the Lambda path. → `references/gcp-cloud-run/setup.md`. ## Working practices @@ -322,6 +322,7 @@ Most questions need 2–3 reference files. |---|---| | Execution model, Activity duration, autoscaling, scale-in, graceful shutdown, or mixed fleets. | `references/gcp-cloud-run/constraints.md` + the selected SDK reference | | Deploy, register, verify, or tear down a Worker Pool. | `references/gcp-cloud-run/setup.md` + the selected SDK reference | +| Hand off, reuse, or rotate the Worker's or the operator's Temporal API key. | `references/gcp-cloud-run/setup.md` | | Permissions, runner vs invoker identity, or Terraform IAM setup. | `references/gcp-cloud-run/iam.md` | | Update, publish a new build, or roll back. | `references/gcp-cloud-run/versioning.md` | | Worker Pool not scaling, Worker not polling, or Workflows not progressing. | `references/gcp-cloud-run/diagnostics.md` | diff --git a/references/gcp-cloud-run/sdk-dotnet.md b/references/gcp-cloud-run/sdk-dotnet.md index b2f6cc4..ffaa940 100644 --- a/references/gcp-cloud-run/sdk-dotnet.md +++ b/references/gcp-cloud-run/sdk-dotnet.md @@ -2,7 +2,7 @@ -Use this reference for .NET-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. +Use this reference for .NET-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-offs, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. ## Install and scaffold diff --git a/references/gcp-cloud-run/sdk-go.md b/references/gcp-cloud-run/sdk-go.md index 29fa0b7..1c2305a 100644 --- a/references/gcp-cloud-run/sdk-go.md +++ b/references/gcp-cloud-run/sdk-go.md @@ -2,7 +2,7 @@ -Use this reference for Go-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. +Use this reference for Go-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-offs, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. ## Install and scaffold diff --git a/references/gcp-cloud-run/sdk-java.md b/references/gcp-cloud-run/sdk-java.md index d858d09..2681e29 100644 --- a/references/gcp-cloud-run/sdk-java.md +++ b/references/gcp-cloud-run/sdk-java.md @@ -2,7 +2,7 @@ -Use this reference for Java-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. +Use this reference for Java-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-offs, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. ## Install and scaffold @@ -174,7 +174,7 @@ Every Workflow needs `VersioningBehavior.PINNED` or `AUTO_UPGRADE`. `Main` sets ## Connection configuration -`addApiKey` takes a **supplier**, called on every request, so a key can be rotated by returning a new value instead of restarting the Worker — worth using on a long-lived pool instance, where a restart is not free. +`addApiKey` takes a **supplier**, called on every request. The sample's supplier returns `TEMPORAL_API_KEY`, which Cloud Run fixes when the instance starts, so it cannot pick up a rotated key; a running instance keeps its starting key until it stops. Rotate as in [`setup.md`](setup.md#rotate-the-temporal-api-keys). Java uses gRPC/Netty and the JVM truststore, so it is unaffected by the `NativeCertsNotFound` failure the Rust-core SDKs hit. diff --git a/references/gcp-cloud-run/sdk-python.md b/references/gcp-cloud-run/sdk-python.md index 5af5b49..d99b2a5 100644 --- a/references/gcp-cloud-run/sdk-python.md +++ b/references/gcp-cloud-run/sdk-python.md @@ -2,7 +2,7 @@ -Use this reference for Python-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. +Use this reference for Python-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-offs, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. ## Install and scaffold diff --git a/references/gcp-cloud-run/sdk-typescript.md b/references/gcp-cloud-run/sdk-typescript.md index 9970ad6..cac8a35 100644 --- a/references/gcp-cloud-run/sdk-typescript.md +++ b/references/gcp-cloud-run/sdk-typescript.md @@ -2,7 +2,7 @@ -Use this reference for TypeScript-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-off, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. +Use this reference for TypeScript-specific Worker construction, versioning behavior, connection configuration, image packaging, and scale-in safety. For shared Cloud Run execution constraints, deployment lifecycle, permissions, versioning, observability, and diagnostics, see `constraints.md`, `setup.md`, `iam.md`, `versioning.md`, `observability.md`, and `diagnostics.md`. Scoping (Namespace, GCP project, region), the API-key hand-offs, IAM, registration, and verification are the same for every SDK and are defined once in `SKILL.md` and `setup.md`; do not vary them per SDK. This guide's sample is one Workflow that takes a string and returns `Hello, !`, which is what `setup.md` Step 8 verifies. ## Install and scaffold diff --git a/references/gcp-cloud-run/setup.md b/references/gcp-cloud-run/setup.md index 36cab8a..26d5a9b 100644 --- a/references/gcp-cloud-run/setup.md +++ b/references/gcp-cloud-run/setup.md @@ -12,11 +12,11 @@ End-to-end: write a standard Worker, containerize it, push the image, create a W - `gcloud` CLI and `jq` installed; authenticate `gcloud` before the preflight. The Google Cloud console or Terraform also work. - **Terraform** installed — Temporal ships the IAM setup as a Terraform module. - A Temporal SDK supported by this skill: Go, Python, TypeScript, Java, or .NET. -- A Temporal Cloud API key that can access the target Namespace. **The user creates it**, either in the Temporal Cloud UI under **Settings → API Keys** or by running `tcld apikey create --name --duration ` in their own terminal. Never run `tcld apikey create` from the agent's shell: it prints the new key. The command creates a key for the current user; add `--service-account-id ` to create it for an existing service account instead. Prefer a service-account-owned key for shared or long-lived Workers and a short-lived user key for a personal test. The key is needed twice, in the operator's Temporal CLI profile and in the Worker's Secret Manager runtime secret; the [API-key hand-off](#hand-off-the-temporal-api-key) sets both from one read. +- Two Temporal Cloud API keys that can access the target Namespace, one for each consumer. The **Worker's key** lives only in the Secret Manager runtime secret. The **operator's key** lives only in the operator's Temporal CLI profile. Both need the Namespace's Write permission, because polling Task Queues and changing Worker Deployment routing are both Write operations. Separate keys therefore do not reduce privilege; they let each key be rotated or revoked without breaking the other. **The user creates both keys**, either in the Temporal Cloud UI under **Settings → API Keys** or by running `tcld apikey create --name --duration ` in their own terminal. Never run `tcld apikey create` from the agent's shell: it prints the new key. The command creates a key for the current user; add `--service-account-id ` to create it for an existing service account instead. For a shared or long-lived Worker, give the Worker's key to a service account scoped to this Namespace, which an account administrator creates with `tcld service-account create-scoped --name --namespace-permission "=Write"`, and let the operator use their own key. For a personal test, the user may enter one short-lived user key in both [hand-offs](#hand-off-the-temporal-api-keys); say that rotating or revoking it then breaks both. -The `temporal` CLI commands in Steps 6–8 authenticate through a named CLI profile that the user creates during the [API-key hand-off](#hand-off-the-temporal-api-key). An exported variable in the user's terminal does not reach the agent's shell, so do not rely on `TEMPORAL_API_KEY` there. Never append `--api-key ` or put the key in an inline assignment. On macOS the default profile file is `~/Library/Application Support/temporalio/temporal.toml`. The examples below pass `--profile ` explicitly; omit it only when using a different already-configured authentication mechanism, including self-hosted mTLS. +The `temporal` CLI commands in Steps 6–8 authenticate through a named CLI profile that holds the operator's key, written by the [operator hand-off](#hand-off-the-temporal-api-keys) or reused from an earlier deployment. An exported variable in the user's terminal does not reach the agent's shell, so do not rely on `TEMPORAL_API_KEY` there. Never append `--api-key ` or put the key in an inline assignment. On macOS the default profile file is `~/Library/Application Support/temporalio/temporal.toml`. The examples below pass `--profile ` explicitly; omit it only when using a different already-configured authentication mechanism, including self-hosted mTLS. **Use the endpoint Temporal Cloud shows for the Namespace.** Copy the gRPC endpoint from the Namespace page in the Cloud UI, or from `tcld namespace get --namespace ` when `tcld` is signed in: `.uri.grpc` is the Namespace endpoint and `.uri.regionalGrpc` the regional API endpoint. Both accept API-key authentication; prefer `.uri.grpc` unless the user already uses the regional one. Do not construct it from the Namespace name or region. Use the same value, written `` below, for the CLI profile and for the pool's `TEMPORAL_ADDRESS`. @@ -54,7 +54,7 @@ gcloud iam service-accounts create \ --project ``` -Create the Secret Manager secret before deploying the pool. This creates only the secret; the key itself is added during the hand-off below: +Create the Secret Manager secret before deploying the pool. This creates only the secret; the Worker's key is added during the hand-off below: ```bash gcloud secrets create \ @@ -62,41 +62,62 @@ gcloud secrets create \ --project ``` -### Hand off the Temporal API key +### Hand off the Temporal API keys -The user runs this once, after the resource list is approved and the secret exists, **in their own interactive terminal**. It reads the key once, then writes both the CLI profile and the secret version. Do not run it through the agent's shell or a `!`-prefixed command: `read -s` needs a terminal, and the key would pass through the agent's session. Steps 1–3 and Step 5's Terraform need no key and can continue meanwhile. **Step 4 must wait:** the pool deploy fails until the secret has an `ENABLED` version. +There are two hand-offs, and each reads one key. The **Worker hand-off** adds the Worker's key to the secret. The **operator hand-off** writes the operator's key to the CLI profile. Skip the operator hand-off when the operator already has a profile that authenticates to this Namespace; see [Reuse what already works](#reuse-what-already-works). + +The user runs each script after the resource list is approved and the secret exists, **in their own interactive terminal**. Do not run either through the agent's shell or a `!`-prefixed command: `read -s` needs a terminal, and the key would pass through the agent's session. Steps 1–3 and Step 5's Terraform need no key and can continue meanwhile. **Step 4 waits for the Worker hand-off:** the pool deploy fails until the secret has an `ENABLED` version. **Step 6 waits for the operator's profile.** + +Worker hand-off: ```bash ( if [ ! -t 0 ]; then printf 'Run this in an interactive terminal; nothing was changed\n' >&2; exit 1 fi - printf 'Temporal API key: ' >&2 + printf 'Worker API key: ' >&2 IFS= read -r -s key printf '\nlen=%s\n' "${#key}" >&2 dots=${key//[^.]/} if [ "${#dots}" -ne 2 ]; then printf 'Expected a JWT-shaped key with two dots; nothing was changed\n' >&2; exit 1 fi - temporal --profile config set --prop address --value "" && - temporal --profile config set --prop namespace --value "" && - temporal --profile config set --prop api_key --value "$key" && printf %s "$key" | gcloud secrets versions add \ --data-file=- --project ) ``` -The parentheses run the script in a subshell, so the key does not stay in the user's shell afterwards. The key is piped with `printf %s` because `gcloud` stores standard input byte for byte: pasting it, pressing Enter, and sending EOF would store a trailing newline and corrupt the credential. `temporal config set` has no standard-input option, so the key is briefly visible in the process list while that command runs; on a shared machine, say so before the user runs it. +Operator hand-off: -**Make the hand-off impossible to miss; it is the same for every SDK.** Write the script above to a file outside the app directory, with every placeholder filled in, such as `/-handoff.sh`. It contains no key. Then end your turn with the user's action first: +```bash +( + if [ ! -t 0 ]; then + printf 'Run this in an interactive terminal; nothing was changed\n' >&2; exit 1 + fi + printf 'Operator API key: ' >&2 + IFS= read -r -s key + printf '\nlen=%s\n' "${#key}" >&2 + dots=${key//[^.]/} + if [ "${#dots}" -ne 2 ]; then + printf 'Expected a JWT-shaped key with two dots; nothing was changed\n' >&2; exit 1 + fi + temporal --profile config set --prop address --value "" && + temporal --profile config set --prop namespace --value "" && + temporal --profile config set --prop api_key --value "$key" +) +``` + +The parentheses run each script in a subshell, so the key does not stay in the user's shell afterwards. The Worker's key is piped with `printf %s` because `gcloud` stores standard input byte for byte: pasting it, pressing Enter, and sending EOF would store a trailing newline and corrupt the credential. `temporal config set` has no standard-input option, so the operator's key is briefly visible in the process list while that command runs; on a shared machine, say so before the user runs it. + +**Make the hand-offs impossible to miss; they are the same for every SDK.** Write each script to its own file outside the app directory and outside any repository, with every placeholder filled in, such as `/-worker-key.sh` and `/-operator-profile.sh`. Neither contains a key. Then end your turn with the user's action first: 1. One line saying the run is waiting on them. -2. The file's absolute path as plain text, and the one command to run it in their own terminal: `bash `. -3. What to reply when it finishes. +2. Each file's absolute path as plain text, Worker hand-off first, and the one command to run it in their own terminal: `bash `. Say which key each one asks for. +3. What to reply when they finish. -Put status, checklists, and notes after that. Never point back to "the script above": whenever the run is still blocked on the hand-off, repeat these steps in full. When the user replies, verify with the commands below rather than trusting the reply. +Put status, checklists, and notes after that. Never point back to "the script above": whenever the run is still blocked on a hand-off, repeat these steps in full. When the user replies, verify with the commands below rather than trusting the reply. -Then verify without reading the key back. The agent may run these: +Then verify without reading either key back. The agent may run these: ```bash temporal --profile config get --prop address @@ -106,9 +127,9 @@ gcloud secrets versions list --project \ --format='table(name,state,createTime)' ``` -Never read back `api_key` with `temporal config get`, and never run `gcloud secrets versions access`. The `worker deployment list` call proves the profile authenticates to the Namespace. Expect exactly one `ENABLED` secret version. If the hand-off ran more than once, keep the newest version, confirm the pool uses `latest`, and disable the older ones with `gcloud secrets versions disable --secret --project `. Secret versions are immutable: replace a bad one by adding a correct version, never by editing it. +Never read back `api_key` with `temporal config get`, and never run `gcloud secrets versions access`. The `worker deployment list` call proves the profile authenticates to the Namespace. Expect exactly one `ENABLED` secret version. If the Worker hand-off ran more than once, keep the newest version and disable the older ones with `gcloud secrets versions disable --secret --project `. Secret versions are immutable: replace a bad one by adding a correct version, never by editing it. -**Check the Temporal-side names now.** The CLI profile did not exist at approval time, so this is the first point at which the deployment name can be checked: +**Check the Temporal-side names now.** If the profile is new, it did not exist at approval time, so this is the first chance to check the deployment name: ```bash temporal --profile worker deployment describe --namespace --name @@ -116,7 +137,11 @@ temporal --profile worker deployment describe --namespace It should report that the deployment does not exist. If it exists, stop and agree a new name with the user before Step 6. -**If the user already has a CLI profile and a secret**, for example from an earlier deployment, reuse them instead of running the hand-off. Confirm the profile with the read-back commands above, always passing `--namespace`: a profile's stored Namespace can differ from the target. Confirm the secret has an `ENABLED` version with `gcloud secrets versions list`, then ask the user to confirm that it holds the same key as the profile; you must not read either value. Record a reused secret as shared, so teardown keeps it. +#### Reuse what already works + +**An existing CLI profile.** If the operator already has a profile for this Namespace, for example from an earlier deployment, skip the operator hand-off, but only after the user confirms that the profile's key is not also stored in a secret. A profile written together with a secret from a single key shares that key with every pool that mounts the secret, so rotating or revoking it breaks them. If the user cannot confirm, run the operator hand-off with a new key; you must not read either value to compare them. Confirm the profile with the read-back commands above, always passing `--namespace`: a profile's stored Namespace can differ from the target. + +**An existing secret.** Reuse a secret only when the user confirms that it holds a key meant for Workers; you must not read it. Do not reuse a secret that holds the operator's own key; run the Worker hand-off into a new secret instead. Confirm a reused secret has an `ENABLED` version with `gcloud secrets versions list`, and record it as shared, so teardown keeps it. Grant only the runner access to that secret. First check whether it already has access, and record the answer in the inventory: teardown removes only a binding this deployment added. This prints the member and exits `0` when the binding already exists: @@ -351,9 +376,24 @@ Confirm from two independent signals: - **Temporal** — Task completions in the Workflow's event history. - **Cloud Run** — pool logs showing Worker startup and Task processing, read with the [pool log query](diagnostics.md#read-the-pool-logs). **A scaled-to-zero pool emits no new logs.** +## Rotate the Temporal API keys + +Rotate each key on its own schedule, by overlap: the user creates the new key, its consumer switches to it, you confirm the consumer works, and only then is the old key disabled, with the user's approval, and later deleted. Never revoke the old key first. + +- **The operator's key.** Run the [operator hand-off](#hand-off-the-temporal-api-keys) with the new key, then confirm with `temporal --profile worker deployment list --namespace `. +- **The Worker's key.** Run the [Worker hand-off](#hand-off-the-temporal-api-keys), which adds a new secret version. Cloud Run reads the secret when an instance starts, so running instances keep the old key. Before the old key is disabled, list every pool that mounts the secret, in every region the project uses, and confirm from each pool's logs that every instance started before the new version has shut down. A requested count of `0` is not that evidence. Never disable the newest secret version. + + ```bash + gcloud run worker-pools list --region --project --format=json \ + | jq -r --arg s '' '.[] + | select([.. | objects | .secretKeyRef? // empty | (.name // .secret // "")] + | any(. == $s or endswith("/secrets/" + $s))) + | .metadata.name // .name' + ``` + ## Teardown -Record what you create as you go: project and region; the Artifact Registry repository, image tag and digest; the pool name; the runner and invoker service accounts, and whether this run created each; whether this run added the runner's secret-access binding; the Terraform state directory; the secret and its versions, and whether it is shared; the deployment name, build ID, and Task Queue; the local CLI profile name; and who created the Temporal API key. +Record what you create as you go: project and region; the Artifact Registry repository, image tag and digest; the pool name; the runner and invoker service accounts, and whether this run created each; whether this run added the runner's secret-access binding; the Terraform state directory; the secret and its versions, and whether it is shared; the deployment name, build ID, and Task Queue; the local CLI profile name, and whether this run wrote it; and, for each Temporal API key, who created it, which user or service account owns it, and whether it is the Worker's, the operator's, or both. Scale the pool to zero before deleting the version so its pollers stop without destroying the pool prematurely. @@ -405,6 +445,6 @@ Scale the pool to zero before deleting the version so its pollers stop without d ``` Keep any binding or runner the inventory does not attribute to this run. -7. Delete the container image from Artifact Registry, and any Secret Manager secret this run created that no other deployment uses; keep a reused or shared secret. Delete the Artifact Registry repository too if this run created it. Ask before revoking a Temporal Cloud API key: it is account-scoped, not deployment-scoped. Once nothing else uses the local CLI profile, remove it with `temporal config delete-profile --profile `. +7. Delete the container image from Artifact Registry, and any Secret Manager secret this run created that no other deployment uses; keep a reused or shared secret. Delete the Artifact Registry repository too if this run created it. Ask before revoking either Temporal Cloud API key: a key is account-scoped, not deployment-scoped, and the Worker's key may also be mounted by another pool. List those pools as in [Rotate the Temporal API keys](#rotate-the-temporal-api-keys) first. Once nothing else uses the local CLI profile, and only if this run wrote it, remove it with `temporal config delete-profile --profile `. Confirm each deletion with a list command rather than `describe`: a recently deleted service account can return `PERMISSION_DENIED` from `describe` instead of `NOT_FOUND`. For example, `gcloud iam service-accounts list --project --filter='email:' --format='value(email)'` prints nothing once the account is gone. diff --git a/references/wci.md b/references/wci.md index 993b510..95bb776 100644 --- a/references/wci.md +++ b/references/wci.md @@ -16,7 +16,7 @@ The WCI responds to sync match failures and Task Queue backlog or rate informati ## Inspect the WCI -Run these with the CLI connection the version was registered with: on Cloud Run, add `--profile ` (the profile the API-key hand-off writes); on AWS Lambda, use the connection configured in `aws-lambda/setup.md`. +Run these with the CLI connection the version was registered with: on Cloud Run, add `--profile ` (the operator's profile from `gcp-cloud-run/setup.md`); on AWS Lambda, use the connection configured in `aws-lambda/setup.md`. List WCI Workflows in the Namespace: