From ef050fd3c4c808aaf6b30c9e38d9254a7332d8e9 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Fri, 6 Nov 2026 08:49:30 -0600 Subject: [PATCH 1/4] Document split Agent Substrate atelet release Update installation, upgrade, export, and operations guides for running atelet as a separate Helm release. Clarify object-store and telemetry configuration, and document the limitation on git and S3 artifact sources. Signed-off-by: Rachael Graham --- .../kagent-docs/snippets/artifact-sources.md | 4 + .../snippets/helm-substrate-path.md | 1 + .../kagent-docs/snippets/image-substrate.md | 1 + .../content/kagent/1.x/examples/skills.md | 1 + .../observability/lightweight-otel-stack.md | 16 +++- .../kagent/1.x/observability/metrics.md | 8 +- .../kagent/1.x/observability/otel-stack.md | 16 +++- .../kagent/1.x/operations/export-to-helm.md | 5 +- .../1.x/operations/multiple-namespaces.md | 2 +- .../1.x/operations/tune-agent-substrate.md | 29 ++++-- .../kagent/1.x/operations/uninstall.md | 2 +- .../content/kagent/1.x/operations/upgrade.md | 91 ++++++++++++++----- .../content/kagent/1.x/setup/installation.md | 51 +++++++++-- 13 files changed, 182 insertions(+), 45 deletions(-) create mode 100644 docs-site/assets/kagent-docs/snippets/helm-substrate-path.md create mode 100644 docs-site/assets/kagent-docs/snippets/image-substrate.md diff --git a/docs-site/assets/kagent-docs/snippets/artifact-sources.md b/docs-site/assets/kagent-docs/snippets/artifact-sources.md index 4800294e..030cacc4 100644 --- a/docs-site/assets/kagent-docs/snippets/artifact-sources.md +++ b/docs-site/assets/kagent-docs/snippets/artifact-sources.md @@ -6,6 +6,10 @@ A skill or a plugin changes what an agent does, so kagent only accepts artifact - **`git`**: A repository URL together with a full commit identifier. An abbreviated commit, a branch, or a tag is rejected. - **`bucket.s3`**: An endpoint, bucket, and key, together with the `versionId` of that exact object version. A region is included where the service requires one for request signing. +> [!WARNING] +> **An agent with a `git` or `bucket.s3` source does not become ready.** The Actor that builds the golden snapshot fetches the source itself. No egress policy applies to that Actor, so Agent Substrate blocks the download, and the Agent's `Ready` condition reports `GoldenActorCrashed`. An EgressPolicy in the Agent's `egressPolicyRefs` field does not resolve the failure, because that field governs Sessions only. Use an `oci` source until this limitation is fixed. + + If a source sets more than one kind, or none of them, the API server rejects the AgentTemplate with the error `exactly one of the fields in [oci git bucket] must be set`. For an `oci` source, the Agent Substrate worker pulls the image with the registry credentials that it uses for workload images. The worker must therefore be able to reach the registry, and a private registry must accept those credentials. The agent's egress policies do not need to grant access to the registry. Agent Substrate mounts the image in the agent sandbox as a read-only volume, and the agent runtime reads the skill or plugin from that volume. diff --git a/docs-site/assets/kagent-docs/snippets/helm-substrate-path.md b/docs-site/assets/kagent-docs/snippets/helm-substrate-path.md new file mode 100644 index 00000000..ae50973b --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/helm-substrate-path.md @@ -0,0 +1 @@ +oci://ghcr.io/kagent-dev/substrate/helm \ No newline at end of file diff --git a/docs-site/assets/kagent-docs/snippets/image-substrate.md b/docs-site/assets/kagent-docs/snippets/image-substrate.md new file mode 100644 index 00000000..726d57e0 --- /dev/null +++ b/docs-site/assets/kagent-docs/snippets/image-substrate.md @@ -0,0 +1 @@ +ghcr.io/kagent-dev/substrate \ No newline at end of file diff --git a/docs-site/content/kagent/1.x/examples/skills.md b/docs-site/content/kagent/1.x/examples/skills.md index d8960f6f..a89257a5 100644 --- a/docs-site/content/kagent/1.x/examples/skills.md +++ b/docs-site/content/kagent/1.x/examples/skills.md @@ -391,6 +391,7 @@ A skill that kagent cannot fetch stops the agent from starting at all, rather th | ------- | ----- | | `pull : ... 401 Unauthorized` | The registry needs credentials that the cluster does not have. | | `pull : ... x509` or a Transport Layer Security (TLS) error | The registry does not serve HTTPS with a certificate that the runtime trusts. | + | `materialize skill "": exit status 128`, after `Could not resolve host` | The source is `git`, which the Actor that builds the golden snapshot cannot download. Use an `oci` source, as described in [Every source is immutable]({{< link path="skills-and-mcp/skills#every-source-is-immutable" >}}). | | `SKILL.md is required` | The artifact was fetched, but the root that `source.path` selects does not contain a `SKILL.md` file. | | `symlink "..." escapes artifact root` | A symlink in the artifact points outside it. | | `artifact contains more than 10000 filesystem entries`, or `artifact exceeds 104857600 bytes` | The artifact exceeds one of the plugin limits. | diff --git a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md index c10a4511..9c3984d9 100644 --- a/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md @@ -296,9 +296,17 @@ Turn on the kagent trace, log, and metrics exporters, and point them at the coll ## Send Agent Substrate telemetry to the collector -Upgrade the Agent Substrate Helm release. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower `samplingRatio` again for production. +Upgrade both Agent Substrate Helm releases. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower `samplingRatio` again for production. ```bash +helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ @@ -372,6 +380,12 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in --set config.telemetry=null \ --set config.runtimeDefaults.telemetry=null \ --set controller.metrics.enabled=false + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ diff --git a/docs-site/content/kagent/1.x/observability/metrics.md b/docs-site/content/kagent/1.x/observability/metrics.md index c834e6a2..f9368c07 100644 --- a/docs-site/content/kagent/1.x/observability/metrics.md +++ b/docs-site/content/kagent/1.x/observability/metrics.md @@ -66,9 +66,15 @@ The metric carries labels for the Agent name, model, and error type when those v ### Export the metrics -Agent Substrate exports metrics over OTLP when its Helm release has an OTLP endpoint. The same setting also turns on Agent Substrate traces and logs. +Agent Substrate exports metrics over OTLP when both of its Helm releases have an OTLP endpoint. The same setting also turns on Agent Substrate traces and logs. ```bash +helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ diff --git a/docs-site/content/kagent/1.x/observability/otel-stack.md b/docs-site/content/kagent/1.x/observability/otel-stack.md index 9f1d4410..255a3db6 100644 --- a/docs-site/content/kagent/1.x/observability/otel-stack.md +++ b/docs-site/content/kagent/1.x/observability/otel-stack.md @@ -443,8 +443,16 @@ Turn on the kagent trace, log, and metrics exporters, and point them at the coll Point Agent Substrate at the collector. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. -1. Upgrade the Agent Substrate Helm release. +1. Upgrade both Agent Substrate Helm releases. ```bash + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \ + --set otel.traces.samplingRatio=1.0 \ + --wait --timeout 10m helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ @@ -553,6 +561,12 @@ Log in to Grafana, and query each backend from the **Explore** view. --set config.runtimeDefaults.telemetry=null \ --set controller.metrics.enabled=false \ --set controller.metrics.serviceMonitor.enabled=false + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system --reuse-values \ + --set otel.endpoint="" \ + --set otel.traces.samplingRatio=0.01 helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ diff --git a/docs-site/content/kagent/1.x/operations/export-to-helm.md b/docs-site/content/kagent/1.x/operations/export-to-helm.md index 9d361c9c..72682cac 100644 --- a/docs-site/content/kagent/1.x/operations/export-to-helm.md +++ b/docs-site/content/kagent/1.x/operations/export-to-helm.md @@ -46,6 +46,7 @@ The export must reproduce the installation that runs in your cluster, so run the ├── kagent-crds.yaml ├── substrate-crds.yaml ├── substrate-podcert.yaml + ├── substrate-atelet.yaml ├── substrate.yaml └── kagent.yaml ``` @@ -56,7 +57,7 @@ The export must reproduce the installation that runs in your cluster, so run the ## Apply the export -The `install.sh` script runs `helm upgrade --install` for each release, in installation order, against your current kubeconfig context. In a cluster that `kagent install` set up, the releases already exist, so the script upgrades each release in place with the same values. +The `install.sh` script runs `helm upgrade --install` for each release, in installation order, against your current kubeconfig context. After the `substrate-atelet` release, the script waits for the `atelet` DaemonSet to roll out. In a cluster that `kagent install` set up, the releases already exist, so the script upgrades each release in place with the same values. 1. Confirm that your kubeconfig points at the cluster that runs kagent. ```bash @@ -73,7 +74,7 @@ The `install.sh` script runs `helm upgrade --install` for each release, in insta sh ./kagent-install/install.sh ``` -4. Confirm that the releases are deployed. The `STATUS` column shows `deployed` for `kagent-crds`, `kagent`, `substrate-crds`, `substrate`, and `substrate-podcert`. +4. Confirm that the releases are deployed. The `STATUS` column shows `deployed` for `kagent-crds`, `kagent`, `substrate-crds`, `substrate-atelet`, `substrate`, and `substrate-podcert`. ```bash helm list -A --filter 'kagent|substrate' ``` diff --git a/docs-site/content/kagent/1.x/operations/multiple-namespaces.md b/docs-site/content/kagent/1.x/operations/multiple-namespaces.md index df971ca8..c93104e7 100644 --- a/docs-site/content/kagent/1.x/operations/multiple-namespaces.md +++ b/docs-site/content/kagent/1.x/operations/multiple-namespaces.md @@ -63,7 +63,7 @@ The following steps prepare the `team-a` namespace. - name: gvisor configRef: name: gvisor-default - workerImage: ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}} + workerImage: {{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} template: serviceAccountName: team-a-worker EOF diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md index 995a312c..90a7b827 100644 --- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md +++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md @@ -49,7 +49,7 @@ Read the current state of the runtime before you change it. The rest of this gui "workerPools": [ { "name": "kagent-default", - "workerImage": "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}", + "workerImage": "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}", "readyReplicas": 1 } ], @@ -218,7 +218,7 @@ spec: replicas: 2 sandboxClasses: - name: gvisor - workerImage: "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}" + workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}" ``` Earlier releases placed Actors with a single `kagent.dev/worker-pool` label, and the chart no longer sets it. A Session or Sandbox that is pinned to a revision from one of those releases still selects that key. Keep the old label on the pool until you recreate those Sessions and Sandboxes. @@ -233,12 +233,13 @@ substrateWorkerPool: Agent Substrate writes every suspended Actor to an object store, so the bucket stores the Session state that a resume restores. The following settings decide where a snapshot lands. -- In the Agent Substrate Helm values, `atelet.storageBackend` selects the storage client. `rustfs.enabled` decides whether the chart supplies credentials for that client. +- In the `substrate` release values, `atelet.storageBackend` selects the storage client of the `ate-api-server`. `rustfs.enabled` decides whether the chart supplies credentials for that client. +- In the `substrate-atelet` release values, `objectStore` selects the storage client of `atelet`, with its endpoint and credentials. - In each {{< gloss "Harness" >}}Harness{{< /gloss >}}, `spec.substrate.snapshotPolicy.location` sets the bucket and prefix that the Harness's Actors write under. The field is required, and it must point into a bucket that the selected client can reach. ### The bundled object store -Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs an in-cluster object store and configures the `atelet` DaemonSet and the `ate-api-server` Deployment to use it. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults. +Because `rustfs.enabled` defaults to `true`, the `substrate` chart installs an in-cluster object store and configures the `ate-api-server` Deployment to use it. The [installation guide]({{< link path="setup/installation#install-agent-substrate" >}}) points the `substrate-atelet` release at the same store through its `objectStore` values. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults. > [!IMPORTANT] > The bundled store is not durable, and every installation receives the same credentials for it. Before a production installation serves real traffic, [move its snapshots to your own bucket](#move-snapshots-to-an-external-bucket). @@ -247,13 +248,19 @@ Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs Replace the bundled store with a bucket in Google Cloud Storage, Amazon S3, or an S3-compatible store. The Agent Substrate backend and every Harness location must point at the same bucket. -1. Save an Agent Substrate values file that disables the bundled store and selects the storage backend. The `s3` value selects the S3 client. Any other value, including an empty one, selects Google Cloud Storage. +1. Save an Agent Substrate values file that disables the bundled store and selects the storage backend. Both Agent Substrate releases read this file. Set `atelet.storageBackend` and `objectStore.backend` to `s3` for Amazon S3 or an S3-compatible store, or to `gcs` for Google Cloud Storage. The empty `objectStore` values clear the endpoint and the credentials of the bundled store from the `substrate-atelet` release. ```bash cat > substrate-storage.yaml <}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --reuse-values \ + -f substrate-storage.yaml helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ @@ -364,7 +377,7 @@ Set the pool's class, its Worker image, and its SandboxConfig together. cat > kagent-sandbox-class.yaml <}}" + workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}" EOF ``` {{% /tab %}} @@ -373,7 +386,7 @@ Set the pool's class, its Worker image, and its SandboxConfig together. cat > kagent-sandbox-class.yaml <}}" + workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-microvm:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}" EOF ``` {{% /tab %}} diff --git a/docs-site/content/kagent/1.x/operations/uninstall.md b/docs-site/content/kagent/1.x/operations/uninstall.md index b0e37cf5..87618a6b 100644 --- a/docs-site/content/kagent/1.x/operations/uninstall.md +++ b/docs-site/content/kagent/1.x/operations/uninstall.md @@ -9,7 +9,7 @@ The `kagent uninstall` command removes everything that `kagent install` created. The command removes: -- The `kagent`, `kagent-crds`, `substrate`, `substrate-podcert`, and `substrate-crds` Helm releases +- The `kagent`, `kagent-crds`, `substrate`, `substrate-atelet`, `substrate-podcert`, and `substrate-crds` Helm releases - Every `kagent.dev` and `ate.dev` custom resource definition, and with them every resource of those kinds in every namespace, such as Agents, Harnesses, ModelConfigs, and SandboxTemplates - The development PostgreSQL database that `kagent install` deployed, together with its PersistentVolumeClaim and the data on it - The `ate-system` and `podcertificate-controller-system` namespaces, and the identity pools and certificate authorities in them diff --git a/docs-site/content/kagent/1.x/operations/upgrade.md b/docs-site/content/kagent/1.x/operations/upgrade.md index db123159..5af9cbf4 100644 --- a/docs-site/content/kagent/1.x/operations/upgrade.md +++ b/docs-site/content/kagent/1.x/operations/upgrade.md @@ -5,7 +5,7 @@ weight: 35 author: kagent.dev --- -An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of five Helm releases. Upgrade the releases in the order that you installed them, starting with the CRDs, then Agent Substrate, and then the kagent control plane. +An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of six Helm releases. Upgrade the kagent CRDs first, then Agent Substrate in the order that it supports, and then the kagent control plane. Upgrade from one minor release to the next, such as from 1.0 to 1.1, and do not skip a minor release. Before you move to the next minor release, upgrade to the latest patch version of your current minor release. Patch releases do not change the database schema, so you can upgrade directly to the latest patch release without the patch releases in between. To move a 0.10.x installation to 1.x, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x" >}}) instead. @@ -21,11 +21,13 @@ Complete every step in this section before you change any release. The steps mak ```bash helm list -A --filter 'kagent|substrate' ``` - The output lists five releases. The `CHART` column shows the version of each release. + The output lists six releases. The `CHART` column shows the version of each release. * The `kagent` and `kagent-crds` releases are in the `kagent` namespace. - * The `substrate` and `substrate-crds` releases are in the `ate-system` namespace. + * The `substrate`, `substrate-atelet`, and `substrate-crds` releases are in the `ate-system` namespace. * The `substrate-podcert` release is in the `podcertificate-controller-system` namespace. + If the output does not list a `substrate-atelet` release, the `substrate` release still runs `atelet`. Move `atelet` to its own release first, as described in [Moving atelet to substrate-atelet](https://github.com/kagent-dev/substrate/releases/tag/v1.0.0-rc4#moving-atelet-to-substrate-atelet), with the kagent-specific changes in the [kagent Helm chart README](https://github.com/kagent-dev/kagent/blob/main/helm/README.md). + 4. Back up the PostgreSQL database that kagent and Agent Substrate share. The upgrade keeps your data. However, the kagent controller and the Agent Substrate API server change the database schema when they start on the new version. To return to the previous version, you must restore the backup or roll the schema back. ```bash pg_dump "postgresql://:@:5432/kagent" \ @@ -42,6 +44,7 @@ Complete every step in this section before you change any release. The steps mak 5. Save the values of each release to a file. The upgrade commands reapply these files, so each release keeps its database settings and model provider settings. The `--reuse-values` flag also reapplies the defaults of the previous chart version, and the new chart can reject a default that it removed. Do not use that flag. ```bash helm get values substrate-podcert -n podcertificate-controller-system -o yaml > substrate-podcert-values.yaml + helm get values substrate-atelet -n ate-system -o yaml > substrate-atelet-values.yaml helm get values substrate -n ate-system -o yaml > substrate-values.yaml helm get values kagent -n kagent -o yaml > kagent-values.yaml ``` @@ -53,9 +56,14 @@ Complete every step in this section before you change any release. The steps mak export SUBSTRATE_VERSION={{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} ``` -## Upgrade the CRDs +7. Save the installed kagent chart version. The [Agent Substrate upgrade](#upgrade-agent-substrate) uses this version to change the Worker image without upgrading kagent. + ```bash + export CURRENT_KAGENT_VERSION=$(helm get metadata kagent -n kagent -o json | jq -r .version) + ``` -Upgrade the kagent and Agent Substrate custom resource definitions (CRDs) before the controllers that read them. Both charts ship the CRDs as templates, so `helm upgrade` updates the CRDs in place. +## Upgrade the kagent CRDs + +Upgrade the kagent custom resource definitions (CRDs) before the controller that reads them. The chart ships the CRDs as templates, so `helm upgrade` updates the CRDs in place. 1. Upgrade the kagent CRDs. ```bash @@ -65,37 +73,72 @@ Upgrade the kagent and Agent Substrate custom resource definitions (CRDs) before --namespace kagent --wait ``` -2. Upgrade the Agent Substrate CRDs. +2. Confirm that the CRD release reports the target version. ```bash - helm upgrade substrate-crds \ - oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \ - --version ${SUBSTRATE_VERSION} \ - --namespace ate-system --wait + helm list -n kagent --filter 'crds' ``` - -3. Confirm that both CRD releases report the target version. - ```bash - helm list -A --filter 'crds' - ``` - In the output, verify that the `STATUS` column reports `deployed` for both releases. The `CHART` column shows `kagent-crds-` with the target kagent version, and `substrate-crds-` with the target Agent Substrate version. + In the output, verify that the `STATUS` column reports `deployed`, and that the `CHART` column shows `kagent-crds-` with the target kagent version. ## Upgrade Agent Substrate -Upgrade the Agent Substrate PodCertificate controller, followed by the Agent Substrate control plane and data plane. The kagent controller connects to the Agent Substrate API server, so Agent Substrate moves to the new version first. - - +Upgrade `atelet` first, then the Worker image of each WorkerPool, then the Agent Substrate CRDs and the `substrate` release. Agent Substrate does not support a Worker on an older `atelet`, or a control plane that is newer than its Workers. A microVM WorkerPool that is left on the previous Worker image cannot start or restore Actors after the `substrate` upgrade. 1. Upgrade the PodCertificate controller. ```bash helm upgrade substrate-podcert \ - oci://ghcr.io/kagent-dev/substrate/helm/substrate-podcert \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-podcert \ --version ${SUBSTRATE_VERSION} \ --namespace podcertificate-controller-system \ --wait --timeout 5m \ -f substrate-podcert-values.yaml ``` -2. Upgrade Agent Substrate. +2. Upgrade `atelet`, and wait for the rollout to finish. Do not continue until the output reports that the `atelet` DaemonSet rolled out. + ```bash + helm upgrade substrate-atelet \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \ + --version ${SUBSTRATE_VERSION} \ + --namespace ate-system \ + --wait --timeout 1h \ + -f substrate-atelet-values.yaml + kubectl rollout status daemonset/atelet -n ate-system --timeout=1h + ``` + Example output: + ```console + daemon set "atelet" successfully rolled out + ``` + +3. Move the `kagent-default` WorkerPool to the new Worker image through the kagent release. The command keeps the installed kagent chart, so the `--reuse-values` flag changes only the Worker image. + * For a microVM WorkerPool, use the `ateworker-microvm` image. + * If Git or a pipeline applies your other WorkerPools, change `workerImage` there too. + ```bash + helm upgrade kagent \ + {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \ + --version ${CURRENT_KAGENT_VERSION} \ + --namespace kagent --reuse-values \ + --set substrateWorkerPool.workerImage={{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v${SUBSTRATE_VERSION} \ + --wait + ``` + +4. Confirm that every WorkerPool lists the new image. + ```bash + kubectl get workerpools -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IMAGE:.spec.workerImage' + ``` + In the output, verify that the `IMAGE` column of every WorkerPool ends in `:v`. + ```console + NAMESPACE NAME IMAGE + kagent kagent-default {{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} + ``` + +5. Upgrade the Agent Substrate CRDs. + ```bash + helm upgrade substrate-crds \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-crds \ + --version ${SUBSTRATE_VERSION} \ + --namespace ate-system --wait + ``` + +6. Upgrade the Agent Substrate control plane and data plane. ```bash helm upgrade substrate \ {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \ @@ -105,7 +148,7 @@ Upgrade the Agent Substrate PodCertificate controller, followed by the Agent Sub -f substrate-values.yaml ``` -3. Confirm that the Agent Substrate pods are running. +7. Confirm that the Agent Substrate pods are running. ```bash kubectl get pods -n ate-system ``` @@ -124,7 +167,7 @@ Upgrade the kagent control plane. The upgrade has no separate migration step. -f kagent-values.yaml \ -f - <}}/ateworker-gvisor:v${SUBSTRATE_VERSION}" EOF ``` @@ -144,7 +187,7 @@ Confirm that every release runs the new version and that the agents are ready. ```bash helm list -A --filter 'kagent|substrate' ``` - For the `kagent` and `kagent-crds` releases, the `CHART` column shows the target kagent version. For the `substrate`, `substrate-crds`, and `substrate-podcert` releases, the `CHART` column shows the target Agent Substrate version. + For the `kagent` and `kagent-crds` releases, the `CHART` column shows the target kagent version. For the `substrate`, `substrate-atelet`, `substrate-crds`, and `substrate-podcert` releases, the `CHART` column shows the target Agent Substrate version. 2. Confirm that the kagent pods are running. ```bash diff --git a/docs-site/content/kagent/1.x/setup/installation.md b/docs-site/content/kagent/1.x/setup/installation.md index 864784f9..5a6ac28f 100644 --- a/docs-site/content/kagent/1.x/setup/installation.md +++ b/docs-site/content/kagent/1.x/setup/installation.md @@ -148,7 +148,7 @@ The kagent and Agent Substrate custom resource definitions (CRDs) install from t 2. Install the Agent Substrate CRDs. ```bash helm upgrade --install substrate-crds \ - oci://ghcr.io/kagent-dev/substrate/helm/substrate-crds \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-crds \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ --namespace ate-system --create-namespace --wait ``` @@ -229,7 +229,7 @@ Agent Substrate authenticates its components with mutual Transport Layer Securit 3. Install the Agent Substrate PodCertificate controller, which signs the certificates that components present to each other. ```bash helm upgrade --install substrate-podcert \ - oci://ghcr.io/kagent-dev/substrate/helm/substrate-podcert \ + {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-podcert \ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ --namespace podcertificate-controller-system --create-namespace \ --wait --timeout 5m @@ -307,9 +307,46 @@ kagent and Agent Substrate each connect with an owner login, which applies migra ### Install Agent Substrate -The `substrate` release installs the Agent Substrate control plane and data plane. The release name must be `substrate`, and the namespace must be `ate-system`. +Agent Substrate installs as two releases in the `ate-system` namespace. The `substrate-atelet` release runs `atelet`, the node agent that starts the Workers on each node. The `substrate` release runs the control plane and the rest of the data plane. The releases must be named `substrate-atelet` and `substrate`. -1. Install Agent Substrate. The command sets the following values. +1. Save the object store values for `atelet`. The `substrate-atelet` release cannot read the settings of the bundled object store from the `substrate` release, so you set them. + 1. Create a Secret with the default `rustfs.accessKey` and `rustfs.secretKey` values of the `substrate` chart. + ```bash + kubectl create secret generic kagent-rustfs-credentials -n ate-system \ + --from-literal=AWS_ACCESS_KEY_ID=rustfsadmin \ + --from-literal=AWS_SECRET_ACCESS_KEY=rustfsadmin + ``` + 2. Save the values that point `atelet` to the bundled object store and the Secret. To use your own bucket instead, see [Move snapshots to an external bucket]({{< link path="operations/tune-agent-substrate#move-snapshots-to-an-external-bucket" >}}). + ```bash + cat > substrate-atelet-values.yaml <}}/substrate-atelet \ + --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \ + --namespace ate-system \ + --wait --timeout 10m \ + -f substrate-atelet-values.yaml + kubectl rollout status daemonset/atelet -n ate-system --timeout=5m + ``` + Example output: + ```console + daemon set "atelet" successfully rolled out + ``` + +3. Install the Agent Substrate control plane and data plane. The command sets the following values. + * `atelet.enabled: false` leaves `atelet` out of the `substrate` release, because the `substrate-atelet` release runs it. * `preview` turns on the `PerTemplateSnapshotLocation` gate, so that Agent Substrate stores the snapshots of each agent at the location that its Harness names. * `ateApi.authorization.bootstrapOwners` grants owner access to the Agent Substrate API to the kagent controller and to the `ate-client` ServiceAccount. The API server authorizes every call, and without the controller entry every Agent reports `RuntimePreparationFailed (PermissionDenied)`. * `atenetRouter.ingressAllowedSPIFFEIDs` lets the kagent controller send traffic to agents through the Agent Substrate router. @@ -321,6 +358,8 @@ The `substrate` release installs the Agent Substrate control plane and data plan --namespace ate-system \ --wait --timeout 10m \ -f - <}}" + workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}" EOF ``` From 84c0fe48af6cbc8b69eee0e681f81c03de1a2fee Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Fri, 6 Nov 2026 08:50:36 -0600 Subject: [PATCH 2/4] Update upgrade.md Signed-off-by: Rachael Graham --- docs-site/content/kagent/1.x/operations/upgrade.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs-site/content/kagent/1.x/operations/upgrade.md b/docs-site/content/kagent/1.x/operations/upgrade.md index 5af9cbf4..5828bdcc 100644 --- a/docs-site/content/kagent/1.x/operations/upgrade.md +++ b/docs-site/content/kagent/1.x/operations/upgrade.md @@ -5,7 +5,7 @@ weight: 35 author: kagent.dev --- -An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of six Helm releases. Upgrade the kagent CRDs first, then Agent Substrate in the order that it supports, and then the kagent control plane. +An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of separate Helm releases for the custom resource definitions (CRDs), Agent Substrate, and the kagent control plane. Upgrade the kagent CRDs first, then Agent Substrate in the order that it supports, and then the kagent control plane. Upgrade from one minor release to the next, such as from 1.0 to 1.1, and do not skip a minor release. Before you move to the next minor release, upgrade to the latest patch version of your current minor release. Patch releases do not change the database schema, so you can upgrade directly to the latest patch release without the patch releases in between. To move a 0.10.x installation to 1.x, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x" >}}) instead. @@ -21,7 +21,7 @@ Complete every step in this section before you change any release. The steps mak ```bash helm list -A --filter 'kagent|substrate' ``` - The output lists six releases. The `CHART` column shows the version of each release. + The `CHART` column shows the version of each release. * The `kagent` and `kagent-crds` releases are in the `kagent` namespace. * The `substrate`, `substrate-atelet`, and `substrate-crds` releases are in the `ate-system` namespace. * The `substrate-podcert` release is in the `podcertificate-controller-system` namespace. @@ -63,7 +63,7 @@ Complete every step in this section before you change any release. The steps mak ## Upgrade the kagent CRDs -Upgrade the kagent custom resource definitions (CRDs) before the controller that reads them. The chart ships the CRDs as templates, so `helm upgrade` updates the CRDs in place. +Upgrade the kagent CRDs before the controller that reads them. The chart ships the CRDs as templates, so `helm upgrade` updates the CRDs in place. 1. Upgrade the kagent CRDs. ```bash From aa15113d9bf22537169780cb05480a7e38a4091c Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Fri, 6 Nov 2026 11:01:42 -0600 Subject: [PATCH 3/4] Update docs-site/content/kagent/1.x/operations/upgrade.md Co-authored-by: Nadine Spies Signed-off-by: Rachael Graham --- docs-site/content/kagent/1.x/operations/upgrade.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-site/content/kagent/1.x/operations/upgrade.md b/docs-site/content/kagent/1.x/operations/upgrade.md index 5828bdcc..e5346711 100644 --- a/docs-site/content/kagent/1.x/operations/upgrade.md +++ b/docs-site/content/kagent/1.x/operations/upgrade.md @@ -5,7 +5,7 @@ weight: 35 author: kagent.dev --- -An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of separate Helm releases for the custom resource definitions (CRDs), Agent Substrate, and the kagent control plane. Upgrade the kagent CRDs first, then Agent Substrate in the order that it supports, and then the kagent control plane. +An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of separate Helm releases for the custom resource definitions (CRDs), Agent Substrate, and the kagent control plane. Upgrade the kagent CRDs first, then Agent Substrate, and then the kagent control plane. Upgrade from one minor release to the next, such as from 1.0 to 1.1, and do not skip a minor release. Before you move to the next minor release, upgrade to the latest patch version of your current minor release. Patch releases do not change the database schema, so you can upgrade directly to the latest patch release without the patch releases in between. To move a 0.10.x installation to 1.x, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x" >}}) instead. From e040a008f160a7336b22158995341571238b2106 Mon Sep 17 00:00:00 2001 From: Rachael Graham Date: Fri, 6 Nov 2026 11:01:54 -0600 Subject: [PATCH 4/4] conventions pass Signed-off-by: Rachael Graham --- .../kagent/1.x/reference/a2a-metadata.md | 72 +++++++------------ 1 file changed, 25 insertions(+), 47 deletions(-) diff --git a/docs-site/content/kagent/1.x/reference/a2a-metadata.md b/docs-site/content/kagent/1.x/reference/a2a-metadata.md index ccc5505d..12a9f89e 100644 --- a/docs-site/content/kagent/1.x/reference/a2a-metadata.md +++ b/docs-site/content/kagent/1.x/reference/a2a-metadata.md @@ -1,23 +1,15 @@ --- -title: A2A Metadata and Extension Reference -description: Reference the canonical A2A metadata keys and versioned extensions that Kagent supports. +title: A2A metadata and extension reference +description: Reference the canonical A2A metadata keys and versioned extensions that kagent supports. weight: 15 author: kagent.dev --- ## Metadata -A2A metadata is a set of key-value pairs on A2A elements that allows the A2A -protocol to carry extra information. +Agent2Agent (A2A) metadata is a set of key-value pairs on A2A elements that lets the A2A protocol transmit extra information. -Kagent runtime adapters translate metadata that a harness produces into -canonical Kagent metadata keys and discard metadata that they cannot translate. -Although these metadata semantics and formats are stable unless otherwise -noted, Kagent depends on individual harnesses to produce them. Harnesses that -the Kagent project does not own might emit different metadata across versions. -The [kagent harness]({{< link path="agents/agent-harness#kagent-harness" >}}) -is owned by the Kagent project, and its emitted metadata is stable across -versions. +kagent runtime adapters translate the metadata that a harness produces into canonical kagent metadata keys, and discard metadata that they cannot translate. These metadata semantics and formats are stable unless otherwise noted. However, kagent depends on each harness to produce them, and a harness that the kagent project does not own might emit different metadata across versions. The kagent project owns the [kagent harness]({{< link path="agents/agent-harness#kagent-harness" >}}), so the metadata from that harness is stable across versions. ### `kagent.dev/a2a/timeline-position` @@ -35,27 +27,22 @@ An RFC 3339 timestamp that records when the task was created. **A2A element:** Data part. -Identifies the semantic kind of a data part. The value depends on the runtime -adapter. The supported values are: +The semantic kind of a data part. The value depends on the runtime adapter that produced the part. Of the following values, the kagent harness recognizes `function_call` and `function_response`. -* `function_call` - The name and arguments of a function call. -* `function_response` - The return value of a function call. -* `executable_code` - A piece of code to execute. -* `code_execution_result` - The result of a code execution. - -The [kagent harness]({{< link path="agents/agent-harness#kagent-harness" >}}) -recognizes `function_call` and `function_response`. +| Value | Meaning | +| --- | --- | +| `function_call` | The name and arguments of a function call. | +| `function_response` | The return value of a function call. | +| `executable_code` | A piece of code to execute. | +| `code_execution_result` | The result of a code execution. | ### `kagent.dev/a2a/usage` **A2A element:** Task event. -Best-effort model usage reporting for one response when the runtime reports it. -The format of this value is harness-specific and may change in the future. For -normalized, task-lifetime usage totals, use the [token-usage extension](#token-usage). +Best-effort model usage for one response, present only if the runtime reports it. The format of this value is harness-specific and might change in later versions. For normalized usage totals over the lifetime of a task, use the [token-usage extension](#token-usage). -The [kagent harness]({{< link path="agents/agent-harness#kagent-harness" >}}) -reports the following camelCase metadata for a single model response: +The kagent harness reports the following camelCase metadata for a single model response: ```json { @@ -73,43 +60,31 @@ reports the following camelCase metadata for a single model response: **A2A element:** Result part. -Binds a structured result to the SHA-256 digest of its compiled output schema. -Consumers can use the digest to identify the schema of the result. +The SHA-256 digest of the compiled output schema of a structured result. Consumers can use the digest to identify the schema of the result. ### `kagent.dev/a2a/thought` **A2A element:** Text part. -A boolean that indicates the text contains model reasoning rather than the -model's answer. +A boolean that indicates that the text contains model reasoning rather than the model's answer. ## A2A extensions -[A2A extensions](https://a2a-protocol.org/latest/topics/extensions/) add -semantics that the core A2A protocol does not define. Agents advertise -extensions in their Agent Cards. Harness support varies by extension. +[A2A extensions](https://a2a-protocol.org/latest/topics/extensions/) add semantics that the core A2A protocol does not define. Agents advertise extensions in their Agent Cards. Harness support varies by extension. ### Human in the loop -The human-in-the-loop (HITL) extension lets an agent pause a task for a -person's decision and resume the same operation. See -[Human in the loop]({{< link path="agents/human-in-the-loop" >}}). +The human-in-the-loop extension lets an agent pause a task for a person's decision and then resume the same operation. For details, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}). ### Token usage -The token-usage extension reports task-lifetime token usage. Its URI is: +The token-usage extension reports token usage over the lifetime of a task. Its URI is: ```text https://kagent.dev/extensions/usage/v1 ``` -The [kagent harness]({{< link path="agents/agent-harness#kagent-harness" >}}) -advertises this optional extension when it supports usage reporting. The -payload is stored under the extension URI in the metadata of the status update -that ends an execution: `completed`, -`input-required`, `failed`, or `canceled`. When a client reads the persisted -task later, it finds the same payload in the task's metadata. The key is -omitted when no model call reported usage. +When the kagent harness supports usage reporting, it advertises this optional extension. The payload appears under the extension URI in the metadata of the status update that ends an execution. An execution ends in the `completed`, `input-required`, `failed`, or `canceled` state. When a client reads the persisted task later, the task's metadata contains the same payload. When no model call reported usage, the metadata does not contain the key. ```json { @@ -141,11 +116,14 @@ omitted when no model call reported usage. } ``` +The top-level counts are totals for the whole task. The `models` array reports the same fields for each model that the task called. + | Field | Meaning | | --- | --- | | `inputTokens` | Prompt tokens sent to the model. | | `outputTokens` | Generated tokens, excluding reasoning tokens. | -| `reasoningTokens` | Reasoning tokens, when the provider reports them. | -| `cachedInputTokens` | Provider-cached input tokens; a subset of `inputTokens`. | -| `totalTokens` | The provider-reported total, or the sum of input, output, and reasoning tokens when the provider reports no total. | +| `reasoningTokens` | Reasoning tokens, if the provider reports them. | +| `cachedInputTokens` | Provider-cached input tokens, a subset of `inputTokens`. | +| `totalTokens` | The provider-reported total. If the provider reports no total, the sum of input, output, and reasoning tokens. | | `models` | The same counts per model, ordered by model name. | +| `models[].model` | The model identifier. |