Document split Agent Substrate atelet release

docs-site/assets/kagent-docs/snippets/artifact-sources.md

@@ -6,6 +6,10 @@ A skill or a plugin changes what an agent does, so kagent only accepts artifact
 - **`git`**: A repository URL together with a full commit identifier. An abbreviated commit, a branch, or a tag is rejected.
 - **`bucket.s3`**: An endpoint, bucket, and key, together with the `versionId` of that exact object version. A region is included where the service requires one for request signing.
 
+> [!WARNING]
+> **An agent with a `git` or `bucket.s3` source does not become ready.** The Actor that builds the golden snapshot fetches the source itself. No egress policy applies to that Actor, so Agent Substrate blocks the download, and the Agent's `Ready` condition reports `GoldenActorCrashed`. An EgressPolicy in the Agent's `egressPolicyRefs` field does not resolve the failure, because that field governs Sessions only. Use an `oci` source until this limitation is fixed.
+<!-- REVIEW: kagent-dev/kagent#3435 fixes this by setting golden_egress_policy, which needs the Agent Substrate GoldenEgressPolicy preview gate. Verified live 2026-11-05 for a git source (SEFK v2026.11.0, which bundles kagent 1.0.0-rc2); S3 was not run. Remove this warning when a release with #3435 ships, and add the gate to the Helm installation path. -->
+
 If a source sets more than one kind, or none of them, the API server rejects the AgentTemplate with the error `exactly one of the fields in [oci git bucket] must be set`.
 
 For an `oci` source, the Agent Substrate worker pulls the image with the registry credentials that it uses for workload images. The worker must therefore be able to reach the registry, and a private registry must accept those credentials. The agent's egress policies do not need to grant access to the registry. Agent Substrate mounts the image in the agent sandbox as a read-only volume, and the agent runtime reads the skill or plugin from that volume.

docs-site/assets/kagent-docs/snippets/helm-substrate-path.md

@@ -0,0 +1 @@
+oci://ghcr.io/kagent-dev/substrate/helm

docs-site/assets/kagent-docs/snippets/image-substrate.md

@@ -0,0 +1 @@
+ghcr.io/kagent-dev/substrate

docs-site/content/kagent/1.x/examples/skills.md

@@ -391,6 +391,7 @@ A skill that kagent cannot fetch stops the agent from starting at all, rather th
    | ------- | ----- |
    | `pull <image>: ... 401 Unauthorized` | The registry needs credentials that the cluster does not have. |
    | `pull <image>: ... x509` or a Transport Layer Security (TLS) error | The registry does not serve HTTPS with a certificate that the runtime trusts. |
+   | `materialize skill "<name>": exit status 128`, after `Could not resolve host` | The source is `git`, which the Actor that builds the golden snapshot cannot download. Use an `oci` source, as described in [Every source is immutable]({{< link path="skills-and-mcp/skills#every-source-is-immutable" >}}). |
    | `SKILL.md is required` | The artifact was fetched, but the root that `source.path` selects does not contain a `SKILL.md` file. |
    | `symlink "..." escapes artifact root` | A symlink in the artifact points outside it. |
    | `artifact contains more than 10000 filesystem entries`, or `artifact exceeds 104857600 bytes` | The artifact exceeds one of the plugin limits. |

docs-site/content/kagent/1.x/observability/lightweight-otel-stack.md

@@ -296,9 +296,17 @@ Turn on the kagent trace, log, and metrics exporters, and point them at the coll
 
 ## Send Agent Substrate telemetry to the collector
 
-Upgrade the Agent Substrate Helm release. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower `samplingRatio` again for production.
+Upgrade both Agent Substrate Helm releases. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower `samplingRatio` again for production.
 
 ```bash
+helm upgrade substrate-atelet \
+  {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \
+  --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+  --namespace ate-system \
+  --reuse-values \
+  --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \
+  --set otel.traces.samplingRatio=1.0 \
+  --wait --timeout 10m
 helm upgrade substrate \
   {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
   --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
@@ -372,6 +380,12 @@ Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in
      --set config.telemetry=null \
      --set config.runtimeDefaults.telemetry=null \
      --set controller.metrics.enabled=false
+   helm upgrade substrate-atelet \
+     {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \
+     --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+     --namespace ate-system --reuse-values \
+     --set otel.endpoint="" \
+     --set otel.traces.samplingRatio=0.01
    helm upgrade substrate \
      {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
      --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \

docs-site/content/kagent/1.x/observability/metrics.md

@@ -66,9 +66,15 @@ The metric carries labels for the Agent name, model, and error type when those v
 
 ### Export the metrics
 
-Agent Substrate exports metrics over OTLP when its Helm release has an OTLP endpoint. The same setting also turns on Agent Substrate traces and logs.
+Agent Substrate exports metrics over OTLP when both of its Helm releases have an OTLP endpoint. The same setting also turns on Agent Substrate traces and logs.
 
 ```bash
+helm upgrade substrate-atelet \
+  {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \
+  --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+  --namespace ate-system \
+  --reuse-values \
+  --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317
 helm upgrade substrate \
   {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
   --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \

docs-site/content/kagent/1.x/observability/otel-stack.md

@@ -443,8 +443,16 @@ Turn on the kagent trace, log, and metrics exporters, and point them at the coll
 
 Point Agent Substrate at the collector. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component.
 
-1. Upgrade the Agent Substrate Helm release.
+1. Upgrade both Agent Substrate Helm releases.
    ```bash
+   helm upgrade substrate-atelet \
+     {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \
+     --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+     --namespace ate-system \
+     --reuse-values \
+     --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \
+     --set otel.traces.samplingRatio=1.0 \
+     --wait --timeout 10m
    helm upgrade substrate \
      {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
      --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
@@ -553,6 +561,12 @@ Log in to Grafana, and query each backend from the **Explore** view.
      --set config.runtimeDefaults.telemetry=null \
      --set controller.metrics.enabled=false \
      --set controller.metrics.serviceMonitor.enabled=false
+   helm upgrade substrate-atelet \
+     {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \
+     --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+     --namespace ate-system --reuse-values \
+     --set otel.endpoint="" \
+     --set otel.traces.samplingRatio=0.01
    helm upgrade substrate \
      {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
      --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \

docs-site/content/kagent/1.x/operations/export-to-helm.md

@@ -46,6 +46,7 @@ The export must reproduce the installation that runs in your cluster, so run the
        ├── kagent-crds.yaml
        ├── substrate-crds.yaml
        ├── substrate-podcert.yaml
+       ├── substrate-atelet.yaml
        ├── substrate.yaml
        └── kagent.yaml
    ```
@@ -56,7 +57,7 @@ The export must reproduce the installation that runs in your cluster, so run the
 
 ## Apply the export
 
-The `install.sh` script runs `helm upgrade --install` for each release, in installation order, against your current kubeconfig context. In a cluster that `kagent install` set up, the releases already exist, so the script upgrades each release in place with the same values.
+The `install.sh` script runs `helm upgrade --install` for each release, in installation order, against your current kubeconfig context. After the `substrate-atelet` release, the script waits for the `atelet` DaemonSet to roll out. In a cluster that `kagent install` set up, the releases already exist, so the script upgrades each release in place with the same values.
 
 1. Confirm that your kubeconfig points at the cluster that runs kagent.
    ```bash
@@ -73,7 +74,7 @@ The `install.sh` script runs `helm upgrade --install` for each release, in insta
    sh ./kagent-install/install.sh
    ```
 
-4. Confirm that the releases are deployed. The `STATUS` column shows `deployed` for `kagent-crds`, `kagent`, `substrate-crds`, `substrate`, and `substrate-podcert`.
+4. Confirm that the releases are deployed. The `STATUS` column shows `deployed` for `kagent-crds`, `kagent`, `substrate-crds`, `substrate-atelet`, `substrate`, and `substrate-podcert`.
    ```bash
    helm list -A --filter 'kagent|substrate'
    ```

docs-site/content/kagent/1.x/operations/multiple-namespaces.md

@@ -63,7 +63,7 @@ The following steps prepare the `team-a` namespace.
        - name: gvisor
          configRef:
            name: gvisor-default
-     workerImage: ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}
+     workerImage: {{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}
      template:
        serviceAccountName: team-a-worker
    EOF

docs-site/content/kagent/1.x/operations/tune-agent-substrate.md

@@ -49,7 +49,7 @@ Read the current state of the runtime before you change it. The rest of this gui
      "workerPools": [
        {
          "name": "kagent-default",
-         "workerImage": "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}",
+         "workerImage": "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}",
          "readyReplicas": 1
        }
      ],
@@ -218,7 +218,7 @@ spec:
   replicas: 2
   sandboxClasses:
     - name: gvisor
-  workerImage: "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}"
+  workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}"
 ```
 
 Earlier releases placed Actors with a single `kagent.dev/worker-pool` label, and the chart no longer sets it. A Session or Sandbox that is pinned to a revision from one of those releases still selects that key. Keep the old label on the pool until you recreate those Sessions and Sandboxes.
@@ -233,12 +233,13 @@ substrateWorkerPool:
 
 Agent Substrate writes every suspended Actor to an object store, so the bucket stores the Session state that a resume restores. The following settings decide where a snapshot lands.
 
-- In the Agent Substrate Helm values, `atelet.storageBackend` selects the storage client. `rustfs.enabled` decides whether the chart supplies credentials for that client.
+- In the `substrate` release values, `atelet.storageBackend` selects the storage client of the `ate-api-server`. `rustfs.enabled` decides whether the chart supplies credentials for that client.
+- In the `substrate-atelet` release values, `objectStore` selects the storage client of `atelet`, with its endpoint and credentials.
 - In each {{< gloss "Harness" >}}Harness{{< /gloss >}}, `spec.substrate.snapshotPolicy.location` sets the bucket and prefix that the Harness's Actors write under. The field is required, and it must point into a bucket that the selected client can reach.
 
 ### The bundled object store
 
-Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs an in-cluster object store and configures the `atelet` DaemonSet and the `ate-api-server` Deployment to use it. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults.
+Because `rustfs.enabled` defaults to `true`, the `substrate` chart installs an in-cluster object store and configures the `ate-api-server` Deployment to use it. The [installation guide]({{< link path="setup/installation#install-agent-substrate" >}}) points the `substrate-atelet` release at the same store through its `objectStore` values. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults.
 
 > [!IMPORTANT]
 > The bundled store is not durable, and every installation receives the same credentials for it. Before a production installation serves real traffic, [move its snapshots to your own bucket](#move-snapshots-to-an-external-bucket).
@@ -247,13 +248,19 @@ Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs
 
 Replace the bundled store with a bucket in Google Cloud Storage, Amazon S3, or an S3-compatible store. The Agent Substrate backend and every Harness location must point at the same bucket.
 
-1. Save an Agent Substrate values file that disables the bundled store and selects the storage backend. The `s3` value selects the S3 client. Any other value, including an empty one, selects Google Cloud Storage.
+1. Save an Agent Substrate values file that disables the bundled store and selects the storage backend. Both Agent Substrate releases read this file. Set `atelet.storageBackend` and `objectStore.backend` to `s3` for Amazon S3 or an S3-compatible store, or to `gcs` for Google Cloud Storage. The empty `objectStore` values clear the endpoint and the credentials of the bundled store from the `substrate-atelet` release.
    ```bash
    cat > substrate-storage.yaml <<EOF
    rustfs:
      enabled: false
    atelet:
      storageBackend: s3
+   objectStore:
+     backend: s3
+     endpoint: ""
+     region: ""
+     pathStyle: false
+     credentialsSecret: ""
    EOF
    ```
 
@@ -286,8 +293,14 @@ Replace the bundled store with a bucket in Google Cloud Storage, Amazon S3, or a
    EOF
    ```
 
-3. Upgrade the Agent Substrate release by using the values file.
+3. Upgrade both Agent Substrate releases by using the values file. Upgrade `substrate-atelet` first.
    ```bash
+   helm upgrade substrate-atelet \
+     {{< reuse "kagent-docs/snippets/helm-substrate-path.md" >}}/substrate-atelet \
+     --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+     --namespace ate-system \
+     --reuse-values \
+     -f substrate-storage.yaml
    helm upgrade substrate \
      {{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
      --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
@@ -364,7 +377,7 @@ Set the pool's class, its Worker image, and its SandboxConfig together.
    cat > kagent-sandbox-class.yaml <<EOF
    substrateWorkerPool:
      sandboxClass: gvisor
-     workerImage: "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}"
+     workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}"
    EOF
    ```
    {{% /tab %}}
@@ -373,7 +386,7 @@ Set the pool's class, its Worker image, and its SandboxConfig together.
    cat > kagent-sandbox-class.yaml <<EOF
    substrateWorkerPool:
      sandboxClass: microvm
-     workerImage: "ghcr.io/kagent-dev/substrate/ateworker-microvm:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}"
+     workerImage: "{{< reuse "kagent-docs/snippets/image-substrate.md" >}}/ateworker-microvm:v{{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}"
    EOF
    ```
    {{% /tab %}}

docs-site/content/kagent/1.x/operations/uninstall.md

@@ -9,7 +9,7 @@ The `kagent uninstall` command removes everything that `kagent install` created.
 
 The command removes:
 
-- The `kagent`, `kagent-crds`, `substrate`, `substrate-podcert`, and `substrate-crds` Helm releases
+- The `kagent`, `kagent-crds`, `substrate`, `substrate-atelet`, `substrate-podcert`, and `substrate-crds` Helm releases
 - Every `kagent.dev` and `ate.dev` custom resource definition, and with them every resource of those kinds in every namespace, such as Agents, Harnesses, ModelConfigs, and SandboxTemplates
 - The development PostgreSQL database that `kagent install` deployed, together with its PersistentVolumeClaim and the data on it
 - The `ate-system` and `podcertificate-controller-system` namespaces, and the identity pools and certificate authorities in them