[release-1.0] Backport installer library work and egress credential providers

charts/substrate/README.md

@@ -86,6 +86,7 @@ See `values.yaml` for the full set; the important keys:
 | `<component>.resources` | see `values.yaml` | Requests and limits for each container, e.g. `ateApi.resources` and `atenetRouter.agentgateway.resources` |
 | `atenetRouter.requestParking.budget` | `60s` | Total budget for actor resolution and resume retries |
 | `atenetRouter.requestTimeout` | `60s` | Timeout for each Substrate ingress API call, including actor resolution |
+| `atenetEgress.extraCredentialProviders` | `[]` | Credential providers you deploy, added after the bundled `k8s.io` provider, each `{name, address}`. `name` is the host of the `ate-secret://` URIs it serves; `address` is the `host:port` the egress gateway dials over mTLS. Its serving certificate must chain to the servicedns CA. See [egress credential injection](../../docs/guides/egress-credential-injection.md#the-plugin-api) |
 | `atenetEgress.agentgateway.logLevel` | `info` | Egress proxy `RUST_LOG` filter, e.g. `debug` or `info,agentgateway=debug` when troubleshooting |
 | `rustfs.enabled` | `true` | Deploy an in-cluster S3-compatible RustFS bucket for snapshots |
 | `ateApi.snapshotLocation` | `""` | Cluster-wide snapshot URI; defaults to `gs://<rustfs.bucket>/` with bundled RustFS |
@@ -94,6 +95,7 @@ See `values.yaml` for the full set; the important keys:
 | `atelet.imageCredentialProviderConfig` | `""` | Host path to the kubelet credential provider config; set together with the bin directory |
 | `atelet.imageCredentialProviderConfigYAML` | `""` | Credential provider config as text, rendered into a ConfigMap, for nodes where that file cannot be authored or must differ from the kubelet's; mutually exclusive with the host path |
 | `atelet.imageCredentialProviderBinDir` | `""` | Host path to the kubelet credential provider binaries; both paths are mounted read-only |
+| `credentialProvider.enabled` | `true` | Deploy the Kubernetes Secret provider and route `k8s.io` credential URIs to it. With no provider, every credential injection fails closed |
 | `credentialProvider.namespacePolicies` | `[]` | Default-deny atespace-to-namespace grants; the chart includes get-only Secret RBAC for the provider |
 | `ateApi.actorJWTIssuer` | `https://<idp-service>.<namespace>.svc` | Issuer placed in actor JWTs; set a stable URL reachable by relying parties |
 | `objectStorePlugin.extraEnv` | `[]` | Object storage credentials/settings shared by both object-store plugins |

charts/substrate/templates/_helpers.tpl

@@ -259,3 +259,45 @@ definition so the fallback cannot drift between pods.
 {{ toYaml . }}
 {{- end }}
 {{- end }}
+
+{{/*
+The egress gateway's credentialProviders: the bundled k8s.io provider when
+credentialProvider.enabled, then atenetEgress.extraCredentialProviders. Renders
+nothing when there are none, so every injection fails closed.
+*/}}
+{{- define "substrate.credentialProviders" -}}
+{{- $providers := list -}}
+{{- if .Values.credentialProvider.enabled -}}
+{{- $providers = append $providers (dict "name" "k8s.io" "address" (printf "%s.%s.svc:50051" (include "substrate.fullname" (list "k8s-credential-provider" .)) .Release.Namespace)) -}}
+{{- end -}}
+{{- $seen := dict -}}
+{{- range .Values.atenetEgress.extraCredentialProviders -}}
+{{- if not .name -}}
+{{- fail "atenetEgress.extraCredentialProviders: each provider needs a name" -}}
+{{- end -}}
+{{- if eq .name "k8s.io" -}}
+{{- fail "atenetEgress.extraCredentialProviders: k8s.io is the bundled provider; set credentialProvider.enabled instead" -}}
+{{- end -}}
+{{- if hasKey $seen .name -}}
+{{- fail (printf "atenetEgress.extraCredentialProviders: duplicate name %q" .name) -}}
+{{- end -}}
+{{- if not (regexMatch "^[^:]+:[0-9]+$" (.address | default "")) -}}
+{{- fail (printf "atenetEgress.extraCredentialProviders: %s needs an address as host:port" .name) -}}
+{{- end -}}
+{{- $_ := set $seen .name true -}}
+{{- $providers = append $providers (dict "name" .name "address" .address) -}}
+{{- end -}}
+{{- if $providers -}}
+credentialProviders:
+{{- range $providers }}
+- uriAuthority: {{ .name | quote }}
+  target:
+    host: {{ .address | quote }}
+    policies:
+      backendTLS:
+        cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
+        key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
+        root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
+{{- end }}
+{{- end }}
+{{- end -}}

charts/substrate/templates/atenet-egress.yaml

@@ -80,15 +80,7 @@ data:
                   cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
                   key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
                   root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
-              credentialProviders:
-              - uriAuthority: k8s.io
-                target:
-                  host: {{ include "substrate.fullname" (list "k8s-credential-provider" .) }}.{{ .Release.Namespace }}.svc:50051
-                  policies:
-                    backendTLS:
-                      cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
+{{- include "substrate.credentialProviders" . | nindent 14 }}
       - protocol: HTTP
         routes:
         - backends:
@@ -102,15 +94,7 @@ data:
                   cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
                   key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
                   root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
-              credentialProviders:
-              - uriAuthority: k8s.io
-                target:
-                  host: {{ include "substrate.fullname" (list "k8s-credential-provider" .) }}.{{ .Release.Namespace }}.svc:50051
-                  policies:
-                    backendTLS:
-                      cert: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      key: /run/podidentity.podcert.ate.dev/credential-bundle.pem
-                      root: /run/servicedns.podcert.ate.dev/trust-bundle.pem
+{{- include "substrate.credentialProviders" . | nindent 14 }}
       - protocol: TLS
         tcpRoutes:
         - backends:

charts/substrate/templates/k8s-credential-provider.yaml

@@ -13,6 +13,7 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 */}}
+{{- if .Values.credentialProvider.enabled }}
 
 # The credential provider: a gRPC service that resolves ate-secret:// URIs
 # of the k8s.io class to Kubernetes Secret values. It is the ONLY
@@ -197,3 +198,4 @@ spec:
     ports:
     - protocol: TCP
       port: 50051
+{{- end }}

charts/substrate/tests/control_plane_test.yaml

@@ -511,3 +511,68 @@ tests:
       content:
         name: RUST_LOG
         value: "info,agentgateway=debug"
+- it: routes k8s.io credentials to the bundled provider on both listeners
+  template: atenet-egress.yaml
+  documentSelector:
+    path: kind
+    value: ConfigMap
+  asserts:
+  - matchRegex:
+      path: data["config.yaml"]
+      pattern: '(?s)credentialProviders:\n\s+- uriAuthority: "?k8s.io"?\n\s+target:\n\s+host: "?custom-k8s-credential-provider.custom-system.svc:50051"?.*credentialProviders:\n\s+- uriAuthority: "?k8s.io"?'
+- it: routes credentials only to extra providers when the bundled one is off
+  templates:
+  - atenet-egress.yaml
+  - k8s-credential-provider.yaml
+  set:
+    credentialProvider.enabled: false
+    atenetEgress.extraCredentialProviders:
+    - name: vault.example.com
+      address: vault-provider.vault.svc:50051
+  asserts:
+  - hasDocuments:
+      count: 0
+    template: k8s-credential-provider.yaml
+  - notMatchRegex:
+      path: data["config.yaml"]
+      pattern: 'k8s.io'
+    template: atenet-egress.yaml
+    documentSelector:
+      path: kind
+      value: ConfigMap
+  - matchRegex:
+      path: data["config.yaml"]
+      pattern: 'uriAuthority: "vault.example.com"\n\s+target:\n\s+host: "vault-provider.vault.svc:50051"'
+    template: atenet-egress.yaml
+    documentSelector:
+      path: kind
+      value: ConfigMap
+- it: configures no credential providers when every provider is off
+  template: atenet-egress.yaml
+  set:
+    credentialProvider.enabled: false
+  documentSelector:
+    path: kind
+    value: ConfigMap
+  asserts:
+  - notMatchRegex:
+      path: data["config.yaml"]
+      pattern: credentialProviders
+- it: rejects an extra provider named k8s.io
+  template: atenet-egress.yaml
+  set:
+    atenetEgress.extraCredentialProviders:
+    - name: k8s.io
+      address: a.b.svc:50051
+  asserts:
+  - failedTemplate:
+      errorMessage: "atenetEgress.extraCredentialProviders: k8s.io is the bundled provider; set credentialProvider.enabled instead"
+- it: rejects an extra provider without a host:port address
+  template: atenet-egress.yaml
+  set:
+    atenetEgress.extraCredentialProviders:
+    - name: vault.example.com
+      address: vault-provider.vault.svc
+  asserts:
+  - failedTemplate:
+      errorMessage: "atenetEgress.extraCredentialProviders: vault.example.com needs an address as host:port"

charts/substrate/values.yaml

@@ -165,6 +165,14 @@ atenetRouter:
         memory: 1Gi
 
 atenetEgress:
+  # Credential providers you deploy, added after the bundled k8s.io provider
+  # when credentialProvider.enabled. name is the host of the ate-secret:// URIs
+  # the provider serves, and address is the host:port the gateway dials over
+  # mTLS. The provider's serving certificate must chain to the servicedns CA
+  # and name the address host.
+  extraCredentialProviders: []
+  # - name: vault.example.com
+  #   address: vault-provider.vault.svc:50051
   agentgateway:
     logLevel: info
     resources:
@@ -178,6 +186,8 @@ atenetEgress:
 # Kubernetes Secret provider and AGW HTTP/HTTPS credential injection.
 # Includes get-only Secret RBAC. HTTPS requires egress-mitm-ca-pool and actor trust.
 credentialProvider:
+  # Deploy the Kubernetes Secret provider and route k8s.io credential URIs to it.
+  enabled: true
   namespacePolicies: []
   # - atespace: team-a
   #   allowedNamespaces: [team-a-secrets]

cmd/kubectl-ate/internal/cmd/install.go

@@ -28,23 +28,33 @@ import (
 	"github.com/spf13/cobra"
 	"golang.org/x/term"
 
-	"go.ate.dev/substrate/cmd/kubectl-ate/internal/install"
-	"go.ate.dev/substrate/cmd/kubectl-ate/internal/install/tui"
+	"go.ate.dev/substrate/pkg/kagent/install"
+	"go.ate.dev/substrate/pkg/kagent/install/tui"
 )
 
 var (
 	configPath    string
 	chartDir      string
 	noTUI         bool
 	upgradeDryRun bool
-	uninstallOpts install.Options
+	uninstallOpts install.UninstallOptions
 	uninstallYes  bool
 )
 
 const configHelp = `--config takes a YAML file, or - for stdin:
 
-  postgres: bundled        # or external
-  objectStore: rustfs      # or s3, gcs
+  postgres:
+    type: bundled          # or external
+  objectStore:
+    type: bundled          # or s3, gcs
+    bucket: my-snapshots   # for s3 or gcs
+    s3:                    # all optional
+      region: us-east-1
+      endpoint: https://s3.example.com
+      pathStyle: true
+      credentialsSecret: s3-creds # in the system namespace
+    gcs:
+      transferServiceAccount: transfer@my-project.iam.gserviceaccount.com
   microvm:
     enabled: true
     assetBucket: my-assets # for s3 or gcs
@@ -79,10 +89,9 @@ a RustFS for snapshots. On a terminal it asks for these choices first and
 prints the matching config.
 
 For anything but evaluation, create the database connection Secrets
-described in charts/substrate/README.md and set postgres: external. To store
-snapshots in a bucket, set objectStore to s3 or gcs and set the charts'
-objectStore.*, ateApi.snapshotLocation and, for GCS,
-objectStorePlugin.transferServiceAccount values.
+described in charts/substrate/README.md and set postgres.type to external.
+To store snapshots in a bucket, set objectStore.type to s3 or gcs and set
+objectStore.bucket. GCS also needs objectStore.gcs.transferServiceAccount.
 
 microvm.enabled downloads the Kata and Cloud Hypervisor guest assets for each
 node architecture, caches them under the user cache directory, uploads them to
@@ -132,9 +141,14 @@ releases, and CRD changes that would strand stored objects.
 			return err
 		}
 		defer closeTerminal(tty)
-		opts.DryRun, opts.Out = upgradeDryRun, cmd.OutOrStdout()
-		return runInstaller(cmd, opts, tty, "upgrade", (*install.Installer).Upgrade,
-			"Agent Substrate is upgraded.")
+		up := install.UpgradeOptions{DryRun: upgradeDryRun, Out: cmd.OutOrStdout()}
+		// The diff goes to stdout, so it can't share the terminal with the TUI.
+		if up.DryRun {
+			tty = nil
+		}
+		return runInstaller(cmd, opts, tty, "upgrade", func(in *install.Installer, ctx context.Context) error {
+			return in.Upgrade(ctx, up)
+		}, "Agent Substrate is upgraded.")
 	},
 }
 
@@ -157,20 +171,24 @@ data.`,
 			return err
 		}
 		defer closeTerminal(tty)
-		opts.IncludeCRDs, opts.Purge = uninstallOpts.IncludeCRDs, uninstallOpts.Purge
+		un := uninstallOpts
 		if !uninstallYes {
-			opts.Confirm = confirmFunc(cmd, tty)
+			un.Confirm = confirmFunc(cmd, tty)
 		}
-		return runInstaller(cmd, opts, tty, "uninstall", (*install.Installer).Uninstall,
-			"Agent Substrate is uninstalled.")
+		return runInstaller(cmd, opts, tty, "uninstall", func(in *install.Installer, ctx context.Context) error {
+			return in.Uninstall(ctx, un)
+		}, "Agent Substrate is uninstalled.")
 	},
 }
 
 // options reads the config and finds the terminal to read keys and answers
 // from. Without --config the config is install.DefaultConfig. The terminal is
 // nil when there is none. Close it with closeTerminal.
 func options(cmd *cobra.Command) (install.Options, *os.File, error) {
-	opts := install.Options{ChartDir: chartDir}
+	var opts install.Options
+	if chartDir != "" {
+		opts.Charts = os.DirFS(chartDir)
+	}
 	var err error
 	switch configPath {
 	case "":
@@ -282,7 +300,7 @@ func newInstallerWith(cmd *cobra.Command, opts install.Options, r install.Report
 // lines otherwise. The TUI shows the error itself, so the returned error is
 // marked as reported.
 func runInstaller(cmd *cobra.Command, opts install.Options, tty *os.File, title string, fn func(*install.Installer, context.Context) error, summary string) error {
-	if !useTUI(cmd, tty) || opts.DryRun {
+	if !useTUI(cmd, tty) {
 		in, err := newInstaller(cmd, opts)
 		if err != nil {
 			return err

docs/dev/releasing.md

@@ -18,7 +18,7 @@ Releases should always be made from a release branch, named `release-X.Y`.
 
 1. Tag the tested commit on the intended release branch and push the tag. Both tag pushes and manual runs of the [`release`](../../.github/workflows/release.yaml) workflow check out that tag:
    1. The `images` job runs `ate-setup publish release-images`, which builds every release image for `linux/amd64` and `linux/arm64`, pushes it to `ghcr.io/<owner>/<repository>/<image>:<tag>`, and then checks that every tag names the image just pushed, for both platforms. The ko images and `envoy-dataplane`, built from its Dockerfile with `docker buildx`, all cross-compile for arm64 without emulation.
-   2. In `kagent-dev/substrate`, the `charts` job publishes all three Helm charts after the images succeed. See [Fork Helm charts](#fork-helm-charts).
+   2. In `kagent-dev/substrate`, the `charts` job publishes all four Helm charts after the images succeed. See [Fork Helm charts](#fork-helm-charts).
    3. Once artifact publishing succeeds, the `draft` job builds the `kubectl-ate` binaries with `ate-setup publish release-binaries`, then runs [`hack/release/draft-release.sh`](../../hack/release/draft-release.sh), which creates a draft release, and attaches the binaries to it. The notes cover every change since the previous `vX.Y.Z` release, say where the release images are published and link to the list below, give the command that installs them, give the steps to check and install `kubectl-ate`, and end with a list of committers. A release candidate is marked as a prerelease.
 
    If images, charts, or the binary build fails, no new draft is created. If attaching the binaries fails, the draft exists without them. Either way, fix the cause and run the workflow again from the Actions tab with the tag as input.
@@ -112,7 +112,8 @@ VERSION=v0.3.0 go run ./cmd/ate-setup publish release-binaries
 
 ## Fork Helm charts
 
-The fork publishes `substrate-crds`, `substrate`, and `substrate-podcert` to
+The fork publishes `substrate-crds`, `substrate`, `substrate-podcert`, and
+`substrate-atelet` to
 `oci://ghcr.io/kagent-dev/substrate/helm`. Their chart version is the release tag
 without `v`; their `appVersion` is the full tag, which selects the matching
 component images. All charts are packaged before any chart is pushed. A failed

manifests/ate-install/sandboxconfig-microvm.yaml.tmpl

@@ -32,7 +32,7 @@
 # (virtio-fs RO lower + guest-tmpfs upper), so it fetches virtiofsd too.
 #
 # cmd/ate-setup/internal/steps/microvm.go and
-# cmd/kubectl-ate/internal/microvmassets read the per-arch sha256 values
+# pkg/kagent/microvmassets read the per-arch sha256 values
 # below to verify the assembled binaries and stages them at these digest-keyed
 # paths so newer releases do not overwrite kept versions.