[FEATURE] Release Layouts #3176

Defines what each release publishes for kustomize users, and the order in which it is applied for fresh installs and for upgrades.

Owner: kustomize release workstream (#3169). Consumers: Helm workstream (#3170), GKE workstream (ai-on-gke/substrate-gke#70), sizing workstream (#3165).

Specification.

Each release commits this tree on its tag; consumers reference it as https://github.com/agent-substrate/substrate//release/<dir>?ref=<tag>.

release/
  crds/                  CRDs and their RBAC
  bootstrap/             namespaces, bootstrap ServiceAccount, RBAC, and Job
  system/                kustomization that includes every stage below (fresh installs)
  stages/
    1-podcert/           podcertificate-controller and its signers
    2-atelet/            atelet DaemonSet
    4-apiserver/         ate-api-server, ate-idp-server, their Services and settings
    5-controller/        ate-controller and its RBAC (CRDs are applied from crds/ just before)
    6-atenet/            atenet-router, atenet-egress, credential provider
    7-sandboxconfig/     SandboxConfig admission policy and the release's default SandboxConfig
    8-rest/              everything else (for example NetworkPolicies)
  components/
    sizing-default/      sizing Component applied by default (#3168)
    sizing-large/        large-cluster sizing Component

• Every directory is a valid kustomization with images pinned to digests and no text placeholders.
• CRDs. All three are in API group ate.dev, version v1alpha1: workerpools.ate.dev (namespaced), sandboxconfigs.ate.dev and csidriverconfigs.ate.dev (cluster-scoped).
• Fresh install applies three layers, each after the previous one is ready:
  1. crds/, then wait for every CRD to be Established;
  2. bootstrap/, then wait for the bootstrap Job to be Complete;
  3. system/, then wait for every rollout. Within this layer no order is required, because pods waiting on Secrets or certificates retry until they are ready.
• Upgrade follows the version skew policy:
  1. bootstrap Job;
  2. 1-podcert;
  3. 2-atelet, node by node;
  4. WorkerPool spec.workerImage updates (operator-owned; the release notes give the image);
  5. 4-apiserver;
  6. crds/ then 5-controller;
  7. 6-atenet;
  8. 7-sandboxconfig;
  9. 8-rest.
• Rollback applies the previous release's stages in reverse and does not roll back 8-rest.
• Bootstrap Job name is ate-bootstrap-<version>, with dots replaced by dashes (for example ate-bootstrap-v0-6-0). It sets ttlSecondsAfterFinished, because a Job's pod template cannot change and each release must create a new Job.
• SandboxConfigs. Each release ships its SandboxConfig under its own name. SandboxConfigs carry helm.sh/resource-policy: keep and are never pruned ([Bug Bash] Install a new SandboxConfig per release instead of updating the class default in place #3253, PR [Bug bash] Make sure each release creates sandbox config to enable rollback #3254).
• Shared configuration. Shared settings objects consumed by components in several stages, such as ate-otel-config, ship in bootstrap/ and change only in ways compatible with both releases' components (Proposed).
• Sample render. Until the render command exists, the kustomize release workstream maintains a sample of this tree for the Helm workstream to build against. The sample is regenerated in the same change as any layout change.

Open points

• Release bundle committed on the tag (above), or published as an OCI artifact ([FEATURE] Release coordinates for images, bases, charts and binaries #3664).
• Where ate-idp-server and bundled Postgres go. Resolved in ate-setup: deploy ate-idp-server with ate-api-server #2891: 4-apiserver.
• If CI shows layer 3 does not converge when certificate-dependent workloads start before the certificate controller, system/ is applied as stages/1-podcert then the rest.

Enforcement

• CI installs a release on kind using only the documented fresh-install commands ([FEATURE] Kind CI: install, upgrade and roll back from release bases with kubectl only #3679).
• CI installs the previous release, upgrades and rolls back stage by stage with actors running, and asserts the previous release's SandboxConfig survives ([FEATURE] Kind CI: install, upgrade and roll back from release bases with kubectl only #3679).
• CI compares the Helm workstream's sample with the real render ([FEATURE] Generate the release layout from ate-setup render #3678).

Implementation: #3678 (layout generation), #3679 (kind CI).

Parent Issue: #3160