Repository navigation
Preserve the ARC retirement fence while migrating it to ValidatingPolicy #5834
Description
🤖 Generated by the Agentic Engineer
Evidence
- Migrate the tenant and organisation validation ClusterPolicies to ValidatingPolicy #4995 needs all twelve tenant and organisation validation policies to be
ValidatingPolicyobjects before Kyverno 1.20 removesClusterPolicy(Migrate Kyverno ClusterPolicy and Policy before Kyverno 1.20 removes them #4984). As of 2026-11-03, 8 of 12 are migrated or queued (refactor(policy): move the GitHub team management guard to ValidatingPolicy #5756).restrict-tenant-network-policiesis parked on Let tenant FQDN egress resolve through the platform DNS floor instead of tenant kube-system rules #5823 (refactor(policy): move the tenant network guard to ValidatingPolicy #5759), and the public listener guard is tracked in Preserve the public listener guard while migrating its validation API #5464. restrict-arc-retirement(k8s/bases/infrastructure/cluster-policies/best-practices/restrict-arc-retirement.yaml) has no tracking issue. Its current behaviour is already pinned:scripts/tests/test-arc-retirement-fence.shevaluates about 75 engine cases throughapiCallResponses(real Namespace annotations, including empty, malformed, 403/404/500 lookups, every phase, the baseline and maximum-1 paths and UID binding), and test(kyverno): pin the ARC listener and retirement policies before their ValidatingPolicy rewrite #5031 added thetests/restrict-arc-retirement-*fixtures. Migrate the tenant and organisation validation ClusterPolicies to ValidatingPolicy #4995 still lists it as untested; that line predates both.- It is the only
ClusterPolicyincluster-policies/that reads live state through anapiCallcontext: six of its eight rules read thearc-runnersNamespace'splatform.devantler.tech/arc-retirementannotation, then decides with a long JMESPath condition. Those six arefence-pool-retirement,fence-scale-set-retirement,fence-listener-retirement,fence-retired-listener-pod,fence-credential-recreationandfence-retired-secret; two more,preserve-retirement-journalandpreserve-retiring-namespace, guard the annotation itself on Namespace UPDATE and DELETE. ItswebhookConfiguration.failurePolicyisFail, and the failure action isEnforcewithallowExistingViolations: false. - ARC is still deployed (
k8s/bases/infrastructure/actions-runners/), and the retirement tooling that writes the annotation (scripts/retire-arc-pool.sh,scripts/retire-arc-pool/) is live. The fence must therefore keep the same behaviour; it cannot simply be dropped.
Audience and problem
Whoever retires or recovers the ARC runner pool. The fence is what stops a Flux reconcile or a manual apply from recreating the release, or reopening runner admission, while a retirement is in progress. A migration that changes its decision even in one phase either blocks a legitimate recovery step or lets a retired pool come back. The existing cases feed the Namespace through a JMESPath apiCall context. A ValidatingPolicy reads it through CEL resource.Get instead, so the harness must give the new kind the same Namespace responses, or the old expectations prove nothing about it.
Hypothesis
If the existing engine cases run unchanged in their expected results against a ValidatingPolicy that reads the Namespace through CEL resource.Get, with the same Namespace responses supplied to it, then any decision the rewrite changes shows up as a failing case before deploy, not as a blocked recovery.
Success signal
scripts/tests/test-arc-retirement-fence.shand the test(kyverno): pin the ARC listener and retirement policies before their ValidatingPolicy rewrite #5031 fixtures pass against the new kind with unchanged expected results, and a deliberate mutation of the new policy turns at least one case red (the harness is not vacuous for the new kind).- The
ValidatingPolicypasses the same test unchanged in its expected results, keepsDeny, the same match scope andfailurePolicy: Fail, and replaces theClusterPolicyin the same PR. - After deploy, a read-only check shows the new policy ready and no
ClusterPolicyof that name.
Guardrail: an unreadable Namespace must still deny (fail closed), as it does today through failurePolicy: Fail.
Smallest useful change
- Spike: confirm how the pinned Kyverno CLI (1.19.2) supplies a Namespace to a
ValidatingPolicy'sresource.Get(theapiCallResponsesmechanism, or another context input), and howrequest.oldObjectand user groups reach it. Record the answer here. - Migration PR: the
ValidatingPolicyand deletion of theClusterPolicy, with the same tests pointed at the new kind.
Activity
devantler commentedon Nov 10, 2026
Folding in #6026, which was filed for the same policy on 10 November and is closed as a duplicate of this issue. What it added:
restrict-arc-retirementis one of the two Migrate the tenant and organisation validation ClusterPolicies to ValidatingPolicy #4995 policies still deployed as aClusterPolicy. Thearc-runnersNamespace carries no retirement journal, so most rules are inactive today; the guard still has to work once a retirement starts.ValidatingPolicycan express each rule (the Namespace lookup, the Namespace DELETE rule and thesystem:masterscheck), and record any rule it cannot express on Migrate Kyverno ClusterPolicy and Policy before Kyverno 1.20 removes them #4984 so it blocks only that rule; keepEnforce, the same match scope andfailurePolicy: Failas recorded intests/policy-failure-actions.json; add the replacement and delete theClusterPolicyin the same change, then confirm after deploy that only the new policy remains.devantler commentedon Nov 11, 2026
Spike (step 1) result, at Kyverno CLI and engine 1.19.2. Short version: the CLI can feed the Namespace to a
ValidatingPolicy, but it cannot feed a previous object, and the retirement tooling depends on the ClusterPolicy's kind and denial format. So the migration is bigger than one PR. I measured each point with small throwaway policies run throughkyverno test:Contextfile (context:in the Test,kind: Context,spec.resources) answersresource.Get('v1','namespaces','','arc-runners'), and it works insidematchConditionstoo.json.unmarshalparses the journal. A bad journal comes back as anerrorrow, and a false match condition asskip, the same results the harness expects today.error(no such key: metadata). It cannot produce a 403 or a 500, so in the CLI the three lookup-failure cases become one "not found" case. The repository's offline engine driver (scripts/tests/tenant-webapp-admission, already used for the tenant and GitHub-team policies) can inject a refused read, so that is where those cases should live.request.oldObject. For aValidatingPolicy,kyverno testmakes the previous object a copy of the new one. Neither global nor per-resourcerequest.oldObjectvalues reacholdObject, and on DELETE it also leaves the old object without the values. The listener-cleanup, credential-rewrite, journal and namespace-delete cases all depend on a separate previous object, so they cannot staykyverno testfixtures. They need the same offline engine driver, given an explicitoldObjectand the user's groups (the driver has no groups field yet; thesystem:masterscheck needs one).request.userInfo.groupsthrough a CLIUserInfofile.scripts/retire-arc-pool.shreads the liveclusterpolicy restrict-arc-retirementand compares it field by field with the declared rules. Both it andscripts/retire-arc-pool/writer.gorecognise a fence denial by the ClusterPolicy webhook name and by therestrict-arc-retirement:/<rule>:lines in the message. Its contract test also requires a rendered ClusterPolicy with eight rules. In the engine source (pkg/webhooks/resource/vpol/handler.go), aValidatingPolicydenial readsPolicy restrict-arc-retirement failed: <message>, with no rule name, and comes from thevpol.validate.kyverno.svcwebhook family. So the migration also has to change those readers, or the tooling would refuse a real retirement. Each validation's message should name its old rule, so the denial stays tied to a specific fence.Suggested split: (a) add groups to the offline engine driver and port the existing ~75 engine cases to it, unchanged in their expected results and still running against today's ClusterPolicy where the CLI allows; (b) the
ValidatingPolicyitself, with the cases pointed at it, the retirement tooling's readers moved to the new kind and denial format,tests/policy-failure-actions.jsonand the fixture-evaluation self-test (which uses these fixtures as its examples) updated, and the ClusterPolicy deleted in the same change. The after-deploy check is still open:kubectl get validatingpolicy restrict-arc-retirementreports ready andkubectl get clusterpolicy restrict-arc-retirementreturns NotFound.I am not starting the migration in this session. The issue stays open.
🤖 Generated with Claude Code