Skip to content

feat: roll out instance pods when the Archive sidecar configuration changes - #187

Open
ermakov-oleg wants to merge 1 commit into
operasoftware:mainfrom
ermakov-oleg:feat-archive-sidecar-rollout
Open

ermakov-oleg wants to merge 1 commit into
operasoftware:mainfrom
ermakov-oleg:feat-archive-sidecar-rollout

Conversation

@ermakov-oleg

Copy link
Copy Markdown
Contributor

Problem

The sidecar is configured from instanceSidecarConfiguration when an instance pod is created. CloudNativePG does not watch Archive objects, so after a change to the sidecar resources, env or security context nothing happens until some unrelated event reconciles the Cluster. The rollout then starts at a random moment, bounded only by the informer resync (10h by default): on a test cluster of ours, changing only the sidecar memory limit in the Archive recreated the replica 17 minutes later, on an unrelated reconciliation. A shared Archive rolls its clusters one by one at unpredictable times. This is #53; plugin-barman-cloud has the same gap (cloudnative-pg/plugin-barman-cloud#824, #918).

The drift detection itself already works: the pod lifecycle hook handles TYPE_EVALUATE, and CloudNativePG compares the init containers of the stored and target pod specs, resources included (original and target PodSpec differ in init-containers: container plugin-pgbackrest differs in resources). Only the trigger is missing.

What this PR does

The Archive controller, a scaffold until now, keeps the pgbackrest.cnpg.opera.com/sidecar-configuration-hash annotation up to date on every Cluster referring to the Archive (archive, recovery or replica source). Changing the annotation triggers the Cluster reconciliation, and CloudNativePG rolls out the pods with an outdated sidecar according to primaryUpdateStrategy: with supervised, replicas are recreated and the primary waits for a switchover, as for any other pod spec change.

  • The hash covers the sidecar configuration of all the Archives the cluster refers to, so it converges to a single value when several of them are reconciled.
  • Updates not touching instanceSidecarConfiguration are filtered out. Create events are processed, so changes made while the plugin was down or during a leader change are caught up on start.
  • Clusters are looked up through a field index on the referred Archive names, so the startup pass does not scan the namespace once per Archive. The archive of a disabled plugin entry is not counted: CloudNativePG does not load that plugin, so the pods carry no sidecar.
  • RBAC: the operator ClusterRole gets get/list/watch/patch on clusters (config/rbac/role.yaml regenerated, manifest.yaml updated).

Visible behaviour changes

  • On the first start of this version, every cluster using the plugin gets the annotation once. Clusters whose pods already run a sidecar differing from the current Archive are rolled out at that point; CloudNativePG spaces the rollouts of different clusters by CLUSTERS_ROLLOUT_DELAY.
  • A change to a shared Archive now triggers the reconciliation of all its clusters when the change is observed, instead of whenever each one happens to be reconciled (see the limitation below for several plugin replicas).
  • A change to catalogMaintenanceIntervalSeconds also updates the hash; it triggers a reconciliation but no rollout, since it is not part of the pod spec.

Known limitation

With several plugin replicas the lifecycle hook may be served by a replica whose informer has not seen the new Archive yet. The target pod spec then matches the old one and the rollout is postponed to the next Cluster reconciliation, as today. Reading the Archive from the API server in the hook would avoid that at the cost of a request per evaluation; happy to change it if you prefer.

Testing

  • Unit tests for the reconciler (fake client with the field index): only the referring clusters with the plugin enabled are annotated, other namespaces are ignored, the annotation is stable while the configuration does not change and converges for a cluster with two Archives; predicate tests.
  • New e2e spec sidecarrollout: changes only the sidecar memory limit of the Archive and waits for the annotation and for every instance pod to be recreated with the new limit.
  • The envtest bootstrap of internal/controller is removed: the package has no envtest-based tests left.

Closes #53

https://claude.ai/code/session_01MUoHHS8cK7mCTdiKN7f739

…hanges

CloudNativePG does not watch Archive objects, so a change to instanceSidecarConfiguration
reached the instance pods only after an unrelated Cluster reconciliation. The Archive
controller now keeps the hash of the sidecar configuration of the referred Archives in the
pgbackrest.cnpg.opera.com/sidecar-configuration-hash annotation of each Cluster, and
CloudNativePG rolls out the pods with an outdated sidecar according to primaryUpdateStrategy.

Closes operasoftware#53

Claude-Session: https://claude.ai/code/session_01MUoHHS8cK7mCTdiKN7f739
Signed-off-by: ermakov-oleg <ermakovolegs@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Controller for Archive CR?

1 participant