Repository navigation
feat: roll out instance pods when the Archive sidecar configuration changes - #187
Open
ermakov-oleg wants to merge 1 commit into
Open
ermakov-oleg wants to merge 1 commit into
ermakov-oleg wants to merge 1 commit into
Conversation
…hanges CloudNativePG does not watch Archive objects, so a change to instanceSidecarConfiguration reached the instance pods only after an unrelated Cluster reconciliation. The Archive controller now keeps the hash of the sidecar configuration of the referred Archives in the pgbackrest.cnpg.opera.com/sidecar-configuration-hash annotation of each Cluster, and CloudNativePG rolls out the pods with an outdated sidecar according to primaryUpdateStrategy. Closes operasoftware#53 Claude-Session: https://claude.ai/code/session_01MUoHHS8cK7mCTdiKN7f739 Signed-off-by: ermakov-oleg <ermakovolegs@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The sidecar is configured from
instanceSidecarConfigurationwhen an instance pod is created. CloudNativePG does not watchArchiveobjects, so after a change to the sidecar resources, env or security context nothing happens until some unrelated event reconciles theCluster. The rollout then starts at a random moment, bounded only by the informer resync (10h by default): on a test cluster of ours, changing only the sidecar memory limit in theArchiverecreated the replica 17 minutes later, on an unrelated reconciliation. A sharedArchiverolls its clusters one by one at unpredictable times. This is #53; plugin-barman-cloud has the same gap (cloudnative-pg/plugin-barman-cloud#824, #918).The drift detection itself already works: the pod lifecycle hook handles
TYPE_EVALUATE, and CloudNativePG compares the init containers of the stored and target pod specs, resources included (original and target PodSpec differ in init-containers: container plugin-pgbackrest differs in resources). Only the trigger is missing.What this PR does
The
Archivecontroller, a scaffold until now, keeps thepgbackrest.cnpg.opera.com/sidecar-configuration-hashannotation up to date on everyClusterreferring to theArchive(archive, recovery or replica source). Changing the annotation triggers theClusterreconciliation, and CloudNativePG rolls out the pods with an outdated sidecar according toprimaryUpdateStrategy: withsupervised, replicas are recreated and the primary waits for a switchover, as for any other pod spec change.instanceSidecarConfigurationare filtered out. Create events are processed, so changes made while the plugin was down or during a leader change are caught up on start.get/list/watch/patchonclusters(config/rbac/role.yamlregenerated,manifest.yamlupdated).Visible behaviour changes
Archiveare rolled out at that point; CloudNativePG spaces the rollouts of different clusters byCLUSTERS_ROLLOUT_DELAY.Archivenow triggers the reconciliation of all its clusters when the change is observed, instead of whenever each one happens to be reconciled (see the limitation below for several plugin replicas).catalogMaintenanceIntervalSecondsalso updates the hash; it triggers a reconciliation but no rollout, since it is not part of the pod spec.Known limitation
With several plugin replicas the lifecycle hook may be served by a replica whose informer has not seen the new
Archiveyet. The target pod spec then matches the old one and the rollout is postponed to the nextClusterreconciliation, as today. Reading theArchivefrom the API server in the hook would avoid that at the cost of a request per evaluation; happy to change it if you prefer.Testing
sidecarrollout: changes only the sidecar memory limit of theArchiveand waits for the annotation and for every instance pod to be recreated with the new limit.internal/controlleris removed: the package has no envtest-based tests left.Closes #53
https://claude.ai/code/session_01MUoHHS8cK7mCTdiKN7f739