You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Changes to ObjectStore spec do not get applied until after the cluster is healthy #918
We had an issue yesterday where we updated QNAP QuObjects, causing some change to the way S3 is handled. The resultant error from barman cloud was:
ERROR: Barman cloud WAL archiver exception: An error occurred (InvalidDigest)
when calling the PutObject operation: The Content-MD5 or checksum value that
you specified is not valid.
The fix is already mentioned in the documentation, where we add the following to the object store:
and I can confirm this works perfectly. However. Because one of the pods is stuck (1/2 Ready, blocked on the error above) the update never gets applied. What I had to do was apply the fix and then manually delete that one stuck pod (3-node cluster, this pod was a replica).
I think this is a bug, just because this configuration change was specifically required to resolve the issue I was having (S3 semantics changed). It's a little more complicated, here, because it seems like I had a primary switchover (perhaps due to similar S3-related errors) which left a bunch of WALs waiting to get uploaded. Would be good to see if there's a way this could be improved such that the barman-cloud-plugin operator is able to detect the S3-related failure and apply outstanding configuration.
Hi @dancollins! I'm Dosu and I’m helping the plugin-barman-cloud team.
This is a confirmed behavioral gap in the plugin. ObjectStore configuration (including instanceSidecarConfiguration.env) is only read at pod creation time via the CNPG-I lifecycle hooks — there's no mechanism to dynamically reload it into running sidecars [1]. Additionally, the ObjectStoreReconciler that could potentially trigger automatic pod restarts on spec changes is currently unimplemented [2].
So the chicken-and-egg scenario you experienced is expected (unfortunately): the config fix can't take effect because the pod won't be recreated, and the pod is stuck because it doesn't have the config fix.
Current workaround: After applying the ObjectStore change, force a rolling restart:
kubectl cnpg restart <cluster-name>
This recreates the pods, which picks up the new environment variables.
A similar issue has been reported with resource limit changes (e.g., OOM kills) where the same manual restart is needed [3]. The underlying improvement would be for the plugin to detect ObjectStore spec drift and trigger pod restarts automatically — but that doesn't exist yet.
I spent some time tracing the code path and the reported behavior appears valid.
From what I found:
ObjectStoreReconciler (internal/controller/objectstore_controller.go) only reconciles RBAC Role rules and does not update pods or enqueue Cluster reconciliation.
instanceSidecarConfiguration is consumed through the CNPG-I lifecycle hooks (collectObjectStoreEnvs -> reconcilePodSpec) when pod specs are created/evaluated.
There does not appear to be a runtime reload path for instanceSidecarConfiguration.env, so environment changes require pod recreation.
CNPG's Cluster controller does not appear to watch ObjectStore resources directly.
Additionally, isInstanceNeedingRollout() skips rollout evaluation for pods that are not Ready, which seems relevant to the reported 1/2 Ready failure scenario.
This matches the reproduction described in the issue: updating the ObjectStore alone does not propagate the new sidecar environment to the failing pod, while deleting/restarting the pod causes the updated configuration to be picked up during recreation.
Unless I'm missing another reconciliation path, this looks like a real propagation/reconciliation gap rather than user error.
We hit the same gap in production with CloudNativePG 1.30.1 and plugin v0.15.1. The code involved is unchanged in v0.16.0 and main.
Here is what happened:
We changed instanceSidecarConfiguration.env in the ObjectStore used by two clusters. Flux applied it at 10:13 UTC.
The plugin's ObjectStoreReconciler processed the change right away, but it only reconciles the RBAC Roles.
Nothing else happened until 19:28 UTC, when the operator reconciled both clusters after an informer resync that had nothing to do with the change. The reconciliation bursts land on multiples of fixed periods counted from the operator start, and we could predict the next one to the second. The operator then rolled out the instances with original and target PodSpec differ in init-containers: container plugin-barman-cloud differs in environment.
One of the clusters has a single instance, so it was down for about 3.5 minutes at a time nobody had planned.
So the drift detection works: since cloudnative-pg/cloudnative-pg#7126 the operator asks the plugin for the target pod spec through the EVALUATE lifecycle hook. What is missing is a trigger for the Cluster reconciliation when an ObjectStore changes. The change is not "never applied", it is applied at an unpredictable time, bounded only by the resync period, so the restart cannot be planned.
Proposal: the ObjectStoreReconciler, which already watches ObjectStores with GenerationChangedPredicate, keeps an annotation with a hash of the instanceSidecarConfiguration on every Cluster referring to the ObjectStore through barmanObjectName, recoveryBarmanObjectName or replicaSourceBarmanObjectName (the three sources read by collectAdditionalEnvs). Updating the annotation triggers the Cluster reconciliation, and the operator rolls out the pods according to primaryUpdateStrategy, as for any other pod spec change. This is the approach proposed for the pgBackRest plugin in operasoftware/cnpg-plugin-pgbackrest#187. It needs patch on clusters in the plugin's RBAC.
Would you accept a PR along these lines? I'd be happy to work on it. Two points I'd like your opinion on before starting:
Should the hash cover only instanceSidecarConfiguration, or every ObjectStore field that ends up in the pod spec?
On the first start of the new version, every Cluster gets the annotation once. I expect this not to roll anything out by itself, only the clusters that already have a pending drift, but I would like to confirm the operator does not react to the annotation alone.
This alone would not fix the scenario described in this issue: isInstanceNeedingRollout() returns early for pods that are not Ready (internal/controller/cluster_upgrade.go), so a pod stuck on a broken S3 configuration would still not get the new one. That part is on the operator side.
Activity
dosubot commentedon May 20, 2026
Hi @dancollins! I'm Dosu and I’m helping the plugin-barman-cloud team.
This is a confirmed behavioral gap in the plugin. ObjectStore configuration (including
instanceSidecarConfiguration.env) is only read at pod creation time via the CNPG-I lifecycle hooks — there's no mechanism to dynamically reload it into running sidecars [1]. Additionally, theObjectStoreReconcilerthat could potentially trigger automatic pod restarts on spec changes is currently unimplemented [2].So the chicken-and-egg scenario you experienced is expected (unfortunately): the config fix can't take effect because the pod won't be recreated, and the pod is stuck because it doesn't have the config fix.
Current workaround: After applying the ObjectStore change, force a rolling restart:
This recreates the pods, which picks up the new environment variables.
A similar issue has been reported with resource limit changes (e.g., OOM kills) where the same manual restart is needed [3]. The underlying improvement would be for the plugin to detect ObjectStore spec drift and trigger pod restarts automatically — but that doesn't exist yet.
To reply, just mention @dosu.
Docs are dead. Just use Dosu.
Utkarsh-sharma47 commentedon Jul 10, 2026
I spent some time tracing the code path and the reported behavior appears valid.
From what I found:
ObjectStoreReconciler(internal/controller/objectstore_controller.go) only reconciles RBAC Role rules and does not update pods or enqueue Cluster reconciliation.instanceSidecarConfigurationis consumed through the CNPG-I lifecycle hooks (collectObjectStoreEnvs->reconcilePodSpec) when pod specs are created/evaluated.instanceSidecarConfiguration.env, so environment changes require pod recreation.ObjectStoreresources directly.isInstanceNeedingRollout()skips rollout evaluation for pods that are not Ready, which seems relevant to the reported 1/2 Ready failure scenario.This matches the reproduction described in the issue: updating the ObjectStore alone does not propagate the new sidecar environment to the failing pod, while deleting/restarting the pod causes the updated configuration to be picked up during recreation.
Unless I'm missing another reconciliation path, this looks like a real propagation/reconciliation gap rather than user error.
DvdChe commentedon Nov 19, 2026
We hit the same gap in production with CloudNativePG 1.30.1 and plugin v0.15.1. The code involved is unchanged in v0.16.0 and
main.Here is what happened:
instanceSidecarConfiguration.envin the ObjectStore used by two clusters. Flux applied it at 10:13 UTC.ObjectStoreReconcilerprocessed the change right away, but it only reconciles the RBAC Roles.original and target PodSpec differ in init-containers: container plugin-barman-cloud differs in environment.So the drift detection works: since cloudnative-pg/cloudnative-pg#7126 the operator asks the plugin for the target pod spec through the
EVALUATElifecycle hook. What is missing is a trigger for the Cluster reconciliation when an ObjectStore changes. The change is not "never applied", it is applied at an unpredictable time, bounded only by the resync period, so the restart cannot be planned.Proposal: the
ObjectStoreReconciler, which already watches ObjectStores withGenerationChangedPredicate, keeps an annotation with a hash of theinstanceSidecarConfigurationon every Cluster referring to the ObjectStore throughbarmanObjectName,recoveryBarmanObjectNameorreplicaSourceBarmanObjectName(the three sources read bycollectAdditionalEnvs). Updating the annotation triggers the Cluster reconciliation, and the operator rolls out the pods according toprimaryUpdateStrategy, as for any other pod spec change. This is the approach proposed for the pgBackRest plugin in operasoftware/cnpg-plugin-pgbackrest#187. It needspatchonclustersin the plugin's RBAC.Would you accept a PR along these lines? I'd be happy to work on it. Two points I'd like your opinion on before starting:
instanceSidecarConfiguration, or every ObjectStore field that ends up in the pod spec?This alone would not fix the scenario described in this issue:
isInstanceNeedingRollout()returns early for pods that are not Ready (internal/controller/cluster_upgrade.go), so a pod stuck on a broken S3 configuration would still not get the new one. That part is on the operator side.