Skip to content

Changes to ObjectStore spec do not get applied until after the cluster is healthy #918

Open

Description

@dancollins

We had an issue yesterday where we updated QNAP QuObjects, causing some change to the way S3 is handled. The resultant error from barman cloud was:

ERROR: Barman cloud WAL archiver exception: An error occurred (InvalidDigest) 
when calling the PutObject operation: The Content-MD5 or checksum value that 
you specified is not valid.

The fix is already mentioned in the documentation, where we add the following to the object store:

---
apiVersion: barmancloud.cnpg.io/v1
kind: ObjectStore
spec:
  configuration:
     ...
  instanceSidecarConfiguration:
      env:
        - name: AWS_REQUEST_CHECKSUM_CALCULATION
          value: when_required
        - name: AWS_RESPONSE_CHECKSUM_VALIDATION
          value: when_required

and I can confirm this works perfectly. However. Because one of the pods is stuck (1/2 Ready, blocked on the error above) the update never gets applied. What I had to do was apply the fix and then manually delete that one stuck pod (3-node cluster, this pod was a replica).

I think this is a bug, just because this configuration change was specifically required to resolve the issue I was having (S3 semantics changed). It's a little more complicated, here, because it seems like I had a primary switchover (perhaps due to similar S3-related errors) which left a bunch of WALs waiting to get uploaded. Would be good to see if there's a way this could be improved such that the barman-cloud-plugin operator is able to detect the S3-related failure and apply outstanding configuration.

Activity

  1. dosubot commented on May 20, 2026

    @dosubot

    Hi @dancollins! I'm Dosu and I’m helping the plugin-barman-cloud team.

    This is a confirmed behavioral gap in the plugin. ObjectStore configuration (including instanceSidecarConfiguration.env) is only read at pod creation time via the CNPG-I lifecycle hooks — there's no mechanism to dynamically reload it into running sidecars [1]. Additionally, the ObjectStoreReconciler that could potentially trigger automatic pod restarts on spec changes is currently unimplemented [2].

    So the chicken-and-egg scenario you experienced is expected (unfortunately): the config fix can't take effect because the pod won't be recreated, and the pod is stuck because it doesn't have the config fix.

    Current workaround: After applying the ObjectStore change, force a rolling restart:

    kubectl cnpg restart <cluster-name>

    This recreates the pods, which picks up the new environment variables.

    A similar issue has been reported with resource limit changes (e.g., OOM kills) where the same manual restart is needed [3]. The underlying improvement would be for the plugin to detect ObjectStore spec drift and trigger pod restarts automatically — but that doesn't exist yet.

    To reply, just mention @dosu.


    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about plugin-barman-cloud Share Dosu with your team

  2. Utkarsh-sharma47 commented on Jul 10, 2026

    @Utkarsh-sharma47

    I spent some time tracing the code path and the reported behavior appears valid.

    From what I found:

    • ObjectStoreReconciler (internal/controller/objectstore_controller.go) only reconciles RBAC Role rules and does not update pods or enqueue Cluster reconciliation.
    • instanceSidecarConfiguration is consumed through the CNPG-I lifecycle hooks (collectObjectStoreEnvs -> reconcilePodSpec) when pod specs are created/evaluated.
    • There does not appear to be a runtime reload path for instanceSidecarConfiguration.env, so environment changes require pod recreation.
    • CNPG's Cluster controller does not appear to watch ObjectStore resources directly.
    • Additionally, isInstanceNeedingRollout() skips rollout evaluation for pods that are not Ready, which seems relevant to the reported 1/2 Ready failure scenario.

    This matches the reproduction described in the issue: updating the ObjectStore alone does not propagate the new sidecar environment to the failing pod, while deleting/restarting the pod causes the updated configuration to be picked up during recreation.

    Unless I'm missing another reconciliation path, this looks like a real propagation/reconciliation gap rather than user error.

  3. DvdChe commented on Nov 19, 2026

    @DvdChe

    We hit the same gap in production with CloudNativePG 1.30.1 and plugin v0.15.1. The code involved is unchanged in v0.16.0 and main.

    Here is what happened:

    • We changed instanceSidecarConfiguration.env in the ObjectStore used by two clusters. Flux applied it at 10:13 UTC.
    • The plugin's ObjectStoreReconciler processed the change right away, but it only reconciles the RBAC Roles.
    • Nothing else happened until 19:28 UTC, when the operator reconciled both clusters after an informer resync that had nothing to do with the change. The reconciliation bursts land on multiples of fixed periods counted from the operator start, and we could predict the next one to the second. The operator then rolled out the instances with original and target PodSpec differ in init-containers: container plugin-barman-cloud differs in environment.
    • One of the clusters has a single instance, so it was down for about 3.5 minutes at a time nobody had planned.

    So the drift detection works: since cloudnative-pg/cloudnative-pg#7126 the operator asks the plugin for the target pod spec through the EVALUATE lifecycle hook. What is missing is a trigger for the Cluster reconciliation when an ObjectStore changes. The change is not "never applied", it is applied at an unpredictable time, bounded only by the resync period, so the restart cannot be planned.

    Proposal: the ObjectStoreReconciler, which already watches ObjectStores with GenerationChangedPredicate, keeps an annotation with a hash of the instanceSidecarConfiguration on every Cluster referring to the ObjectStore through barmanObjectName, recoveryBarmanObjectName or replicaSourceBarmanObjectName (the three sources read by collectAdditionalEnvs). Updating the annotation triggers the Cluster reconciliation, and the operator rolls out the pods according to primaryUpdateStrategy, as for any other pod spec change. This is the approach proposed for the pgBackRest plugin in operasoftware/cnpg-plugin-pgbackrest#187. It needs patch on clusters in the plugin's RBAC.

    Would you accept a PR along these lines? I'd be happy to work on it. Two points I'd like your opinion on before starting:

    1. Should the hash cover only instanceSidecarConfiguration, or every ObjectStore field that ends up in the pod spec?
    2. On the first start of the new version, every Cluster gets the annotation once. I expect this not to roll anything out by itself, only the clusters that already have a pending drift, but I would like to confirm the operator does not react to the annotation alone.

    This alone would not fix the scenario described in this issue: isInstanceNeedingRollout() returns early for pods that are not Ready (internal/controller/cluster_upgrade.go), so a pod stuck on a broken S3 configuration would still not get the new one. That part is on the operator side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions