Repository navigation
feat: backups from a standby over the pgBackRest TLS server - #188
ermakov-oleg wants to merge 1 commit into
Conversation
Signed-off-by: ermakov-oleg <ermakovolegs@gmail.com>
|
Could we agree on the TLS trust model for standby backups before continuing the follow-ups in #104? Is reusing the CNPG replication certificate acceptable for pgBackRest access given the additional privileges it grants? If so, what isolation should #188 require when clusters share a client CA and use the same replication CN? Should reuse require a per-cluster client CA or a unique replication identity under a shared CA, and can we enforce that rather than only document it? If reuse cannot provide that isolation, should we require dedicated pgBackRest credentials, as proposed in #104? I'd like to settle this before we build out both approaches. |
|
Thanks for raising this. My view: Reuse stays the default. With CNPG-generated certificates every cluster has its own client CA, so a replication certificate of another cluster is not accepted. It needs no extra secrets, mounts or pod restarts, which is the direction @Agalin preferred in #103. What reuse cannot isolate. pgBackRest authorises on CA and CN only, and CNPG pins the replication CN to What I propose. For the first case, enforce it: the server trusts only the CA certificates that chain the cluster's own replication certificate, not the whole bundle. The second case cannot be detected from inside the sidecar, so I would add opt-in dedicated credentials: I can add the issuer pinning to this PR now. Maintainers, would you want |
Problem
CloudNativePG prefers a standby for a backup by default (
target: prefer-standby) and falls back to the primary only when no standby is available. With this plugin a backup dispatched to a standby fails: pgBackRest needs the primary to start and stop the backup and to build the manifest, and the sidecar of a standby cannot reach it. Clusters with more than one instance have to pintarget: primaryand take the whole backup load on the primary. See #103.What this PR does
Adds an opt-in
backupStandbyblock to theArchiveconfiguration. When it is enabled:pgbackrest server(the pgBackRest TLS server) on port 8432. It runs on every instance because the primary changes.--pg2-host ... --backup-standby=y). pgBackRest starts and stops the backup on the primary and copies the data files from the standby.serverTLSSecretthe issuer has to include the name. The default name is<cluster>-pgbackrest;backupStandby.serviceNameoverrides it.No new certificates, secrets, mounts, RBAC or pod spec changes. Enabling or disabling does not restart pods: each sidecar checks the
Clusterand theArchiveroughly every five minutes. Before a backup the standby waits up to 90 seconds for the primary's server to accept connections; if the server is still unreachable when that wait expires, the backup fails and can be retried.Design choices
Certificates are reused. The sidecar already sees the CloudNativePG instance certificates in
/controller/certificates. The server presents the cluster's server certificate; the client presents the streaming replication certificate, and the server authorises the common name read from that certificate, so a custom replication CN works as long as pgBackRest can express it in its authorisation rule (no=). pgBackRest verifies the host name against the certificate and this cannot be turned off, which is why the Service name has to be in the certificate.This widens what the replication certificate can do: its holder, with network access to port 8432, can run SQL as
postgresand access the data directory through the pgBackRest protocol. A certificate of another cluster is accepted as well if it is signed by the same client CA and has the same common name. The README says so, and the feature is opt-in for that reason. Dedicated pgBackRest credentials are a possible later extension.The server is long-running, not on-demand. The backup RPC only reaches the sidecar of the standby; starting a server in the primary's sidecar for the duration of a backup would need a coordination channel that does not exist today.
The server is supervised by the sidecar. It starts once the certificates are present and form a consistent key pair. After that, each time it observes a valid configuration and certificate set, a changed certificate, key or CA triggers a reload (SIGHUP) and a changed authorised CN or stanza triggers a restart; an unusable certificate set leaves the running server as it is. A log level change does not restart it, since a restart terminates running backups. The server runs in its own process group; the group is killed on stop and when the sidecar detects that the server exited, and a server that exited is started again while the feature is enabled. A missing
ClusterorArchivedisables the server; other read errors keep the previous state.The feature reads the
Archivestraight from the API. The supervisor and the backup decision use the manager's API reader instead of the sidecar's caching client. The cache keeps anArchivefor 10 seconds, so a backup right after enabling could fail as "not enabled", and it holds one mutex across the API call, so an unrelated slow read could stall the supervisor past its own deadline.The polling is slow on purpose. Every instance of every cluster that uses the plugin has to poll, enabled or not, to learn that the feature was turned on. The supervisor reads the
Clusterand theArchiveroughly every five minutes, with jitter. Per 1,000 instances the expected steady-state load is about 6 GETs per second, where a one-minute poll would be about 33.Not covered
archive-timeoutbefore reaching the backup start LSN, the backup fails and is not retried on the primary.The e2e suite gets a
standbybackupspec: two instances, a backup on the standby, a restore from it, a switchover, and a second backup on the former primary.Closes #103. #104 explores the same problem with a different certificate model; this PR is an independent implementation.