Skip to content

feat: backups from a standby over the pgBackRest TLS server - #188

Open
ermakov-oleg wants to merge 1 commit into
operasoftware:mainfrom
ermakov-oleg:feat-backup-from-standby
Open

ermakov-oleg wants to merge 1 commit into
operasoftware:mainfrom
ermakov-oleg:feat-backup-from-standby

Conversation

@ermakov-oleg

Copy link
Copy Markdown
Contributor

Problem

CloudNativePG prefers a standby for a backup by default (target: prefer-standby) and falls back to the primary only when no standby is available. With this plugin a backup dispatched to a standby fails: pgBackRest needs the primary to start and stop the backup and to build the manifest, and the sidecar of a standby cannot reach it. Clusters with more than one instance have to pin target: primary and take the whole backup load on the primary. See #103.

What this PR does

Adds an opt-in backupStandby block to the Archive configuration. When it is enabled:

  • The sidecar of every instance runs pgbackrest server (the pgBackRest TLS server) on port 8432. It runs on every instance because the primary changes.
  • A backup dispatched to a standby adds the primary as a second pgBackRest host (--pg2-host ... --backup-standby=y). pgBackRest starts and stops the backup on the primary and copies the data files from the standby.
  • The standby reaches the primary through a Service that selects the primary and exposes port 8432. A CloudNativePG managed service is enough: CloudNativePG creates it and adds its names to the server certificate it generates, without restarting pods. With a user-provided serverTLSSecret the issuer has to include the name. The default name is <cluster>-pgbackrest; backupStandby.serviceName overrides it.
  • A backup dispatched to the primary stays a local backup. A backup dispatched to a standby while the feature is disabled now fails with a message that names the fix, instead of pgBackRest exit 56.

No new certificates, secrets, mounts, RBAC or pod spec changes. Enabling or disabling does not restart pods: each sidecar checks the Cluster and the Archive roughly every five minutes. Before a backup the standby waits up to 90 seconds for the primary's server to accept connections; if the server is still unreachable when that wait expires, the backup fails and can be retried.

Design choices

Certificates are reused. The sidecar already sees the CloudNativePG instance certificates in /controller/certificates. The server presents the cluster's server certificate; the client presents the streaming replication certificate, and the server authorises the common name read from that certificate, so a custom replication CN works as long as pgBackRest can express it in its authorisation rule (no =). pgBackRest verifies the host name against the certificate and this cannot be turned off, which is why the Service name has to be in the certificate.

This widens what the replication certificate can do: its holder, with network access to port 8432, can run SQL as postgres and access the data directory through the pgBackRest protocol. A certificate of another cluster is accepted as well if it is signed by the same client CA and has the same common name. The README says so, and the feature is opt-in for that reason. Dedicated pgBackRest credentials are a possible later extension.

The server is long-running, not on-demand. The backup RPC only reaches the sidecar of the standby; starting a server in the primary's sidecar for the duration of a backup would need a coordination channel that does not exist today.

The server is supervised by the sidecar. It starts once the certificates are present and form a consistent key pair. After that, each time it observes a valid configuration and certificate set, a changed certificate, key or CA triggers a reload (SIGHUP) and a changed authorised CN or stanza triggers a restart; an unusable certificate set leaves the running server as it is. A log level change does not restart it, since a restart terminates running backups. The server runs in its own process group; the group is killed on stop and when the sidecar detects that the server exited, and a server that exited is started again while the feature is enabled. A missing Cluster or Archive disables the server; other read errors keep the previous state.

The feature reads the Archive straight from the API. The supervisor and the backup decision use the manager's API reader instead of the sidecar's caching client. The cache keeps an Archive for 10 seconds, so a backup right after enabling could fail as "not enabled", and it holds one mutex across the API call, so an unrelated slow read could stall the supervisor past its own deadline.

The polling is slow on purpose. Every instance of every cluster that uses the plugin has to poll, enabled or not, to learn that the feature was turned on. The supervisor reads the Cluster and the Archive roughly every five minutes, with jitter. Per 1,000 instances the expected steady-state load is about 6 GETs per second, where a one-minute poll would be about 33.

Not covered

  • Certificate rotation or a switchover while a backup is running. Neither is tested; the PR does not promise that an in-flight backup survives them.
  • If the standby makes no replay progress for pgBackRest's archive-timeout before reaching the backup start LSN, the backup fails and is not retried on the primary.
  • Which standby runs the backup is decided by CloudNativePG.
  • The server listens on IPv4 only; IPv6-only pod networks are not supported.

The e2e suite gets a standbybackup spec: two instances, a backup on the standby, a restore from it, a switchover, and a second backup on the former primary.

Closes #103. #104 explores the same problem with a different certificate model; this PR is an independent implementation.

Signed-off-by: ermakov-oleg <ermakovolegs@gmail.com>
@melancholictheory

Copy link
Copy Markdown
Contributor

Could we agree on the TLS trust model for standby backups before continuing the follow-ups in #104?

Is reusing the CNPG replication certificate acceptable for pgBackRest access given the additional privileges it grants? If so, what isolation should #188 require when clusters share a client CA and use the same replication CN? Should reuse require a per-cluster client CA or a unique replication identity under a shared CA, and can we enforce that rather than only document it?

If reuse cannot provide that isolation, should we require dedicated pgBackRest credentials, as proposed in #104? I'd like to settle this before we build out both approaches.

@ermakov-oleg

Copy link
Copy Markdown
Contributor Author

Thanks for raising this. My view:

Reuse stays the default. With CNPG-generated certificates every cluster has its own client CA, so a replication certificate of another cluster is not accepted. It needs no extra secrets, mounts or pod restarts, which is the direction @Agalin preferred in #103.

What reuse cannot isolate. pgBackRest authorises on CA and CN only, and CNPG pins the replication CN to streaming_replica through pg_ident, so a unique identity under a shared CA is not really available. Two cases remain: the client CA bundle holds additional CAs, or one CA issues replication certificates for several clusters. In both cases PostgreSQL already accepts the foreign certificate for streaming replication; pgBackRest raises that from read access to superuser.

What I propose. For the first case, enforce it: the server trusts only the CA certificates that chain the cluster's own replication certificate, not the whole bundle. The second case cannot be detected from inside the sidecar, so I would add opt-in dedicated credentials: backupStandby.tlsSecret, a user-provided secret that the sidecar reads through the API. No pod spec change, and the plugin does not have to run a CA.

I can add the issuer pinning to this PR now. Maintainers, would you want tlsSecret in this PR or as a follow-up?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: support backups taken from a standby (pgBackRest TLS multi-host + --backup-standby)

2 participants