Depending on your business requirements, you may only need the default asynchronous behavior, or you may need to configure one or more synchronous replicas. Fortunately, PostgreSQL lets you choose and provides options for tuning the consistency and performance (latency) behavior depending on your requirements.

See the PostgreSQL documentation for more details on streaming replication.

Preparation

Name your instances. Life will be simpler. You do that by setting a configuration parameter in each instance's postgresql.conf file. We'll see why it makes things easier later using the pg_stat_replication table's contents on the primary. Since you use the name of a replica to configure it as synchronous, if each replica has a unique name, you can configure individual replicas as synchronous or asynchronous.

For example, on one of the replicas in postgresql.conf

(Note: If your cluster is managed or created by Patroni or Crunchy HA PostgreSQL, it will manage the contents of postgresql.conf, so make the changes in the Patroni config, which will generate the postgresql.conf file that is used by the servers in your cluster)

cluster_name = 'replica2' # added to process titles if nonempty

Creating a Replica

The first step in creating a replica is to clone the primary. There are several ways to do that, but the straightforward way is to use pg_basebackup, which clones a running primary PostgreSQL instance. For the simple test case for this article, to create two replicas, as user postgres

/usr/lib/postgresql/12/bin/pg_basebackup -Xs -D ~/12/replica1 -R -p 5433 -h localhost -U replicant
/usr/lib/postgresql/12/bin/pg_basebackup -Xs -D ~/12/replica2 -R -p 5443 -h localhost -U replicant

These connect to the primary as a client as the user replicant which must exist as a user in the primary DB instance and have REPLICATION privileges.

Note the -Xs argument which will open a second connection to the primary to capture WAL changes to the DB as it operates normally. The -R argument will tell pg_basebackup to create the recovery configuration in the data directory for the new replica.

Other ways to clone a primary are:

• pgBackrest to backup the primary and restore to the new replica location.
• OS level backups or snapshots, but you must be certain that you get a consistent copy of the primary. A safe way to do this is to stop the instance first.

Example replica settings

(Note that synchronous replication is independent of replication slots. Either can be used with or without the other.)

On a replica:

Note that on the replica's configuration, including the recovery section, there is no indication that it's sync or async. Whether a replica is synchronous or async is determined by the primary's configuration.

This was generated by pg_basebackup to clone the primary and create a replica.

cat /var/lib/postgresql/12/replica2/postgresql.auto.conf

Do not edit this file manually!

• It will be overwritten by the ALTER SYSTEM command.
• On PostgreSQL versions prior to 12, this information is stored in the recovery.conf file
• Recovery settings generated by pgBackRest restore on 2020-02-13 13:11:08
• standby_mode on # only on pg versions <12. Replaced by the standby.signal file in pg12

recovery_target_timeline 'latest'
recovery_target_action = 'promote'
primary_conninfo = 'user=replication_user passfile=''/var/lib/postgresql/.pgpass'' port=5433 host=''localhost'' user=''replicant'''

Make a replica synchronous

Once you have streaming replication working, on the primary add a replica name to synchronous_standby_names in postgresql.conf:

synchronous_standby_names = 'replica2'

and tell the postgresql primary to reload the config (e.g. - pg_ctl reload -D $PGDATA or the method your HA support uses). Now the instance named replica2 is a synchronous replica. That's all. Really.

Changing a replica from synchronous back to the default asynchronous is similar. Just remove that replica's name from the list in synchronous_standby_names in the primary's config and tell the primary to reload its configuration.

Reviewing and checking the current replication cluster

Back on the primary:

  pid  | usesysid | usename  | application_name | client_addr | client_hostname | client_port |         backend_start         | backend_xmin |   state   |  sent_lsn  | write_lsn  | flush_lsn  | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
-------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+------------+------------+------------+------------+-----------+-----------+------------+---------------+------------+-------------------------------
 28488 |       10 | replicant | walreceiver      | 127.0.0.1   |                 |       43221 | 2020-08-25 08:25:22.658642-07 |              | streaming | 0/2E000060 | 0/2E000060 | 0/2E000060 | 0/2E000060 |           |           |            |             0 | async      | 2020-08-25 08:25:22.400688-07
 15936 |    24794 | replicant | replica2         | 127.0.0.1   |                 |       50772 | 2020-08-25 08:25:06.760228-07 |              | streaming | 0/2E000060 | 0/2E000060 | 0/2E000060 | 0/2E000060 |           |           |            |             0 | sync       | 2020-08-25 08:25:56.915357-07
(2 rows)

• Note that we have a sync and async replica.
• Note that one of the replicas has cluster_name unset/defaulting, so walreceiver is its name.
• Recall this setting in postgresql.conf on the primary.

synchronous_standby_names = 'replica2'

Life will be much easier if the replicas don't all default to cluster_name unset, where all the replicas will have the default name walreceiver.

Changing a replica to/from synchronous

Changing a replica from synchronous to asynchronous, or vice versa, is easy. Just add the replica name to synchronous_standby_names in the primary's postgresql.conf and tell PostgreSQL to reload the configuration; no DB restart needed.

How synchronous is it? Waiting for storage

How synchronous is it? The value of synchronous_commit on the primary determines this.

synchronous_commit = on  # the default

In order of increasing "safety" (durability), and increasing latency, values of synchronous_commit on the primary:

• When we set synchronous_commit = off, a COMMIT does not wait for the transaction record to be flushed to disk.
• When we set synchronous_commit = local, a COMMIT waits until the transaction record is flushed to the local disk.
• When we set synchronous_commit = on, a COMMIT will wait until the server(s) specified by synchronous_standby_names confirm that the transaction record was safely written to disk.
  • Note: When synchronous_standby_names is empty, this setting behaves same as synchronous_commit = local.
• When we set synchronous_commit = remote_write, a COMMIT will wait until the server(s) specified by synchronous_standby_names confirm write of the transaction record to the operating system but has not necessarily reached the disk on the replica.
• When we set synchronous_commit = remote_apply, a COMMIT will wait until the server(s) specified by synchronous_standby_names confirm that the transaction record was applied to the replica's database.

How much data might not be replicated in a case where the replica loses connectivity with the primary, with the faster and less durable options? That depends on more PostgreSQL settings - wal_writer_delay and wal_writer_flush_after. The first one flushes WAL after a specified time period (200ms default), the second one flushes if the specified number of WAL files are created since the last flush. If you set synchronous_commit to off, then these two settings will limit how much WAL remains uncommitted.

Setting synchronous behavior in a session (client)

Since the synchronous commit behavior is related to a transaction, it can be changed by a client for the session and during a session, so a client can set different values for each statement. You can set the synchronous behavior at any of these levels:

• Single statement / transaction - SET LOCAL synchronous_commit =
• Session - `SET synchronous_commit = ``
• User - ALTER USER someuser SET synchronous_commit =
• Database - ALTER DATABASE SET synchronous_commit =
• And of course cluster wide by updating postgreql.conf

Adding priority or quorum to the list of synchronous replicas

In addition to specifying how synchronous a remote replica, you can also create a list of synchronous replicas by priority - FIRST , or a quorum of replicas - ANY.

Quorum is an important aspect of distributed computing. You may already know what it is, but if not, here’s a simplified explanation. In this case we are concerned with consistency of the DB data across multiple DB nodes. When a number of nodes - that you choose - all have the same data committed, then the cluster is considered to be in a consistent state. The number of nodes can be all the nodes in the cluster, or it can be a subset of the nodes. The number of nodes is the quorum number and is chosen depending on your business requirements for data consistency. In this case, the nodes “vote” for quorum by replying to the primary that they have received and applied the replicated data. It’s common to have an odd number of nodes in a cluster and then define quorum as the majority of nodes with consistent copies of the DB data (e.g. - 2 of 3 nodes or 3 of 5 nodes). Quorum is used for other purposes in distributed computing. A common case is electing a new primary from a cluster of nodes when the current primary fails or is unavailable.

There are more options in the primary's postgresql.conf setting of synchronous_standby_names to support priority and simple quorum.

For the priority case, the FIRST keyword:

synchronous_standby_names = ‘FIRST num (standby_name [, …])’

The synchronous commit will wait for a reply from at least num number of standbys listed in the order of priority.

For the quorum case, the ANY keyword:

synchronous_standby_names = ‘ANY num (standby_name [, …])’

The same rules as above apply. So, for example setting synchronous_standby_names = 'ANY 2 (\*)' will cause synchronous commit to wait for reply from any 2 standby servers. Double check your syntax and test that the settings implement your business rules for consistency.

Configure so you don't wait forever

With synchronous replication, you've built in a dependency that a transaction is not committed on the primary until it's written to the synchronous replica, so depending on the configuration options above, your primary can hang forever if the replica (or quorum of replicas) is not reachable by the primary. Obviously, you're dependent on the connection between primary and replicas.

If you have a single synchronous replica and it is unavailable, your primary will wait for it to return, and will block until it does. To avoid that you want to have at least two replicas and use the FIRST or ANY options to synchronous_standby_names described above. You could disable synchronous replication by commenting out synchronous_standby_names but then of course, you don't have a synchronous replica.

The TL;DR

To convert a streaming binary replica to synchronous, add its name to the primary's postgresql.conf setting synchronous_standby_names, and reload the primary.

To convert a synchronous replica to asynchronous, remove its name from the primary's postgresql.conf setting synchronous_standby_names, and reload the primary.

These will be much easier if you've added a unique name to the replica's cluster_name setting in its postgresql.conf

References

As always, the PostgreSQL documentation is the place to look for more information - https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION ]]> https://blog.crunchydata.com/blog/synchronous-replication-in-postgresql Wed, 30 Sep 2020 05:00:00 EDT 2020-09-30T09:00:00.000Z 2020-09-30T09:00:00.000Z https://www.crunchydata.com/blog/how-to-recover-when-postgresql-is-missing-a-wal-file Creation and clean up of WAL files in the primary's pg_wal folder (pg_xlog prior to PG10) is a normal part of PostgreSQL operation. The WAL files on the primary are used to ensure data consistency during crash recovery. Use of write-ahead logs (also called redo logs or transaction logs in other products) is common for data stores that must provide durability and consistency of data when writing to storage. The same technique is used in modern journaling and log-structured filesystems.

As the DB is operating, blocks of data are first written serially and synchronously as WAL files, then some time later, usually a very short time later, written to the DB data files. Once the data contained in these WAL files has been flushed out to their final destination in the data files, they are no longer needed by the primary. At some point, depending on your configuration, the primary will remove or recycle the WAL files whose data has been committed to the DB. This is necessary to keep the primary's disk from filling up. However, these WAL files are also what streaming replicas read when they are replicating data from the primary. If the replica is able to keep up with the primary, using these WAL files generally isn't an issue.

If the replica falls behind or is disconnected from the primary for an extended period of time, the primary may have already removed or recycled the WAL file(s) that a replica needs (but see Streaming Replication Slots below). A replica can fall behind on a primary with a high write rate. How far the replica falls behind will depend on network bandwidth from the primary, as well as storage performance on the replica.

To account for this possibility, we recommend keeping secondary copies of the WAL files in another location using a WAL archiving mechanism. This is known as WAL archiving and is done by ensuring archive_mode is turned on and a value has been set for the archive_command. These are variables contained in the postgresql.conf file.

Whenever the primary generates a WAL file, this command is run to make a secondary copy of it. Until that archive_command succeeds, the primary will keep that WAL file, so you must monitor for this command failing. Otherwise the primary's disk may fill. Once WAL archiving is in place, you can then configure your replicas to use that secondary location for WAL replay if they ever lose their connection to the primary.

This process is explained more in the PostgreSQL documentation.

Note that configuration details for creating a cluster have changed starting in PostgreSQL 12. In particular, the recovery.conf file on a replica instance no longer exists and those configuration lines are now part of postgresql.conf. If you're using PostgreSQL 12 or newer, read the documentation carefully and note the new files recovery.signal and standby.signal.

Crunchy Data provides the pgBackRest tool which provides full WAL archiving functionality as well as maintaining binary backups. We do not recommend the simple copy mechanism given as an example in the documentation since that does not provide the resiliency typically required for production databases. pgBackRest provides full, differential and incremental backups as well as integrated WAL file management.

WAL archiving and backups are typically used together since this then provides point-in-time recovery (PITR) where you can restore a backup to any specific point in time as long as you have the full WAL stream available between all backups.

pgBackRest is an integral part of the Crunchy HA and Crunchy PostgreSQL Operator products, and is what we recommend for a binary backup and archive tool.

See https://pgbackrest.org/ for more information on pgBackRest.

Environment

The most common case where PostgreSQL won't start because it can't find a WAL file is in a replicated cluster where a replica has been disconnected from the cluster for some time. Most of the rest of this article will discuss ways to diagnose and recover this case.

For most of this article we will discuss the cluster case, where:

• Cluster of 2 or more PostgreSQL hosts

• Using WAL archiving via the PostgreSQL archive_command configuration, plus binary backups, likely and preferably pgBackRest

• On replicas, the recovery.conf (or postgresql.auto.conf in PG 12 and newer, see also standby.signal and recovery.signal) has a line with restore_command = that pulls WAL files from a binary backup location and applies them to the replica.

• If using pgBackRest, the line may look like:

restore_command = 'pgbackrest --stanza=demo archive-get %f "%p"'

• It's common to use both PostgreSQL binary streaming replication and WAL file archiving together.
• It's common to use a tool like pgBackRest for both binary backups and WAL archiving. Combining the two gives you the opportunity to restore a DB to a specific point in time (PITR).

Symptoms

• A replica will doesn't start completely and won't accept read-only connections.
• If using Crunchy HA (or Patroni), patronictl list may show no leader, or Lag in DB Unknown and cluster members stopped.
• This is a common symptom when the primary has already recycled/removed the requested WAL file:

2020-03-13 09:32:22.572 EDT [101800] ERROR: requested WAL segment 00000002000000050000007C has already been removed

• You may see log entries in the pg_log logs like:

2020-04-17 14:29:49.479 P00 INFO: unable to find 0000001600000000000000C2 in the archive,

and/or

2020-04-17 14:29:49 EDT [379]: [6-1], , FATAL: requested timeline 23 does not contain minimum recovery point 0/C2A56FC0 on timeline 22

and especially these:

2020-04-17 14:29:49 EDT [376]: [5-1], , LOG: database system is shut down

2020-04-17 14:29:49 EDT [459]: [1-1], , LOG: database system was interrupted while in recovery at log time 2020-04-17 14:19:28 EDT

2020-04-17 14:29:49 EDT [459]: [2-1], ,

HINT: If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.

Common Causes

The underlying cause for a replica getting out of sync and not able to replay WAL (either via streaming replication or from the WAL archive / backup) is almost always an infrastructure issue and almost always network connectivity interruptions. To have a HA cluster you must have reliable network connectivity between the cluster members and from each of the cluster members to the WAL archive (backup) location.

It's worth a reminder that time synchronization across cluster member hosts is critically important for correct cluster operation. Always check and confirm that all nodes in the cluster have a NTP service running (e.g. - ntpd or chronyd), and that the nodes are correctly synced to each other and to a master time source.

It is common to use the same tools for binary backups and WAL archiving. A common configuration in a HA cluster is to use pgBackRest as both the backup tool and the WAL archiving and playback tool. With pgBackRest and other binary backup tools, you will likely have a backup schedule that does a full backup of the cluster primary server periodically and differential or incremental backups periodically between the full backups. Along with the backup schedule are backup retention settings. For example, you may have configured your cluster to retain the last three full backups and the last three incremental or differential backups.

In addition to the backups, pgBackRest will retain WAL files that are needed to do a point-in-time recovery from your full, differential and incremental backups. So if a replica has been disconnected long enough (several backup cycles) for the archived WAL files to be past their retention period, when reconnected, PostgreSQL will be far behind the current state and will attempt to restore archived WAL files that no longer exist. In that case, you will need to reset or reinitialize the replica from current backups and the WAL files that are relative to them.

If the DB data disk fills completely on the replica, the replica will stop accepting and applying WAL updates from the primary. If this isn't caught and repaired for some time, the primary may have removed older WAL files. The length of time will depend on the configuration on the primary and the change rate on the primary. See the documentation for wal_keep_segments to retain more older WAL files.

Another failure mode is when you have intermittent network connectivity among hosts, and the cluster fails over and fails back several times. For each failover, a replica is promoted to primary (failover or switchover), and the WAL timeline is incremented. If one of the replicas can't communicate with the cluster for some time, its local DB will be based on an earlier timeline, and when it attempts to restore a WAL file from the earlier timeline, that WAL file won't be found in the archive, which will default to the current primary's timeline. Note that there is an option to specify the timeline in the recovery.conf file but you probably want to fix the replica to be on the current primary's timeline. See here for more information.

This is simplified when using pgBackRest, which is the recommended method and the method used by Crunchy HA, and the backup-standby option is enabled. With this option enabled, backups are done from a replica/standby host rather than the primary. There is more explanation here in the pgBackRest documentation.

Check and confirm that the former primary was properly re-synch'ed to the new primary. This should have been done automatically by your cluster software, and involves cloning the new primary's data directory to the new replica using a tool like pg_rewind or pgBackRest's delta restore (or full restore). If the new replica (former primary) is logging messages about being on an older timeline, the re-synch may not have happened or may not have been done correctly.

Repairing / Fixing a Replica

It's likely that the best and only option will be to restore the replica from the backup/archive server. We recommend pgBackRest. Crunchy Data products, including the Crunchy PostgreSQL Operator, Crunchy HA and others include and use pgBackRest.

• pgBackRest restore
  • full restore vs delta restore. A pgBackRest delta restore can save time and resources. It checks the PostgreSQL destination and restore only files that are needed and don't exist in the destination.

If you're using Crunchy HA or the Crunchy PostgreSQL Operator, you want to use the HA layer tools rather than using pgBackRest directly.

First use

patronictl reinit

to restore a cluster member node (PostgreSQL instance) from the backup/archive system.

Repairing / Fixing a Standalone or Failed Primary

• This one is potentially more serious and will be a reminder of why you want regular, reliable and carefully tested backups, especially the standalone server case, and why a properly managed replica and backups are a very good idea for data you like.

• This can be caused by broken or failing hardware, misconfigured storage, power failures, or mistakenly using OS level commands to modify the DB directory contents.

• Avoid the temptation to just run pg_resetwal (pg_resetxlog in versions prior to PG 10). More on this below, and please read the man page and/or pg docs on pg_resetwal risks.

• The first thing to do is properly and completely stop the failing standalone or primary.

• If your primary is part of a cluster, and your cluster uses software to manage auto-failover, like Crunchy HA, Patroni, or pacemaker/corosync, your cluster control software should detect a failure, and properly failover to one of the replicas. Occasionally it doesn't and you will have to manually stop the failed primary/standalone. If you're not using software to manage your cluster, failover is a manual process, where you first shut down the failed primary, then promote one of the replicas to be the new primary. If you are using software for auto-failover, and it hasn't automatically failed over, what you need to do is first, shutdown the failed primary instance, preferably using the same mechanism that was used to start it - patronictl if it's using Patroni or Crunchy HA, or pcs or crmsh if you're using pcs/corosync/pacemaker clustering, or an OS-level utility like systemd's systemctl or the Linux service, or pg_ctl. Even if those complete successfully, you want to check to be certain that all the PostgreSQL processes have stopped. Occasionally, none of those stop all the PostgreSQL processes. In that case, you need to use the Linux kill command. To stop the postmaster, the three signals that correspond to "smart", "fast" and "immediate" shutdown are SIGTERM, SIGINT and SIGQUIT. The very last option to use is SIGKILL (kill -9); avoid doing that unless it's a true emergency.

• Once all the PostgreSQL processes are no longer running, make a copy of the DB's data directory using your favorite OS-level snapshot utility - tar, cpio, rsync or your infrastructure's disk snapshotting tool. This is a safe copy in case attempts to repair the DB cause more damage.

• To get the DB back to healthy, the best option is to recreate the DB from a good backup. If you have a good replica (now likely the primary), make the failed primary a replica from the new primary, after fixing the underlying cause.

• In the worst case, where you have only the broken DB directory, you may be able to get it working again but you will likely lose data. First make an OS level copy of the DB directory. Avoid using OS level commands to move or delete DB files; you will likely make things worse.

• It may be tempting to just run pg_resetwal. While it may be possible to recover from some errors using pg_resetwal, you will likely lose data in the process. You can also do more damage to the DB. If you need to minimize data loss, don't have a replica to restore or a good backup, and don't have detailed knowledge of PostgreSQL, it's time to ask for help. It may be possible to recover most of your DB but read the documentation and note this advice from the documentation: "It should be used only as a last resort, when the server will not start due to such corruption."

Streaming Replication Slots

Streaming replication slots are a feature available since PostgreSQL 9.4. Using replication slots will cause the primary to retain WAL files until the primary has been notified that the replica has received the WAL file.

There is a tradeoff for keeping the WAL files on the primary until the primary is notified that the replica has received the WAL file. If the replica is unavailable for any reason, WAL files will accumulate on the primary until it can deliver them. On a busy database, with replicas unavailable, disk space can be consumed very quickly as unreplicated WAL files accumulate. This can also happen on a busy DB with relatively slow infrastructure (network and storage). Streaming replication slots are a very useful feature; if you do use them, carefully monitor disk space on the primary.

There is much more on replication slots and replication configuration generally.

Multi-datacenter

If you have deployed HA across multiple data centers, there's another layer of complexity. Crunchy HA provides example Ansible files for several MDC configurations. In the case where the primary data center fails and the DR data center has been promoted to primary, extra steps are required when you recover the original failed primary. In particular, when you start the recovered original primary, be certain that you start it as a DR data center. You will likely need to copy (rsync or other reliable mechanism) the pgBackRest repo from the new primary data center (old DR data center) when recovering the original primary data center.

What not to do / Don't do this

• Again, avoid just running pg_resetwal (pg_resetxlog). You may need to use it as a last resort, but don't start by running it.
• Do not try to cherry-pick or copy individual files from a backup / archive repo directly to the PostgreSQL the pg_wal (pg_xlog in earlier versions) directory. That is part of the database runtime that is managed by PostgreSQL. Deleting or copying things there can break your DB. Let the tools do their jobs.
• Never use OS tools to manually delete, modify or add files to the PostgreSQL pg_wal (pg_xlog) directory.
• Never use OS tools to manually delete, modify or add files to the pgBackRest repo. The pgBackRest repo contains state and metadata about current backups and which WAL files depend on which backups. (Note that listing or viewing the files in a pgBackRest repo can be helpful to diagnose replicas and restore issues. It's also OK to copy the repo in its entirety as long as you use a copy utility that verifies that the files have been copied correctly, for example rsync.)

Since PgBouncer is located logically between the client and PostgreSQL you have the option of using TLS and cert authentication from client to PgBouncer and from PgBouncer to PostgreSQL. In this brief blog post, we’ll describe configuring securing the client-to-PgBouncer transport first, then build on that to use client certificate authentication to PgBouncer.

A central part of this is TLS and tools for creating and maintaining keys, certificates, signing requests, signing and more. For this talk we use the widely used open source software OpenSSL, but any utilities that produce valid keys and certificates could be used.

The client certificates will need to be signed by the same CA (certificate authority) that signed the PgBouncer certificate. For testing and for this article we’ll use self-signed certificates but for production you should at least create a local CA, or preferably, use a public CA, though the latter can get expensive if you have many client certificates. Both PgBouncer and PostgreSQL have a configuration option that determines the level of root certificate verification, ranging from no verification to strict verification. This accommodates a range of uses, including self-signed certificates for internal use to more secure environments that must use certs signed by a public CA.

Testing for this post was done with PgBouncer 1.12.0 on Linux.

Creating a TLS certificate for PgBouncer

We’ll use openssl to create a certificate for PgBouncer, to enable TLS transport security. Here are the steps:

1. Generate a private key (you must provide a passphrase).

   openssl genrsa -des3 -out server.key 1024
   

2. Remove the passphrase (but remember it).

   openssl rsa -in server.key -out server.key
   

3. Set appropriate permission and owner on the private key file.

   chmod 400 server.key
   chown postgres.postgres server.key
   

4. Create the server certificate signing request. Note that this is where the process differs depending on whether you use self-signed certificates (like here) or create a CSR (certificate signing request) and send it to a CA to be signed, and who will return you the signed certificate, private key and root (or intermediate) certificates for your new signed cert. Note the -x509 below that produces a self-signed certificate instead of a CSR, and -subj is a shortcut to avoid prompting for the info and typing it interactively.

• Creating a self-signed cert with the -x509 argument. You probably don't want to do this in production:

  openssl req -new -key server.key -days 3650 -out server.crt -x509 -subj '/C=US/ST=Washington/L=Redmond/O=Crunchy Data/CN=crunchy-testuser1/emailAddress=testuser1@example.com'
  

• or instead, generate a CSR (certificate signing request) for a real certificate, use this and send the .csr file to your CA to be signed:

  openssl req -new -key server.key -out server.csr -subj '/C=US/ST=Washington/L=Redmond/O=Crunchy Data/CN=crunchy-testuser1/emailAddress=testuser1@example.com'
  

  Your CA will return a signed certificate and key to you.

  Change the -subj arg details for your environment of course, and note that the CN= part of the cert's needs to be the hostname of your PgBouncer host. You can use SAN (Subject Alt Names) to define more than one hostname in the cert, but that's outside the scope of this post.

At this point, you have a signed certificate, its private key and a root certificate from the signing CA, or your self-signed cert.

Configuring PgBouncer to use TLS transport security (prerequisite for cert authentication)

Once you have a signed certificate for PgBouncer, configuring for TLS transport security is pretty straightforward.

You need to set these options in your pgbouncer.ini file (/etc/pgbouncer/pgbouncer.ini on Linux):

    client_tls_sslmode = require
    client_tls_ca_file = /etc/pgbouncer/root.crt
    client_tls_key_file = /etc/pgbouncer/server.key
    client_tls_cert_file = /etc/pgbouncer/server.crt
    client_tls_ciphers = normal

Note that there are stricter checking options for client_tls_sslmode but the require value will not allow non-TLS/SSL connections from clients. And we can restrict the allowed TLS cipher suites and versions with client_tls_ciphers but one step at a time.

For now, leave auth_type to something other than cert, for example md5.

Restart PgBouncer using whatever your platform requires, for example systemctl restart pgbouncer on a Linux system that uses systemd.

And let’s test that it’s working. Or not. Here’s a simple test using psql with auth_type = trust in pgbouncer.ini:

bash> psql "sslmode=require host=localhost port=6432"
Password for user testuser1:
psql (12.1)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.testuser1=# \q<

If the client requests a SSL connection, it succeeds.

bash> psql "host=localhost port=6432"
psql: ERROR:  SSL required
bash>

If the client does not request a SSL connection, it fails.

OK. encrypted TLS connections between client and PgBouncer are working, and using a fairly secure cipher suite and TLS version. With client_tls_sslmode = require, if the client doesn’t request a TLS/SSL connection, it’s denied. So we have the transport layer from client to PgBouncer using TLS.

Note that this process is almost identical to configuring PostgreSQL to use TLS/SSL transport authentication. Later, in a follow up blog post, we’ll see how to configure PgBouncer to use TLS/SSL from PgBouncer to the DB. But first, let’s continue to configure TLS client cert authentication to PgBouncer.

Client Certs and options for CA's

At this point, we have TLS transport encryption between client and PgBouncer configured and working. What's next is to create and deploy client certificates and enable cert authentication in PgBouncer.

These steps closely mirror the procedure described here in and earlier blog post on how to set up TLS authentication within Docker containers.

An important consideration and a choice to make when doing this is what you will use for a CA (Certificate Authority). There are at least three options, depending on your needs and requirements for security.

1. The simplest is to not have a CA and use self-signed certificates. This requires that you use one of the less strict verification options for client_tls_sslmode. It's also not recommended for use in production. For this method, we create a self-signed cert for the PgBouncer server, then use its private key to sign the client certs, and the PgBouncer cert is also the root CA cert.

2. Build and manage your own local CA.

   For help building a local, private CA, see one of these:

   • the open source minica project https://github.com/jsha/minica
   • the new open source mkcert project https://github.com/FiloSottile/mkcert
   • Vault (if you’re already using it) can be a local CA
   • Most Linux distros have tools and instructions for creating and managing a local CA, see for example the easy-rsa package, also see the official Ubuntu docs here: https://help.ubuntu.com/lts/serverguide/certificates-and-security.html

3. Use a public CA, though this can get expensive and complicated to administer if you have many clients (or clients and servers).

The high-level steps are:

1. Create the client certificate and Certificate Signing Request. The key thing here is that the CN (Common Name) in the client cert must be a valid user in the PostgreSQL instance.

   openssl req -newkey rsa:4096 -keyout testuser1_key.pem -out testuser1_csr.pem -nodes -days 365 -subj "/CN=testuser1"
   

2. Sign the client certificate with the root certificate that's installed in PgBouncer. For the example here, we're using self-signed certs, so we'll use the private key for the PgBouncer server certificate to sign the client cert.

   openssl x509 -req -in testuser1_csr.pem -CA server.crt -CAkey server.key -out testuser1_cert.pem -set_serial 01 -days 365
   

3. Install the signed client cert on the client(s).

   cp server.crt ~/.postgresql/root.crt
   cp testuser1_cert.pem ~/.postgresql/postgresql.crt
   cp testuser1_key.pem ~/.postgresql/postgresql.key
   chmod 400 ~/.postgresql/postgresql.key
   

   Note that there are number of connection parameters you can set for SSL/TLS and related environment variables.

   Also note that these are for clients that use the libpq PostgreSQL library. If your client does not use it, see the docs for your client for TLS/SSL support and location of client certs and keys.

4. Update the PgBouncer config to use certificate authentication with an acceptable level of signature verification. Edit your pgbouncer.ini file and set

   auth_type = cert
   ;; required for cert auth
   client_tls_sslmode = verify-full
   

   Restart PgBouncer

5. Test

   1. Simple psql test

      testuser1-bash> psql -U testuser1 -p 6432 -h crunchy-testuser1
      psql (12.1)
      SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
      Type "help" for help.
      
      testuser1=# \q
      

   2. Test with PGSSLMODE=require vs PGSSLMODE=verify-ca vs PGSSLMODE=verify-full client-side environment variables.

   3. Try different DB username (fails)

      testuser1-bash> psql -U postgres -p 6432 -h crunchy-testuser1
      psql: error: could not connect to server: ERROR:  TLS certificate name mismatch
      

   4. Copy client cert and key from user testuser1 to different user's ~/.postgresql and try connecting as that user (fails)

      testuser-bash> PGSSLMODE=verify-full psql -h crunchy-testuser1 -p 6432
      psql: error: could not connect to server: ERROR:  TLS certificate name mismatch
      

      but note this, logged in as user testuser and you have installed the key and cert for user

      testuser1
      

      testuser-bash> PGSSLMODE=verify-full psql -U testuser1 -h crunchy-testuser1 -p 6432
      psql (12.1)
      SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
      Type "help" for help.
      
      testuser1=#
      

The moral for this test: Protect your key and password.

Note on this test environment

For this article, we're implementing to a common configuration, where PgBouncer and PostgreSQL are both running on the same host, and we use only local connections (Unix domain sockets) from PgBouncer to PostgreSQL.

Here's the pgbouncer.ini from the test environment:

[databases]
;; Three DB's, with PgBouncer connecting only on local/UDS
testuser1 =
template1 =
postgres =
[users]
[pgbouncer]
logfile = /var/log/postgresql/pgbouncer.log
pidfile = /var/run/postgresql/pgbouncer.pid
listen_addr = *
listen_port = 6432
unix_socket_dir = /var/run/postgresql
client_tls_sslmode = verify-full
client_tls_ca_file = /etc/pgbouncer/root.crt
client_tls_key_file = /etc/pgbouncer/server.key
client_tls_cert_file = /etc/pgbouncer/server.crt
client_tls_ciphers = normal
auth_type = cert
auth_file = /etc/pgbouncer/userlist.txt
admin_users = testuser1

Contents of /etc/pgbouncer/pgbouncer.ini:

bash> cat /etc/pgbouncer/userlist.txt
"testuser1" "<hashed password from pg_shadow for user testuser1 here>"
"testuser2" "<hashed password from pg_shadow for user testuser2 here>"

And contents of ${PGDATA}/pg_hba.conf. Note that this PostgreSQL instance is listening only on localhost and on a local Unix domain socket:

# Database administrative login by Unix domain socket
local   all             postgres                                peer
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             testuser1                               md5
local   all             testuser2                               md5
# "local" is for Unix domain socket connections only
local   all             all                                     peer
#
# IPv4 local connections:# IPv6 local connections:
host    all             all             127.0.0.1/32            md5
host    all             all             ::1/128                 md5

Note that for user postgres, peer auth works because PgBouncer is running as user postgres in this environment.

For authenticating from PgBouncer to PostgreSQL, we're still relying on passwords (md5 hashed in this case).

Connection from PgBouncer to PostgreSQL

At this point, we have a secure transport from the app client to PgBouncer, the client-supplied DB username (which can be specified in the client) must match the CN in the client certificate, and the hostname parameter of the connection string from the client must match the signer (Issuer) in the client cert. On the client/app side, you can adjust how strict validation is using the PGSSLMODE environment variable for apps that use libpq.

But we're still relying on the credentials in PgBouncer's userlist.txt auth_file parameter together with PostgreSQL's pg_hba.conf file to authenticate from PgBouncer to PostgreSQL.

In a follow up blog post, we will describe how to reduce the overhead of managing passwords in the auth_file using the method described in Doug Hunley's post here: https://hunleyd.github.io/posts/pgbouncer-and-auth-pass-thru/

Certificate Authentication from PgBouncer to PostgreSQL

Another common PgBouncer configuration is where the PgBouncer service does not reside on the DB server. It might be on a dedicated server, or there may be a PgBouncer service on each of several app or web servers.

In that case you will have one or more TCP connections from each PgBouncer service to the PostgreSQL server that can be secured using TLS transport as well as configured to user cert auth to the PostgreSQL DB. The configuration is very similar to the description above but in this case, you configure the PgBouncer services to each have a client cert and use hostssl ... cert authentication in the PostgreSQL server after you have enabled SSL in the DB server and configured TLS certs for it and the PgBouncer clients.

That's a topic for another blog post.

To prepare, and to enable SSL/TLS in the PostgreSQL DB, start with https://www.postgresql.org/docs/current/ssl-tcp.html and https://www.crunchydata.com/blog/ssl-certificate-authentication-postgresql-docker-containers. ]]> https://blog.crunchydata.com/blog/improving-pgbouncer-security-with-tlsssl Mon, 16 Dec 2019 04:00:00 EST 2019-12-16T09:00:00.000Z 2019-12-16T09:00:00.000Z