David Steele | CrunchyData Blog

pgBackRest File Bundling and Block Incremental Backup

David.Steele@crunchydata.com (David Steele) — Fri, 09 Jun 2023 09:00:00 EDT

Crunchy Data is proud to support the pgBackRest project, an essential production grade backup tool used in our fully managed and self managed Postgres products. pgBackRest is also available as an open source project.

pgBackRest provides:

Full, differential, and incremental backups
Checksum validation of backup integrity
Point-in-Time recovery

pgBackRest recently released v2.46 with support for block incremental backup, which saves space in the repository by storing only changed parts of files. File bundling, released in v2.39, combines smaller files together for speed and cost savings, especially on object stores.

Efficiently storing backups is a major priority for the pgBackRest project but we also strive to balance this goal with backup and restore performance. The file bundling and block incremental backup features improve backup and, in many cases, restore performance while also saving space in the repository.

In this blog we will provide working examples to help you get started with these exciting features.

File bundling

combines smaller files together
improves speed on object stores like S3, Azure, GCS

Block incremental backup

saves space by storing only changed file parts
improves efficiency of delta restore

Sample repository set up

To demonstrate these features we will create two repositories. The first repository will use defaults. The second will have file bundling and block incremental backup enabled.

Configure both repositories:

[global]
log-level-console=info
start-fast=y

repo1-path=/var/lib/pgbackrest/1
repo1-retention-full=2

repo2-path=/var/lib/pgbackrest/2
repo2-retention-full=2
repo2-bundle=y
repo2-block=y

[demo]
pg1-path=/var/lib/postgresql/12/demo

Create the stanza on both repositories:

pgbackrest --stanza=demo stanza-create

The block incremental backup feature is best demonstrated with a larger dataset. In particular, we would prefer to have at least one table that is near the maximum segment size of 1GB. This can be accomplished by creating data with pgbench:

/usr/lib/postgresql/12/bin/pgbench -i -s 65

PostgreSQL splits tables into segment files of 1GB each, so the main table that pgbench created above will be contained in a single file. The format PostgreSQL uses to store tables on disk will be important in the examples below.

File bundling

File bundling stores data in the repository more efficiently by combining smaller files together. This results in fewer files overall in the backup which improves the speed of all repository operations, especially on object stores like S3, Azure, and GCS. There may also be cost savings on repositories that have a cost per operation since there will be fewer lists, deletes, etc.

To demonstrate this we'll make a backup on repo1, which does not have bundling enabled:

pgbackrest --stanza=demo --type=full --repo=1 backup

Now we check the number of files in repo1 for the latest backup:

$ find /var/lib/pgbackrest/1/backup/demo/latest/ -type f | wc -l

991

This is pretty normal for a small database without bundling enabled since each file is stored separately. There are also a few metadata files that pgBackRest uses to track the backup.

Now we'll perform the same actions on repo2, which has file bundling enabled:

$ pgbackrest --stanza=demo --type=full --repo=2 backup
$ find /var/lib/pgbackrest/2/backup/demo/latest/ -type f | wc -l

7

This time there are far fewer files. The small files have been bundled together and zero-length files are stored only in the manifest.

The repo-bundle-size option can be used to control the maximum size of bundles before compression and other operations are applied. The repo-bundle-limit option limits the files that will be added to bundles. It is not a good idea to set these options too large because any failure in the bundle on backup or restore will require the entire bundle to be retried. The goal of file bundling is to combine small files -- there is very seldom any benefit in combining larger files.

Block incremental backup

Block incremental backup saves space in the repository by storing only the parts of the file that have changed since the last backup. The block size depends on the file size and when the file was last modified, i.e. larger, older files will get larger block sizes. Blocks are compressed and encrypted into super blocks that can be retrieved independently to make restore more efficient.

To demonstrate the block incremental feature, we need to make some changes to the database. With pgbench we can update 100 random rows in the main table, which is about 1GB in size.

/usr/lib/postgresql/12/bin/pgbench -n -b simple-update -t 100

On repo1 the time to make an incremental backup is very similar to making a full backup. As previously discussed, PostgreSQL breaks tables up into 1GB segments so in our case the main table consists of a single file that contains most of the data in our database.

$ pgbackrest --stanza=demo --type=incr --repo=1 backup

<...>
INFO: backup command end: completed successfully (12525ms)

Here we can see that the incremental backup is nearly as large as the full backup, 52.8MB vs 55.5MB. This is expected since the bulk of the database is contained in a single file and by default incremental backups copy the entire file if any part of the file has changed.

$ pgbackrest --stanza=demo --repo=1 info

full backup: 20230520-082323F
    database size: 995.7MB, database backup size: 995.7MB
    repo1: backup size: 55.5MB

incr backup: 20230520-082323F_20230520-082934I
    database size: 995.7MB, database backup size: 972.8MB
    repo1: backup size: 52.8MB

However, on repo2 with block incremental enabled, the backup is significantly faster.

$ pgbackrest --stanza=demo --type=incr --repo=2 backup

<...>
INFO: backup command end: completed successfully (3589ms)

And also much smaller, 943KB vs 52.8MB on the repo without block incremental enabled. This is more than 50x improvement in backup size! Note that the block incremental backup feature also works with differential backups.

$ pgbackrest --stanza=demo --repo=2 info

full backup: 20230520-082438F
    database size: 995.7MB, database backup size: 995.7MB
    repo2: backup size: 56MB

incr backup: 20230520-082438F_20230520-083027I
    database size: 995.7MB, database backup size: 972.8MB
    repo2: backup size: 943.3KB

The block incremental feature also improves the efficiency of the delta restore command. Here we stop the cluster and perform a delta restore back to the full backup in repo 1:

$ pg_ctlcluster 12 demo stop
$ pgbackrest --stanza=demo --delta --repo=1 --set=20230526-053458F restore

<...>
INFO: restore command end: completed successfully (3697ms)

As we saw above the main table is contained in a single file, so the restore must copy and decompress the entire file from repo 1 (compressed size 30.4MB) because it was changed since the full backup.

To test a delta restore of the full backup in repo 2 we need to first restore the cluster to the most recent backup in repo 2:

pgbackrest --stanza=demo --delta --repo=2 restore

And then perform a delta restore back to the full backup in repo 2:

$ pgbackrest --stanza=demo --delta --repo=2 --set=20230526-053406F restore

<...>
INFO: restore command end: completed successfully (1536ms)

This is noticeably faster even on our fairly small demo database. When storage latency is high (e.g. S3) the performance improvement will be more pronounced. With block incremental enabled, delta restore only had to copy 3.5MB of the main table file from repo 2, as compared to 30.4MB from repo 1.

It is best to avoid long chains of block incremental backups since they can have a negative impact on restore performance. In this case pgBackRest may be forced to pull from many backups to restore a file.

Conclusion

Block incremental and file bundling both help make backup and restore more efficient and they are a powerful combination when used together. In general you should consider enabling both on all your repositories, with the caveat that these features are not backward compatible with older versions of pgBackRest.

Introducing pgBackRest Multiple Repository Support

David.Steele@crunchydata.com (David Steele) — Fri, 09 Apr 2021 05:00:00 EDT

The pgBackRest team is pleased to announce the introduction of multiple repository support in v2.33. Backups already provide redundancy by creating an offline copy of your PostgreSQL cluster that can be used in disaster recovery. Multiple repositories allow you to have copies of your backups and WAL archives in separate locations to increase your redundancy and provide even more protection for your data. This feature is the culmination of many months of hard work, so let's delve into why we think multiple repositories are so important and how they can help preserve your data.

If you are unfamiliar with pgBackRest, general repository configuration, or configuring PostgreSQL to work with pgBackRest, please read the pgBackRest Quick Start before proceeding.

Configuration

Up to four repositories may be configured and each one can be any repo type, e.g. S3 or Posix. The configuration below defines two repositories. repo1 is Posix and stored on locally-mounted NFS volume; the repository should always be located off the PostgreSQL server in case disaster strikes. repo2 is stored on Azure.

$ cat /etc/pgbackrest/pgbackrest.conf

[demo]
pg1-path=/var/lib/postgresql/13/demo

[global]
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2

repo2-type=azure
repo2-azure-account=pgbackrest
repo2-azure-container=demo-container
repo2-azure-key=YXpLZXk=
repo2-path=/demo-repo
repo2-retention-full=8

Note that the retention has been configured differently on each repository. repo1 has a shorter retention to save space but still provide several backups and plenty of WAL on local storage that is both fast and cheap (in terms of bandwidth cost) to access. repo2 has a longer retention and is stored in Azure where storage is cheap but retrieving data is both slower and more expensive than repo1. repo2 is both a fail-safe in case something goes wrong with repo1 and a resource to restore backups from remote sites.

pgBackRest will treat repo1 with higher priority than repo2 for certain commands like restore and archive-get. In general, the lower-numbered repositories should be faster and/or cheaper than the higher-numbered repositories.

Usage

Now that the repositories are configured, run stanza-create. This will initialize the demo stanza in each repository.

$ pgbackrest --stanza=demo stanza-create

<...>
INFO: stanza-create for stanza 'demo' on repo1
INFO: stanza-create for stanza 'demo' on repo2
<...>

Once the repositories have been initialized it is a good idea to run check to ensure that everything is working. Note that WAL segments are being pushed to both repositories.

$ pgbackrest --stanza=demo check

<...>
INFO: check repo1 configuration (primary)
INFO: check repo2 configuration (primary)
INFO: check repo1 archive for WAL (primary)
INFO: WAL segment 000000010000000000000003 successfully archived to '/var/lib/pgbackrest/archive/demo/13-1/0000000100000000/000000010000000000000003-c981b4ddc8c1437c539eda05427a6aa454a0923e.gz' on repo1
INFO: check repo2 archive for WAL (primary)
INFO: WAL segment 000000010000000000000003 successfully archived to '/demo-repo/archive/demo/13-1/0000000100000000/00000001000000000000000

Now a backup should be run for each repository. Most commands operate automatically on all repos but backup requires the repo to be specified. Each repo is likely to have different retention and backup schedules so backups should be run independently.

pgbackrest --stanza=demo --repo=1 backup
pgbackrest --stanza=demo --repo=2 backup

Info for the repositories is shown in a unified fashion with the organizing unit being the stanza. This way all the backups for a stanza can be seen together and the most recent backup easily identified since the list is sorted oldest to newest.

$ pgbackrest info

stanza: demo
    status: ok
    cipher: none

    db (current)
        wal archive min/max (13): 000000010000000000000003/000000010000000000000006

        full backup: 20210331-155726F
            timestamp start/stop: 2021-03-31 15:57:26 / 2021-03-31 15:57:30
            wal start/stop: 000000010000000000000005 / 000000010000000000000005
            database size: 23.1MB, database backup size: 23.1MB
            repo1: backup set size: 2.8MB, backup size: 2.8MB

        full backup: 20210331-155736F
            timestamp start/stop: 2021-03-31 15:57:36 / 2021-03-31 15:57:41
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 23.1MB, database backup size: 23.1MB
            repo2: backup set size: 2.8MB, backup size: 2.8MB

It is also possible to get info for a single repository by specifying the --repo option.

$ pgbackrest --repo=2 info

stanza: demo
    status: ok
    cipher: none

    db (current)
        wal archive min/max (13): 000000010000000000000003/000000010000000000000006

        full backup: 20210331-155736F
            timestamp start/stop: 2021-03-31 15:57:36 / 2021-03-31 15:57:41
            wal start/stop: 000000010000000000000006 / 000000010000000000000006
            database size: 23.1MB, database backup size: 23.1MB
            repo2: backup set size: 2.8MB, backup size: 2.8MB

When restoring, pgBackRest will automatically select the best backup from the repositories based on your criteria. Here is an example of time-based recovery:

pgbackrest --stanza=demo --delta --type=time --target="2021-03-31 15:57:31-04" restore

<...>
INFO: repo1: restore backup set 20210331-155726F
<...>

In this case only the backup from repo1 was valid because of the time restriction. Even if a later time was specified that would seem to favor the later backup in repo2, the same backup from repo1 will be selected. This is because pgBackRest tends to prefer the repo with higher priority, i.e. repo1 over repo2, under the assumption that it will be faster and/or cheaper than a repository with lower priority.

If you want to restore from a specific repository then simply specify the preferred repository using --repo. The archive-get command generated for recovery will always search all repositories in priority order for WAL segments.

Conclusion

Multiple repositories allow for more redundancy as well as cost savings and performance improvements by allowing a repository to be located close to the PostgreSQL clusters while also having a repository located safely out in the cloud or in another data center.

For more information about multiple repository support, see the User Guide.

pgBackRest - Reliable PostgreSQL Backup & Restore

David.Steele@crunchydata.com (David Steele) — Tue, 20 Sep 2016 05:00:00 EDT

In our ongoing series of blog posts designed to help you better run, manage, and support PostgreSQL, today we have a post discussing pgBackRest, a powerful open source tool for managing backups and restores of PostgreSQL databases...

Introduction

pgBackRest aims to be a simple, reliable backup and restore system that can seamlessly scale up to the largest databases and workloads.

Instead of relying on traditional backup tools like tar and rsync, pgBackRest implements all backup features internally and uses a custom protocol for communicating with remote systems. Removing reliance on tar and rsync allows for better solutions to database-specific backup challenges. The custom remote protocol allows for more flexibility and limits the types of connections that are required to perform a backup which increases security.

Features

Multi-process Backup & Restore

Compression is usually the bottleneck during backup operations but, even with now ubiquitous multi-core servers, most database backup solutions are still single-process. pgBackRest solves the compression bottleneck with multi-processing.

Utilizing multiple cores for compression makes it possible to achieve 1TB/hr raw throughput even on a 1Gb/s link. More cores and a faster network lead to even higher throughput.

Local or Remote Operation

A custom protocol allows pgBackRest to backup, restore, and archive locally or remotely via SSH with minimal configuration. An interface to query PostgreSQL is also provided via the protocol layer so that remote access to PostgreSQL is never required, which enhances security.

Full, Incremental, & Differential Backups

Full, differential, and incremental backups are supported. pgBackRest is not susceptible to the time resolution issues of rsync, making differential and incremental backups completely safe.

Backup from a Standby Cluster

Performing backups on a standby host greatly reduces CPU and IO load on the master host. pgBackRest copies the majority of the files from the standby and only a few from the master, while still producing a backup exactly as if it were performed entirely on the master.

Backup Rotation & Archive Expiration

Retention polices can be set for full and differential backups to create coverage for any timeframe. WAL archive can be maintained for all backups or strictly for the most recent backups. In the latter case WAL required to make older backups consistent will be maintained in the archive.

Backup Integrity

Checksums are calculated for every file in the backup and rechecked during a restore. After a backup finishes copying files, it waits until every WAL segment required to make the backup consistent reaches the repository.

Backups in the repository are stored in the same format as a standard PostgreSQL cluster (including tablespaces). If compression is disabled and hard links are enabled it is possible to snapshot a backup in the repository and bring up a PostgreSQL cluster directly on the snapshot. This is advantageous for terabyte-scale databases that are time consuming to restore in the traditional way.

All operations utilize file and directory level fsync to ensure durability.

Backup Resume

An aborted backup can be resumed from the point where it was stopped. Files that were already copied are compared with the checksums in the manifest to ensure integrity. Since this operation can take place entirely on the backup server, it reduces load on the database server and saves time since checksum calculation is faster than compressing and retransmitting data.

Streaming Compression & Checksums

Compression and checksum calculations are performed in stream while files are being copied to the repository, whether the repository is located locally or remotely.

If the repository is on a backup server, compression is performed on the database server and files are transmitted in a compressed format and simply stored on the backup server. When compression is disabled a lower level of compression is utilized to make efficient use of available bandwidth while keeping CPU cost to a minimum.

Delta Restore

The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are taken for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Multi-processing can lead to a dramatic reduction in restore times.

Advanced Archiving

Dedicated commands are included for both pushing WAL to the archive and retrieving WAL from the archive.

The push command automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. The push and get commands both ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This precludes the possibility of misconfiguring the WAL archive location.

Asynchronous archiving allows compression and transfer to be offloaded to another process which maintains a continuous connection to the remote server, improving throughput significantly. This can be a critical feature for databases with extremely high write volume.

Selective Restore

Selected databases can be restored from a cluster backup to save space when not all the databases are required. WAL replay during restore takes places for all databases so some space will be used, but generally far less than if the unneeded databases were restored completely. After recovery completes the unrestored databases will not be accessible but can be dropped in the usual way.

Tablespace & Link Support

Tablespaces are fully supported and on restore tablespaces can be remapped to any location. It is also possible to remap all tablespaces to one location with a single command which is useful for development restores.

File and directory links are supported for any file or directory in the PostgreSQL cluster. When restoring it is possible to restore all links to their original locations, remap some or all links, or restore some or all links as normal files or directories within the cluster directory.

Compatibility with PostgreSQL >= 8.3

pgBackRest includes support for versions down to 8.3, since older versions of PostgreSQL are still regularly utilized.

Additional Resources

Download pgBackRest here.
Documentation for pgBackRest can be found here.
User Guide for pgBackRest is here.

Photo Credit: Evan-Amos