Keith Fiske | CrunchyData Blog https://www.crunchydata.com/blog/author/keith-fiske https://www.crunchydata.com/build/_assets/keith-fiske.png-RFVAA3IN.webp Keith Fiske | CrunchyData Blog https://www.crunchydata.com/blog/author/keith-fiske 600 750 PostgreSQL experts from Crunchy Data share advice, performance tips, and guides on successfully running PostgreSQL and Kubernetes solutions en-us Fri, 06 Dec 2024 08:30:00 EST 2024-12-06T13:30:00.000Z en-us hourly 1 <![CDATA[ Postgres Partitioning with a Default Partition ]]> https://www.crunchydata.com/blog/postgres-partitioning-with-a-default-partition Partitioning is an important database maintenance strategy for a growing application backed by PostgreSQL. As one of the main authors of pg_partman and an engineer here at Crunchy Data, I spend a lot of my time helping folks implement partitioning. One of the nuances of PostgreSQL’s partitioning implementation is the default partition, which I’ll dig into in this post and discuss how to use it effectively.

Why default partitions are important

The default partition is pretty much what it sounds like; you can make a special partition designated as the DEFAULT, which will capture any and all data that does not have an existing partition with matching boundary constraints.

If you’re new to partitioning, you might be making partitions a week in advance. But after monitoring you realize you need to make them more like 2 weeks in advance. Default partitions can help you learn how to manage and when to create your child partitions.

Default partitions are also there to catch mistakes. Maybe there’s an issue in application code putting timestamps a hundred years into the future instead of one year. Maybe there’s just some bad data getting created. Your default partition can help you spot that.

While having a default partition is a good idea, you don’t actually want to leave data in there. I’ll show you some tips later on about how to monitor the default for the presence of any rows. When you find data in there, you’ll want to evaluate whether the data is valid, and if it is, create the relevant child partitions and move the data there.

Adding a default

PostgreSQL declarative partitioning does not create any child partitions automatically, including the default. pg_partman can help with that and we’ll discuss that later.

Here we have a daily partition set that has been created but does not yet have a default.

                                      Partitioned table "partman_test.time_taptest_table"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Partitions: partman_test.time_taptest_table_p20241118 FOR VALUES FROM ('2024-11-18 00:00:00-05') TO ('2024-11-19 00:00:00-05'),
            partman_test.time_taptest_table_p20241119 FOR VALUES FROM ('2024-11-19 00:00:00-05') TO ('2024-11-20 00:00:00-05'),
            partman_test.time_taptest_table_p20241120 FOR VALUES FROM ('2024-11-20 00:00:00-05') TO ('2024-11-21 00:00:00-05'),
            partman_test.time_taptest_table_p20241121 FOR VALUES FROM ('2024-11-21 00:00:00-05') TO ('2024-11-22 00:00:00-05'),
            partman_test.time_taptest_table_p20241122 FOR VALUES FROM ('2024-11-22 00:00:00-05') TO ('2024-11-23 00:00:00-05'),
            partman_test.time_taptest_table_p20241123 FOR VALUES FROM ('2024-11-23 00:00:00-05') TO ('2024-11-24 00:00:00-05'),
            partman_test.time_taptest_table_p20241124 FOR VALUES FROM ('2024-11-24 00:00:00-05') TO ('2024-11-25 00:00:00-05'),
            partman_test.time_taptest_table_p20241125 FOR VALUES FROM ('2024-11-25 00:00:00-05') TO ('2024-11-26 00:00:00-05'),
            partman_test.time_taptest_table_p20241126 FOR VALUES FROM ('2024-11-26 00:00:00-05') TO ('2024-11-27 00:00:00-05'),

If you try to insert data for 2024-12-25, that will be outside the existing child partition boundaries, you will get an error, and the data is lost.

INSERT INTO partman_test.time_taptest_table (col3) VALUES ('2024-12-25'::date);
ERROR:  no partition of relation "time_taptest_table" found for row
DETAIL:  Partition key of the failing row contains (col3) = (2024-12-25 00:00:00-05).

Adding a DEFAULT partition is very easy:

CREATE TABLE partman_test.time_taptest_table_default PARTITION OF partman_test.time_taptest_table DEFAULT;

\d+ partman_test.time_taptest_table
                                      Partitioned table "partman_test.time_taptest_table"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Partitions: partman_test.time_taptest_table_p20241118 FOR VALUES FROM ('2024-11-18 00:00:00-05') TO ('2024-11-19 00:00:00-05'),
            partman_test.time_taptest_table_p20241119 FOR VALUES FROM ('2024-11-19 00:00:00-05') TO ('2024-11-20 00:00:00-05'),
            partman_test.time_taptest_table_p20241120 FOR VALUES FROM ('2024-11-20 00:00:00-05') TO ('2024-11-21 00:00:00-05'),
            partman_test.time_taptest_table_p20241121 FOR VALUES FROM ('2024-11-21 00:00:00-05') TO ('2024-11-22 00:00:00-05'),
            partman_test.time_taptest_table_p20241122 FOR VALUES FROM ('2024-11-22 00:00:00-05') TO ('2024-11-23 00:00:00-05'),
            partman_test.time_taptest_table_p20241123 FOR VALUES FROM ('2024-11-23 00:00:00-05') TO ('2024-11-24 00:00:00-05'),
            partman_test.time_taptest_table_p20241124 FOR VALUES FROM ('2024-11-24 00:00:00-05') TO ('2024-11-25 00:00:00-05'),
            partman_test.time_taptest_table_p20241125 FOR VALUES FROM ('2024-11-25 00:00:00-05') TO ('2024-11-26 00:00:00-05'),
            partman_test.time_taptest_table_p20241126 FOR VALUES FROM ('2024-11-26 00:00:00-05') TO ('2024-11-27 00:00:00-05'),
            partman_test.time_taptest_table_default DEFAULT

Now when we try and insert the data that failed before, it succeeds and we can see it is in the default table.

INSERT INTO partman_test.time_taptest_table (col3) VALUES ('2024-12-25'::date);
INSERT 0 1

SELECT * FROM partman_test.time_taptest_table_default;
  col1  | col2  |          col3
--------+-------+------------------------
 «NULL» | stuff | 2024-12-25 00:00:00-05
(1 row)

Constraints with partition tables

The constraint on a normal partition is as you’d expect it to be, showing the lower and upper bounds.

keith@keith=# \d partman_test.time_taptest_table_p20241124
            Table "partman_test.time_taptest_table_p20241124"
 Column |           Type           | Collation | Nullable |    Default
--------+--------------------------+-----------+----------+---------------
 col1   | integer                  |           |          |
 col2   | text                     |           |          | 'stuff'::text
 col3   | timestamp with time zone |           | not null | now()
Partition of: partman_test.time_taptest_table FOR VALUES FROM ('2024-11-24 00:00:00-05') TO ('2024-11-25 00:00:00-05')

If we look at the default partition, we see that the constraint set up is not so simple.

\d+ partman_test.time_taptest_table_default

Table "partman_test.time_taptest_table_default"
Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
col1   | integer                  |           |          |               | plain    |             |              |
col2   | text                     |           |          | 'stuff'::text | extended |             |              |
col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition of: partman_test.time_taptest_table DEFAULT
Partition constraint: (NOT ((col3 IS NOT NULL) AND (((col3 >= '2024-11-18 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-19 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-19 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-20 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-20 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-21 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-21 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-22 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-22 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-23 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-23 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-24 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-24 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-25 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-25 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-26 00:00:00-05'::timestamp with time zone)) OR ((col3 >= '2024-11-26 00:00:00-05'::timestamp with time zone) AND
(col3 < '2024-11-27 00:00:00-05'::timestamp with time zone)))))

The constraint of a default partition in PostgreSQL can basically be thought of as an anti-constraint of all the other currently existing partitions. When a new partition is added, that anti-constraint is automatically updated to account for the new partition’s boundaries.

But what happens if we try to add a new partition that matches data in the default?

CREATE TABLE partman_test.time_taptest_table_p20241225 PARTITION OF partman_test.time_taptest_table FOR VALUES FROM ('2024-12-25') TO ('2024-12-26');

ERROR:  updated partition constraint for default partition "time_taptest_table_default" would be violated by some row

We get a constraint violation because there is already data in the default partition that would match the new partition’s boundaries. PostgreSQL cannot allow there to be two possible partition routes for the same values.

Moving default data to a new child table

Because of these constraint violations, we must develop a process to be able to keep our data and get it moved to the proper partition: remove that data from the default partition, add the new child partition, then reinsert the data back via the parent so the data routes to the new partition. Thanks to PostgreSQL’s transactional DDL, this can all be done in a single transaction making it transparent to your users.

Here is an example of moving the data from the default to a new partition in a single transaction.

BEGIN;

CREATE TEMP TABLE clean_default_temp (LIKE partman_test.time_taptest_table_default);

WITH partition_data AS (
    DELETE FROM partman_test.time_taptest_table_default RETURNING *
)
INSERT INTO clean_default_temp (col1, col2, col3) SELECT col1, col2, col3 FROM partition_data;

CREATE TABLE partman_test.time_taptest_table_p20241225 PARTITION OF partman_test.time_taptest_table FOR VALUES FROM ('2024-12-25') TO ('2024-12-26');

WITH partition_data AS (
    DELETE FROM clean_default_temp RETURNING *
)
INSERT INTO partman_test.time_taptest_table (col1, col2, col3) SELECT col1, col2, col3 FROM partition_data;

DROP TABLE clean_default_temp;

COMMIT;

SELECT * FROM partman_test.time_taptest_table_default ;
 col1 | col2 | col3
------+------+------
(0 rows)

SELECT * FROM partman_test.time_taptest_table;
  col1  | col2  |          col3
--------+-------+------------------------
 «NULL» | stuff | 2024-12-25 00:00:00-05
(1 row)

SELECT * FROM partman_test.time_taptest_table_p20241225 ;
  col1  | col2  |          col3
--------+-------+------------------------
 «NULL» | stuff | 2024-12-25 00:00:00-05
(1 row)

Large amounts of data in the default

This above example was rather simple for just a single row. However, if you have a large amount of data in the default, this could cause a noticeable disturbance to your users since these rows that are being moved will be locked until the transaction commits. This can be done in smaller batches, but to stay completely transparent to your users, the smallest transactional batch you could do would be the interval size of the partition set, in this case 1 day. You could do it in smaller transactional batches, but that would have to be done to a permanent table that you’re moving the data to and that data would be inaccessible to your users via their normal means. This is because you cannot add that new child partition until ALL the data that would go into it has been removed from the default. However, this isn’t even the most serious problem with data going into the default.

The way that PostgreSQL is able to tell you that you cannot add that new child partition is because at the time you try and attach one, PostgreSQL does a scan of the entire default partition to see if the new child partition’s boundaries match any data there. Even if you have an index on the default, PostgreSQL is going to have to scan the entire table, and most likely be using a costly sequential scan anyway. This means the lock obtained on the parent table to add a partition is held for the duration of the attach command’s transaction. If you’ve got billions of rows, this could possibly take minutes or even longer. This is why it is critical to keep an eye on any data going into any default partition table and move or remove it as soon as possible.

pg_partman

pg_partman is an open source extension for managing partitioning in PostgreSQL and adds several features to PostgreSQL’s built-in, declarative partitioning including automatically creating child partitions, including a default partition for every partitioned table set.

pg_partman check default

The pg_partman extension has a utility to check the default table for rows with the check_default() function. Let’s say we have 4 rows in our default table. Passing no parameters to this function will do a full count on all default partitions of all partition sets managed by pg_partman and return how many rows it found in each partition set.

SELECT * FROM partman.check_default();
              default_table              | count
-----------------------------------------+-------
 partman_test.time_taptest_table_default |     4

However, if you pass false to this function, it will not do a full count and simply return a 1 if even a single row is encountered in any default partition (using a LIMIT 1 clause). This usage of the function can be used for regular monitoring of your partition sets.

SELECT * FROM partman.check_default(false);
              default_table              | count
-----------------------------------------+-------
 partman_test.time_taptest_table_default |     1

Default row cleanup with pg_partman

The pg_partman procedure partition_data_proc() will automatically clean up your default partition. This procedure does the same steps shown above for native partitioning: moving the data to a temporary table, creating the necessary child tables based on the data found, then moving the data back.

Let’s look at our default partition with 4 rows of data.

select * from partman_test.time_taptest_table_default;
  col1  | col2  |          col3
--------+-------+------------------------
 «NULL» | stuff | 2024-12-25 00:00:00-05
 «NULL» | stuff | 2024-12-26 00:00:00-05
 «NULL» | stuff | 2024-12-27 00:00:00-05
 «NULL» | stuff | 2024-12-28 00:00:00-05

When we call the function partition_data_proc(), it commits after each child partition is created. If you do not give this procedure a source table, it assumes you are moving data out of the default partition for the given partition set.

CALL partman.partition_data_proc('partman_test.time_taptest_table');
NOTICE:  Loop: 1, Rows moved: 1
NOTICE:  Loop: 2, Rows moved: 1
NOTICE:  Loop: 3, Rows moved: 1
NOTICE:  Loop: 4, Rows moved: 1
NOTICE:  Total rows moved: 4
NOTICE:  Ensure to VACUUM ANALYZE the parent (and source table if used) after partitioning data

Notice that the last line of advice is very important to ensure the statistics for your partition set have been updated and old rows cleaned up properly.

VACUUM ANALYZE partman_test.time_taptest_table;

Gaps in child partitions

Now we can see that the new child partitions have been made, the data has been moved to them, and the default partition is empty.

\d+ partman_test.time_taptest_table
                                      Partitioned table "partman_test.time_taptest_table"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Foreign-key constraints:
    "fk_test" FOREIGN KEY (col2) REFERENCES partman_test.fk_test_reference(col2)
Partitions: partman_test.time_taptest_table_p20241118 FOR VALUES FROM ('2024-11-18 00:00:00-05') TO ('2024-11-19 00:00:00-05'),
            partman_test.time_taptest_table_p20241119 FOR VALUES FROM ('2024-11-19 00:00:00-05') TO ('2024-11-20 00:00:00-05'),
            partman_test.time_taptest_table_p20241120 FOR VALUES FROM ('2024-11-20 00:00:00-05') TO ('2024-11-21 00:00:00-05'),
            partman_test.time_taptest_table_p20241121 FOR VALUES FROM ('2024-11-21 00:00:00-05') TO ('2024-11-22 00:00:00-05'),
            partman_test.time_taptest_table_p20241122 FOR VALUES FROM ('2024-11-22 00:00:00-05') TO ('2024-11-23 00:00:00-05'),
            partman_test.time_taptest_table_p20241123 FOR VALUES FROM ('2024-11-23 00:00:00-05') TO ('2024-11-24 00:00:00-05'),
            partman_test.time_taptest_table_p20241124 FOR VALUES FROM ('2024-11-24 00:00:00-05') TO ('2024-11-25 00:00:00-05'),
            partman_test.time_taptest_table_p20241125 FOR VALUES FROM ('2024-11-25 00:00:00-05') TO ('2024-11-26 00:00:00-05'),
            partman_test.time_taptest_table_p20241126 FOR VALUES FROM ('2024-11-26 00:00:00-05') TO ('2024-11-27 00:00:00-05'),
            partman_test.time_taptest_table_p20241225 FOR VALUES FROM ('2024-12-25 00:00:00-05') TO ('2024-12-26 00:00:00-05'),
            partman_test.time_taptest_table_p20241226 FOR VALUES FROM ('2024-12-26 00:00:00-05') TO ('2024-12-27 00:00:00-05'),
            partman_test.time_taptest_table_p20241227 FOR VALUES FROM ('2024-12-27 00:00:00-05') TO ('2024-12-28 00:00:00-05'),
            partman_test.time_taptest_table_p20241228 FOR VALUES FROM ('2024-12-28 00:00:00-05') TO ('2024-12-29 00:00:00-05'),
            partman_test.time_taptest_table_default DEFAULT

But, you will notice that we now have a gap between Nov 26 and Dec 25th.

pg_partman will only make new partitions based on the NEWEST partition and row data, in this case Dec 28th. It will not automatically fill in gaps to avoid potentially expensive automatic maintenance operations. However in many cases, you will be expecting data for these missing child partitions and will want to fill in the gaps.

pg_partman has a utility that you can run manually to do this: partition_gap_fill.

SELECT * FROM partman.partition_gap_fill('partman_test.time_taptest_table');
 partition_gap_fill
--------------------
                 28

After running this, PostgreSQL returns the number of partitions that were created and, as you can see below, we now have a daily partitioned set fully covered from Nov 18, 2024 to Dec 28, 2024.

\d+ partman_test.time_taptest_table
                                      Partitioned table "partman_test.time_taptest_table"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Foreign-key constraints:
    "fk_test" FOREIGN KEY (col2) REFERENCES partman_test.fk_test_reference(col2)
Partitions: partman_test.time_taptest_table_p20241118 FOR VALUES FROM ('2024-11-18 00:00:00-05') TO ('2024-11-19 00:00:00-05'),
            partman_test.time_taptest_table_p20241119 FOR VALUES FROM ('2024-11-19 00:00:00-05') TO ('2024-11-20 00:00:00-05'),
            partman_test.time_taptest_table_p20241120 FOR VALUES FROM ('2024-11-20 00:00:00-05') TO ('2024-11-21 00:00:00-05'),
            partman_test.time_taptest_table_p20241121 FOR VALUES FROM ('2024-11-21 00:00:00-05') TO ('2024-11-22 00:00:00-05'),
            partman_test.time_taptest_table_p20241122 FOR VALUES FROM ('2024-11-22 00:00:00-05') TO ('2024-11-23 00:00:00-05'),
            partman_test.time_taptest_table_p20241123 FOR VALUES FROM ('2024-11-23 00:00:00-05') TO ('2024-11-24 00:00:00-05'),
            partman_test.time_taptest_table_p20241124 FOR VALUES FROM ('2024-11-24 00:00:00-05') TO ('2024-11-25 00:00:00-05'),
            partman_test.time_taptest_table_p20241125 FOR VALUES FROM ('2024-11-25 00:00:00-05') TO ('2024-11-26 00:00:00-05'),
            partman_test.time_taptest_table_p20241126 FOR VALUES FROM ('2024-11-26 00:00:00-05') TO ('2024-11-27 00:00:00-05'),
            partman_test.time_taptest_table_p20241127 FOR VALUES FROM ('2024-11-27 00:00:00-05') TO ('2024-11-28 00:00:00-05'),
            partman_test.time_taptest_table_p20241128 FOR VALUES FROM ('2024-11-28 00:00:00-05') TO ('2024-11-29 00:00:00-05'),
            partman_test.time_taptest_table_p20241129 FOR VALUES FROM ('2024-11-29 00:00:00-05') TO ('2024-11-30 00:00:00-05'),
            partman_test.time_taptest_table_p20241130 FOR VALUES FROM ('2024-11-30 00:00:00-05') TO ('2024-12-01 00:00:00-05'),
            partman_test.time_taptest_table_p20241201 FOR VALUES FROM ('2024-12-01 00:00:00-05') TO ('2024-12-02 00:00:00-05'),
            partman_test.time_taptest_table_p20241202 FOR VALUES FROM ('2024-12-02 00:00:00-05') TO ('2024-12-03 00:00:00-05'),
            partman_test.time_taptest_table_p20241203 FOR VALUES FROM ('2024-12-03 00:00:00-05') TO ('2024-12-04 00:00:00-05'),
            partman_test.time_taptest_table_p20241204 FOR VALUES FROM ('2024-12-04 00:00:00-05') TO ('2024-12-05 00:00:00-05'),
            partman_test.time_taptest_table_p20241205 FOR VALUES FROM ('2024-12-05 00:00:00-05') TO ('2024-12-06 00:00:00-05'),
            partman_test.time_taptest_table_p20241206 FOR VALUES FROM ('2024-12-06 00:00:00-05') TO ('2024-12-07 00:00:00-05'),
            partman_test.time_taptest_table_p20241207 FOR VALUES FROM ('2024-12-07 00:00:00-05') TO ('2024-12-08 00:00:00-05'),
            partman_test.time_taptest_table_p20241208 FOR VALUES FROM ('2024-12-08 00:00:00-05') TO ('2024-12-09 00:00:00-05'),
            partman_test.time_taptest_table_p20241209 FOR VALUES FROM ('2024-12-09 00:00:00-05') TO ('2024-12-10 00:00:00-05'),
            partman_test.time_taptest_table_p20241210 FOR VALUES FROM ('2024-12-10 00:00:00-05') TO ('2024-12-11 00:00:00-05'),
            partman_test.time_taptest_table_p20241211 FOR VALUES FROM ('2024-12-11 00:00:00-05') TO ('2024-12-12 00:00:00-05'),
            partman_test.time_taptest_table_p20241212 FOR VALUES FROM ('2024-12-12 00:00:00-05') TO ('2024-12-13 00:00:00-05'),
            partman_test.time_taptest_table_p20241213 FOR VALUES FROM ('2024-12-13 00:00:00-05') TO ('2024-12-14 00:00:00-05'),
            partman_test.time_taptest_table_p20241214 FOR VALUES FROM ('2024-12-14 00:00:00-05') TO ('2024-12-15 00:00:00-05'),
            partman_test.time_taptest_table_p20241215 FOR VALUES FROM ('2024-12-15 00:00:00-05') TO ('2024-12-16 00:00:00-05'),
            partman_test.time_taptest_table_p20241216 FOR VALUES FROM ('2024-12-16 00:00:00-05') TO ('2024-12-17 00:00:00-05'),
            partman_test.time_taptest_table_p20241217 FOR VALUES FROM ('2024-12-17 00:00:00-05') TO ('2024-12-18 00:00:00-05'),
            partman_test.time_taptest_table_p20241218 FOR VALUES FROM ('2024-12-18 00:00:00-05') TO ('2024-12-19 00:00:00-05'),
            partman_test.time_taptest_table_p20241219 FOR VALUES FROM ('2024-12-19 00:00:00-05') TO ('2024-12-20 00:00:00-05'),
            partman_test.time_taptest_table_p20241220 FOR VALUES FROM ('2024-12-20 00:00:00-05') TO ('2024-12-21 00:00:00-05'),
            partman_test.time_taptest_table_p20241221 FOR VALUES FROM ('2024-12-21 00:00:00-05') TO ('2024-12-22 00:00:00-05'),
            partman_test.time_taptest_table_p20241222 FOR VALUES FROM ('2024-12-22 00:00:00-05') TO ('2024-12-23 00:00:00-05'),
            partman_test.time_taptest_table_p20241223 FOR VALUES FROM ('2024-12-23 00:00:00-05') TO ('2024-12-24 00:00:00-05'),
            partman_test.time_taptest_table_p20241224 FOR VALUES FROM ('2024-12-24 00:00:00-05') TO ('2024-12-25 00:00:00-05'),
            partman_test.time_taptest_table_p20241225 FOR VALUES FROM ('2024-12-25 00:00:00-05') TO ('2024-12-26 00:00:00-05'),
            partman_test.time_taptest_table_p20241226 FOR VALUES FROM ('2024-12-26 00:00:00-05') TO ('2024-12-27 00:00:00-05'),
            partman_test.time_taptest_table_p20241227 FOR VALUES FROM ('2024-12-27 00:00:00-05') TO ('2024-12-28 00:00:00-05'),
            partman_test.time_taptest_table_p20241228 FOR VALUES FROM ('2024-12-28 00:00:00-05') TO ('2024-12-29 00:00:00-05'),
            partman_test.time_taptest_table_default DEFAULT

Summary

  • PostgreSQL does not make any child partitions automatically, including a default partition. If you’re using partitioning, it is recommended to have a default partition to catch mistakes in application code or in child partition creation. However it is very important to monitor the contents of those default partitions.
  • pg_partman manages automatically creating child partitions for you, including the default partition if desired. The check_default function can help you monitor the contents of default partitions.
  • If rows are found in the default, it is important to ensure these are reviewed ASAP. If the rows are invalid, they can simply be deleted. If they are important, devise a process to move them to the proper child partitions. pg_partman’s partition_data_proc can assist with this.

The default partition is an incredibly useful tool to ensure you do not lose important data that may not be covered by existing child partitions. If you see data frequently going into the default partition, I highly advise reviewing your partition maintenance to ensure it is keeping up with the window of data that is regularly being ingested into that partition set. If the necessary child partitions always exist, you will have the best performance with the least maintenance. ]]> 315a9020b320411eb8023215d7492428204b85b43c4846f482b27b6a2dcdaa6c Fri, 06 Dec 2024 08:30:00 EST 2024-12-06T13:30:00.000Z 2024-12-06T13:30:00.000Z <![CDATA[ Announcing an Open Source Monitoring Extension for Postgres with pgMonitor ]]> https://www.crunchydata.com/blog/announcing-an-open-source-monitoring-extension-for-postgres-with-pgmonitor Crunchy Data is pleased to announce a new open source pgMonitor Extension. Crunchy Data has worked on a pgMonitor tool for several years as part of our Kubernetes and self-managed Postgres deployments and recently we’ve added an extension to the tool set.

Two primary scenarios motivated the creation of the pgMonitor extension :

  1. Quicker Metrics: Monitoring metrics often need quick response times to allow for frequent updates. We've noticed that certain metrics become slower as the database grows. This impacts not only common metrics but also more complex business metrics that could require several minutes to generate.
  2. Version Compatibility: New PostgreSQL versions can break existing metrics due to changes in the catalogs. Managing different metric sets for various PostgreSQL versions is tedious and can be challenging.

Benefits of the pgMonitor extension

The pgMonitor extension focuses on improving query performance and simplifying metric collection:

  • Efficient Metrics Storage: the pgMonitor extension queries PostgreSQL internal tables and stores metrics data in materialized views. This setup reduces the overhead on the database and allows for frequent polling without performance degradation.
  • Flexible Refresh Intervals: The extension uses a background worker to refresh materialized views on the database at intervals you choose, ensuring up-to-date metrics with minimal impact.
  • High Performance: By leveraging materialized views, the pgMonitor extension can handle slower metric queries efficiently, keeping the overall metrics scrape process fast.
  • Broad Compatibility: the pgMonitor extension integrates seamlessly with various monitoring systems like Prometheus, Icinga/Nagios, etc., allowing for simpler configuration and flexibility.
  • Future-Proofing: With query definitions contained within the database, the pgMonitor extension can adapt more easily to changes in Postgres versions, simplifying support for multiple versions.

Getting started with pgMonitor

Install the extension code

make install

CREATE SCHEMA pgmonitor_ext;
CREATE EXTENSION pgmonitor SCHEMA pgmonitor_ext;

Add to shared libraries

shared_preload_libraries = 'pgmonitor_bgw'     # (change requires restart)

Set the database(s) for the background worker to run on

pgmonitor_bgw.dbname = 'proddb,staging'

Set the role for the background worker to use

pgmonitor_bgw.role = 'postgres'

Background workers and pgMonitor configuration

pgMonitor supports a refresh of the materialized views with a background worker. The extension has configuration tables that help manage all the objects mentioned above. The names of all views and materialized views are stored in the metric_views configuration table.

                              Table "pgmonitor_ext.metric_views"
       Column       |           Type           | Collation | Nullable |        Default
--------------------+--------------------------+-----------+----------+-----------------------
 view_schema        | text                     |           | not null | 'pgmonitor_ext'::text
 view_name          | text                     |           | not null |
 materialized_view  | boolean                  |           | not null | true
 concurrent_refresh | boolean                  |           | not null | true
 run_interval       | interval                 |           | not null | '00:10:00'::interval
 last_run           | timestamp with time zone |           |          |
 last_run_time      | interval                 |           |          |
 active             | boolean                  |           | not null | true
 scope              | text                     |           | not null | 'global'::text

Key configs:

  • Concurrent Refresh: The materialized_view column indicates if a view is a materialized view, and concurrent_refresh specifies if a concurrent refresh can be done. This avoids locking issues during refreshes.
  • Refresh Interval: The run_interval column sets how often a materialized view should be refreshed, ensuring timely updates.
  • Monitoring Metrics: Columns like last_run and last_run_time track the last refresh time and duration, useful for alerting if metrics are not refreshed as expected.
  • Custom Table Refresh: The metric_tables configuration table allows defining custom SQL statements for refreshing data, enabling versatile data collection methods.

Out of the box metrics

The pgMonitor comes with many metrics already built into it. Some examples are below and users are able to add their own views and materialized views and to meet their own business needs.

pgMonitor Views:

  • Recovery and uptime
  • Postgres version
  • Monitor status of exhausted transaction ids
  • Status of successful WAL archiving
  • List of pending settings requiring a restart
  • Replication lag
  • Connection statistics
  • Locks
  • Query times

pgMonitor Materialized views

  • Table information
  • Table size
  • Database size
  • pgBackrest monitoring
  • Background worker statistics
  • Database statistics
  • Transaction ID exhaustion/wraparound
  • Checksum settings

You can also see a list of some of the metrics below from the pgmonitor_ext.metric_views table.

SELECT view_schema, view_name, materialized_view, scope FROM pgmonitor_ext.metric_views ;

  view_schema  |          view_name           | materialized_view |  scope
---------------+------------------------------+-------------------+----------
 pgmonitor_ext | ccp_transaction_wraparound   | f                 | global
 pgmonitor_ext | ccp_archive_command_status   | f                 | global
 pgmonitor_ext | ccp_postmaster_uptime        | f                 | global
 pgmonitor_ext | ccp_replication_lag          | f                 | global
 pgmonitor_ext | ccp_connection_stats         | f                 | global
 pgmonitor_ext | ccp_replication_lag_size     | f                 | global
 pgmonitor_ext | ccp_stat_user_tables         | t                 | database
 pgmonitor_ext | ccp_table_size               | t                 | database
 [...]

Another example is an included function pg_stat_statements_func() that can be used instead of querying pg_stat_statements directly. This provides consistent compatibility across all supported versions of PostgreSQL without having to worry about catalog differences between the major versions. Crunchy Data plans on providing version compatibility functions like this within the extension should the need ever arise in the future for any other metric.

In addition to the included metrics, users are free to add their own metrics for the pgMonitor extension to manage as well. Simply add the name of the materialized view to the table with the additional configuration options as needed.

Conclusion

The pgMonitor extension will advance the state of the art for PostgreSQL monitoring, by offering performance improvements and simplifying the management of metrics across different PostgreSQL versions. By addressing these challenges, the pgMonitor extension ensures reliable, efficient, and scalable data monitoring. It also will enable easier management of custom monitoring queries, allowing you to easily customize your monitoring stack to capture and track the data that is important to you.

We invite you to try the pgMonitor extension and experience the benefits firsthand. For more details and installation instructions, visit our pgMonitor GitHub repository. If you have questions or are interested in Crunchy Data's integrated solutions that leverage the pgMonitor extension, contact us at info@crunchydata.com ]]> 5917c5e15303511c115660e45fd9d26b4d88e2c85a3fe5c350f043fbb5208b92 Tue, 27 Aug 2024 10:00:00 EDT 2024-08-27T14:00:00.000Z 2024-08-27T14:00:00.000Z <![CDATA[ Time Partitioning and Custom Time Intervals in Postgres with pg_partman ]]> https://www.crunchydata.com/blog/time-partitioning-and-custom-time-intervals-in-postgres-with-pg_partman Whether you are managing a large table or setting up automatic archiving, time based partitioning in Postgres is incredibly powerful. pg_partman’s newest versions support a huge variety of custom time internals. Marco just published a post on using pg_partman with our new database product for doing analytics with Postgres, Crunchy Data Warehouse. So I thought this would be a great time to review the basic and complex options for the time based partitioning.

Time partitioning intervals

When I first started designing pg_partman for time-based partitioning, it only had preset intervals that users could choose. Currently, pg_partman supports all of Postgres’ time base interval values. The partitioning interval is set during the initial parent creation, in the p_interval field.

SELECT partman.create_parent(
    p_parent_table := 'partman_test.time_taptest_table'
    , p_control := 'col3'
    , p_interval := '1 day'
    , p_template_table := 'partman_test.time_taptest_table_template'
);

Additional examples like:

p_interval := '1 month'
p_interval := '1 year'

Even with allowing these intervals, some common intervals used in business, like weekly and quarterly, can be a little tricky. But thankfully pg_partman still has options to make these intervals easy and now even more flexible. So let’s dig into these examples.

Weekly partitioning

Weekly partitioning was and still is a fairly popular partitioning interval. When I started working on it, I’d thankfully found the ISO week date standard to allow me to tackle the more difficult issues of handling weeks (leap years, starting days, 53 week years) when I’d wanted to label the children with the week number . However with declarative partitioning I found an opportunity to allow this to be more flexible when redesigning things for version 5 of partman. While the result did get rid of the nice weekly numbering pattern I had liked for this interval (IYYYwIW which came out to something like “2024w15”), the new method lets people start their week on whichever day they desired. However, with flexibility always comes a little more complexity.

When you set your partitioning interval to 1 week in pg_partman, the day that starts that weekly pattern will be whatever day of the week it is when you run create_parent(). So today being a Wednesday when I’m writing this blog post, my partition naming pattern AND constraints for the child tables would be as follows:

CREATE TABLE time_stuff(id int GENERATED ALWAYS AS IDENTITY, created_at timestamptz NOT NULL) PARTITION BY RANGE (created_at);

SELECT partman.create_parent('public.time_stuff', 'created_at', '1 week');
 create_parent
---------------
 t
(1 row)

\d+ time_stuff
                                                      Partitioned table "public.time_stuff"
   Column   |           Type           | Collation | Nullable |           Default            | Storage | Compression | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------+---------+-------------+--------------+-------------
 id         | integer                  |           | not null | generated always as identity | plain   |             |              |
 created_at | timestamp with time zone |           | not null |                              | plain   |             |              |
Partition key: RANGE (created_at)
Partitions: time_stuff_p20240327 FOR VALUES FROM ('2024-03-27 00:00:00-04') TO ('2024-04-03 00:00:00-04'),
            time_stuff_p20240403 FOR VALUES FROM ('2024-04-03 00:00:00-04') TO ('2024-04-10 00:00:00-04'),
            time_stuff_p20240410 FOR VALUES FROM ('2024-04-10 00:00:00-04') TO ('2024-04-17 00:00:00-04'),
            time_stuff_p20240417 FOR VALUES FROM ('2024-04-17 00:00:00-04') TO ('2024-04-24 00:00:00-04'),
            time_stuff_p20240424 FOR VALUES FROM ('2024-04-24 00:00:00-04') TO ('2024-05-01 00:00:00-04'),
            time_stuff_p20240501 FOR VALUES FROM ('2024-05-01 00:00:00-04') TO ('2024-05-08 00:00:00-04'),
            time_stuff_p20240508 FOR VALUES FROM ('2024-05-08 00:00:00-04') TO ('2024-05-15 00:00:00-04'),
            time_stuff_p20240515 FOR VALUES FROM ('2024-05-15 00:00:00-04') TO ('2024-05-22 00:00:00-04'),
            time_stuff_p20240522 FOR VALUES FROM ('2024-05-22 00:00:00-04') TO ('2024-05-29 00:00:00-04'),
            time_stuff_default DEFAULT

I ran these statements on Wednesday, April 24, 2024 so you can see the partition time_stuff_p20240424 with the minimum value of that same day. And each subsequent child table is 7 days later, starting on every Wednesday. So while we’ve accomplished our weekly partitioning goal, this is not a common day to start the week. There is thankfully a very easy solution with pg_partman: we tell it the date to start making partitions. Say we wanted our weeks to start on Sunday. Just pick any Sunday date that would work for child tables we’d like to have initially created

SELECT partman.create_parent('public.time_stuff', 'created_at', '1 week', p_start_partition => '2024-04-17');
 create_parent
---------------
 t
(1 row)

\d+ time_stuff
                                                      Partitioned table "public.time_stuff"
   Column   |           Type           | Collation | Nullable |           Default            | Storage | Compression | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------+---------+-------------+--------------+-------------
 id         | integer                  |           | not null | generated always as identity | plain   |             |              |
 created_at | timestamp with time zone |           | not null |                              | plain   |             |              |
Partition key: RANGE (created_at)
Partitions: time_stuff_p20240417 FOR VALUES FROM ('2024-04-17 00:00:00-04') TO ('2024-04-24 00:00:00-04'),
            time_stuff_p20240424 FOR VALUES FROM ('2024-04-24 00:00:00-04') TO ('2024-05-01 00:00:00-04'),
            time_stuff_p20240501 FOR VALUES FROM ('2024-05-01 00:00:00-04') TO ('2024-05-08 00:00:00-04'),
            time_stuff_p20240508 FOR VALUES FROM ('2024-05-08 00:00:00-04') TO ('2024-05-15 00:00:00-04'),
            time_stuff_p20240515 FOR VALUES FROM ('2024-05-15 00:00:00-04') TO ('2024-05-22 00:00:00-04'),
            time_stuff_p20240522 FOR VALUES FROM ('2024-05-22 00:00:00-04') TO ('2024-05-29 00:00:00-04'),
            time_stuff_default DEFAULT

We don’t have the partitions prior to our starting date created, but you can just pick an even earlier Sunday if you need more older partitions to start with. You can see that April 17, 2024 is a Sunday and every subsequent child table has its lower boundary on a Sunday as well. So using this method you can start on any day of the week you desire. And simply using the day of the lower boundary for the suffix name got rid of the complexities of trying to use week numbers that previously required ISO weeks to solve.

Quarterly partitioning

I’d always liked the idea of quarterly partitioning since it seemed to be a nice balance between larger and smaller partitioning intervals. PostgreSQL does have some limited quarterly timestamp formatting options, but if you go back and look at the partman source code for older versions, you’ll see it was way more complex than I’d expected it to be. And it pretty much locked the quarters into 4 pre-defined month blocks. With version 5.x of pg_partman, I decided to do the same as I did with weekly and simply allow any arbitrary 3 month interval people may want. So while it lost the nicer quarterly suffix pattern (YYYYq#, 2024q2), it’s now much more flexible.

The problem and solution for quarterly is the same as weekly. It’s not quite as bad of a problem in that the child lower boundaries are always rounded to the first of the month, but the quarter will default to start in the month that create_parent() runs. So running in April 2024 results in:

keith=# SELECT partman.create_parent('public.time_stuff', 'created_at', '3 months');
 create_parent
---------------
 t
(1 row)

keith=# \d+ time_stuff
                                                      Partitioned table "public.time_stuff"
   Column   |           Type           | Collation | Nullable |           Default            | Storage | Compression | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------+---------+-------------+--------------+-------------
 id         | integer                  |           | not null | generated always as identity | plain   |             |              |
 created_at | timestamp with time zone |           | not null |                              | plain   |             |              |
Partition key: RANGE (created_at)
Partitions: time_stuff_p20230401 FOR VALUES FROM ('2023-04-01 00:00:00-04') TO ('2023-07-01 00:00:00-04'),
            time_stuff_p20230701 FOR VALUES FROM ('2023-07-01 00:00:00-04') TO ('2023-10-01 00:00:00-04'),
            time_stuff_p20231001 FOR VALUES FROM ('2023-10-01 00:00:00-04') TO ('2024-01-01 00:00:00-05'),
            time_stuff_p20240101 FOR VALUES FROM ('2024-01-01 00:00:00-05') TO ('2024-04-01 00:00:00-04'),
            time_stuff_p20240401 FOR VALUES FROM ('2024-04-01 00:00:00-04') TO ('2024-07-01 00:00:00-04'),
            time_stuff_p20240701 FOR VALUES FROM ('2024-07-01 00:00:00-04') TO ('2024-10-01 00:00:00-04'),
            time_stuff_p20241001 FOR VALUES FROM ('2024-10-01 00:00:00-04') TO ('2025-01-01 00:00:00-05'),
            time_stuff_p20250101 FOR VALUES FROM ('2025-01-01 00:00:00-05') TO ('2025-04-01 00:00:00-04'),
            time_stuff_p20250401 FOR VALUES FROM ('2025-04-01 00:00:00-04') TO ('2025-07-01 00:00:00-04'),
            time_stuff_default DEFAULT

This does start on the quarterly months people typically expect but only by coincidence. To start your quarters in whichever month you’d like, simply set the starting partition as we did with weekly. The day doesn’t really matter, just the month.

SELECT partman.create_parent('public.time_stuff', 'created_at', '3 months', p_start_partition => '2024-03-15');
 create_parent
---------------
 t
(1 row)

\d+ time_stuff
                                                      Partitioned table "public.time_stuff"
   Column   |           Type           | Collation | Nullable |           Default            | Storage | Compression | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------+---------+-------------+--------------+-------------
 id         | integer                  |           | not null | generated always as identity | plain   |             |              |
 created_at | timestamp with time zone |           | not null |                              | plain   |             |              |
Partition key: RANGE (created_at)
Partitions: time_stuff_p20240301 FOR VALUES FROM ('2024-03-01 00:00:00-05') TO ('2024-06-01 00:00:00-04'),
            time_stuff_p20240601 FOR VALUES FROM ('2024-06-01 00:00:00-04') TO ('2024-09-01 00:00:00-04'),
            time_stuff_p20240901 FOR VALUES FROM ('2024-09-01 00:00:00-04') TO ('2024-12-01 00:00:00-05'),
            time_stuff_p20241201 FOR VALUES FROM ('2024-12-01 00:00:00-05') TO ('2025-03-01 00:00:00-05'),
            time_stuff_p20250301 FOR VALUES FROM ('2025-03-01 00:00:00-05') TO ('2025-06-01 00:00:00-04'),
            time_stuff_default DEFAULT

Any Arbitrary Interval

While solving for these two custom intervals isn’t too complicated, I did run into issues with allowing any arbitrary custom interval before 5.0. The issue is with how partman rounds the intervals to give the normally expected lower boundaries depending on the length of the interval: daily rounds to midnight, monthly rounds to the first of the month, etc. What if we wanted to partition by 9 week intervals and we wanted it to start on Mondays beginning with April 22, 2024?

keith=# SELECT partman.create_parent('public.time_stuff', 'created_at', '9 weeks', p_start_partition => '2024-04-22');
 create_parent
---------------
 t
(1 row)

keith=# \d+ time_stuff
                                                      Partitioned table "public.time_stuff"
   Column   |           Type           | Collation | Nullable |           Default            | Storage | Compression | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------+---------+-------------+--------------+-------------
 id         | integer                  |           | not null | generated always as identity | plain   |             |              |
 created_at | timestamp with time zone |           | not null |                              | plain   |             |              |
Partition key: RANGE (created_at)
Partitions: time_stuff_p20240401 FOR VALUES FROM ('2024-04-01 00:00:00-04') TO ('2024-06-03 00:00:00-04'),
            time_stuff_p20240603 FOR VALUES FROM ('2024-06-03 00:00:00-04') TO ('2024-08-05 00:00:00-04'),
            time_stuff_p20240805 FOR VALUES FROM ('2024-08-05 00:00:00-04') TO ('2024-10-07 00:00:00-04'),
            time_stuff_p20241007 FOR VALUES FROM ('2024-10-07 00:00:00-04') TO ('2024-12-09 00:00:00-05'),
            time_stuff_p20241209 FOR VALUES FROM ('2024-12-09 00:00:00-05') TO ('2025-02-10 00:00:00-05'),
            time_stuff_default DEFAULT

That doesn’t look right! The issue here is that since the interval is less than 1 year but greater than or equal to 1 month, partman always tries to round to the first day of the month. What we want partman to do is round to the nearest week instead since our interval is based on an arbitrary amount of weeks. As I said, this was an issue before 5.0 but fixed only fairly recently thanks to a bug report from a user. This was solved in 4.6.0 by adding another option to create_parent().

keith=# SELECT partman.create_parent('public.time_stuff', 'created_at', '9 weeks', p_start_partition => '2024-04-22', p_date_trunc_interval => 'week');
 create_parent
---------------
 t
(1 row)

keith=# \d+ time_stuff
                                                      Partitioned table "public.time_stuff"
   Column   |           Type           | Collation | Nullable |           Default            | Storage | Compression | Stats target | Description
------------+--------------------------+-----------+----------+------------------------------+---------+-------------+--------------+-------------
 id         | integer                  |           | not null | generated always as identity | plain   |             |              |
 created_at | timestamp with time zone |           | not null |                              | plain   |             |              |
Partition key: RANGE (created_at)
Partitions: time_stuff_p20240422 FOR VALUES FROM ('2024-04-22 00:00:00-04') TO ('2024-06-24 00:00:00-04'),
            time_stuff_p20240624 FOR VALUES FROM ('2024-06-24 00:00:00-04') TO ('2024-08-26 00:00:00-04'),
            time_stuff_p20240826 FOR VALUES FROM ('2024-08-26 00:00:00-04') TO ('2024-10-28 00:00:00-04'),
            time_stuff_p20241028 FOR VALUES FROM ('2024-10-28 00:00:00-04') TO ('2024-12-30 00:00:00-05'),
            time_stuff_p20241230 FOR VALUES FROM ('2024-12-30 00:00:00-05') TO ('2025-03-03 00:00:00-05'),
            time_stuff_default DEFAULT

The p_date_trunc_interval parameter takes values that are valid for the PostgreSQL built-in function date_trunc. This tells partman how to round the boundaries to get the values you’re more likely expecting. One unfortunate thing that is unique for the weekly option here is that date_trunc('week', <timetamptz>) always rounds to a Monday. So in this case you wouldn’t be able to have an arbitrary amount of weeks that start on a Sunday or any other day of the week.

If you cannot use more common partition intervals (daily, monthly, etc), you’ll likely have to experiment with this feature to see if it allows you do do what you need. I would personally recommend trying to stick with more common intervals if at all possible, but business requirements sometimes require the uncommon.

Conclusion

pg_partman provides comprehensive support of time based intervals to serve a wide variety of needs. Hopefully this blog post has helped to show both basic and advanced features and how to handle some more complex partitioning requirements. ]]> 097174af368a889d3f27f1e3261b63d76ac8028e74061883199a5bbb0a2063b1 Thu, 09 May 2024 13:00:00 EDT 2024-05-09T17:00:00.000Z 2024-05-09T17:00:00.000Z <![CDATA[ Auto-archiving and Data Retention Management in Postgres with pg_partman ]]> https://www.crunchydata.com/blog/auto-archiving-and-data-retention-management-in-postgres-with-pg_partman You could be saving money every month on databases costs with a smarter data retention policy. One of the primary reasons, and a huge benefit of partitioning is using it to automatically archive your data. For example, you might have a huge log table. For business purposes, you need to keep this data for 30 days. This table grows continually over time and keeping all the data makes database maintenance challenging. With time-based partitioning, you can simply archive off data older than 30 days.

The nature of most relational databases means that deleting large volumes of data can be very inefficient and that space is not immediately, if ever, returned to the file system. PostgreSQL does not return the space it reserves to the file system when normal deletion operations are run except under very specific conditions:

  1. the page(s) at the end of the relation are completely emptied
  2. a VACUUM FULL/CLUSTER is run against the relation (exclusively locking it until complete)

If you find yourself needing that space back more immediately, or without intrusive locking, then partitioning can provide a much simpler means of removing old data: drop the table. The removal is nearly instantaneous (barring any transactions locking the table) and immediately returns the space to the file system. pg_partman, the Postgres extension for partitioning, provides a very easy way to manage this for time and integer based partitioning.

pg_partman daily partition example

Recently pg_partman 5.1 was released that includes new features such as list partitioning for single value integers, controlled maintenance run ordering, and experimental support for numeric partitioning. This new version also includes several bug fixes, so please update to the latest release when possible! All examples were done using this latest version.

https://github.com/pgpartman/pg_partman

First lets get a simple, time-based daily partition set going

CREATE TABLE public.time_stuff
    (col1 int
        , col2 text default 'stuff'
        , col3 timestamptz NOT NULL DEFAULT now() )
    PARTITION BY RANGE (col3);

SELECT partman.create_parent('public.time_stuff', 'col3', '1 day');

\d+ time_stuff
                                             Partitioned table "public.time_stuff"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Partitions: time_stuff_p20240408 FOR VALUES FROM ('2024-04-08 00:00:00-04') TO ('2024-04-09 00:00:00-04'),
            time_stuff_p20240409 FOR VALUES FROM ('2024-04-09 00:00:00-04') TO ('2024-04-10 00:00:00-04'),
            time_stuff_p20240410 FOR VALUES FROM ('2024-04-10 00:00:00-04') TO ('2024-04-11 00:00:00-04'),
            time_stuff_p20240411 FOR VALUES FROM ('2024-04-11 00:00:00-04') TO ('2024-04-12 00:00:00-04'),
            time_stuff_p20240412 FOR VALUES FROM ('2024-04-12 00:00:00-04') TO ('2024-04-13 00:00:00-04'),
            time_stuff_p20240413 FOR VALUES FROM ('2024-04-13 00:00:00-04') TO ('2024-04-14 00:00:00-04'),
            time_stuff_p20240414 FOR VALUES FROM ('2024-04-14 00:00:00-04') TO ('2024-04-15 00:00:00-04'),
            time_stuff_p20240415 FOR VALUES FROM ('2024-04-15 00:00:00-04') TO ('2024-04-16 00:00:00-04'),
            time_stuff_p20240416 FOR VALUES FROM ('2024-04-16 00:00:00-04') TO ('2024-04-17 00:00:00-04'),
            time_stuff_default DEFAULT

Setting data retention policies

This partition set was created on April 12, 2024, so a default setup will create 4 partitions before and 4 partitions after. The first setting to configure for retention, and the only one that is required, is the retention column in the part_config table. For this example, we’ll set a retention of 2 days. We’re also going to increase the premake value just to see that normal maintenance is working as well.

UPDATE partman.part_config SET retention = '2 days', premake = 6 WHERE parent_table = 'public.time_stuff';

By default, pg_partman also does not create new child tables if there is no data in the partition set, so lets add some data in as well.

INSERT INTO public.time_stuff (col1, col3)
VALUES (generate_series(1,10), CURRENT_TIMESTAMP);

SELECT * FROM partman.part_config
WHERE parent_table = 'public.time_stuff';

-[ RECORD 1 ]--------------+-----------------------------------
parent_table               | public.time_stuff
control                    | col3
partition_interval         | 1 day
partition_type             | range
premake                    | 6
automatic_maintenance      | on
template_table             | partman.template_public_time_stuff
retention                  | 2 days
retention_schema           |
retention_keep_index       | t
retention_keep_table       | t
epoch                      | none
constraint_cols            |
optimize_constraint        | 30
infinite_time_partitions   | f
datetime_string            | YYYYMMDD
jobmon                     | t
sub_partition_set_full     | f
undo_in_progress           | f
inherit_privileges         | f
constraint_valid           | t
ignore_default_data        | t
default_table              | t
date_trunc_interval        |
maintenance_order          |
retention_keep_publication | f
maintenance_last_run       |

In pg_partman, retention management is handled at the same time as new partition creation. So a simple call to run_maintenance_proc() will handle both.

CALL partman.run_maintenance_proc();

\d+ time_stuff
                                             Partitioned table "public.time_stuff"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Partitions: time_stuff_p20240410 FOR VALUES FROM ('2024-04-10 00:00:00-04') TO ('2024-04-11 00:00:00-04'),
            time_stuff_p20240411 FOR VALUES FROM ('2024-04-11 00:00:00-04') TO ('2024-04-12 00:00:00-04'),
            time_stuff_p20240412 FOR VALUES FROM ('2024-04-12 00:00:00-04') TO ('2024-04-13 00:00:00-04'),
            time_stuff_p20240413 FOR VALUES FROM ('2024-04-13 00:00:00-04') TO ('2024-04-14 00:00:00-04'),
            time_stuff_p20240414 FOR VALUES FROM ('2024-04-14 00:00:00-04') TO ('2024-04-15 00:00:00-04'),
            time_stuff_p20240415 FOR VALUES FROM ('2024-04-15 00:00:00-04') TO ('2024-04-16 00:00:00-04'),
            time_stuff_p20240416 FOR VALUES FROM ('2024-04-16 00:00:00-04') TO ('2024-04-17 00:00:00-04'),
            time_stuff_p20240417 FOR VALUES FROM ('2024-04-17 00:00:00-04') TO ('2024-04-18 00:00:00-04'),
            time_stuff_p20240418 FOR VALUES FROM ('2024-04-18 00:00:00-04') TO ('2024-04-19 00:00:00-04'),
            time_stuff_default DEFAULT

Now you can see the two partitions older than 2 days ago have been removed and two new partitions have been created to include 6 days ahead. There are some other more advanced options for retention available in pg_partman as well. You’ll see above that the retention_keep_table option is set to true by default. This means that while the child tables are no longer part of the retention set, those tables do still exist in the database. pg_partman tries to keep all default options set in a manner to reduce accidental data loss.

\dt public.time_stuff*
                     List of relations
 Schema |         Name         |       Type        | Owner
--------+----------------------+-------------------+-------
 public | time_stuff           | partitioned table | keith
 public | time_stuff_default   | table             | keith
 public | time_stuff_p20240408 | table             | keith
 public | time_stuff_p20240409 | table             | keith
 public | time_stuff_p20240410 | table             | keith
 public | time_stuff_p20240411 | table             | keith
 public | time_stuff_p20240412 | table             | keith
 public | time_stuff_p20240413 | table             | keith
 public | time_stuff_p20240414 | table             | keith
 public | time_stuff_p20240415 | table             | keith
 public | time_stuff_p20240416 | table             | keith
 public | time_stuff_p20240417 | table             | keith
 public | time_stuff_p20240418 | table             | keith

Dropping tables and indexes

If you’d like these tables to actually be dropped, you can set the retention_keep_table to false. Or if you’d like to keep the tables live in the database, but don’t need the indexes taking up space anymore, you can leave retention_keep_table set to true, but set retention_keep_index false instead. In the example below, I have reset the partition set back to its original state after running create_parent() and then running this update.

UPDATE partman.part_config
SET retention = '2 days', premake = 6, retention_keep_table = false
WHERE parent_table = 'public.time_stuff';

CALL partman.run_maintenance_proc();

Now if we look at the tables that actually exist, we can see the oldest two tables are gone.

\dt public.time*
                     List of relations
 Schema |         Name         |       Type        | Owner
--------+----------------------+-------------------+-------
 public | time_stuff           | partitioned table | keith
 public | time_stuff_default   | table             | keith
 public | time_stuff_p20240410 | table             | keith
 public | time_stuff_p20240411 | table             | keith
 public | time_stuff_p20240412 | table             | keith
 public | time_stuff_p20240413 | table             | keith
 public | time_stuff_p20240414 | table             | keith
 public | time_stuff_p20240415 | table             | keith
 public | time_stuff_p20240416 | table             | keith
 public | time_stuff_p20240417 | table             | keith
 public | time_stuff_p20240418 | table             | keith

Retention outside the database

Another scenario is if you don’t need the data live in the database, but you still want to keep a backup of it outside of the database. In this case, we’re going to use the retention_schema option which detaches the child tables from the partition set and then moves them to the schema named in this option. Again, the partition set has been reset to the initial state after create_parent() and then we run this:

CREATE SCHEMA old_tables;

UPDATE partman.part_config
SET retention = '2 days', retention_schema = 'old_tables'
WHERE parent_table = 'public.time_stuff';

CALL partman.run_maintenance_proc();

Now we can see that the old tables are no longer in the partition set, but are now in the old_tables schema.

\d+ time_stuff
                                             Partitioned table "public.time_stuff"
 Column |           Type           | Collation | Nullable |    Default    | Storage  | Compression | Stats target | Description
--------+--------------------------+-----------+----------+---------------+----------+-------------+--------------+-------------
 col1   | integer                  |           |          |               | plain    |             |              |
 col2   | text                     |           |          | 'stuff'::text | extended |             |              |
 col3   | timestamp with time zone |           | not null | now()         | plain    |             |              |
Partition key: RANGE (col3)
Partitions: time_stuff_p20240410 FOR VALUES FROM ('2024-04-10 00:00:00-04') TO ('2024-04-11 00:00:00-04'),
            time_stuff_p20240411 FOR VALUES FROM ('2024-04-11 00:00:00-04') TO ('2024-04-12 00:00:00-04'),
            time_stuff_p20240412 FOR VALUES FROM ('2024-04-12 00:00:00-04') TO ('2024-04-13 00:00:00-04'),
            time_stuff_p20240413 FOR VALUES FROM ('2024-04-13 00:00:00-04') TO ('2024-04-14 00:00:00-04'),
            time_stuff_p20240414 FOR VALUES FROM ('2024-04-14 00:00:00-04') TO ('2024-04-15 00:00:00-04'),
            time_stuff_p20240415 FOR VALUES FROM ('2024-04-15 00:00:00-04') TO ('2024-04-16 00:00:00-04'),
            time_stuff_p20240416 FOR VALUES FROM ('2024-04-16 00:00:00-04') TO ('2024-04-17 00:00:00-04'),
            time_stuff_default DEFAULT

\dt old_tables.*
                 List of relations
   Schema   |         Name         | Type  | Owner
------------+----------------------+-------+-------
 old_tables | time_stuff_p20240408 | table | keith
 old_tables | time_stuff_p20240409 | table | keith

To store these tables “offline” outside of the database, we can use a python script provided by pg_partman to dump all tables in a given schema. It’s not tied in any way to the partition configuration or the partition set, so this script can be used to dump any tables in any schema.

$ python3 dump_partition.py -c"host=localhost" --schema=old_tables
DROP TABLE IF EXISTS"old_tables"."time_stuff_p20240409"
DROP TABLE IF EXISTS"old_tables"."time_stuff_p20240408

$ ls -l old*
-rw-rw-r-- 1 keith keith  168 Apr 12 18:17 old_tables.time_stuff_p20240408.hash
-rw-rw-r-- 1 keith keith 1410 Apr 12 18:17 old_tables.time_stuff_p20240408.pgdump
-rw-rw-r-- 1 keith keith  168 Apr 12 18:17 old_tables.time_stuff_p20240409.hash
-rw-rw-r-- 1 keith keith 1410 Apr 12 18:17 old_tables.time_stuff_p20240409.pgdump

By default it creates dump files in the custom dump format as well as providing a SHA-512 hash of the dump file to provide long-term data integrity checks. This backup option can either be run as part of a regularly scheduled script or as a one off backup.

Summary

Keeping data that doesn’t need to actually exist inside the database is a key part of keeping it running efficiently. Hopefully this has provided a guide to using both basic and advanced retention management options available in pg_partman. ]]> 05b05bdd5553484106957495fd50b99eefe32ddb0fb78c9f66976c65501e8e00 Fri, 19 Apr 2024 09:00:00 EDT 2024-04-19T13:00:00.000Z 2024-04-19T13:00:00.000Z <![CDATA[ Five Great Features of the PostgreSQL Partition Manager ]]> https://www.crunchydata.com/blog/five-great-features-of-postgres-partition-manager After much testing and work the PostgreSQL Partition Manager, pg_partman, version 5 is now available for public release. Thanks to everyone involved for helping me get here!

My recent post discusses many of the big changes, so please see that post or the CHANGELOG for a full summary of version 5.

What I'd like to do today is take a step back and review five notable features that make pg_partman an important tool for managing large tables in PostgreSQL:

  • Retention
  • Background Worker
  • Additional Constraint Exclusion
  • Epoch Partitioning
  • Template Table

Retention

One of the primary reasons to partition tables in PostgreSQL is to avoid the potentially very heavy overhead associated with deleting many rows in large batches. PostgreSQL's MVCC system means that when you delete rows, they're not actually gone until the VACUUM system comes through to mark them as reusable space for new rows. Even then that does not immediately (and may likely never) return the space to the file system. Vacuuming such large volumes of old rows can put a heavy strain on an already stressed system, which is what you may have been trying to alleviate by removing those rows.

Instead of running delete statements to remove your old data, partitioning let's you slice the table up and then simply drop the old partitions that have data you don't need in the database anymore. This avoids the need to vacuum to clean up old rows and, more importantly, immediately returns that space to the file system with very little overhead. Another thing to note here is that if retention is your reason for partitioning, you don't have to make your partition interval very small. It just has to be enough to cover your retention policy for how much data you need to keep. If your policy is 30 days, you can try setting the partition interval to monthly. That doesn't necessarily mean you'll only have one month of data around (Ex. February). But it does meet your retention policy of at least 30 days, so you might just be keeping 2 months of data around to meet that and the 3rd oldest table gets dropped. It's not exact, but it meets your needs and can vastly reduce the resource requirements of keeping years of data in a single table.

To configure this in partman, simply set the retention column in the part_config table to either an interval or integer value. For time-based partitioning, this is as easy as setting an expected interval value: 30 days, 3 weeks, 1 year, etc. If a given older partition contains ONLY data that is older than that interval, then it will be dropped.

UPDATE partman.part_config SET retention = '30 days' WHERE parent_table = 'public.testing_plans';

Numeric-based partitioning is not quite as straight forward but a simple example hopefully helps: the integer value will set that any partitions with a value less than the current maximum value minus the retention value will be dropped. For example, if the current max value is 100 and the retention value is 30, any partitions with values less than 70 will be dropped. The current maximum value at the time the drop function is run is always used.

UPDATE partman.part_config SET retention = '10000', retention_keep_table = false WHERE parent_table = 'public.testing_plans';

Some other features involving retention available in partman are that you can configure the old tables to either be detached (the default) or actually dropped, which the above example shows. The default is to detach (retention_keep_table = true) to help avoid losing data accidentally. Many of partman's features that involve any sort of data removal or rejection typically default to a method that does not lose the data as a safety precaution. We'd like you to have to make extra effort to tell partman that yes, I actually want to ignore/remove data. You can also tell partman to move the old tables to a different schema as part of the retention maintenance option. And lastly, there is a Python script available to dump out any tables in a given schema to compressed, checksummed files using pg_dump. This, combined with the schema migration feature, gives you the option of storing old tables out-of-bounds of the database but still available in a much-reduced size to be restored if needed.

UPDATE partman.part_config SET retention = '10 days', retention_schema = 'plan_archives' WHERE parent_table = 'public.testing_plans';

python3 dump_partition.py --connection="host=localhost user=admin dbname=testingdb" --schema="plan_archives" --output="/opt/testing/archive_dumps/092023_testing_plans_archive.pgr.tar.gz" --dump_database="testingdb" --dump_host="localhost" --dump_username="admin"

Example output using the table schema given below in the Epoch Partitioning section given the above retention on Sept 27, 2023.

$ pwd
/opt/testing/archive_dumps

$ ls -l
total 48
-rw-rw-r-- 1 keith keith  174 Sep 27 11:04 plan_archives.testing_plans_p20230731.hash
-rw-rw-r-- 1 keith keith 1874 Sep 27 11:04 plan_archives.testing_plans_p20230731.pgdump
-rw-rw-r-- 1 keith keith  174 Sep 27 11:04 plan_archives.testing_plans_p20230807.hash
-rw-rw-r-- 1 keith keith 1874 Sep 27 11:04 plan_archives.testing_plans_p20230807.pgdump
-rw-rw-r-- 1 keith keith  174 Sep 27 11:04 plan_archives.testing_plans_p20230814.hash
-rw-rw-r-- 1 keith keith 1874 Sep 27 11:04 plan_archives.testing_plans_p20230814.pgdump
-rw-rw-r-- 1 keith keith  174 Sep 27 11:04 plan_archives.testing_plans_p20230821.hash
-rw-rw-r-- 1 keith keith 1874 Sep 27 11:04 plan_archives.testing_plans_p20230821.pgdump
-rw-rw-r-- 1 keith keith  174 Sep 27 11:04 plan_archives.testing_plans_p20230828.hash
-rw-rw-r-- 1 keith keith 1874 Sep 27 11:04 plan_archives.testing_plans_p20230828.pgdump
-rw-rw-r-- 1 keith keith  174 Sep 27 11:04 plan_archives.testing_plans_p20230904.hash
-rw-rw-r-- 1 keith keith 1874 Sep 27 11:04 plan_archives.testing_plans_p20230904.pgdump

Background Worker

Partitioning schemes based on time or numbers generally are not a once-and-done set up. They need continual work to create new partitions for new data and possibly remove old data (see Retention). For many, this means using some sort of external scheduler like cron to periodically call some function to perform maintenance and add/remove child tables. pg_partman takes advantage of a feature in PostgreSQL called a background worker. These have been around since PostgreSQL 9.3, but many people are still unaware of them. Anyone can write a background worker for PostgreSQL and it will run automatically when the database runs and basically do whatever you tell it to do. In the case of pg_partman, this can be the maintenance process to manage the partition sets.

Enabling and configuring this is very easy. Install pg_partman however your environment requires (package, manual build, etc) and enable its shared library in the postgresql.conf

shared_preload_libraries = 'pg_partman_bgw'

You'll also need to tell it which databases pg_partman is installed and how often you'd like it to run. The default is once per hour (3600 seconds), but can be any valid interval value. You'll want to make sure it's running at least often enough for your smallest partition set's interval, usually at least twice within that interval. So if you're doing daily partitioning at the smallest, 12 hours is usually a good value. If no partitions need to be made when it's called, it won't do anything, but you don't want it needlessly using up max_worker_processes.

pg_partman_bgw.dbname = 'alphadb, betadb'
pg_partman_bgw.interval = '12 hours'

See the documentation for other BGW settings that are available.

Note that the time the actual BGW process runs can drift due to the nature of calling a process on an interval vs at a specific time. If you need partition maintenance to run at specific times, then you will need some sort of scheduler instead of partman's BGW. There is another helpful extension that also makes use of BGWs called pg_cron that can help you with this. Also, you can run the maintenance for all partition sets at once:

SELECT partman.run_maintenance();

Or you can run maintenance for specific partition sets. Note that if you also call the above command but don't want these specific partition sets to run when that is done, you'll need to set the automatic_maintenance flag for those partition sets to false.

UPDATE partman.part_config SET automatic_maintenance = false WHERE parent_table = 'public.testing_plans';

Then set your schedule to call this set specifically

SELECT partman.run_maintenance('public.testing_plans');

Additional Constraint Exclusion

One of the first advanced features I added to pg_partman was the ability to create additional constraints on columns other than the partition control column. Why would you do this? Well, for one of the same reasons you use partitioning in the first place in PostgreSQL: constraint exclusion.

What if you could tell PostgreSQL to ignore all the tables in the partition set that don't have any data that you want as part of your query conditions? Well, that's what partitioning allows PostgreSQL to automatically do. Now, in the case of the partition column itself it uses something more advanced called partition pruning (PG 12+). Normally, a constraint exclusion has to at least visit each child table to see if the constraint covers the requested data range. Partition pruning is even smarter than that and uses the partition definitions instead, but we're not going to get much deeper than that for now. Even so, simply evaluating a constraint could be tremendously more efficient than scanning an index or performing a filter. So how could we take advantage of that on columns other than the partition key?

pg_partman's feature uses the constraint_cols and optimize_constraint columns from the part_config table to set this up. constraint_cols is an array list of one or more columns that you want to consider for constraint exclusion. optimize_constraint is a count of how many partitions back from the "current" one to go to set things up for constraint exclusion using the given columns. So let's say we're partitioning monthly and after 3 months, that data is never changed anymore on the evaluation_date column in our public.testing_plans table. We'd update the part_config table like so

UPDATE partman.part_config SET constraint_cols = ARRAY['evaluation_date'], optimize_constraint = '3' WHERE parent_table = 'public.testing_plans';

What this tells partman to do is to go back and look at the tables before the 3rd oldest one relative to the current and create a constraint based on the existing minimum and maximum values in those given columns, no matter the data type as long as that is a valid aggregation. Now, if you run a query with conditionals on these columns, PostgreSQL may be able to take advantage of constraint exclusion for all your tables older than 3 months and exclude them from query plans. Note this does mean that this data most likely can never change unless the new value keeps it within that constraint. So again, this is a feature mainly for older, static data. However, this could potentially have huge query planning benefits depending on your data patterns.

Epoch Partitioning

Many tools and applications out there need a simpler value to store and evaluate time. What could be simpler than an integer? But what integer? Enter the epoch into computer science. Most are familiar with the UNIX/POSIX epoch of the number of seconds since January 1, 1970. So how can we partition on this value but still consider it a time in all but its literal value?

pg_partman can do this with a simple flag when creating your partition set:

SELECT partman.create_parent(
    'public.testing_plans'
    , 'created_at'
    , '1 week'
    , p_epoch := 'seconds'
    , p_start_partition := to_char(date_trunc('week',CURRENT_TIMESTAMP)-'8 weeks'::interval, 'YYYY-MM-DD HH24:MI:SS')
);

This tells pg_partman that the column itself is allowed to be any integer type, but for all maintenance purposes it is considered a time value. Normally the epoch is the number of seconds, but I've had requests for millisecond and nanosecond support over the years, so that is why the flag column isn't just a simple boolean and can be one of these 3 values. We've also used another handy feature here in partman to tell it that we actually want the "first" partition in this set to be eight weeks in the past from now. So, as of the time of this writing the above would configure a partition set that looks like this. Note that I did run this on a Monday, so the first day of the week here is a Monday. I'll be doing a followup blog post on how to ensure which specific day a given partition interval starts on for things like weekly, quarterly, or any other interval that needs to be told to start on a specific timeframe.

\d+ public.testing_plans
      Column       |           Type           | Collation | Nullable |                                 Default                                  | Storage  | Compression | Stats target | Description
-------------------+--------------------------+-----------+----------+--------------------------------------------------------------------------+----------+-------------+--------------+-------------
 testid            | integer                  |           |          |                                                                          | plain    |             |              |
 plan              | text                     |           |          |                                                                          | extended |             |              |
 created_at        | integer                  |           | not null | EXTRACT(epoch FROM date_trunc('week'::text, CURRENT_TIMESTAMP))::integer | plain    |             |              |
 evaluation_date   | timestamp with time zone |           |          |                                                                          | plain    |             |              |
 evaluation_result | boolean                  |           |          |                                                                          | plain    |             |              |
Partition key: RANGE (created_at)
Partitions: testing_plans_p20230731 FOR VALUES FROM (1690776000) TO (1691380800),
            testing_plans_p20230807 FOR VALUES FROM (1691380800) TO (1691985600),
            testing_plans_p20230814 FOR VALUES FROM (1691985600) TO (1692590400),
            testing_plans_p20230821 FOR VALUES FROM (1692590400) TO (1693195200),
            testing_plans_p20230828 FOR VALUES FROM (1693195200) TO (1693800000),
            testing_plans_p20230904 FOR VALUES FROM (1693800000) TO (1694404800),
            testing_plans_p20230911 FOR VALUES FROM (1694404800) TO (1695009600),
            testing_plans_p20230918 FOR VALUES FROM (1695009600) TO (1695614400),
            testing_plans_p20230925 FOR VALUES FROM (1695614400) TO (1696219200),
            testing_plans_p20231002 FOR VALUES FROM (1696219200) TO (1696824000),
            testing_plans_p20231009 FOR VALUES FROM (1696824000) TO (1697428800),
            testing_plans_p20231016 FOR VALUES FROM (1697428800) TO (1698033600),
            testing_plans_p20231023 FOR VALUES FROM (1698033600) TO (1698638400),
            testing_plans_default DEFAULT

Template Table

The final feature I'll be discussing is actually a feature I hope will eventually disappear! Early on in PG10 and 11, there were many features that had been possible with the old, trigger-based partitioning that hadn't made it to the built-in declarative system (Ex. index and foreign key inheritance). At that time, I implemented a template table system to provide a way to have these features with declarative partitioning: apply the feature to a separate template table and pg_partman would handle applying that feature to the child tables. Since then, many features have been built into declarative partitioning and I've migrated them away from something that is taken from the template table. However, some features still remain on the template table:

  • Unique indexes that do not include the partition key
  • UNLOGGED tables
  • Table properties set with ``ALTER TABLE ... SET``` commands (autovacuum settings, storage properties, etc)

And there are some additional ones that are currently in open issues that I'll be investigating for future versions. However, as I said, I hope these issues are addressed by the core team so that hopefully this feature is no longer required in pg_partman. And who knows, maybe core could potentially make this entire extension obsolete some day and I'd be 100% fine with that!

Conclusion

Hopefully this overview of features in pg_partman shows you the power that the extension system can bring to PostgreSQL. The core database system can sometimes be a slow moving beast and there's nothing wrong with that since that keeps it the stable and reliable database that PostgreSQL is known to be. But thankfully users like us can contribute by making new features available a little sooner and help drive that core development forward! ]]> 26cb77bf8542a212d2a4bcd8b876857fc4002e44dc4eec112bfbb307e53bf251 Fri, 29 Sep 2023 09:00:00 EDT 2023-09-29T13:00:00.000Z 2023-09-29T13:00:00.000Z