<?xml version="1.0" encoding="UTF-8" ?>
<rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" version="2.0"><channel><title>Greg Smith | CrunchyData Blog</title>
<atom:link href="https://www.crunchydata.com/blog/author/greg-smith/rss.xml" rel="self" type="application/rss+xml" />
<link>https://www.crunchydata.com/blog/author/greg-smith</link>
<image><url>https://www.crunchydata.com/build/_assets/default.png-W4XGD4DB.webp</url>
<title>Greg Smith | CrunchyData Blog</title>
<link>https://www.crunchydata.com/blog/author/greg-smith</link>
<width>256</width>
<height>256</height></image>
<description>PostgreSQL experts from Crunchy Data share advice, performance tips, and guides on successfully running PostgreSQL and Kubernetes solutions</description>
<language>en-us</language>
<pubDate>Tue, 19 Nov 2024 09:30:00 EST</pubDate>
<dc:date>2024-11-19T14:30:00.000Z</dc:date>
<dc:language>en-us</dc:language>
<sy:updatePeriod>hourly</sy:updatePeriod>
<sy:updateFrequency>1</sy:updateFrequency>
<item><title><![CDATA[ Loading the World! OpenStreetMap Import In Under 4 Hours ]]></title>
<link>https://www.crunchydata.com/blog/loading-the-world-openstreetmap-import-in-under-4-hours</link>
<description><![CDATA[ Greg has a full OSM load for the entire world running in record time. He digs into turning and recent software and hardware updates that make a full planet run in less than 4 hours. ]]></description>
<content:encoded><![CDATA[ <p>The OpenStreetMap (OSM) database builds almost 750GB of location data from a single file download. <a href=https://www.openstreetmap.org/>OSM</a> notoriously takes a full day to run. A fresh open street map load involves both a massive write process and large index builds. It is a great performance stress-test bulk load for any Postgres system. I use it to stress the latest PostgreSQL versions and state-of-the-art hardware. The stress test validates new tuning tricks and identifies performance regressions.<p>Two years ago, I presented (<a href="https://www.youtube.com/watch?v=BCMnu7xay2Y">video</a> / <a href=https://www.slideshare.net/slideshow/speedrunnin-the-open-street-map-osm2pgsql-loader/254313657>slides</a>) at PostGIS Day on challenges of this workload. In honor of this week’s <a href=https://www.crunchydata.com/community/events/postgis-day-2024>PostGIS Day 2024</a>, I’ve run the same benchmark on Postgres 17 and the very latest hardware. The findings:<ul><li><strong>PostgreSQL</strong> keeps getting better! Core improvements sped up index building in particular.<li>The <strong>osm2pgsql</strong> loader got better too! New takes on indexing speed things up.<li><strong>Hardware</strong> keeps getting better! It has been two years since my last report and the state-of-the-art has advanced.</ul><h2 id=tune-your-instrument><a href=#tune-your-instrument>Tune Your Instrument</a></h2><p>First, we are using bare metal hardware—a server with 128GB RAM—so so let’s tune Postgres for loading and to match that server:<pre><code>max_wal_size = 256GB
shared_buffers = 48GB
effective_cache_size = 64GB
maintenance_work_mem = 20GB
work_mem = 1GB
</code></pre><p>Second, let’s prioritize bulk load. The following settings do not make sense for a live system under read/write load, but they will improve performance for this bulk load scenario:<pre><code class=language-bash>checkpoint_timeout = 60min
synchronous_commit = off
# if you don't have replication:
wal_level = minimal
max_wal_senders = 0
# if you believe my testing these make things
# faster too
fsync = off
autovacuum = off
full_page_writes = off
</code></pre><p>It’s also possible to tweak the <a href=https://www.postgresql.org/docs/current/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-BACKGROUND-WRITER>background writer</a> for the particular case of massive data ingestion, but for bulk loads without concurrency it doesn’t make a large difference.<h2 id=how-postgresql-has-improved><a href=#how-postgresql-has-improved>How PostgreSQL has Improved</a></h2><p>In 2022, testing that year's new AMD AM5 hardware loaded the data in just under 8 hours with Postgres 14. Today the amount of data in the OSM Planet files has grown another 14%. Testing with Postgres 17 still halves the load time, with the biggest drops coming from software improvements in the PG14-16 time-frame.<p><img alt="osm building time"loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/d4f15c2b-2a9f-4d95-f397-00101bc60900/public><p>The benchmark orchestration and metrics framework here is my <a href=https://github.com/gregs1104/pgbench-tools>pgbench-tools</a>. Full hardware details are published to <a href=https://browser.geekbench.com/user/232126>GeekBench</a>.<h3 id=gist-index-building-in-postgresql-15><a href=#gist-index-building-in-postgresql-15>GIST Index Building in PostgreSQL 15</a></h3><p>The biggest PostgreSQL speed gains are from <a href="https://www.youtube.com/watch?v=TG28lRoailE">improvements in the GIST index building code</a>.<p>The new code pre-sorts index pages before merging them, and for large GIST index builds the performance speed-up can be substantial, as <a href=https://osm2pgsql.org/news/2023/01/22/faster-with-postgresql15.html>reported by the author</a> of osm2pgsql.<p>My tests showed going from PostgreSQL 14 to 15 delivered:<ul><li>16% speedup<li>15% size reduction<li>86% GIST index build speedup!</ul><p><img alt="osm index building time"loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/7fd95086-d103-449e-1336-23566d73b500/public><p>There have been further improvements in PostgreSQL 16 and 17 in <a href=https://www.crunchydata.com/blog/real-world-performance-gains-with-postgres-17-btree-bulk-scans>B-Tree index building</a>, but this osm2pgsql benchmark does not really show them. The GIST index time build times wash out the other index builds.<h2 id=how-osm2pgsql-has-improved><a href=#how-osm2pgsql-has-improved>How osm2pgsql has improved</a></h2><p>In Q3 2022, osm2pgsql 1.7 made a technique called the <a href=https://osm2pgsql.org/doc/manual-v1.html#bucket-index-for-slim-mode>Middle Way Node Index ID Shift</a> the new <strong>default</strong>.<p><a href=https://osm2pgsql.org/doc/manual-v1.html#bucket-index-for-slim-mode>Middle Way Node Index ID Shift</a> is a clever design approach that compresses the database's largest index, trading off lookup and update performance for a smaller footprint. It uses a Partial Index to merge nearby values together into less fine grained sections. When an index is used frequently, this would waste too many CPU cycles. Similar to hash bucket collision, partial indexes have to constantly exclude non-matched items. That chews through extra CPU on every read. In addition, because individual blocks hold so many more values, the locking footprint for updates increases proportionately. However, for large but infrequently used indexes like this one, those are satisfactory trade-offs.<p>Applying that improvement dropped my loading times by 37% and plummeted the database size from 1000GB to under 650GB. Total time at the terabyte size had crept upward to near 10 hours. The speed-up drove it back below 6 hours.<p>The osm2pgsql manual shows the details in its <a href=https://osm2pgsql.org/doc/manual-v1.html#update-for-expert-users>Update for Expert Users</a>. I highly recommend that section and its <a href=https://blog.jochentopf.com/2023-07-25-improving-the-middle-of-osm2pgsql.html>Improving the middle</a> blog entry. It's a great study of how PG's skinnable indexing system lets applications optimize for their exact workload.<h2 id=how-hardware-has-improved><a href=#how-hardware-has-improved>How hardware has improved</a></h2><h3 id=ssd-write-speed><a href=#ssd-write-speed>SSD Write Speed</a></h3><p>During data import, the osm2pgsql workload writes heavily at medium <a href=https://www.techtarget.com/searchstorage/definition/queue-depth>queue depths</a> for hours. The best results come from SSDs with oversized <a href=https://www.advantech.com/en/resources/news/maximizing-ssd-performance-with-slc-cache#1>SLC caches</a> that also balance cleanup compaction of that cache. The later CREATE TABLE AS (CTAS) sections of the build reach its peak read/write speeds.<p>I saw 11GB/s from a <a href="https://www.google.com/search?client=safari&#38rls=en&#38q=Crucial+T705+PCIe+5.0&#38ie=UTF-8&#38oe=UTF-8">Crucial T705</a> PCIe 5.0 drive the week (foreshadowing!) I was running that with an Intel i9-14900K:<p><img alt="read write for osm"loading=lazy src=https://imagedelivery.net/lPM0ntuwQfh8VQgJRu0mFg/3a260fb6-37cc-43e3-f9ad-930ffe535200/public><p>osm2pgsql has a tuning parameter named <code>--number-processes</code> that guides how many parallel operations the code tries to spawn.<p>For the server and memory I used in this benchmark, increasing<code>--number-processes</code>from my earlier 2 to 5 worked well. However, be careful: you can easily go too far! Bumping up this parameter increases memory usage too. Going wild on the concurrent work will run you out of memory and put you into the hands of the Linux Out of Memory (OOM) killer.<h3 id=processor-advances><a href=#processor-advances>Processor advances</a></h3><p>Obviously, every year processors get a little better, but they do so in different ways and at different rates.<p>For later 2023 and testing against PostgreSQL 15 and 16, an <strong><a href=https://www.intel.com/content/www/us/en/products/sku/230493/intel-core-i513600k-processor-24m-cache-up-to-5-10-ghz/specifications.html>Intel i7-13600K</a></strong> overtook the earlier <strong>AMD R5 7700X</strong>. There was another small bump in 2024 upgrading to an <strong>i9-14900K</strong>.<p>But this is a demanding regression test workload, and it only took a few weeks of running the OSM workload to trigger the <strong>i9-14900K</strong>’s <a href=https://community.intel.com/t5/Blogs/Tech-Innovation/Client/Intel-Core-13th-and-14th-Gen-Desktop-Instability-Root-Cause/post/1633239>voltage bugs</a> to the point where my damaged CPU could not even finish the test.<p>Thankfully I was able to step away from those issues when <strong>AMD's 9600X</strong> launched. Here's the latest results from PG17 on an AMD 9600X, with the same SK41 2TB drive as I tested in 2022 for my <a href="https://www.youtube.com/watch?v=BCMnu7xay2Y">PostGIS Day talk</a>.<h2 id=my-best-osm-import-results-to-date><a href=#my-best-osm-import-results-to-date>My best OSM import results to date</a></h2><pre><code>2024-10-15 10:03:41  [00] Reading input files done in 7851s (2h 10m 51s).
2024-10-15 10:03:41  [00]   Processed 9335778934 nodes in 490s (8m 10s) - 19053k/s
2024-10-15 10:03:41  [00]   Processed 1044011263 ways in 4301s (1h 11m 41s) - 243k/s
2024-10-15 10:03:41  [00]   Processed 12435485 relations in 3060s (51m 0s) - 4k/s
2024-10-15 10:03:41  [00] Overall memory usage: peak=158292MByte current=157746MByte...
2024-10-15 11:32:13  [00] osm2pgsql took 13162s (3h 39m 22s) overall. f
</code></pre><p>Completed in <strong>less than 4 hours</strong>!<p>PostgreSQL 17 is about 3% better on this benchmark than PostgreSQL 16 when replication is used, thanks to improvements in the WAL infrastructure in PostgreSQL 17.<p>I look forward to following up on this benchmark in more detail, after my scorched Intel system is fully running again! Like the speed of the Postgres ecosystem, the pile of hardware I've benchmarked to death grows every year. ]]></content:encoded>
<category><![CDATA[ Spatial ]]></category>
<category><![CDATA[ Postgres 17 ]]></category>
<author><![CDATA[ Greg.Smith@crunchydata.com (Greg Smith) ]]></author>
<dc:creator><![CDATA[ Greg Smith ]]></dc:creator>
<guid isPermalink="false">9e2c4eee6da51e862d11c5257ce74d187841b9a7deed0a7f62b44557857a70d8</guid>
<pubDate>Tue, 19 Nov 2024 09:30:00 EST</pubDate>
<dc:date>2024-11-19T14:30:00.000Z</dc:date>
<atom:updated>2024-11-19T14:30:00.000Z</atom:updated></item>
<item><title><![CDATA[ PostgreSQL on Linux: Counting Committed Memory ]]></title>
<link>https://www.crunchydata.com/blog/postgresql-on-linux-counting-committed-memory</link>
<description><![CDATA[ By default Linux uses a controversial (for databases) memory extension feature called Overcommit. How that interacts with PostgreSQL is covered in the Managing Kernel Resources section of the PG manual. ]]></description>
<content:encoded><![CDATA[ <p>By default Linux uses a controversial (for databases) memory extension feature called <a href=https://www.kernel.org/doc/Documentation/vm/overcommit-accounting>overcommit</a>. How that interacts with PostgreSQL is covered in the <a href=https://www.postgresql.org/docs/current/kernel-resources.html>Managing Kernel Resources</a> section of the PG manual.<p>Overcommit allows clients to pre-allocate virtual memory beyond even server RAM. They are only nailed down to a real allocation, <strong>committed</strong> to use its terminology, when it's actually used. This lets applications have a flatter memory model without having to grapple with virtual memory coding. This model improves how effectively swap can work as well.<p>If you upgraded PostgreSQL or increased your server's <code>shared_buffers</code> setting recently, you may find a larger chunk of memory is now listed in Linux's "Committed" section that wasn't noticeable before. Let's walk through enough of this area to interpret the associated system memory metrics.<h2 id=shared-memory-history><a href=#shared-memory-history>Shared memory history</a></h2><p>In PostgreSQL versions up to 9.2, the shared memory block needed to run the server was allocated directly as UNIX System V shared memory. Documentation from that era gave an estimate of memory needed in that block. The <a href=https://www.postgresql.org/docs/9.2/kernel-resources.html>9.2 Kernel Resources</a> has it in Table 17-2 "PostgreSQL Shared Memory Usage".<p>Starting in PostgreSQL 9.3, <em>"PostgreSQL normally allocates a very small amount of System V shared memory, as well as a much larger amount of POSIX (mmap) shared memory"</em>, quoting the <a href=https://www.postgresql.org/docs/10/kernel-resources.html>10.0 Kernel Resources</a>. The system then commits the <code>shared_buffers</code> memory to pin them down and initialize. That's why the shared/committed balance of newer Postgres servers will look very different from older versions. The memory use formula numbers were made largely obsolete by this change, and that table was impossible to maintain well in the documentation anyway. That's why the level of detail was reduced when switching to the new <em>mmap</em> allocation style.<h2 id=pg10-example><a href=#pg10-example>PG10 example</a></h2><p>This example uses the PostgreSQL 10 included with Ubuntu 18.04; you can use any Linux distribution albeit with different service control scripts. Start with the server down (more on the right syntax below) and look at the memory use:<pre><code class=language-shell>$ service postgresql stop
$ cat /proc/meminfo | grep Commit
CommitLimit:    10252072 kB
Committed_AS:     806928 kB
</code></pre><p>On this 16GB RAM server, that gives <code>CommitLimit</code>=10252072kB <mjx-container class=MathJax jax=SVG><svg focusable=false height=0.968ex role=img style=vertical-align:.124ex viewBox="0 -483 778 428"width=1.76ex xmlns=http://www.w3.org/2000/svg xmlns:xlink=http://www.w3.org/1999/xlink><defs><path d="M55 319Q55 360 72 393T114 444T163 472T205 482Q207 482 213 482T223 483Q262 483 296 468T393 413L443 381Q502 346 553 346Q609 346 649 375T694 454Q694 465 698 474T708 483Q722 483 722 452Q722 386 675 338T555 289Q514 289 468 310T388 357T308 404T224 426Q164 426 125 393T83 318Q81 289 69 289Q55 289 55 319ZM55 85Q55 126 72 159T114 210T163 238T205 248Q207 248 213 248T223 249Q262 249 296 234T393 179L443 147Q502 112 553 112Q609 112 649 141T694 220Q694 249 708 249T722 217Q722 153 675 104T555 55Q514 55 468 76T388 123T308 170T224 192Q164 192 125 159T83 84Q80 55 69 55Q55 55 55 85Z"id=MJX-1-TEX-N-2248 /></defs><g fill=currentColor stroke=currentColor stroke-width=0 transform=scale(1,-1)><g data-mml-node=math><g data-mml-node=mo><use data-c=2248 xlink:href=#MJX-1-TEX-N-2248 /></g></g></g></svg></mjx-container> <em>10GB</em>. Currently locked down, committed RAM <code>Committed_AS</code>=806928kB <mjx-container class=MathJax jax=SVG><svg focusable=false height=0.968ex role=img style=vertical-align:.124ex viewBox="0 -483 778 428"width=1.76ex xmlns=http://www.w3.org/2000/svg xmlns:xlink=http://www.w3.org/1999/xlink><defs><path d="M55 319Q55 360 72 393T114 444T163 472T205 482Q207 482 213 482T223 483Q262 483 296 468T393 413L443 381Q502 346 553 346Q609 346 649 375T694 454Q694 465 698 474T708 483Q722 483 722 452Q722 386 675 338T555 289Q514 289 468 310T388 357T308 404T224 426Q164 426 125 393T83 318Q81 289 69 289Q55 289 55 319ZM55 85Q55 126 72 159T114 210T163 238T205 248Q207 248 213 248T223 249Q262 249 296 234T393 179L443 147Q502 112 553 112Q609 112 649 141T694 220Q694 249 708 249T722 217Q722 153 675 104T555 55Q514 55 468 76T388 123T308 170T224 192Q164 192 125 159T83 84Q80 55 69 55Q55 55 55 85Z"id=MJX-2-TEX-N-2248 /></defs><g fill=currentColor stroke=currentColor stroke-width=0 transform=scale(1,-1)><g data-mml-node=math><g data-mml-node=mo><use data-c=2248 xlink:href=#MJX-2-TEX-N-2248 /></g></g></g></svg></mjx-container> <em>800MB</em>. This is memory dedicated to the core Linux operating system and its utilities. You might conclude that this OS as configured requires at least 1GB to run at all, which is accurate.<p>On this server, starting the database correctly means I have to drop back to my user account to use <code>sudo</code>. You can easily give those powers to the <code>postgres</code> Linux account instead, it's just not necessary on my test system. The proper <em>systemd</em> call to stop and start the database on this server uses <code>systemctl</code>. Here are some alternate forms of startup lines you might need to use instead:<pre><code class=language-shell>gsmith@hydra:~$ sudo systemctl start postgresql@10-main
postgres@hydra:~$ service postgresql start
postgres@hydra:~$ pg_ctlcluster 10 main start # Debian/Ubuntu, PG10
</code></pre><p>Confirm the database just restarted:<pre><code class=language-shell>postgres@hydra:~$ ps -eaf | grep postgres
postgres  8022     1  0 06:40 ?        00:00:00 /usr/lib/postgresql/10/bin/postgres -D /var/lib/postgresql/10/main -c config_file=/etc/postgresql/10/main/postgresql.conf
postgres  8024  8022  0 06:40 ?        00:00:00 postgres: 10/main: checkpointer process
postgres  8025  8022  0 06:40 ?        00:00:00 postgres: 10/main: writer process
postgres  8026  8022  0 06:40 ?        00:00:00 postgres: 10/main: wal writer process
postgres  8027  8022  0 06:40 ?        00:00:00 postgres: 10/main: autovacuum launcher process
postgres  8028  8022  0 06:40 ?        00:00:00 postgres: 10/main: stats collector process
postgres  8029  8022  0 06:40 ?        00:00:00 postgres: 10/main: bgworker: logical replication launcher
postgres@hydra:~$ date
Sat May  1 06:42:45 EDT 2021
</code></pre><p>And check the biggest user of committed memory, <code>shared_buffers</code>:<pre><code class=language-shell>postgres@hydra:~$ psql -c "show shared_buffers"
 shared_buffers
----------------
 4GB
</code></pre><p>Now let's look at memory again:<pre><code class=language-shell>postgres@hydra:~$  cat /proc/meminfo | grep Commit
CommitLimit:    10252072 kB
Committed_AS:    5115160 kB
</code></pre><p><code>Committed_AS</code> jumped to 5115160 kB<mjx-container class=MathJax jax=SVG><svg focusable=false height=1.505ex role=img style=vertical-align:-.186ex viewBox="0 -583 778 665"width=1.76ex xmlns=http://www.w3.org/2000/svg xmlns:xlink=http://www.w3.org/1999/xlink><defs><path d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"id=MJX-3-TEX-N-3D /></defs><g fill=currentColor stroke=currentColor stroke-width=0 transform=scale(1,-1)><g data-mml-node=math><g data-mml-node=mo><use data-c=3D xlink:href=#MJX-3-TEX-N-3D /></g></g></g></svg></mjx-container><em>4.9GB</em>. Since it was 800MB before, that means the database server committed a new 4308232kb<mjx-container class=MathJax jax=SVG><svg focusable=false height=1.505ex role=img style=vertical-align:-.186ex viewBox="0 -583 778 665"width=1.76ex xmlns=http://www.w3.org/2000/svg xmlns:xlink=http://www.w3.org/1999/xlink><defs><path d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"id=MJX-4-TEX-N-3D /></defs><g fill=currentColor stroke=currentColor stroke-width=0 transform=scale(1,-1)><g data-mml-node=math><g data-mml-node=mo><use data-c=3D xlink:href=#MJX-4-TEX-N-3D /></g></g></g></svg></mjx-container><em>4.1GB</em> on startup. That's the shared memory block, which includes <code>shared_buffers</code> plus some overhead for clients and other shared state.<h2 id=digging-into-the-memory><a href=#digging-into-the-memory>Digging into the memory</a></h2><p>You can see more about where the memory is going when using the <code>pmap</code> utility. While most of the bytes are <code>shared_buffers</code>, the bulk of the text output is linking to various shared libraries. Here's a grep command that screens most of the trivia out:<pre><code class=language-shell>postgres@hydra:~$ pmap -x 8022 | egrep -v "anon|lib|ld|locale"
Address           Kbytes     RSS   Dirty Mode  Mapping
00005637cb721000    7012    3492       0 r-x-- postgres
00005637cb721000       0       0       0 r-x-- postgres
00005637cbffa000     136     136     136 r---- postgres
00005637cbffa000       0       0       0 r---- postgres
00005637cc01c000      52      52      52 rw--- postgres
00005637cc01c000       0       0       0 rw--- postgres
00007f919cb75000 4317408  108240  108240 rw-s- zero (deleted)
00007f919cb75000       0       0       0 rw-s- zero (deleted)
00007f92ae841000       8       4       4 rw-s- PostgreSQL.158420325
00007f92ae841000       0       0       0 rw-s- PostgreSQL.158420325
00007f92ae843000       4       4       4 rw-s-   [ shmid=0x48000 ]
00007f92ae843000       0       0       0 rw-s-   [ shmid=0x48000 ]
00007ffec9539000     132      32      32 rw---   [ stack ]
00007ffec9539000       0       0       0 rw---   [ stack ]
---------------- ------- ------- -------
total kB         4492416  124396  110252
</code></pre><p>The key block is obviously this one:<pre><code class=language-txt>00007f919cb75000 4317408  108240  108240 rw-s- zero (deleted)
</code></pre><p>That shows 4317408k is the zeroed out buffer space holding shared_buffers, while 108240k is nailed down using an old SysV <dfn>resident memory allocation</dfn> (<abbr>RSS</abbr>). That RSS chunk is the overhead Postgres needs to run, things similar to what the old documentation put into the "Shared Memory Usage" table.<p>Most people find this information easier to track on a hot server using the <code>top</code> command. For Postgres <code>top -c</code> is recommended because it will decode what all the database processes are doing. <code>top</code> output from this server shows the big virtual memory block in the <code>VIRT</code> column:<pre><code class=language-txt>  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 8025 postgres  20   0 4492412  36020  33992 S   0.0  0.2   0:00.15 postgres: 10/main: writer process
 8026 postgres  20   0 4492412  21412  19388 S   0.0  0.1   0:00.42 postgres: 10/main: wal writer process
 8028 postgres  20   0  175124   4396   2260 S   0.0  0.0   0:00.26 postgres: 10/main: stats collector process
 8024 postgres  20   0 4493760  62308  59052 S   0.0  0.4   0:00.65 postgres: 10/main: checkpointer process
 8029 postgres  20   0 4492724   4984   2840 S   0.0  0.0   0:00.00 postgres: 10/main: bgworker: logical replication launcher
 8027 postgres  20   0 4492816   6800   4552 S   0.0  0.0   0:00.14 postgres: 10/main: autovacuum launcher process
</code></pre><p>Dealing with shared memory on modern PostgreSQL and Linux versions is far improved from the old days when you had to endlessly tweak kernel parameters just to make a database run. There is still another level of work to support <em>Huge Pages</em>, which I'll demonstrate next time.</p><style>mjx-container[jax=SVG]{direction:ltr}mjx-container[jax=SVG]>svg{overflow:visible;min-height:1px;min-width:1px}mjx-container[jax=SVG]>svg a{fill:blue;stroke:blue}mjx-container[jax=SVG][display=true]{display:block;text-align:center;margin:1em 0}mjx-container[jax=SVG][display=true][width=full]{display:flex}mjx-container[jax=SVG][justify=left]{text-align:left}mjx-container[jax=SVG][justify=right]{text-align:right}g[data-mml-node=merror]>g{fill:red;stroke:red}g[data-mml-node=merror]>rect[data-background]{fill:yellow;stroke:none}g[data-mml-node=mtable]>line[data-line],svg[data-table]>g>line[data-line]{stroke-width:70px;fill:none}g[data-mml-node=mtable]>rect[data-frame],svg[data-table]>g>rect[data-frame]{stroke-width:70px;fill:none}g[data-mml-node=mtable]>.mjx-dashed,svg[data-table]>g>.mjx-dashed{stroke-dasharray:140}g[data-mml-node=mtable]>.mjx-dotted,svg[data-table]>g>.mjx-dotted{stroke-linecap:round;stroke-dasharray:0,140}g[data-mml-node=mtable]>g>svg{overflow:visible}[jax=SVG] mjx-tool{display:inline-block;position:relative;width:0;height:0}[jax=SVG] mjx-tool>mjx-tip{position:absolute;top:0;left:0}mjx-tool>mjx-tip{display:inline-block;padding:.2em;border:1px solid #888;font-size:70%;background-color:#f8f8f8;color:#000;box-shadow:2px 2px 5px #aaa}g[data-mml-node=maction][data-toggle]{cursor:pointer}mjx-status{display:block;position:fixed;left:1em;bottom:1em;min-width:25%;padding:.2em .4em;border:1px solid #888;font-size:90%;background-color:#f8f8f8;color:#000}foreignObject[data-mjx-xml]{font-family:initial;line-height:normal;overflow:visible}mjx-container[jax=SVG] path[data-c],mjx-container[jax=SVG] use[data-c]{stroke-width:3}</style> ]]></content:encoded>
<category><![CDATA[ Production Postgres ]]></category>
<author><![CDATA[ Greg.Smith@crunchydata.com (Greg Smith) ]]></author>
<dc:creator><![CDATA[ Greg Smith ]]></dc:creator>
<guid isPermalink="false">https://blog.crunchydata.com/blog/postgresql-on-linux-counting-committed-memory</guid>
<pubDate>Fri, 11 Jun 2021 05:00:00 EDT</pubDate>
<dc:date>2021-06-11T09:00:00.000Z</dc:date>
<atom:updated>2021-06-11T09:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ PostgreSQL 13 Benchmark: Memory Speed vs. TPS ]]></title>
<link>https://www.crunchydata.com/blog/postgresql-13-benchmark-memory-speed-vs.-tps</link>
<description><![CDATA[ Today I'm changing the memory speed on my main test system, going from 2133MHz to 3200MHz, and measuring how that impacts PostgreSQL SELECT results. I'm seeing a 3% gain on this server, but as always with databases that's only on a narrow set of in-memory use cases. ]]></description>
<content:encoded><![CDATA[ <p>Some people are obsessed with sports or cars. I follow <a href=/blog/postgresql-benchmarks-apple-arm-m1-macbook-pro-2020>computer hardware</a>. The PC industry has overclocking instead of nitrous, plexi cases instead of chrome, and RGB lighting as its spinning wheels.<p>The core challenge I enjoy is cascading small improvements to see if I can move a bottleneck. The individual improvements are often just a few percent. Percentage gains can compound as you chain them together.<p>Today I'm changing the memory speed on my main test system, going from 2133MHz to 3200MHz, and measuring how that impacts PostgreSQL SELECT results. I'm seeing a 3% gain on this server, but as always with databases that's only on a narrow set of in-memory use cases. Preview:<p><img alt=scaling-sets loading=lazy src=https://f.hubspotusercontent00.net/hubfs/2283855/scaling-sets.png><h2 id=why-more-benchmarks><a href=#why-more-benchmarks>Why more benchmarks?</a></h2><p>The industry around PC gaming has countless performance tests at the micro and macro level. A question I took on this year is how to take some useful metrics or test approaches from PC benchmarking and apply them even to virtual database instances.<p>There was a big constraint: I could only use SQL. As much as I enjoy the tinkering side of real hardware, a lot of the customers we support at Crunchy Data are provided a virtual database instance instead of a dedicated server. For PostgreSQL, I call these "Port 5432" installs, because the only access to the server is a connection to the database's standard port number. Disk seek test? You can't run <code>fio</code> or <code>iozone</code> on port 5432. Memory speed? There's no <code>STREAM</code> or <code>Aida</code> on 5432. You can tunnel system calls through the PostgreSQL's many server-side languages. That only goes so far when the software in each database container is shrunk to a minimum viable installation.<p>The improvements I've put into <a href=https://github.com/gregs1104/pgbench-tools>pgbench-tools</a> this year let me chug through an entire grid of client/size workloads, and my <a href=/blog/postgresql-13-upgrade-and-performance-check-on-ubuntu>last blog</a> went over upgrading to PostgreSQL 13 on this AMD Ryzen 9 3950X server. Part of what I'm doing here today is proving to myself the toolkit is good enough to measure a small gain, given that pgbench itself is not the most consistent benchmark.<h2 id=memory-tweaking-theory><a href=#memory-tweaking-theory>Memory tweaking theory</a></h2><p>On a lot of server installs tuning memory is something only the hardware vendor ever does. To respect that my initial PG13 comparisons left memory at its platform default speed: 2133MHz. The memory I'm using, G.SKILL F4-3600C19-16GVRB, can in theory run at 3600MHz.<p>Most desktop class motherboards have 4 RAM slots and run fastest when only two are used. Effectively this G.SKILL pair can bond into a dual-channel at 3600MHz. But the minute I try to fill all four slots, that speed is impossible. Performance doesn't scale up to quad channels; instead you get dual channels that are each split across two DIMMs. Juggling that adds just enough latency that the motherboard and CPU can't run at the maximum dual-channel speed anymore. Fully loaded with RAM, the best I can do on this hardware is running memory at 3200MHz. There's a similar but even worse trade-off buying big servers, as the buffering needed to handle very large amounts of RAM adds enough latency to pull down single core results.<p>Pounding a server with pgbench generates enough heat and random conditions that I rejected outright overclocking some years ago. I once lost my entire winter holiday chasing a once per day PG crash on my benchmark server, all from the CPU overheating just enough to <a href=https://postgrespro.com/list/thread-id/1813321>flip one bit</a>.<h2 id=quantifying-speed-improvements><a href=#quantifying-speed-improvements>Quantifying speed improvements</a></h2><p>The graph above shows an even increase in speed across all the sizes tested (up to 256GB=4X RAM), which is a nice start. Looking instead at the client count gives a different pattern:<p><img alt=clients-sets loading=lazy src=https://f.hubspotusercontent00.net/hubfs/2283855/clients-sets.png><p>There is a clear trend that high client counts are getting more of a boost from the faster memory than low ones. That is exactly what you'd expect and hope for. More clients means more pressure to move memory around, and anything you can do to accelerate that helps proportionately.<p><code>pgbench-tools</code> puts all the results in a database, so I can write simple SQL to analyze the workload grid:<pre><code class=language-pgsql>SELECT set,clients,ROUND(AVG(tps)) FROM test_stats WHERE set>10
GROUP BY set,clients ORDER BY set,clients
\crosstabview
</code></pre><table><thead><tr><th>clients<th>1<th>2<th>4<th>8<th>16<th>32<th>64<th>128<tbody><tr><td>2133<td>8354<td>16508<td>34366<td>64859<td>106305<td>165437<td>196002<td>219186<tr><td>3000<td>8293<td>16574<td>34267<td>65612<td>107867<td>169757<td>202697<td>231303<tr><td>3200<td>8459<td>16954<td>35221<td>67392<td>109688<td>170635<td>203348<td>232132</table><p>To make these examples cleaner, in the first column I replaced the set identifier number with the actual speed in MHz. The 128 client results are notably better. At 1 client the run to run variation noise was bigger than the regression, showing the bizarre result that 3000MHz memory worked slower than 2133MHz. I can make problems like that go away by running a lot more tests until the averages settle down; that didn't seem necessary here. I have a follow-up article coming where I dig into single client speeds more carefully.<p>I also like to look at the maximum rate any test runs at. Averages can hide changes to a distribution . You can't fake legitimately running faster than ever before. Considering only the best out of the runs that fit into each summary cell, which normally is the scale=100 1.6GB result, gives:<pre><code class=language-pgsql>SELECT set,clients,max(tps) FROM test_stats WHERE set>10
GROUP BY set,clients ORDER BY set,clients
\crosstabview
</code></pre><table><thead><tr><th>clients<th>1<th>2<th>4<th>8<th>16<th>32<th>64<th>128<tbody><tr><td>2133<td>22319<td>41535<td>81216<td>149678<td>233058<td>377269<td>347553<td>367855<tr><td>3000<td>22361<td>42217<td>83650<td>153980<td>234985<td>379450<td>348910<td>369827<tr><td>3200<td>23213<td>42806<td>84694<td>157181<td>237486<td>386854<td>352344<td>375003</table><p>Re-scaling to percentages and eliminating the 3000MHz middle step:<table><thead><tr><th>2133-3200<th>1<th>2<th>4<th>8<th>16<th>32<th>64<th>128<th>Median<tbody><tr><td>Avg TPS<td>+1.3%<td>+2.7%<td>+2.5%<td>+3.9%<td>+3.2%<td>+3.1%<td>+3.7%<td>+5.9%<td>+3.2%<tr><td>Max TPS<td>+4.0%<td>+3.1%<td>+4.3%<td>+5.0%<td>+1.9%<td>+2.5%<td>+1.4%<td>+1.9%<td>+2.8%</table><p>Since increasing memory speed gives a 2.8-3.2% gain overall depending on how you slice the results, I'm happy to call that a solid 3% gain across the grid. Light client counts gain the least, with a low at 1 client of only a 1.3% average gain. When overloaded with a full 128 clients, average throughput increased by up to 5.9%.<p>If you'd like to read another perspective on this topic, Puget Systems has a nice article on <a href=https://www.pugetsystems.com/labs/articles/RealityCapture-CPU-Performance-AMD-Ryzen-9-3950X-1607/>CPU Performance: AMD Ryzen 9 3950X</a>. They find a similarly sized gain to what I measured here, and their commentary about larger memory capacity is in line with my comments above. ]]></content:encoded>
<category><![CDATA[ Production Postgres ]]></category>
<author><![CDATA[ Greg.Smith@crunchydata.com (Greg Smith) ]]></author>
<dc:creator><![CDATA[ Greg Smith ]]></dc:creator>
<guid isPermalink="false">https://blog.crunchydata.com/blog/postgresql-13-benchmark-memory-speed-vs.-tps</guid>
<pubDate>Wed, 30 Dec 2020 04:00:00 EST</pubDate>
<dc:date>2020-12-30T09:00:00.000Z</dc:date>
<atom:updated>2020-12-30T09:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ PostgreSQL Benchmarks: Apple ARM M1 MacBook Pro 2020 ]]></title>
<link>https://www.crunchydata.com/blog/postgresql-benchmarks-apple-arm-m1-macbook-pro-2020</link>
<description><![CDATA[ This week Apple started delivering Macs using their own Apple Silicon chips, starting with a Mac SOC named the M1. M1 uses the ARM instruction set and claims some amazing acceleration for media workloads. I wanted to know how it would do running PostgreSQL, an app that's been running on various ARM systems for years. The results are great! ]]></description>
<content:encoded><![CDATA[ <p>This week Apple started delivering Macs using their own Apple Silicon chips, starting with a Mac SOC named the M1. M1 uses the ARM instruction set and claims some amazing acceleration for media workloads. I wanted to know how it would do running PostgreSQL, an app that's been running on various ARM systems for years. The results are great!<p>The OSS community around the homebrew project already qualified their PostgreSQL package as working on M1, and with some recompiling work that all worked as expected:<pre><code class=language-shell>$ /opt/homebrew/bin/psql -c "select version()"
PostgreSQL 13.0 on arm-apple-darwin20.1.0, compiled by
Apple clang version 12.0.0 (clang-1200.0.32.28), 64-bit
</code></pre><p>I need some additional software for my benchmark toolkit, and the only compile problems I saw were Qt and Python's numpy; those two seemed straightforward to fix once someone gets to them.<p>My last <a href=/blog/postgresql-benchmarks-apple-intel-macbook-pro-2011-2019>blog entry</a> introduced my basic method of using <a href=https://github.com/gregs1104/pgbench-tools>pgbench-tools</a> to look at past MacBook Pro models. I said there Apple needed to exceed "15K TPS single/60K TPS all core" on PostgreSQL to fully embarrass Intel. Well, they outperformed expectations:<p><img alt=MacYearly-M1 loading=lazy src=https://f.hubspotusercontent00.net/hubfs/2283855/MacYearly-M1.png><p>32K single/92K all core is so fast for a laptop, I need to pull in some other hardware to put it into perspective. Here's a data table for all the results behind the graph, plus adding two generations of AMD's Ryzen desktop hardware:<table><thead><tr><th>server<th>1<th>2<th>4<th>8<th>16<th>32<tbody><tr><td>2011 16GB MacBookPro8,2<td>7252<td>14644<td>20471<td>30749<td>32894<td>32647<tr><td>2012 16GB MacBookPro9,1<td>7781<td>15861<td>22380<td>34743<td>38294<td>36754<tr><td>2015 16GB MacBookPro11,4<td>9770<td>17795<td>22372<td>38341<td>45048<td>43497<tr><td>2017 16GB Intel NUC7i3BNB<td>12789<td>20870<td>33649<td>31053<td>32029<td>32409<tr><td>2019 16GB MacBookPro16,1<td>14353<td>27588<td>43784<td>45089<td>61603<td>58705<tr><td>2019 64GB MacBookPro16,1<td>14105<td>28733<td>46836<td>61167<td>62083<td>69101<tr><td>2019 16GB Intel NUC10i5FNB<td>15444<td>27496<td>43341<td>70015<td>61927<td>62584<tr><td>2020 8GB MacBookPro17,1<td>32198<td>52828<td>96536<td>97042<td>95130<td>92663<tr><td>2018 64GB Ryzen 7 2700X<td>11624<td>22153<td>41648<td>69399<td>138431<td>123466<tr><td>2019 64GB Ryzen 9 3950X<td>37768<td>69162<td>133943<td>206684<td>258722<td>306185</table><p>This graph is amazing to me:<p><img alt=MacAMD loading=lazy src=https://f.hubspotusercontent00.net/hubfs/2283855/MacAMD.png>Of course Intel has Xeon processors that have pushed single core performance higher than these laptop-oriented Intel results. But look at that big cluster below 5 clients, showing how long they've been stuck in the same performance range when power and heat is limited. I mentioned last time Intel had only doubled performance in the 8 years of MacBook models I looked at, which is not industry leading performance.<p>AMD has been doing a lot better, getting their single core boost competitive in their 3000 series. Even last year's 3950X with its mandatory water cooling is barely faster than the M1 until you hit 8 clients.<p>If Apple can push the M1 design into larger amounts of memory and add a few more cores, it could be a fierce midsize server competitor. That's not going to disrupt the big industry push toward hosting things on giant cloud systems, where data centers want >=48 processors for a server to be worth installing. There are cloud scale ARM servers out there, and Apple's ARM instruction set Macs make developing for that platform easier. I'm looking forward to the competition of a four way race between Intel, AMD, Apple, and the other ARM designers.<p>The M1 is a great step forward for developers who can take advantage of it. Let's hope the obvious virtualization issues are sorted out in the near future. A lot of developers need tools like Docker and VMs to build modern cloud software. Until that area is sorted out, the M1 Macs aren't suitable for everyone. Make sure you understand your requirements and what's supported before you consider buying one.<p>To make this article complete, here's the detailed list of the hardware I tested for this tour of benchmark results, and you can drill into detail about the systems I have here by digging into my <a href=https://browser.geekbench.com/user/gregs1104>Geekbench Profile</a>.<table><thead><tr><th>System<th>CPU Model<th>CPUs<tbody><tr><td>2011 16GB MacBookPro8,2<td>Intel i7-2860QM CPU @ 2.50GHz<td>8<tr><td>2012 16GB MacBookPro9,1<td>Intel i7-3615QM CPU @ 2.30GHz<td>8<tr><td>2015 16GB MacBookPro11,4<td>Intel i7-4770HQ CPU @ 2.20GHz<td>8<tr><td>2017 16GB Intel NUC7i3BNB<td>Intel i3-7100U CPU @ 2.40GHz<td>4<tr><td>2019 16GB MacBookPro16,1<td>Intel i7-9750H CPU @ 2.60GHz<td>12<tr><td>2019 64GB MacBookPro16,1<td>Intel i9-9980HK CPU @ 2.40GHz<td>16<tr><td>2019 16GB Intel NUC10i5FNB<td>Intel i5-10210U CPU @ 1.60GHz<td>8<tr><td>2020 8GB MacBookPro17,1<td>Apple M1<td>8<tr><td>2018 64GB Ryzen X470<td>AMD Ryzen 7 2700X<td>16<tr><td>2019 64GB Ryzen X570<td>AMD Ryzen 9 3950X<td>32</table> ]]></content:encoded>
<category><![CDATA[ Production Postgres ]]></category>
<author><![CDATA[ Greg.Smith@crunchydata.com (Greg Smith) ]]></author>
<dc:creator><![CDATA[ Greg Smith ]]></dc:creator>
<guid isPermalink="false">https://blog.crunchydata.com/blog/postgresql-benchmarks-apple-arm-m1-macbook-pro-2020</guid>
<pubDate>Fri, 20 Nov 2020 04:00:00 EST</pubDate>
<dc:date>2020-11-20T09:00:00.000Z</dc:date>
<atom:updated>2020-11-20T09:00:00.000Z</atom:updated></item>
<item><title><![CDATA[ PostgreSQL Benchmarks: Apple Intel MacBook Pro, 2011-2019 ]]></title>
<link>https://www.crunchydata.com/blog/postgresql-benchmarks-apple-intel-macbook-pro-2011-2019</link>
<description><![CDATA[ Apple's Intel-based laptops are very popular among developers, and that's as true of people who work on PostgreSQL as other groups. Tomorrow, the first shipping Apple laptops running on ARM CPUs instead of Intel are expected. ]]></description>
<content:encoded><![CDATA[ <p>Apple's Intel-based laptops are very popular among developers, and that's as true of people who work on PostgreSQL as other groups. Tomorrow, the first shipping Apple laptops running on ARM CPUs instead of Intel are expected. That is likely to include at least a 13" MacBook Pro. I decided to prepare for that with a survey of PostgreSQL performance on my small herd of Apple laptops. Mine are all the 15" or newer 16" models.<p>Crunchy Data has already started digging into PostgreSQL on ARM performance as part of <a href=https://crunchybridge.com/>Crunchy Bridge</a>, such as <a href=https://www.postgresql.org/message-id/CACN56%2BP1astF5zvocrT7--Mu2dQWFS0eQ31xNmX%3Db%3D98y9fMSw%40mail.gmail.com>microbenchmarking on AWS Graviton 2</a>. The OSS community around the Mac Homebrew tools seems <a href=https://github.com/Homebrew/brew/issues/7857>ready for the ARM transition</a> too. I'm hopeful that with some work, the new Apple ARM hardware can be as performant running Postgres as the Intel chips they replace. My results here say that ideally, Apple Silicon would hit 15K TPS single/60K TPS all core, or at least get close. Who wants to make an over/under bet?<p>I benchmarked them all with the consistent toolchain and method of my <a href=https://github.com/gregs1104/pgbench-tools>pgbench-tools</a> software. That runs lots of PostgreSQL performance tests at various database sizes and client counts. For these MacBook CPU tests, the most useful tests I found for general CPU performance used single row "point" SELECT statements against a 1.6GB database, which is 100 on pgbench's size scale factor. Performance on that specific benchmark hasn't changed a lot (for laptop sized workloads) in the last few versions of PostgreSQL. Some of these results on older systems were using PG11, most are using PG12.<p>Here's the hardware I tested for this tour of benchmark results, and you can drill into detail about the systems I have here by digging into my <a href=https://browser.geekbench.com/user/gregs1104>Geekbench Profile</a>.<table><thead><tr><th>System<th>CPU Model<th>CPUs<tbody><tr><td>2011 16GB MacBookPro8,2<td>Intel i7-2860QM CPU @ 2.50GHz<td>8<tr><td>2012 16GB MacBookPro9,1<td>Intel i7-3615QM CPU @ 2.30GHz<td>8<tr><td>2015 16GB MacBookPro11,4<td>Intel i7-4770HQ CPU @ 2.20GHz<td>8<tr><td>2017 16GB Intel NUC7i3BNB<td>Intel i3-7100U CPU @ 2.40GHz<td>4<tr><td>2019 16GB MacBookPro16,1<td>Intel i7-9750H CPU @ 2.60GHz<td>12<tr><td>2019 64GB MacBookPro16,1<td>Intel i9-9980HK CPU @ 2.40GHz<td>16<tr><td>2019 16GB Intel NUC10i5FNB<td>Intel i5-10210U CPU @ 1.60GHz<td>8</table><p>Since I stayed away from laptops with Apple's butteryfly keyboard, I noticed a gap in my data around 2017. Intel had its own problems during this period too. I did pick up a cheap Intel 7th generation i3 CPU NUC in 2017, so I substituted that onto the chart. The single core performance fit midway in performance between the 2015 and 2019 MacBook models I had. The NUC hardware isn't a proper laptop chip, but the thermal limits of the form factor make them perform more like laptop CPUs than desktop one. I also have a 2019 10th gen i5 NUC I added for comparison. The two NUCs are running Ubuntu Linux instead of Mac OS.<p><img alt="CPU comparison"loading=lazy src=https://f.hubspotusercontent00.net/hubfs/2283855/CPU%20comparison.png><p>I like data tables as much as charts, and parts of the story here are easier to see that way:<table><thead><tr><th>server<th>1<th>2<th>4<th>8<th>16<th>32<tbody><tr><td>2011 16GB MacBookPro8,2<td>7252<td>14644<td>20471<td>30749<td>32894<td>32647<tr><td>2012 16GB MacBookPro9,1<td>7781<td>15861<td>22380<td>34743<td>38294<td>36754<tr><td>2015 16GB MacBookPro11,4<td>9770<td>17795<td>22372<td>38341<td>45048<td>43497<tr><td>2017 16GB Intel NUC7i3BNB<td>12789<td>20870<td>33649<td>31053<td>32029<td>32409<tr><td>2019 16GB MacBookPro16,1<td>14353<td>27588<td>43784<td>45089<td>61603<td>58705<tr><td>2019 64GB MacBookPro16,1<td>14105<td>28733<td>46836<td>61167<td>62083<td>69101<tr><td>2019 16GB Intel NUC10i5FNB<td>15444<td>27496<td>43341<td>70015<td>61927<td>62584</table><p>It's nice to see that both single-core and multi-core results have doubled during this 8 year stretch of time. ]]></content:encoded>
<category><![CDATA[ Production Postgres ]]></category>
<author><![CDATA[ Greg.Smith@crunchydata.com (Greg Smith) ]]></author>
<dc:creator><![CDATA[ Greg Smith ]]></dc:creator>
<guid isPermalink="false">https://blog.crunchydata.com/blog/postgresql-benchmarks-apple-intel-macbook-pro-2011-2019</guid>
<pubDate>Mon, 09 Nov 2020 04:00:00 EST</pubDate>
<dc:date>2020-11-09T09:00:00.000Z</dc:date>
<atom:updated>2020-11-09T09:00:00.000Z</atom:updated></item></channel></rss>