Just Postgres in a browser

All of our curated tutorials exist directly on the main Crunchy Data Developer area. However, you can visit an empty Postgres playground directly at: https://www.crunchydata.com/developers/playground. This is a blank slate Postgres instance, utilizing compute and memory directly from your web browser. It is not persistent and not connected to a remote session.

Creating your own SQL files

If you pass in a parameter for a sql file it, the playground will automatically run that once the Postgres session is started.

We can take a very basic set of SQL, something like this:

CREATE TABLE example (
  id serial,
  data text);

INSERT INTO example (data) VALUES ('hello');
INSERT INTO example (data) VALUES ('world');

You can build longer SQL files for full data inputs. If you run a pg_dump from an existing database, that file will also work.

Like a restore file, you can also list SET statements for the session

SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET xmloption = content;
SET client_min_messages = warning;
SET row_security = off;

Load your SQL file into a url

The easiest way to host a sql file online is to use the GitHub gist feature. You’ll create a new gist, give it a name. Then view the raw link. The raw view link is the URL we want for working with the playground.

My example my file is:

 https://gist.githubusercontent.com/craigkerstiens/62e00c80df31700302726f77595b7661/raw/3aa83426c148ac77bbed0f80ef31f433cf7e5053/example.sql

Build your own playground URLs

Now we can pass that into the playground and build a longer url with:

1 - The base playground URL: https://www.crunchydata.com/developers/playground

2 - The sql parameter ?sql=

3 - Your full gist URL https://gist.githubusercontent.com/craigkerstiens/62e00c80df31700302726f77595b7661/raw/3aa83426c148ac77bbed0f80ef31f433cf7e5053/example.sql

Resulting in this:

https://www.crunchydata.com/developers/playground?sql=https://gist.githubusercontent.com/craigkerstiens/62e00c80df31700302726f77595b7661/raw/3aa83426c148ac77bbed0f80ef31f433cf7e5053/example.sql

I’ve been really excited to see how many people are using the Postgres playground for education and training. Let us know how you’re using the playground on X @crunchydata. ]]> 94a3fc4862f5f58d72d67a639cbc519b4a34820351617ab500b4fc0235888fff Fri, 15 Dec 2023 08:00:00 EST 2023-12-15T13:00:00.000Z 2023-12-15T13:00:00.000Z https://www.crunchydata.com/blog/rolling-the-dice-with-postgres-random-function Generating random numbers is a surprisingly common task in programs, whether it's to create test data or to provide a user with a random entry from a list of items.

PostgreSQL comes with just a few simple foundational functions that can be used to fulfill most needs for randomness.

Almost all your random-ness needs will be met with the random() function.

Uniformity

The random() function returns a double precision float in a continuous uniform distribution between 0.0 and 1.0.

What does that mean? It means that you could get any value between 0.0 and 1.0, with equal probability, for each call of random().

Here's five uniform random numbers between 0.0 and 1.0.

SELECT random() FROM generate_series(1, 5)

0.3978842227698167
0.7438732417540841
0.3875091442400458
0.4108009373061563
0.5524543763568912

Yep, those look pretty random! But, maybe not so useful?

Random Numbers

Most times when people are trying to generate random numbers, they are looking for random integers in a range, not random floats between 0.0 and 1.0.

Say you wanted random integers between 1 and 10, inclusive. How do you get that, starting from random()?

Start by scaling an ordinary random() number up be a factor of 10! Now you have a continuous distribution between 0 and 10.

SELECT 10 * random() FROM generate_series(1, 5)

3.978842227698167
7.438732417540841
3.875091442400458
4.108009373061563
5.5245437635689125

Then, if you push every one of those numbers down to the nearest integer using floor() you'll end up with a random integer between 0 and 9.

SELECT floor(10 * random()) FROM generate_series(1, 5)

4
8
4
5
6

If you wanted a random integer between 1 and 10, you just need to add 1 to the zero-base number.

SELECT floor(10 * random()) + 1 FROM generate_series(1, 5)

3
7
3
4
5

Random Rows and Values

Sometimes the things you are trying to do randomly aren't numbers. How do you get a random entry out of a string? Or a random row from a table?

We already saw how to get one-based integers from random() and we can apply that technique to the problem of pulling an entry from an array.

WITH f AS (
    SELECT ARRAY[
        'apple',
        'banana',
        'cherry',
        'pear',
        'peach'] AS fruits
)
SELECT fruits[ceil(array_length(fruits,1) * random())] AS snack
FROM f;

 snack
-------
 peach

Getting a random row involves some tradeoffs and thinking. For a random value from a small table, the naive way to get a single random value is this.

SELECT *
FROM fruits
ORDER BY random()
LIMIT 1

As you can imagine, this gets quite expensive if the fruits table gets too large, since it sorts the whole table every time.

If you only need a single random row, one way to achieve that is to add a random column to your table and index it.

CREATE TABLE fruits (
  id SERIAL PRIMARY KEY,
  fruit TEXT NOT NULL,
  random FLOAT8 DEFAULT random()
  );

INSERT INTO fruits (fruit)
  VALUES ('apple'),('banana'),('cherry'),('pear'),('peach');

CREATE INDEX fruits_random_x ON fruits (random);

Then when it's time to search, use the random function to generate a starting search location and find the next highest value.

SELECT *
FROM fruits
WHERE random > random()
ORDER BY random ASC
LIMIT 1;

 id | fruit  |       random
----+--------+--------------------
  8 | banana | 0.1997961574379754

Be careful using this trick for more than one row though: since the values in the random column are fixed, the sequences of rows returned will be deterministic, even if the start row is random.

If you want to pull large portions of a table into a query (for random sampling, for example) look at the TABLESAMPLE clause of the SELECT command.

Random Groups

Suppose I wanted the entire contents of the fruits collection, but returned in two random groups? This is actually much like getting a single random value: order the whole set randomly, and then use that ordering to determine grouping.

WITH random_fruits AS (
    SELECT id, fruit
    FROM fruits
    ORDER BY random()
)
SELECT row_number() over () % 2 AS group,
       id, fruit
FROM random_fruits
ORDER BY 1;

 group | id | fruit
-------+----+--------
     0 | 11 | peach
     0 |  8 | banana
     1 | 10 | pear
     1 |  7 | apple
     1 |  9 | cherry

The '2' in the example above is the number of groups desired.

random_normal

So far we have just been looking at ways to permute the uniform distribution offered by the random() function. But there is in fact an infinite number of other probability distributions that random numbers could be a part of.

Of that infinite collection, by far the most frequently used in practice is the "normal distribution" also known as the "Gaussian distribution" or "bell curve".

Rather than having a hard cut-off point, the normal distribution has a frequent center and then ever lower probability of values out to infinity in both directions.

The position of the center of the distribution is the "mean" and the rate of probability decay is controlled by the "standard deviation".

To generate normally distributed data in PostgreSQL, use the random_normal(mean, stddev) function that was introduced in version 16.

SELECT random_normal(0, 1)
FROM generate_series(1,10)
ORDER BY 1

 -0.8147201382612904
 -0.5751449000210354
 -0.4643454485382744
 -0.0630592935151314
 0.26438942114339203
 0.39298889191244274
  0.4946046063256206
  0.8560911955145666
  1.3534309793797454
   1.664493506727331

It's kind of hard to appreciate that the data have a central tendency without generating a lot more of them and counting how many fall within each bin.

SELECT random_normal()::integer,
       Count(*)
FROM generate_series(1,1000)
GROUP BY 1
ORDER BY 1

The cast to integer rounds the values towards the nearest integer, so you can see how the data are mostly between the first two standard deviations of the mean.

 random_normal | count
---------------+-------
            -3 |     5
            -2 |    65
            -1 |   233
             0 |   378
             1 |   246
             2 |    67
             3 |     5
             4 |     1

Seeds and Pseudo-randomness

If you looked very closely at the examples in the first section you'll have noticed that they all started from the same, allegedly random values.

If random() truly is random, how did I get the same starting values four times in a row?

The answer, shockingly, is that random() is actually "pseudo-random".

A pseudorandom sequence of numbers is one that appears to be statistically random, despite having been produced by a completely deterministic and repeatable process.

With a pseudo-random number generator and a known starting point, I will always get the same sequence of numbers, at least on the same computer.

The reason most computer programs use pseudo-random number generators is that generating truly random numbers is actually quite an expensive operation (relatively speaking).

So programs instead generate one truly random number, and use that as a "seed" for a generator.

PostgreSQL uses the Blackman/Vigna "xoroshiro128 1.0" pseudo-random number generator.

By default, on start-up PostgreSQL sets up a seed value by calling an external random number generator, using an appropriate method for the platform:

• Using OpenSSL RAND_bytes() if available, or
• using Windows CryptGenRandom() on that platform, or
• using the operating system /dev/urandom if necessary.

So if you are interested in a random number, just calling random() will get you one every time.

But if you want to put your finger on the scales, you can use the setseed() function to cause your random() and random_normal() functions to generate a deterministic series of random numbers, starting from a seed value you specify. ]]> 0b769e6091f1f43bae7e2cf18e86b756008f4fb621f9b308fb6c46f206282cb5 Fri, 22 Sep 2023 09:00:00 EDT 2023-09-22T13:00:00.000Z 2023-09-22T13:00:00.000Z https://www.crunchydata.com/blog/xkcd-bad-map-projection-with-postgis Last week, Randall Munroe dropped his latest XKCD "Bad Map Projection", number six, "ABS(Longitude)", which looks like this:

Truly this is a bad map projection, on a par with the previous five:

• Liquid Resize
• Time Zones
• South America
• Greenland Special
• Madagascator

The last two are just applications of common map projections with very uncommon projection parameters that accentuate certain areas of the globe, a cartographic version of the classic "View of the World from 9th Avenue".

A colleague asked me if we could recreate ABS(Longitude) and I figured it was worth a try!

Getting Data

At a minimum, we want a countries layer and some independent place labels to provide context, which is available at the first stop for basic global data, Natural Earth.

We have been playing with ogr2ogr and weird remote access tricks lately, and we can use ogr2ogr to load the data in one step.

# Load the countries and places directly from the remote
# zip file into the working PostgreSQL database

ogr2ogr \
 -f PostgreSQL \
 -nlt PROMOTE_TO_MULTI \
 -lco OVERWRITE=yes \
 -lco GEOMETRY_NAME=geom \
 postgresql://postgres@localhost/xkcd \
 /vsizip//vsicurl/https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_admin_0_countries.zip

ogr2ogr \
 -f PostgreSQL \
 -lco OVERWRITE=yes \
 -lco GEOMETRY_NAME=geom \
 postgresql://postgres@localhost/xkcd \
 /vsizip//vsicurl/https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/ne_110m_populated_places.zip

Preparing the Data

Now we have the data in the database, read to go!

The process we are going to apply will be transforming the shapes of one polygon at a time, and the Natural Earth data models the countries with one MultiPolygon per country.

Canada, for example, is one country, but 30 polygons.

We want a table with just one row for each polygon, so we "dump" all the multi-polygons using ST_Dump().

CREATE SEQUENCE country_id;

CREATE TABLE countries AS
  SELECT nextval('country_id') AS id,
    name,
    (ST_Dump(geom)).geom::geometry(Polygon, 4326) AS geom
  FROM ne_110m_admin_0_countries;

Next, because we are going to be processing western shapes with negative longitude different from eastern shapes we have to solve the problem: what to do with shapes that straddle the prime meridian?

The answer: ST_Split()!

First we create a prime meridian geometry to use as a "splitting blade".

CREATE TABLE lon_0 AS
  SELECT ST_SetSrid(
    ST_MakeLine(
      ST_MakePoint(0,90),
      ST_MakePoint(0,-90)),
    4326)::geometry(LineString, 4326) AS geom;

Then we apply that blade to all the shapes that fall under it.

CREATE TABLE split_at_0 AS
  SELECT id, name, ST_CollectionHomogenize(
    ST_Split(c.geom, lon_0.geom))::geometry(MultiPolygon, 4326) AS geom
  FROM countries c
  JOIN lon_0
  ON ST_Intersects(c.geom, lon_0.geom);

Surprisingly few countries end up chopped by the meridian.

The output of the split is, for each input polygon, a multi-polygon of the components. But we want to operate on the shapes one polygon at a time, so again, we must dump the multi-polygon into its components.

A slightly longer query dumps the split shapes, and stores them in a table with the rest of the un-split polygons, labeling each shape depending on whether it is "west" or "east" of the prime meridian.

CREATE TABLE countries_split AS
WITH split AS (
  SELECT id, name, (ST_Dump(geom)).geom::geometry(Polygon, 4326) AS geom
  FROM split_at_0
)
SELECT c.id, c.name, c.geom,
  CASE WHEN ST_X(ST_StartPoint(c.geom)) >= 0 THEN 'east' ELSE 'west' END AS side
FROM countries c
LEFT JOIN split s
USING (id)
WHERE s.id IS NULL
UNION ALL
SELECT s.id, s.name, s.geom,
  CASE WHEN ST_X(ST_StartPoint(s.geom)) >= 0 THEN 'east' ELSE 'west' END AS side
FROM split s;

We have divided the west from the east, and are ready for the final step.

Flipping the West

Now we are ready to apply a transformation to all the "west" countries, to turn their negative longitudes into positive ones.

To do this, we will use the powerful ST_Affine() function.

The two-dimensional form of the function looks like this:

ST_Affine(geom, a, b, d, e, xoff, yoff)

Where the parameters correspond to an affine transformation matrix:

Or, in equation form:

From the equation it is pretty clear, we want to negate the input x and leave everything else alone.

In order to get a pretty map, we'd like the output data to be centered on the prime meridian again, so:

And in SQL like this:

CREATE TABLE countries_affine AS
  SELECT id, name,
    CASE WHEN SIDE = 'west'
         THEN ST_Affine(geom, -1, 0, 0, 1, -90, 0)
         ELSE ST_Affine(geom, 1, 0, 0, 1, -90, 0)
    END AS geom
  FROM countries_split;

CREATE TABLE places_affine AS
  SELECT ogc_fid AS id, name,
    CASE WHEN ST_X(geom) < 0
         THEN ST_Affine(geom, -1, 0, 0, 1, -90, 0)
         ELSE ST_Affine(geom, 1, 0, 0, 1, -90, 0)
    END AS geom
  FROM ne_110m_populated_places
  ORDER BY pop_max DESC;

And the final result on the map looks like the XKCD map, without the pretty hand-labeling and mountains:

The bad map projections aren't the only cartographic cartoons XKCD explored. If you liked this one, take a look at:

• Map Projections, all real!
• World According to Americans
• Upside Down
• Map Age Guide, fascinating history!

Do you have time to whip up a web app? No! You have a database to feed, and events are spiraling out of control... what to do?

How about a Google Sheet? The data layout is obvious, you can even enforce things like data types and required columns using locking and protecting, and unlike an Excel or LibreOffice document, it's always online, so you can hook the data into your system directly.

Access Sheets Data Remotely

You can pull data in CSV format from a (public) Google Sheets workbook just by plugging the sheet ID into a magic access URL.

The URL format looks like this:

https://docs.google.com/spreadsheets/d/{sheetId}/gviz/tq?tqx=out:csv&sheet={sheetName}

The sheetId you can pull out of the URL at the top of your browser. The sheetName is at the bottom of the page. You can read one sheet at a time out of a Sheets workbook.

This example sheet has 22 rows of population projection data in it.

$ curl "https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections"

"Year","18 to 19","Total"
"2024","120107","5485084"
"2025","123484","5563798"
"2026","128627","5641925"
"2027","132540","5719109"
"2028","134067","5796302"
...

Remote Access with COPY

The following examples all use the Crunchy Bridge database-as-a-service. Not all services will support these methods.

To use the COPY command for remote loading you will first need to create a table to load the data into. Our table structure is just three integer columns.

CREATE TABLE popn_copy (
year integer,
age_18_to_19 integer,
all_ages integer
);

We are going to use the PROGRAM option of copy to fire up the curl utility and pull the CSV data from Google, then stream that into the default PostgreSQL CSV reader.

In order to use COPY with the PROGRAM option you must be logged in as the postgres superuser.

COPY popn_copy FROM PROGRAM
'curl "https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections"'
WITH (
 FORMAT csv,
 HEADER true,
 ENCODING utf8
 );

Just like that, 22 rows loaded!

SELECT Count(*) FROM popn_copy;

The COPY approach is really the simplest one available, and to refresh your data, you can just run a scheduled TRUNCATE and then re-run the COPY.

However, it does have the disadvantage of requiring a superuser login.

Remote Access with HTTP

The http extension for PostgreSQL allows users to run web requests and fetch data from any URL. Sounds like exactly what we need!

You can check if you have the extension by querying the pg_available_extensions table.

SELECT *
  FROM pg_available_extensions
  WHERE name = 'http';

name              | http
default_version   | 1.5
installed_version |
comment           | HTTP client for PostgreSQL, allows web page retrieval inside the database.

If you have it, enable the http extension, and create a target table:

CREATE EXTENSION http;

CREATE TABLE popn_http (
year integer,
age_18_to_19 integer,
all_ages integer
);

If we use the http_get() function, we can pull the content from the remote URL in one step.

SELECT content AS row
FROM http_get('https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections')

What we get back is all the data, but in one big string. We would prefer 22 rows of data. Fortunately, PostgreSQL string processing can help us condition the data before inserting it into our table.

First, cut the string up using new-line characters as a delimiter.

SELECT unnest(string_to_array(content, E'\n')) AS row
FROM http_get('https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections')

Now we have 23 rows (22 data rows and one header row), which we can parse the numeric pieces out of using regexp_match():

INSERT INTO popn_http
WITH rows AS (
  SELECT unnest(string_to_array(content, E'\n')) AS row
  FROM http_get('https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections')
),
cols AS (
  SELECT regexp_match(row, '"([0-9]+)","([0-9]+)","([0-9]+)"') AS col FROM rows
)
SELECT col[1]::integer AS year,
       col[2]::integer AS age_18_to_19,
       col[3]::integer AS all_ages
  FROM cols
 WHERE col[1] IS NOT NULL

Whenever you want to refresh, just TRUNCATE the table and re-run the population query. Unlike the COPY method, this doesn't require super-user access to implement.

Remote Access with PL/Python

The http extension is simple, but it leaves a lot of work on the server side. PL/Python is capable of doing remote HTTP access, and it also has a lot of nice string parsing tools that could maybe result in a more pleasant output.

CREATE EXTENSION IF NOT EXISTS plpython3u;

CREATE OR REPLACE FUNCTION read_csv_from_url(url TEXT)
RETURNS SETOF RECORD
AS $$
  import csv
  import requests
  from io import StringIO

  # Make a GET request to the URL and retrieve the CSV data
  response = requests.get(url)
  response.raise_for_status()
  csv_data = response.text

  # Parse the CSV data
  reader = csv.reader(StringIO(csv_data))

  # Skip the header row
  next(reader, None)

  # Yield each row as a result
  for row in reader:
      yield tuple(row)
$$ LANGUAGE 'plpython3u';

Using a set returning function, it is much easier to integrate the function into the database infrastructure, so we can make the data available via a MATERIALIZED VIEW.

CREATE MATERIALIZED VIEW popn_python AS
SELECT *
  FROM read_csv_from_url('https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections')
    AS f(year integer, age_18_to_19 integer, all_ages integer);

Updating the remote data just needs a REFRESH MATERIALIZED VIEW, and super user access is not required.

Remote Access with FDW

Our last remote access trick uses a "foreign data wrapper", specifically the OGR FDW which exposes the multi-format access capabilities of the GDAL library to PostgreSQL.

While this example shows CSV file reading, the OGR FDW extension can be used to access a huge number of different formats, both local and remote.

The hardest part of using the OGR FDW driver is figuring out the correct server string to use in setting up the connection. It is best to start by downloading a copy of GDAL to your workstation and trying out various options using the ogrinfo tool.

With some trial and error, I found that a working URL involved:

• using the vsicurl remote access driver (check out the other "virtual file system" drivers provided to get a feel for just how flexible GDAL is for remote data access),
• prepending CSV to hint to GDAL what format driver to use, and
• appending &/popn to the URL to trick GDAL into using "popn" as the layer name instead of something much less attractive.

The result can connect to the remote source and understand the CSV file contents.

$ ogrinfo CSV:"/vsicurl/https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections&/popn"

INFO: Open of `CSV:/vsicurl/https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections&/popn'
      using driver `CSV' successful.
1: popn (None)

For most sources, like remote databases and so on, the URL will be a lot simpler and obvious. Even a remote CSV file will usually be easier, because it will have a CSV file name at the end of the URL, which GDAL uses to hint the correct driver.

CREATE SERVER myserver
  FOREIGN DATA WRAPPER ogr_fdw
  OPTIONS (
    datasource 'CSV:/vsicurl/https://docs.google.com/spreadsheets/d/1pBbCabAK6u6EIuyu_2XUul4Yxvf2w_Od6QYC_yEc4q4/gviz/tq?tqx=out:csv&sheet=Population_projections&/popn',
    format 'CSV');

Now that we have a "server", we can import the one layer that exists in that server.

If our server was something more sophisticated, like a database, there could potentially be multiple tables that would be imported using this method.

CREATE SCHEMA fdw;

IMPORT FOREIGN SCHEMA ogr_all
	FROM SERVER myserver
	INTO fdw;

SELECT * FROM fdw.popn;

 fid | year | n18_to_19 |  total
-----+------+-----------+---------
   1 | 2024 | 120107    | 5485084
   2 | 2025 | 123484    | 5563798
   3 | 2026 | 128627    | 5641925
   4 | 2027 | 132540    | 5719109
   5 | 2028 | 134067    | 5796302

We have data! Unfortunately, since it is coming from a text CSV file, without any column type mapping, we need to do a little bit of type coercion to get a clean table of integers.

CREATE MATERIALIZED VIEW popn_fdw AS
  SELECT year::integer      AS year,
         n18_to_19::integer AS age_18_to_19,
         total::integer     AS all_ages
  FROM fdw.popn;

Using a materialized view keeps our database from constantly hitting the remote table every time we access the FDW table. Since we have a materialized view, the refresh method is a little prettier than truncating and reloading, though it is effectively the same thing: just refresh the view.

REFRESH MATERIALIZED VIEW popn_fdw;

As with the http approach and unlike using COPY, the FDW approach does not require a superuser to do the refresh step.

Conclusions

• There are lots of ways to access remote data!
  • Using COPY with PROGRAM is simple but requires superuser powers and only reads CSV.
  • Using the http extension is simple to get data but requires parsing it yourself on the database side.
  • Using a PL/Python function results in a short and reusable piece of code, and does not require superuser powers.
  • Using the ogr_fdw extension involves some fiddly setup but is nice and clean once it is up and running. It can also read vastly more different file formats and data services.

• Can I use Postgres for building AI related apps? Absolutely.
• Can Postgres become an autonomous database with AI? Maybe.
• Should you use AI to help you troubleshoot issues? Perhaps, but better left to experts.
• Should I avoid AI when working with my database? Not at all!

We've already talked about the first item previously here. Today we want to drill into the last item on the list and look at some very practical use cases for using it. In particular ChatGPT can help you be more effective when working with your database.

The basics

ChatGPT is absolutely wonderful at the basics of helping you generate schema, queries, and basic Postgres commands. The PostgreSQL docs themselves are wonderful if you know what to look for. They are wonderful reference docs. But if you don't know what to look for (you mean common table expressions and window functions aren't in your everyday lexicon?) then you find yourself hunting and pecking. But before I get too far ahead of myself, let’s get started with the very basics. I want a schema we can use as an example:

I asked Chat GPT: Can you help me generate a schema for a multi-tenant CRM app in Postgres?

And here's what it gave me:

CREATE TABLE tenants (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    tenant_id INTEGER NOT NULL,
    email VARCHAR(255) NOT NULL UNIQUE,
    password VARCHAR(255) NOT NULL,
    first_name VARCHAR(255) NOT NULL,
    last_name VARCHAR(255) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
    FOREIGN KEY (tenant_id) REFERENCES tenants(id) ON DELETE CASCADE
);

CREATE TABLE accounts (
    id SERIAL PRIMARY KEY,
    tenant_id INTEGER NOT NULL,
    name VARCHAR(255) NOT NULL,
    website VARCHAR(255),
    phone VARCHAR(255),
    address VARCHAR(255),
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
    FOREIGN KEY (tenant_id) REFERENCES tenants(id) ON DELETE CASCADE
);

CREATE TABLE contacts (
    id SERIAL PRIMARY KEY,
    tenant_id INTEGER NOT NULL,
    first_name VARCHAR(255) NOT NULL,
    last_name VARCHAR(255) NOT NULL,
    email VARCHAR(255) NOT NULL UNIQUE,
    phone VARCHAR(255),
    address VARCHAR(255),
    account_id INTEGER NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
    FOREIGN KEY (tenant_id) REFERENCES tenants(id) ON DELETE CASCADE,
    FOREIGN KEY (account_id) REFERENCES accounts(id) ON DELETE CASCADE
);

CREATE TABLE deals (
    id SERIAL PRIMARY KEY,
    tenant_id INTEGER NOT NULL,
    name VARCHAR(255) NOT NULL,
    amount DECIMAL(10,2),
    close_date DATE,
    stage VARCHAR(255),
    account_id INTEGER NOT NULL,
    contact_id INTEGER NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
    FOREIGN KEY (tenant_id) REFERENCES tenants(id) ON DELETE CASCADE,
    FOREIGN KEY (account_id) REFERENCES accounts(id) ON DELETE CASCADE,
    FOREIGN KEY (contact_id) REFERENCES contacts(id) ON DELETE CASCADE
);

In fact it gave me even more tables including activities. I personally wanted to trim this example so our blog post wasn't so long, but you'd lose quite a bit. Now is this schema perfect? No. But did it save me a ton of time? 💯 it very much did. A quick look at changes I might make:

• Would use bigint likely for the primary key
• May or may not have ON DELETE CASCADE in a lot of cases logical deletes can be nice
• Lots of use of VARCHAR(255) in places that may not be ideal such as phone, address, email.

ChatGPT did do some things I especially like on this schema which includes:

• Putting the tenant_id on all the tables
• Setting up foreign keys

Onward to data

Let’s keep progressing, this is maybe my favorite use for ChatGPT which is generating fake data. The tables above aren't entirely complex, but it takes some time to generate data. I simply asked ChatGPT:

Can you help me generate some sample records for the above schema?

And viola:

INSERT INTO tenants (name, created_at) VALUES ('Acme Inc.', NOW());
INSERT INTO tenants (name, created_at) VALUES ('Globex Corporation', NOW());

INSERT INTO users (tenant_id, email, password, first_name, last_name, created_at, updated_at)
VALUES (1, 'johndoe@acme.com', 'password123', 'John', 'Doe', NOW(), NOW());
INSERT INTO users (tenant_id, email, password, first_name, last_name, created_at, updated_at)
VALUES (1, 'janedoe@acme.com', 'password456', 'Jane', 'Doe', NOW(), NOW());
INSERT INTO users (tenant_id, email, password, first_name, last_name, created_at, updated_at)
VALUES (2, 'bobsmith@globex.com', 'password789', 'Bob', 'Smith', NOW(), NOW());

INSERT INTO accounts (tenant_id, name, website, phone, address, created_at, updated_at)
VALUES (1, 'Acme Corporation', '<https://www.acme.com>', '555-1234', '123 Main St., Anytown, USA', NOW(), NOW());
INSERT INTO accounts (tenant_id, name, website, phone, address, created_at, updated_at)
VALUES (2, 'Globex Inc.', '<https://www.globex.com>', '555-5678', '456 Broad St., Anytown, USA', NOW(), NOW());

INSERT INTO contacts (tenant_id, first_name, last_name, email, phone, address, account_id, created_at, updated_at)
VALUES (1, 'Tom', 'Jones', 'tjones@acme.com', '555-4321', '456 Oak St., Anytown, USA', 1, NOW(), NOW());
INSERT INTO contacts (tenant_id, first_name, last_name, email, phone, address, account_id, created_at, updated_at)
VALUES (2, 'Alice', 'Johnson', 'ajohnson@globex.com', '555-8765', '789 Maple St., Anytown, USA', 2, NOW(), NOW());

INSERT INTO deals (tenant_id, name, amount, close_date, stage, account_id, contact_id, created_at, updated_at)
VALUES (1, 'New Contract', 10000.00, '2023-06-01', 'Proposal', 1, 1, NOW(), NOW());
INSERT INTO deals (tenant_id, name, amount, close_date, stage, account_id, contact_id, created_at, updated_at)
VALUES (2, 'Expansion Deal', 5000.00, '2023-07-01', 'Negotiation', 2, 2, NOW(), NOW());

INSERT INTO activities (tenant_id, type, subject, due_date, completed, deal_id, contact_id, user_id, created_at, updated_at)
VALUES (1, 'Call', 'Follow up on proposal', '2023-05-15', false, 1, 1, 1, NOW(), NOW());
INSERT INTO activities (tenant_id, type, subject, due_date, completed, deal_id, contact_id, user_id, created_at, updated_at)
VALUES (2, 'Email', 'Send contract terms', '2023-05-20', false, 2, 2, 3, NOW(), NOW());

Is it just me or does it feel like asking nicely "Can you" is a smart choice vs. just demanding?

At first ChatGPT only gave me data for 2 tenants, so I asked it for 10 tenants, and it kindly obliged. I only mention that because tenants number 3 and 4 I especially loved Wayne Enterprises and Stark Industries.

Indexing

In general the best plan for indexing is not to overcomplicate it. Wait to see some of the workload of your application, then add indexes accordingly. One of the best tools for monitoring what consumes time in your database is via the pg_stat_statements extension. But let’s still see what ChatGPT can do to help us here:

What would a good indexing scheme be for common queries if I were to build an app on top of this?

To determine a good indexing scheme for common queries, it's important to
understand the types of queries that will be executed frequently in the
application.

Here are some common queries that may be executed in a CRM app:

Retrieve all the customers for a given tenant.
Retrieve all the orders for a given customer.
Retrieve all the orders for a given tenant.
Retrieve all the products for a given tenant.
Retrieve all the users for a given tenant.
Retrieve all the tasks for a given user.
Retrieve all the tasks for a given customer.
Based on these queries, here are some suggested indexes:

Index on the customers table: (tenant_id).
Index on the orders table: (customer_id).
Index on the orders table: (tenant_id).
Index on the products table: (tenant_id).
Index on the users table: (tenant_id).
Index on the tasks table: (user_id).
Index on the tasks table: (customer_id).

It's important to keep in mind that adding too many indexes can slow down write
performance, so it's important to only add indexes that are necessary for
frequently executed queries. Additionally, you should monitor the performance of
your queries and adjust your indexing scheme as necessary.

At first glance we’ve got a decent set of inputs here and could easily create indexes on this, and ChatGPT could help us with the commands for creating those indexes. If you do follow the above approach of creating a bunch of stock indexes just make sure to clean up your unused indexes over time.

Reporting

How’s your SQL? Do you love writing SQL and use it as the first tool you reach for when you want to know something about your data? No? You’re not alone, the majority of developers aren’t SQL experts, that doesn’t mean SQL isn’t the right tool for the job. Just because you’re not an expert at SQL doesn’t mean you should be scared of it. ChatGPT is great at helping you become more familiar with SQL by expressing in plain english what you want and getting SQL out.

I want to create a report that shows me open opportunities that are expected to close this quarter, can you help me write a query that would generate that report?

SELECT *
FROM opportunities
WHERE status = 'Open'
AND close_date BETWEEN DATE_TRUNC('quarter', CURRENT_DATE) AND DATE_TRUNC('quarter', CURRENT_DATE) + INTERVAL '3 months' - INTERVAL '1 day';

In cases ChatGPT’s SQL may not be perfect, it may pick wrong columns that don’t actually exist or do the wrong calculation. While ChatGPT isn’t perfect it does offer a good method for quickly iterating often faster than having to go reference SQL that you’re looking to create.

More SQL usage

As we talked about earlier PostgreSQL has great reference documentation–IF you know what you’re looking for. But if you don’t know the name of the function to truncate a date is date_trunc then you may find yourself spending extra time to guess how Postgres references it or how that is classified in the docs. When it comes to how do I do X in SQL ChatGPT is great for this:

How do I round a date to the nearest date in Postgres?

ChatGPT:

To round a date to the nearest date in PostgreSQL, you can use the date_trunc function in combination with the round function. Here's an example:

SELECT round(date_trunc('day', your_date_column)::date) AS rounded_date
FROM your_table;

In this example, replace your_date_column with the actual name of the column that contains the date you want to round, and your_table with the name of the table where the column resides.

The date_trunc function is used to truncate the time portion of the date, keeping only the year, month, and day. The round function is then used to round the truncated date to the nearest day.

By casting the result of date_trunc to date using ::date, we ensure that the result is a proper date data type.

After executing this query, you will get a result set with a column named rounded_date, which contains the rounded dates for each row in the table.

SQL is your friend and ChatGPT is it’s side kick

You can have a lot of fun with ChatGPT and go down a lot of rabbit holes. Want to have ChatGPT create a four course menu themed around Postgres? Go for it, we won’t stop you. In fact, you can use our Postgres Playground to load data and play with Postgres queries. We’ve seen a lot of takes ranging from I’m smarter than ChatGPT because I found out something it said wrong to how it’s going change anything.

The simple reality is ChatGPT can help you work better with your database. We hope these practical examples make you a little less afraid of SQL and your database. ]]> 3210e3467efc6d7baa04240fcf9e6d3e80552721b4883b0552a8167a9efa9f29 Wed, 17 May 2023 09:00:00 EDT 2023-05-17T13:00:00.000Z 2023-05-17T13:00:00.000Z