To recap, Postgres replication is the process of copying data from one database server (primary) to another (standby). While this blog post will focus on streaming replication, there is another option with logical replication as well.

The general steps are the same in Windows as in Linux but for the sake of convenience the commands and code samples in this post are tailored specifically to a Windows environment. We are assuming that you already have two servers with Postgres installed on them. Our testing environment was Postgres 11.9 on Windows Server 2012.

Keep in mind that when we say "primary," we mean the Postgres node that you already have running and have configured for read/writes (standbys are not available for writing).

Now, let’s get started.

Update Postgres.conf and pg_hba.conf

On your primary, you would need to edit your Postgres.conf to append the following settings:

wal_level = hot_standby
full_page_writes = on
wal_log_hints = on
max_wal_senders = 6
max_replication_slots = 6
hot_standby = on
hot_standby_feedback = on

You will also need to edit your pg_hba.conf on your primary and add the external IP addresses of both the servers. Yes, you will need to put both IP addresses. Below is an example:

host replication repl_user IP_of_primary/32 scram-sha-256
host replication repl_user IP_of_standby/32 scram-sha-256

This file will be replicated to the standby. When that node is promoted to a primary, it will already be configured to allow the old primary to connect.

Create a replication user called repl_user

You may be wondering: who is the repl_user? This is the user we are going to create that has the replication privilege that is needed. We should probably explain why we do this. On your primary, in the command prompt, run the following:

psql  -d postgres   -U postgres   -c "CREATE ROLE repl_user LOGIN REPLICATION ENCRYPTED PASSWORD 'secretpasswordhere';"

As you might have noticed in our configuration changes above (max_replication_slots = 6 ), we are using replication slots for this setup as well. Replication slots avoid issues with not being able to find write-ahead log (WAL) segments when a standby goes down and is brought back up at a later time.

There is one thing to keep in mind about replication slots: when the standby is down, the primary will continue to accumulate WAL since the replication slot isn't being updated. This will cause the WAL directory to fill until the standby is back up and receiving the WAL or until the replication slot is removed. Postgres 13 did bring in a parameter to help with replication slots and disk usage. Postgres 13 now has the max_slot_wal_keep_size parameter which will allow you to specify the amount of WAL replication slots retained. While this will prevent Postgres from filling your disk with WAL waiting for the missing standby, it also means WAL will be discarded and your standby will need to be rebuilt due to the necessary WAL being absent.

On the primary, create the replication slot for the standby by running the following:

psql -d postgres -U postgres  -c "SELECT * FROM pg_create_physical_replication_slot('standby1', true);"

Perform a base backup of primary to standby

After you have completed the above, you will need to wipe your existing standby. To do this you will first need to stop the cluster.

pg_ctl -D "your\data\directory" stop -mi

After stopping the cluster you will need to delete the data directory. You can do this by either using the command prompt and running:

rmdir /s \your\data\directory

Or you can delete it from File Explorer.

Once you’ve deleted the data directory on your standby, you will now run pg_basebackup on the standby to copy your primary’s data directory to it.

pg_basebackup -D \your\data\directory -Fp -R -Xs -c fast -l 'initial clone' -P -v -h IP_of_master -U repl_user

Keep in mind that you will need to enter the repl_user’s password when asked.

Update standby server settings

Lastly, once the base backup has completed for Postgres 11 and below, you will need to edit your standby’s recovery.conf and add the replication slot name. For Postgres 12 and up, the recovery.conf file no longer exists and the settings for recovery.conf are now moved to Postgres.conf. (Take a look at the release notes for more details.) Using your favorite text editing tool append the below to either your recovery.conf or Postgres.conf depending on your version.

primary_slot_name = 'standby1'

Once you’ve edited the above you can start the standby by either using pg_ctl -D [data dir] start as the Postgres user, or using the net start command if you created a service entry on your Windows server.

There are many ways to check that replication is working properly. First, in the logs you should see:

Started streaming from primary at [wal] on timeline [number]

You can also run the following commands:

• On the primary

  SELECT * FROM pg_stat_replication;
  

• On the replica

  SELECT * FROM pg_stat_wal_receiver;
  

You have now successfully set up streaming replication - congratulations!

Once again a huge shout out to Douglas Hunley and his blog post on streaming replication for Linux environments. As always, it is recommended that you review the Postgres documentation. With that said, you might be interested in checking out more blog posts about replication by my Crunchy colleagues. Hope you enjoy:

1. Synchronous Replication in Postgres
2. Guard Against Transaction Loss with Postgres Synchronous Replication

As the name suggests, ora2pg is a tool that migrates Oracle (or MySQL) databases to PostgreSQL by generating compatible SQL files As the documentation states, ora2pg “connects your Oracle database, scans it automatically and extracts its structure or data, it then generates SQL scripts that you can load into your PostgreSQL database.”

ora2pg also has a bunch of neat features that you can see below. To see all of the features and to learn more about ora2pg please visit https://ora2pg.darold.net/start.html.

• Export full database schema (tables, views, sequences, indexes), with unique, primary, foreign key and check constraints.
• Export grants/privileges for users and groups.
• Export range/list partitions and subpartitions.
• Migration cost assessment of an Oracle database.
• Migration difficulty level assessment of an Oracle database.
• Migration cost assessment of PL/SQL code from a file.

Summary

The process for installing ora2pg is fairly straightforward, but the number of steps involved tend to seem a bit daunting. Below is a summary of the steps we will go through:

1. Install necessary prerequisites
2. Build and install DBD::Oracle
3. Build and install ora2pg
4. Configure ora2pg
5. Test and use ora2pg

Demo Setup

For this demo on how to install ora2pg, we are using a Centos 7.7 Environment with Oracle 12.1 installed. We will also be using ora2pg 20.0. ora2pg can either be run on the PostgreSQL server or you can run it on the Oracle server. ora2pg was run on my Oracle server. In this demo I used a virtual box to create my environment, I will assume that you have your Oracle environment already setup.

If you don't want to install ora2pg on the Oracle server and instead on the PostgreSQL server there are a couple of extra steps that need to be done. First you will need to have PostgreSQL installed. If you need help installing PostgreSQL please look at the previously published blog post.

In your PostgreSQL server you will need to install the following prerequisites and will need to install the Oracle Instant Client SDK.

Prerequisites: After installing PostgreSQL and making sure everything is running properly you will need to run

sudo yum install -y gcc postgresql11-devel \ perl-ExtUtils-MakeMaker perl-DBI perl-DBD-Pg perl-tests

Oracle Instant Client SDK: These are the following steps that need to be taken to install the Instant Client in your PostgreSQL server.

• You will need to download the Oracle 12.2 Basic & SDK Instant Client RPMs, sqlplus suggested from the following site. https://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html

To install the packages simply run

sudo yum localinstall -y /path/to/oracle-instantclient12*.rpm

You will need to include the below in your ~/.bash_profile:

# InstantClient
export ORACLE_HOME=/usr/lib/oracle/12.2/client64
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PAT
export PATH=$ORACLE_HOME/bin:$PATH

Then source the new env vars by running

source ~/.bash_profile

You will then test the connection to your Oracle server (if you've installed the sqlplus instantclient package simply run the below):

sqlplus hr/hr1234@//192.168.99.21:1521/ORCLPDB1

Keep in mind that you only have to do the above if you are going to install ora2pg in your PostgreSQL server. After you do the above you can continue with the Build/Install DBD:oracle section.

Prerequisites for Building

Before getting started there were some prerequisites my environment needed in order to build DBD::Oracle and ora2pg. I needed to install a couple of perl modules in order to get everything up and running. I did so by running:

sudo yum install perl-devel

sudo yum install perl-DBI.x86_64

Build/Install DBD::Oracle

Before we build/install ora2pg you need to build/install the Perl module DBD::Oracle. DBD::Oracle is a Perl module that works with the DBI module to provide access to Oracle databases. If you would like to learn more about it please click the following link https://metacpan.org/pod/DBD::Oracle.

As you can see below there are two ways you can build/install DBD::Oracle. In my environment, I did it the first way. You can see in the first step to run the wget command to get the tar file from the site. If you don’t have wget and are using a Centos or RHEL OS, you can install it by running

sudo yum install wget
wget https://cpan.metacpan.org/authors/id/Z/ZA/ZARQUON/DBD-Oracle-1.76.tar.gz
tar -xvzf DBD-Oracle-1.76.tar.gz
cd DBD-Oracle-1.76
perl Makefile.PL
make && make test
sudo make install

As mentioned previously, if you don’t want to do the above steps, you can simply do the below two steps to build/install DBD::Oracle.

sudo yum install perl-CPAN
perl -MCPAN -e 'install DBD::Oracle' # configure for sudo

Build/Install ora2pg

After you have installed DBD::Oracle, you will then need to build/install ora2pg. The steps below are similar to the steps in the documentation of ora2pg which you can find in this link https://ora2pg.darold.net/. The only difference is the first step, where you need to wget the tar file from ora2pg Github archive.

wget https://github.com/darold/ora2pg/archive/v20.0.tar.gz
tar -xvzf v20.0.tar.gz
cd ora2pg-20.0
perl Makefile.PL
make
sudo make install

Configure ora2pg

After you have ora2pg built and installed you can now configure the ora2pg.conf to work with your Oracle server. The settings that you see below are the settings in my environment. The ora2pg.conf file has many configuration settings that you can change. Below are the configuration changes that you will need to apply to your ora2pg.conf in order to be able to use ora2pg. Please keep in mind that you will need to change the settings to match your environment. If you would like to learn more about the other configurations please look at the documentation: https://ora2pg.darold.net/documentation.html#CONFIGURATION.

In ora2pg-20.0/ora2pg.conf:
PG_VERSION      11
ORACLE_DSN      dbi:Oracle:host=192.168.99.21;port=1521;service_name=ORCLPDB1
ORACLE_USER     hr
ORACLE_PWD      hr1234
USER_GRANTS	    1

EXPORT_SCHEMA   1
SCHEMA          hr
TRANSACTION     READONLY

# This is needed for Oracle XE 11 with ora2pg <= v20, but safe
# to add for any version
SYSUSERS        APEX_040000

Testing ora2pg

Now that ora2pg has been configured you can finally test connectivity. By testing the connectivity you will see that you have configured ora2pg correctly, if you didn’t you will receive errors. You can test by running the below command:

ora2pg -t SHOW_VERSION -c ora2pg.conf

Estimation report

Ora2pg has an amazing feature in that it can give you an estimate of the migration cost from Oracle to PostgreSQL. It is important that you use this feature before actually migrating. Per the documentation, “To obtain a good assessment of this migration cost, Ora2Pg will inspect all database objects, all functions and stored procedures to detect if there's still some objects and PL/SQL code that can not be automatically converted by Ora2Pg.”

As it says above generating an estimation report shows if there are any database objects, PL/SQL code, etc that won’t be automatically converted by ora2pg. This means that you will need to do some manual work. This may be something you didn’t want to hear, but only in a perfect world can a tool do all the work for you. To learn more about the estimation report please look at the documentation.

There are many ways to generate an estimation report as you can see in the link above. Below is the command that will give you the report as well as an estimate of the migration cost.

ora2pg -t SHOW_REPORT --estimate_cost -c ora2pg.conf

The following link shows a great example of the report feature that ora2pg has: https://ora2pg.darold.net/report.html.

As you can see in the link this feature gives a pretty detailed report and even provides workarounds when an object isn’t available in PostgreSQL. For example, if you look for the following objects SEQUENCE and SYNONYM in the report the comments say the below.

SEQUENCE: Sequences are fully supported, but all call to sequence_name.NEXTVAL or sequence_name.CURRVAL will be transformed into NEXTVAL('sequence_name') or CURRVAL('sequence_name').

SYNONYM: SYNONYMs will be exported as views. SYNONYMs do not exist with PostgreSQL but a common workaround is to use views or set the PostgreSQL search_path in your session to access objects outside the current schema.

The report feature is valuable for planning your migration to ora2pg.

Work that you will have to do

Below I would like to provide bullet points for you to take into consideration when migrating from Oracle to PostgreSQL.

• Keep in mind that before you do the migration, your team will need PostgreSQL training. More PostgreSQL knowledge before starting to migrate will lead to a more efficient migration and you will learn more about both Oracle and PostgreSQL during the migration.
• You should do an in depth analysis of your Oracle database architecture. As mentioned previously, ora2pg doesn’t migrate every single object or function. It is best to analyze your Oracle database and determine if you can change anything in there to make your migration easier.
• Think about your applications that connect to the database. You need to make sure that you don't have a third party application that does work with PostgreSQL.
• Finally make sure that you do your research. It can be tedious but it's better to do the research than to have everything break. By reading this article, you’re already researching.
• Transactions in PostgreSQL are implemented differently than in Oracle. Oracle supports nested transactions, PostgreSQL does not. Often you can use PostgreSQL SAVEPOINTs to achieve similar behavior, but if your Oracle deployment uses PL/SQL extensively and uses nested transactions, you will likely have some manual porting to do.
• Schemas are different in PostgreSQL vs Oracle. In PostgreSQL, schema objects are used to add namespaces or scope to DB’s. In PostgreSQL, a schema is independent of a user, group or role object.
• In PostgreSQL, roles, users and groups are global objects - a role/user/group is not part of a database, they are independent of all databases in a PostgreSQL instance. A role/user/group can be granted privileges to any DB in an instance. A user instance can be directly connected to one database in a cluster at a time.
• Oracle provides hundreds of packages. If you use Oracle packages extensively, you will likely have some manual porting ahead.
• Both Oracle and PostgreSQL provide stored procedures and user-defined functions. PostgreSQL supports multiple embedded languages in the server. The most commonly used procedural server-side languages - PL/SQL in Oracle and PL/pgSQL in PostgreSQL - are similar enough that ora2pg will translate some functions or UDF’s directly. It will identify those that it can’t translate directly and identify problems.
• There are common alternatives in PostgreSQL to Oracle’s Native Dynamic SQL, EXECUTE IMMEDIATE - see using the dblink PostgreSQL extension.
• PostgreSQL provides a way to extend the core - PostgreSQL extensions. There is a large and growing number of extensions. It’s worth reviewing extensions to see if there are some that can help you migrate.

There are a lot of great reasons to migrate to PostgreSQL, and if you haven't looked at PostgreSQL lately, it would be a good time to do so again given the various improvements in PostgreSQL 12.

That said, migrating from one database to another is inherently new technology and can raise a multitude of questions. To help ease the transition, we are providing a few frequently asked questions and answers from users transitioning from Oracle to PostgreSQL, based on real life scenarios, to serve as a helpful resource.

Common Questions, Differences and Solutions

How To Install Orafce

Orafce is a useful extension that allows you to implement some functions from Oracle in PostgreSQL. For example, if you are used to DATE functions in Oracle, this extension allows you to use those functions. For additional information about Orafce: https://github.com/orafce/orafce.

Simply follow these steps to get Orafce up and running in a PostgreSQL 12 and RHEL 7 environment.

Typically, the process to build Orafce from source code is relatively user-friendly, but requires a number of dependencies. First, it is necessary to have the postgresql12-devel package installed, as it contains the binary for pg_config. Assuming postgresql12-devel is installed, you may proceed to the following steps to build Orafce and create the extension.

1. Install the dependencies:

   sudo yum -y install flex bison readline-devel zlib-devel openssl-devel wget libicu-devel install llvm5.0-devel llvm-toolset-7-clang gcc-c++
   

2. Download the full Orafce source code, available on GitHub. If you are able to connect to GitHub directly, you may use the following command:

   git clone git@github.com:orafce/orafce.git
   

3. Make sure you have pg_config in your path. You may use echo $PATH to check if /usr/pgsql-12/bin is present. If not, do the following: export PATH=$PATH:/usr/pgsql-12/bin/

4. Build the source code. From within the orafce directory, run the following command:

   make all
   

5. Install the source code. From within the orafce directory, run the following command:

   make install
   

6. Create the orafce extension inside the database. Connect to the database as a user with extension creating privileges and use the following command:

   CREATE EXTENSION orafce;
   

You will also need to have rhel-7-server-devtools-rpms enabled in order to access the llvm-toolset-7-clang package. This repo can be enabled by running the following command as superuser: subscription-manager repos --enable=rhel-7-server-devtools-rpms.

Having performed all of these steps, you will have successfully created the orafce extension for your PostgreSQL database.

How To Disable and Enable Constraints

As you may know, Oracle allows you to disable and enable a constraint as many times as needed. This is something that is not commonly done in PostgreSQL and generally isn’t recommended in any database instance. Even though Oracle allows users to disable and enable constraints, this can cause you to run into data corruption if not handled with great care.

In PostgreSQL, instead of disabling constraints, one typically creates the constraints as deferrable and then uses the SET CONSTRAINTS command to defer them. If the constraint isn't currently deferrable then it will need to be dropped and recreated as deferrable. When creating a constraint, the deferrable clause specifies the default time to check the constraint.

It may also possible to alter the constraint and make it deferrable, avoiding the need to drop and recreate. Note that all DDL in PostgreSQL is transactional, so if you wish to drop and recreate things without letting users enter potentially bad data, you can put all of the DDL in a transaction denoted by the BEGIN/COMMIT block. The tables will be locked for the transaction.

How To Disable ‘NOT NULL’ Constraint

Similar to the question above, we were asked how to disable NOT NULL constraint in PostgreSQL. In Oracle, when you run the command DISABLE CONSTRAINT it disabled all of the constraints including NOT NULL. As mentioned before it is not recommended to disable and enable constraints because the constraints can allow bad data into the database table without warning or notice. If this happens there would be no way to tell how long queries will have been returning necessarily insufficient and/or incorrect results based on bad data.

Fortunately, it is currently not possible to disable/enable NOT NULL in PostgreSQL. If you are required to do this, a better way is to drop and re-add the constraint. The command ALTER TABLE tablename ALTER COLUMN columnname DROP NOT NULL; is how you drop the NOT NULL constraint.

To re-add the NOT NULL constraint you will use the command ALTER TABLE tablename ALTER COLUMN columnname SET NOT NULL;. Re-adding the NOT NULL constraint will cause the constraint to be validated again, so this is not an instant operation. However, dropping and re-adding the NOT NULL constraints might be faster than having the constraint evaluated on every write during your process.

The GRANT Command

The GRANT command in PostgreSQL operates very similarly to Oracle. There are two basic variants to the command. It can grant privileges on a database object and grant membership to a role. A common question is how to grant create procedure or trigger to a role in PostgreSQL.

In PostgreSQL, you can grant the TRIGGER privilege to a table which gives the ability to create triggers on that table, not to use them. So, if trigger creation is all you are trying to grant, that is the only privilege you need to grant. You do not have to grant any special privileges to roles other than normal write privileges (INSERT,UPDATE,DELETE) in order to be able to then use the triggers on that table. As long as a role has normal write privileges to that table the triggers will automatically fire as needed.

All triggers in PostgreSQL use FUNCTIONS as the underlying object that performs the trigger action, not PROCEDURES. PROCEDURES code> did not exist in PostgreSQL prior to version 11 and as of version 11 they are two distinct object types.

The command syntax for CREATE TRIGGER requires some consideration. Prior to version 11, the clause to the CREATE TRIGGER command used the phrase EXECUTE PROCEDURE to name the object that the trigger will fire. As of version 11, it allows you to use the clause EXECUTE PROCEDURE or FUNCTION, however a function is still the only object allowed to be given here as the argument. As the current documentation for the command states:

"In the syntax of CREATE TRIGGER, the keywords FUNCTION and PROCEDURE are equivalent, but the referenced function must in any case be a function, not a procedure. The use of the keyword PROCEDURE here is historical and depreciated."

Additional information regarding CREATE TRIGGER can be found here.

You can also do mass grants of specific privileges on existing objects, to grant all privileges to all procedures in the given schema. So to grant trigger creation privileges on all tables in a given schema you can do:
GRANT TRIGGER ON ALL TABLES IN SCHEMA <schema_name> TO ;

Note that it does not give the privilege to then drop triggers. Only the owner of a table can drop them. This command only does that for existing objects and not any future objects that may be created.

Is it possible to drop database objects in PostgreSQL?

In PostgreSQL, only the owner of the database or a super user is allowed to drop objects.

As per the following documentation, “The right to drop or alter an object, is not treated as a grantable privilege. The owner inherits this privilege. (However, a similar effect can be obtained by granting or revoking membership in the role that owns the object) The owner implicitly has all grant options for the object, too.”

Please note: If your application or service depends on the Oracle ability to drop objects, it is possible that you might need to rewrite or reconfigure how this action is performed.

How To Check for NOT NULL

In Oracle, if you want to know which columns in a table are NOT NULL you can use the command CHECK (<col_name> IS NOT NULL). PostgreSQL does this a little differently. Here’s how to check for this.

There is a NOT NULL constraint column in the pg_attribute systems catalog. The pg_attribute catalog stores information about table columns. As stated in the documentation (https://www.postgresql.org/docs/current/catalog-pg-attribute.html), “there will be exactly one pg_attribute row for every column in every table in the database.” attnotnull is the column name in pg_attribute that represents NOT NULL constraints.

If you are wondering where other constraints are stored in the system catalog, you can look at the following documentation (https://www.postgresql.org/docs/11/catalog-pg-constraint.html).

For an example query of how to link catalogs together to find not null constraint information, the query below that shows all user tables in the database that have not-null columns along with which columns they are.
If you also want to see these columns in the system catalogs, you can remove the WHERE condition that excludes the system schemas. Many links in system catalogs are managed via "oid" values and which tables they relate to are explicitly mentioned in the documentation for that system catalog (Ex: pg_attribute.attrelid relates to pg_class.oid).

SELECT n.nspname as schemaname, c.relname as tablename, a.attname as columnname
FROM pg_class c
JOIN pg_namespace n ON c.relnamespace = n.oid
JOIN pg_attribute a ON c.oid = a.attrelid
WHERE a.attnotnull IS NOT NULL
AND n.nspname NOT IN ('information_schema', 'pg_catalog', 'pg_toast')
AND a.attnum > 0
ORDER BY 1,2,3;

ROWID, CTID and Identity columns

Oracle has a pseudocolumn called ROWID, which returns the address of the row in a table. PostgreSQL has something similar to this called CTID. The only problem is that the CTID gets changed after every VACUUM function. Fortunately, there is a good alternative for this: identity columns.

Since there is no ROWID in PostgreSQL we suggest using self-generated unique identifiers. This can be achieved in the form of identity columns. Identity columns will help you because they are generated when the row is created and will never change during the life of that row. It is important to know that the way IDENTITY is implemented means that you cannot pre-allocate values. IDENTITY also has additional logic controlling their generation/application, even though it is backed by a sequence. Use the following syntax to create an identity column:

column_name type GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY[ ( sequence_option ) ]

In the syntax you can see the additional logic as how you would generate the identity column. GENERATED ALWAYS instructs PostgreSQL to always generate a value for the identity column. If you try to insert/update a value in this GENERATED ALWAYS column, PostgreSQL will give you a warning. This is because the values are system generated and GENERATED ALWAYS means it can only have the system generated values.

GENERATED BY DEFAULT also instructs PostgreSQL to generate a value for the identity column. However, with GENERATED BY DEFAULT you can insert or update a value into the column, and PostgreSQL will use that value for the identity column instead of using the system-generated value.

We hope that this overview of a few common issues in transitioning from Oracle to PostgreSQL eases the process and gives long time Oracle users greater comfort as they evaluate PostgreSQL as an alternative.

For those just getting started with the migration consideration, no blog post on Oracle to PostgreSQL migration would be complete without a mention of ora2pg - a great open source Oracle migration tool that can help evaluate the of difficulty in your migration. Of course Crunchy Data is always here and happy to assist as well! ]]> https://blog.crunchydata.com/blog/migrating-from-oracle-to-postgresql-questions-and-considerations Thu, 23 Jan 2020 04:00:00 EST 2020-01-23T09:00:00.000Z 2020-01-23T09:00:00.000Z