{"id":1704,"date":"2018-09-02T20:04:18","date_gmt":"2018-09-03T00:04:18","guid":{"rendered":"http:\/\/blogs.sungeek.net\/unixwiz\/?p=1704"},"modified":"2018-09-02T20:04:18","modified_gmt":"2018-09-03T00:04:18","slug":"centos-7-postgresql-10-patroni","status":"publish","type":"post","link":"https:\/\/blogs.sungeek.net\/unixwiz\/2018\/09\/02\/centos-7-postgresql-10-patroni\/","title":{"rendered":"CentOS 7 + PostgreSQL 10 + Patroni"},"content":{"rendered":"<p>Lately I have been doing a lot of learning about PostgreSQL and how it can be used to replace other databases, such as Oracle RDBMS, MySQL, etc. One of the things that I have been looking to most recently is how to make PostgreSQL highly available. In Oracle you would use RAC, however in PostgreSQL you have streaming replication, but that only leaves for a single master server and technically an unlimited number of slaves. In reality most web based applications these days are 90% read and about 10% write so having tons of slaves for read-only queries is awesome, but what if your master goes down?<\/p>\n<p>That is where Patroni comes in. Patroni is a framework that handles the auto failover of the master instance of PostgreSQL between multiple servers. However Patroni alone won&#8217;t do this for you, you will need some other software as well. The other two pieces of software that I used was etcd and haproxy. etcd will be the quorum system and haproxy will be the &#8220;load balancer&#8221; so that your applications only have to have one hostname to point to.<\/p>\n<p>As a small example setup, I used 3 machines running PostgreSQL 10, 1 machine running etcd and one machine running haproxy. What the rest of this post will be about is setting up the different machines and software on them.<\/p>\n<p>&nbsp;<\/p>\n<h1>How to setup Centos 7 + PostgreSQL 10 + Patroni<\/h1>\n<h2>All Machines Software Installs:<\/h2>\n<p>The first thing to do is install CentOS 7, fully patch it and record the IP address of each machine if they are on DHCP. I used the minimal install so there wasn&#8217;t a lot of extra software on the machines.<\/p>\n<p>&nbsp;<\/p>\n<h2>PostgreSQL 10 Servers:<\/h2>\n<p>For all the PostgreSQL servers the following packages will need to be installed: gcc, python-devel, epel-release<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y gcc python-devel epel-release<\/pre>\n<p>After you have those installed (in particular the epel-release one) you can install the following : python2-pip<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y python2-pip<\/pre>\n<p>Next you need to add the PostgreSQL Yum Repo from:\u00a0<a href=\"https:\/\/download.postgresql.org\/pub\/repos\/yum\/10\/redhat\/rhel-7-x86_64\/pgdg-centos10-10-2.noarch.rpm\">https:\/\/download.postgresql.org\/pub\/repos\/yum\/10\/redhat\/rhel-7-x86_64\/pgdg-centos10-10-2.noarch.rpm<\/a><\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y https:\/\/download.postgresql.org\/pub\/repos\/yum\/10\/redhat\/rhel-7-x86_64\/pgdg-centos10-10-2.noarch.rpm<\/pre>\n<p>Once you have that RPM installed you can install the following packages: postgresql10-server postgresql10<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y postgresql10-server postgrseql10<\/pre>\n<p>&nbsp;<\/p>\n<h2>etcd server:<\/h2>\n<p>The following packages needs to be installed on the etcd server: gcc,python-devel, epel-release<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y gcc python-devel epel-release<\/pre>\n<p>After epel is installed, you can then install etcd:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">yum install -y etcd <\/pre>\n<p>&nbsp;<\/p>\n<h2>haproxy server:<\/h2>\n<p>The following packages are needed on the haproxy server: epel-release and then haproxy.<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y epel-release<\/pre>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">yum install -y haproxy<\/pre>\n<p>&nbsp;<\/p>\n<h1>Software Configuration<\/h1>\n<h2>Postgresql Servers<\/h2>\n<p>Now that the software has been installed, it is time to configure the components.\u00a0 (Everything here is ran as root)<\/p>\n<p>We will start with the PostgreSQL Servers, mine are named pg01 (10.0.2.124), pg02 (10.0.2.125), and pg03(10.0.2.126) respectively.\u00a0 The following needs to be done on each of the 3 servers:<\/p>\n<p>First some pip items:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">pip install --upgrade setuptools\r\npip install patroni\r\npip install python-etcd\r\npip install psycopg2-binary\r\n<\/pre>\n<p>Next we are going to create a systemd service for patroni. So edit the file\u00a0\/etc\/systemd\/system\/patroni.service to contain the following:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\n\r\n[Unit]\r\nDescription=Runners to orchestrate a high-availability PostgreSQL\r\nAfter=syslog.target network.target\r\n\r\n[Service]\r\nType=simple\r\n\r\nUser=postgres\r\nGroup=postgres\r\n\r\nExecStart=\/bin\/patroni \/etc\/patroni.yml\r\n\r\nKillMode=process\r\n\r\nTimeoutSec=30\r\n\r\nRestart=no\r\n\r\n[Install]\r\nWantedBy=multi-user.targ\r\n\r\n<\/pre>\n<p>Next create a \/etc\/patroni.yml file. This file will control the startup\/shutdown, etc of the postgres instance. Here is an example of mine from my pg01 server. In it, you will need to replace IP addresses for your own servers and usernames and passwords.<\/p>\n<p>&nbsp;<\/p>\n<pre>scope: postgres\r\nname: pg01\r\n\r\nrestapi:\r\n    listen: 10.0.2.124:8008\r\n    connect_address: 10.0.2.124:8008\r\n\r\netcd:\r\n    host: 10.0.2.128:2379\r\n\r\nbootstrap:\r\n    dcs:\r\n        ttl: 30\r\n        loop_wait: 10\r\n        retry_timeout: 10\r\n        maximum_lag_on_failover: 1048576\r\n        postgresql:\r\n            use_pg_rewind: true\r\n\r\n    initdb:\r\n    - encoding: UTF8\r\n    - data-checksums\r\n\r\n    pg_hba:\r\n    - host replication replicator 127.0.0.1\/32 md5\r\n    - host replication replicator 10.0.2.124\/0 md5\r\n    - host replication replicator 10.0.2.125\/0 md5\r\n    - host replication replicator 10.0.2.126\/0 md5\r\n    - host all all 0.0.0.0\/0 md5\r\n\r\n    users:\r\n        admin:\r\n            password: admin\r\n            options:\r\n                - createrole\r\n                - createdb\r\n\r\npostgresql:\r\n    listen: 10.0.2.124:5432\r\n    bin_dir: \/usr\/pgsql-10\/bin\r\n    connect_address: 10.0.2.124:5432\r\n    data_dir: \/data\/patroni\r\n    pgpass: \/tmp\/pgpass\r\n    authentication:\r\n        replication:\r\n            username: replicator\r\n            password: PASSWORD\r\n        superuser:\r\n            username: postgres\r\n            password: PASSWORD\r\n    parameters:\r\n        unix_socket_directories: '.'\r\n\r\ntags:\r\n    nofailover: false\r\n    noloadbalance: false\r\n    clonefrom: false\r\n    nosync: false\r\n<\/pre>\n<h2>etcd server<\/h2>\n<p>Now lets move on to the etcd server. The only thing on the etcd server that needs edited is the \/etc\/etcd\/etcd.conf file. Here is what I changed in mine:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nETCD_LISTEN_PEER_URLS=&quot;http:\/\/10.0.2.128:2380&quot;\r\nETCD_LISTEN_CLIENT_URLS=&quot;http:\/\/localhost:2379,http:\/\/10.0.2.128:2379&quot;\r\nETCD_NAME=&quot;etcd0&quot;\r\nETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;http:\/\/10.0.2.128:2380&quot;\r\nETCD_ADVERTISE_CLIENT_URLS=&quot;http:\/\/10.0.2.128:2379&quot;\r\nETCD_INITIAL_CLUSTER=&quot;etcd0=http:\/\/10.0.2.128:2380&quot;\r\nETCD_INITIAL_CLUSTER_TOKEN=&quot;cluster1&quot;\r\nETCD_INITIAL_CLUSTER_STATE=&quot;new&quot;\r\n<\/pre>\n<p>Some of the above lines may be commented out, if so uncomment them and replace the values.<\/p>\n<p>HAProxy Server<\/p>\n<p>Now to config the HAProxy Server. Replace the \/etc\/haproxy\/haproxy.cfg with the following:<\/p>\n<p>&nbsp;<\/p>\n<pre>global\r\n        maxconn 100\r\n        log     127.0.0.1 local2\r\n\r\ndefaults\r\n        log global\r\n        mode tcp\r\n        retries 2\r\n        timeout client 30m\r\n        timeout connect 4s\r\n        timeout server 30m\r\n        timeout check 5s\r\n\r\nlisten stats\r\n        mode http\r\n        bind *:7000\r\n        stats enable\r\n        stats uri \/\r\n\r\nlisten postgres\r\n        bind *:5000\r\n        option httpchk\r\n        http-check expect status 200\r\n        default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions\r\n        server postgresql_pg01_5432 10.0.2.124:5432 maxconn 100 check port 8008\r\n        server postgresql_pg02_5432 10.0.2.125:5432 maxconn 100 check port 8008\r\n        server postgresql_pg03_5432 10.0.2.126:5432 maxconn 100 check port 8008<\/pre>\n<p>&nbsp;<\/p>\n<p>N.B. The logging setup in the haproxy config requires that you set up rsyslog to allow logging to local files and to local2.*<\/p>\n<p>&nbsp;<\/p>\n<h1>Starting it all up<\/h1>\n<p>So assuming that I haven&#8217;t missed anything above, and there are no typo&#8217;s on my or your part, we can start starting things up. But first some notes. Something in SElinux breaks some of this and I haven&#8217;t had enough time to look in to it, so disable SElinux for now (setenforce 0). In addition I did not add any ports to the firewalld, as this was just a test setup, so I disabled the firewall on all 5 machines. You can leave it enabled and then using the configs above add the appropriate ports to be allowed between the appropriate machines.<\/p>\n<p>So easy starts first:<\/p>\n<p>Start the etcd server: <\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">systemctl start etcd<\/pre>\n<p>Start the haproxy server: <\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">systemctl start haproxy<\/pre>\n<p>Now for the PostgreSQL servers there are a few things to do before we start them. On each PostgreSQL host do the following:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\nmkdir -p \/data\/patroni\r\nchown -R postgres:postgres \/data\r\nchmod -R 700 \/data\r\n<\/pre>\n<p>Now we can start Patroni and see if everything works. When you start Patroni, it will start PostgreSQL in the background and create the first database and set the username and password for the postgres user to what you specify in the \/etc\/patroni.yml file.<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">systemctl start patroni<\/pre>\n<p>On the first host that you start Patroni on if you run systemctl status patroni you should see that it is running and active. In the \/var\/log\/messages you should see something like :<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nSep 2 19:53:08 pg01 patroni: 2018-09-02 19:53:08,628 INFO: Lock owner: pg01; I am pg01\r\nSep 2 19:53:08 pg01 patroni: 2018-09-02 19:53:08,636 INFO: no action. i am the leader with the lock\r\n<\/pre>\n<p>When you start the two slaves you should see something similar to this in the \/var\/log\/messages:<\/p>\n<pre class=\"brush: plain; title: ; notranslate\" title=\"\">\r\nSep 2 19:54:17 pg02 patroni: 2018-09-02 19:54:17,828 INFO: Lock owner: pg01; I am pg02\r\nSep 2 19:54:17 pg02 patroni: 2018-09-02 19:54:17,829 INFO: does not have lock\r\nSep 2 19:54:17 pg02 patroni: 2018-09-02 19:54:17,836 INFO: no action. i am a secondary and i am following a leader\r\n<\/pre>\n<p>Now if you switch to the web interface of HAProxy you should see something like this:<\/p>\n<figure id=\"attachment_1718\" aria-describedby=\"caption-attachment-1718\" style=\"width: 1254px\" class=\"wp-caption alignnone\"><a href=\"http:\/\/blogs.sungeek.net\/unixwiz\/wp-content\/uploads\/2018\/09\/haproxy.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1718\" src=\"http:\/\/blogs.sungeek.net\/unixwiz\/wp-content\/uploads\/2018\/09\/haproxy.png\" alt=\"\" width=\"1254\" height=\"452\" srcset=\"https:\/\/blogs.sungeek.net\/unixwiz\/wp-content\/uploads\/2018\/09\/haproxy.png 1254w, https:\/\/blogs.sungeek.net\/unixwiz\/wp-content\/uploads\/2018\/09\/haproxy-300x108.png 300w, https:\/\/blogs.sungeek.net\/unixwiz\/wp-content\/uploads\/2018\/09\/haproxy-768x277.png 768w, https:\/\/blogs.sungeek.net\/unixwiz\/wp-content\/uploads\/2018\/09\/haproxy-1024x369.png 1024w\" sizes=\"auto, (max-width: 767px) 89vw, (max-width: 1000px) 54vw, (max-width: 1071px) 543px, 580px\" \/><\/a><figcaption id=\"caption-attachment-1718\" class=\"wp-caption-text\">haproxy web interface<\/figcaption><\/figure>\n<p>What this image shows is that the pg01 server is the active master server. If you were to shut down pg01, then one of the other two will become the new master and remain the master until you either promote another server or the new master goes down.<\/p>\n<p>&nbsp;<\/p>\n<p>Now that everything is running, you would point your clients to the haproxy address on port 5000.<\/p>\n<h1>Closing Comments<\/h1>\n<p>While this describes how to setup an HA Environment for PostgreSQL, there are 2 single points of failure with this example setup. One is the etcd server, which should be clustered, the other is the haproxy which should be clustered as well. In addition I didn&#8217;t cover setting up read-only slaves which I may do at some point in the future.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Lately I have been doing a lot of learning about PostgreSQL and how it can be used to replace other databases, such as Oracle RDBMS, MySQL, etc. One of the things that I have been looking to most recently is how to make PostgreSQL highly available. In Oracle you would use RAC, however in PostgreSQL &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/blogs.sungeek.net\/unixwiz\/2018\/09\/02\/centos-7-postgresql-10-patroni\/\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;CentOS 7 + PostgreSQL 10 + Patroni&#8221;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[660,35],"tags":[383,668,667,666,477],"class_list":["post-1704","post","type-post","status-publish","format-standard","hentry","category-linux","category-postgresql","tag-centos","tag-etcd","tag-haproxy","tag-patroni","tag-postgresql"],"_links":{"self":[{"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/posts\/1704","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/comments?post=1704"}],"version-history":[{"count":15,"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/posts\/1704\/revisions"}],"predecessor-version":[{"id":1720,"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/posts\/1704\/revisions\/1720"}],"wp:attachment":[{"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/media?parent=1704"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/categories?post=1704"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.sungeek.net\/unixwiz\/wp-json\/wp\/v2\/tags?post=1704"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}