This was a main topic of the whole conference, and several solutions have been illustrated during the day. A common theme has been that no solution fits all the use cases, and that each solution comes with its cost; hence you have to choose the solution that your use case can afford.

An obvious advantage is that you don't need to alter the well-scrutinised PostgreSQL code; a drawback is that using multiple database servers with their independent timelines you are losing some useful properties. Two examples: the partial loss of transactional semantics generates conflicts; pre-parsing each query outside the database introduces limitations on the accepted queries.

The discussion was quite interesting, and when Dimitri Fontaine mentioned remote tablespaces I started wondering around a related but distinct idea, namely: whether a lower-level approach to the problem of resource pooling would really be impractical. Before I could elaborate on the details the workshop ended, and I could only sketch the idea to some of the people that were around the whiteboard (among which Gabriele Bartolini, Nic Ferrier, Marko Kreen, Hannu Krosing, Greg Smith) together with the basic questions "does it look feasible?" and "does that resemble something you already know?".

A brief sketch: an application stack can be represented in this way

(application) --> (connection) --> (db server) --> (resources)

where the resources used by the database include storage, RAM and CPUs. The purpose is to allow the application to command more resources in order to increase capacity and speed. "Clever" applications that manage several databases can be represented as

(application) --> (connection) --> (db server) --> (resources)
      |
      +---------> (connection) --> (db server) --> (resources)

while "connection pooling" solutions can be represented as

(application) --> (connection) --> (db server) --> (resources)
                       |
                       +---------> (db server) --> (resources)

by "lower-level" solutions I mean something like

(application) --> (connection) --> (db server) --> (resources)
                                       |
                                       +---------> (resources)

which might resemble something familiar, but it is not what I am proposing here. To explain the difference I can increase the detail and write

(resources) = (virtual resources) --> (physical resources)

to represent the fact that at the lowest level you can have a non-trivial mapping between physical objects and virtual ones. For instance, SAN storage or RAID striping can provide larger virtual disks by joining together smaller physical disks. Such cases could be pictured as

(application) --> (connection) --> (db server) --> (virt.res.) --> (ph.res.)
                                                        |
                                                        +--------> (ph.res.)

My proposal is to pool resources at the database server level, so that we can have a more efficient "virtualisation" by using the knowledge of the specific use cases for each resource (CPU, RAM, disk), and at the same time we can avoid may of the difficulties of the transactional paradigm. The picture would be:

(application) --> (connection) --> (db server) --> (virt.res.) --> (ph.res.)
                                        |
                                        +--------> (virt.res.) --> (ph.res.)

The advantage is that we don't need to manage all the possible use cases for each virtual resource; we just have to manage (and optimise for) the use cases that are actually needed by PostgreSQL. For instance: WAL should still be written in local "unvirtualised" storage, the bgwriter will access local and remote resources (RAM and disk), etc.

Some final words about reliability. To operate properly the whole system needs each subsystem; partial failures are not managed, because this architecture is not redundant. It is a distributed system, but not shared. If this architecture could provide cheap and simple scalability via a virtual database server which is functionally equivalent to a physical server with larger resources, then high availability could be obtained in the standard way by setting up two identical virtual servers in a Hot Standby configuration.

Network quality has a large impact on the overall performance; this design might be useful only if you have an array of machines in the same LAN, not only for speed reasons but also because a network failure would actually be a system failure. Even with these restrictions, my opinion is that having this option would be quite useful.

This is still a sketch, to be used as a reference for further discussion. Next possible steps:

• to make a detailed list of the resource use cases
• to decide which technologies can help best in each use case
• to estimate the actual performance/development costs

Secure your place via the on-line booking registration at http://www.char10.org.

CHAR(10) stands for Clustering, High availability, Replication, though includes all forms of Parallel, Distributed and grid architectures.

An Open Source coding day is followed by a full 2-day schedule of international speakers, covering new and existing technologies of PostgreSQL.

Optional tutorial days are available from 5^th to 9^th July taught by Simon Riggs and Greg Smith.

Visit our http://www.2ndquadrant.com/postgresql-training/.

Although it is not easy to install different PostgreSQL versions on the same RedHat Linux server using RPMs, it is much simpler to install several instances of PostgreSQL (servers) and, at the same time, take advantage of the services infrastructure.

• /etc/init.d/postgresql: init script for the PostgreSQL server
• /etc/sysconfig/pgsql/postgresql: system configuration for the postgresql service

By default, PostgreSQL data directory (PGDATA) points to the /var/lib/pgsql/data directory. It is possible to change it by modifying the /etc/sysconfig/pgsql/postgresql file.

Let's suppose we want to install two PostgreSQL servers on the same RedHat Linux, by adding a second server to the default one which will be used for development purposes. We will call this postgresql-devel. It will be installed in the /var/lib/pgsql/data-devel directory and will run on the 5433 port.

We create a symbolic link to the main postgresql init script, and call it postgresql-devel:

cd /etc/init.d/
ln -s postgresql postgresql-devel

Then we start filling the postgresql-devel configuration file in the /etc/sysconfig/pgsql directory. It is important to note that the init script and the system configuration file have the same name.

cat < /etc/sysconfig/pgsql/postgresql-devel
PGDATA=/var/lib/pgsql/data-devel
PGPORT=5433
PGLOG=/var/lib/pgsql/pgstartup.\${PGPORT}.log
EOF

Once this is done, you can initialise the data directory by running: /etc/init.d/postgresql-devel initdb or simply service postgresql-devel initdb.

Similarly you can control the startup and the shutdown of the service, by running - respectively:

• service postgresql-devel start
• service postgresql-devel stop

You can add/remove the script from the startup and the shutdown of the system by using chkconfig the same way you would with other services.

The PostgreSQL wiki contains a detailed page about this topic, and I suggest that you read it along with this one. However, this simple article shows you how to easiliy integrate multiple PostgreSQL instances on the same Linux server, and manage them using the standard RedHat services infrastructure (thanks to the great job done by Devrim Gunduz).

• ext3 ordered:  39-58MB/s write, 44-72MB/s read
• ext3 journal:  25-30MB/s write, 49-67MB/s read
• xfs:  68-72MB/s write, 72-77MB/s read

In order to do this, I need a multi-CPU environment. While still waiting to get our new servers installed here in our data centre in Italy, I decided to look at Amazon's Elastic Compute Cloud (EC2) infrastructure. My intention is to do some benchmarking and spot the main differences in terms of performances between Greenplum Single Node Edition and PostgreSQL 8.4, my favourite DBMS.

If you wish to follow this article, you need to have an Amazon AWS account with a valid credit card. Do not worry, this test will only cost you a couple of dollars!

The Single Node edition of Greenplum is a freely distributed version of Greenplum which can be installed on a single node. On a multi-processor architecture, Greenplum Single Node Edition allows to create multiple segments (usually one per core) and hence to take advantage of parallel processing. Greenplum Single Node Edition can be downloaded for free from the main website.

My intention is to install it on a Large Instance running CentOS Linux 5.4 on Amazon. EC2's large instance has the following characteristics:

• 7.5 GB of memory
• 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
• 850 GB of local instance storage
• 64-bit platform

I also decided to get a 10GB volume of Elastic Block Store (1 dollar a month), which I will format using the XFS file system. This volume will contain Greenplum data directories (this time I will try with just one single volume - next time I will try with a volume per segment).

The first step is to log into your Amazon AWS management console. Get your 10GB EBS volume and then launch a large instance using the ami-ebe4cf9f AMI file (AMI stands for Amazon Machine Image), a CentOS 5.4 image file distributed by RightScale for a 64 bit architecture. You may have a different code, as I use a Europe based server.

I then attach the created volume to the instance I just started. The management console informs me that the volume has been attached on /dev/sdf. I grab the public DNS information and connect to the server via ssh as root, using my EC2 identity.

I install the YUM packages for XFS support, by running:

yum install kmod-xfs.x86_64 xfsprogs xfsdump

I create a primary partition on /dev/sdf using fdisk and format it:

mkfs -t xfs /dev/sdf1 

I then add the entry to /etc/fstab:

/dev/sdf1 /greenplum xfs noatime 0 0

and mount the partition on the /greenplum mount point:

mkdir /greenplum
mount /greenplum

Download Greenplum's Quickstart guide from the download area. Grab the URL of the 64bit RedHat installation of Greenplum and download it from the EC2 server using wget (or upload it from your computer using scp).

Follow the instructions on the quickstart guide about preparing your system to Greenplum (in particular kernel settings and limits).

Unzip the Greenplum's zip file and execute the .bin file. Answer yes to all the questions and Greenplum at the end of the process is installed in the /usr/local/greenplum-db directory.

Create the gpadmin user and set the password:

useradd gpadmin
passwd gpadmin

Prepare the data directories for the master and the segments:

mkdir -p /greenplum/master
mkdir -p /greenplum/segment1
mkdir -p /greenplum/segment2
chown -R gpadmin:gpadmin /greenplum

Become gpadmin using the su command and include source /usr/local/greenplum-db/greenplum_path.sh into gpadmin's ~/.bashrc file. Load these settings. Edit the ~/single_host_file file, add localhost to its contents and launch:

gpssh-exkeys -f ~/single_host_file

Create the ~/gp_init_config file with the following content:

ARRAY_NAME="Greenplum" 
MACHINE_LIST_FILE=/home/gpadmin/single_host_file
SEG_PREFIX=gp
PORT_BASE=50000
declare -a DATA_DIRECTORY=(/greenplum/segment1 /greenplum/segment2)
MASTER_HOSTNAME=localhost
MASTER_DIRECTORY=/greenplum/master
MASTER_PORT=5432
ENCODING=UNICODE

Finally launch:

gpinitsystem -c ~/gp_init_config

At the end of the process, Greenplum SNE edition is installed on your Amazon's EC2 server running CentOS 5.4. On this server you can test the solution at quite a reasonable price (I was on the server for 7 hours today and I spent only 3 dollars).

I will post a few more articles on this topic in the next few days, and hopefully I will be able to post the first benchmarks too. Enjoy!

Standard Long-running Queries

One of the traditional problems in a database using MVCC, like PostgreSQL, is that a long-running query has to keep open a resource--referred to as a snapshot in the current Postgres implementation--to prevent the database from removing data the query needs to operate. For example, just because another client has deleted a row and committed, if an already running query needs that row to complete you can't actually wipe the physical disk blocks related to that row out just yet. You have to wait until no open queries that expect that row to be visible are still around.

Hot Standby Limitations

If you have a long-running query you want Hot Standby to execute, there are a couple of types of bad things that can happen when the recovery process is applying updates. These are described in detail in the Hot Standby Documentation. Some of these bad things will cause queries running on the standby to be canceled for reasons that might not be intuitively obvious:

• A HOT update or VACUUM related update arrives to delete something that query expects to be visible
• A B-tree deletion appears
• There is a locking issue between the query you're running and what locks are required for the update to be processed.

The lock situation is difficult to deal with, but not very likely to happen in practice for all that long if you're just running read-only queries on the standby, because those will be isolated via MVCC. The other two are not hard to run into. The basic thing to understand is that any UPDATE or DELETE on the master can lead to interrupting any query on the standby; doesn't matter if the changes even relate to what the query is doing.

Good, fast, cheap: pick two

Essentially, there are three things people might want to prioritize:

1. Avoid master limiting: Allow xids and associated snapshots to advance unbounded on the master, so that VACUUM and similar cleanup isn't held back by what the standby is doing
2. Unlimited queries: Run queries on the slave for any arbitrary period of time
3. Current recovery: Keep the recovery process on the standby up to date with what's happening on the master, allowing fast fail-over for HA

In any situation with Hot Standby, it's literally impossible to have all three at once. You can only pick your trade-off. The tunable parameters available already let you optimize a couple of ways:

• Disabling all these delay/defer settings optimizes for always current recovery, but then you'll discover queries are more likely to be canceled than you might expect.
• max_standby_delay optimizes for longer queries, at the expense of keeping recovery current. This delays applying updates to the standby once one that will cause a problem (HOT, VACUUM, B-tree delete, etc.) appears.
• vacuum_defer_cleanup_age and some snapshot hacks can introduce some master limiting to improve on the other two issues, but with a weak UI to do that. vacuum_defer_cleanup_age is in units of transaction IDs. You need to have some idea the average amount of xid churn on your system per unit of time to turn the way people think about this problem ("defer by at least 1 hour so my reports will run") into a setting for this value. xid consumption rate just isn't a common or even reasonable thing to measure/predict. Alternately, you can open a snapshot on the primary before starting a long-running query on the standby. dblink is suggested in the Hot Standby documentation as a way to accomplish that. Theoretically a daemon on the standby could be written in user-land, living on the primary, to work around this problem too (Simon has a basic design for one). Basically, you start a series of processes that each acquire a snapshot and then sleep for a period before releasing it. By spacing out how long they each slept for you could ensure xid snapshots never advanced forward too quickly on the master. It should already sound obvious how much of a terrible hack this would be.

Potential Improvements

The only one of these you can really do something about cleanly is tightening up and improving the UI for the master limiting. That turns this into the traditional problem already present in the database: a long-running query holds open a snapshot (or at least limits the advance of visibility related transaction IDs) on the master, preventing the master from removing things needed for that query to complete. You might alternately think of this as an auto-tuning vacuum_defer_cleanup_age.

The question is how to make the primary respect the needs of long running queries on the standby. This might be possible if more information about the transaction visibility requirements of the standby were shared with the master. Doing that sort of exchange would really be something more appropriate for the new Streaming Replication implementation to share. The way a simple Hot Standby server is provisioned does not provide any feedback toward the master suitable for this data to be exchanged, besides approaches like the already mentioned dblink hack.

With PostgreSQL 9.0 just reaching a fourth alpha release, there may still be time to see some improvements in this area yet before the 9.0 release. It would be nice to see Hot Standby and Streaming Replication really integrated together in a way that accomplishes things that neither is fully capable of doing on their own before coding on this release completely freezes.

The problem with checkpoints is that they can be very intensive, because to complete one requires writing every single bit of changed data in the database's buffer cache out to disk. There were a number of features added to PostgreSQL 8.3 that allow you to better monitor the checkpoint overhead, and to lower it by spreading the activity over a longer period of time. I wrote a long article about those changes called Checkpoints and the Background Writer that goes over what changed, but it's pretty dry reading.

What you probably want to know is how to monitor checkpoints on your production system, and how to tell if they're happening too often. Even though things have improved, "checkpoint spikes" where disk I/O becomes really heavy are still possible even in current PostgreSQL versions. And it doesn't help that the default configuration is tuned for very low disk space and fast crash recovery rather than performance. The checkpoint_segments parameter that's one input on how often a checkpoint happens defaults to 3, which forces a checkpoint after only 48MB of writes.

You can find out checkpoint frequency two ways. You can turn on log_checkpoints and watch what happens in the logs. You can also use the pg_stat_bgwriter view, which gives a count of each of the two sources for checkpoints (time passing and writes occurring) as well as statistics about how much work they did.

The main problem with making that easier to do is that until recently, it's been impossible to reset the counters inside of pg_stat_bgwriter. That means you have to take a snapshot with a timestamp on it, wait a while, take another snapshot, then subtract all the values to derive any useful statistics from the data. That's a pain.

Enough of a pain that I wrote a patch to make it easier. With the current development version of the database, you can now call pg_stat_reset_shared('bgwriter') and pop all these values back to 0 again. This allows following a practice that used to be common on PostgreSQL. Before 8.3, there was a parameter named stats_reset_on_server_start you could turn on. That reset all of the server's internal statistics each time you started it. That meant that you could call the handy pg_postmaster_start_time() function, compare with the current time, and always have an accurate count in terms of operations/second of any statistic available on the system.

It's still not automatic, but now that resetting these shared pieces is possible you can do it yourself. The first key is to integrate statistics clearing into your server startup sequence. A script like this will work:

pg_ctl start -l $PGLOG -w
psql -c "select pg_stat_reset();"
psql -c "select pg_stat_reset_shared('bgwriter');"

Note the "-w" on the start command there--that will make pg_ctl wait until the server is finished starting before it returns, which is vital if you want to immediately execute a statement against it.

If you've done that, and your server start time is essentially the same as when the background writer stats started collection, you can now use this fun query:

SELECT 
  total_checkpoints,
  seconds_since_start / total_checkpoints / 60 AS minutes_between_checkpoints
FROM 
  (SELECT 
      EXTRACT(EPOCH FROM (now() - pg_postmaster_start_time())) AS seconds_since_start
      (checkpoints_timed+checkpoints_req) AS total_checkpoints 
    FROM pg_stat_bgwriter
  ) AS sub;

And get a simple report of exactly how often checkpoints are happening on your system. The output looks like this:

total_checkpoints           | 9
minutes_between_checkpoints | 3.82999310740741

What you do with this information is stare at the average time interval and see if it seems too fast. Normally, you'd want a checkpoint to happen no more than every five minutes, and on a busy system you might need to push it to ten minutes or more to have a hope of keeping up. With this example, every 3.8 minutes is probably too fast--this is a system that needs checkpoint_segments to be higher.

Using this technique to measure the checkpoint interval lets you know if you need to increase the checkpoint_segments and checkpoint_timeout parameters in order to achieve that goal. You can compute the numbers manually right now, and once 9.0 ships it's something you can consider making completely automatic--so long as you don't mind your stats going away each time the server restarts.

There are some other interesting ways to analyze the data the background writer provides for you in pg_stat_bgwriter, but I'm not going to give away all of my tricks today.