The two talks I found the most informative were the updates on the states of pgpool-II and pgmemcache. Both those tools have that slightly frustrating combination of being really useful and a bit underdocumented relative to how complicated they are (in English at least!), so getting additional insight into them from those actually working on the code was great.
Markus's discussion of MVCC and clustering also had a fun twist to it. His talk ended with a performance analysis of his Postgres-R against pgpool-II, Postgres-XC, and PostgreSQL 9 using Streaming Replication plus Hot Standby, all used in cluster configurations to accelerate dbt2 test results. I don't quite agree with his premise there that network congestion is the most vital cluster component because "overall computing power, memory and storage capacity scale easily"--that's not always true--but it was satisfying to see that the PG9 HS/SR pairing is efficient in that regard.
The conference set aside two sessions to talk about general clustering topics in a relatively unstructured way. The more heated discussion talked about what would make PostgreSQL deployments into cloud computing infrastructure easier to deal with. That stirred up enough ideas to generate two blog entries from my coworkers already.
One of the ideas from that session I found particularly interesting was noting that if you have a deployment where nodes are added in the "elastic" way people like to discuss in relation to the cloud concept, there's a manageability gap there right now in terms of making it easy for applications to talk to that node set. If you can put pgpool-II or pgBouncer between your application and the set of nodes, you can abstract away exactly what's behind the nodes a bit right now. But now you've added another layer and therefore a potential bottleneck to the whole thing. That's the opposite of what elastic cloud deployments are supposed to be about: just adding capacity as needed with minimal management work.
A solution approach suggested was making it easier to build a database routing directory at the application level, so that apps can just ask for the type of node needed and get one to directly connect to. Nodes can just register themselves to the directory as they are brought online (or are taken down). This has similarities to some components that are already floating around. The directory lookup part you might put into LDAP; PostgreSQL servers can already announce themselves via ZeroConf AKA Bonjour. It's not hard to imagine bolting those two together, putting an application layer that does LDAP lookups connected to a routing backend that tracks available nodes via any number of protocols. As usual, the devil's in the details. Things like timing out failed nodes, distinguishing between read and write traffic (pgpool-II does it by actually parsing the SQL, which is expensive), and making the resulting directory broadcasts cached for high performance while also featuring cache invalidation are all tricky implementation details to get right.
With PostgreSQL 9.0 featuring more ways than ever to scale upward database architecture, this problem isn't going away though. I'm not sure what form yet people are going to solve it in, but it's a common enough problem that it's worth solving.
]]>One of the traditional problems in a database using MVCC, like PostgreSQL, is that a long-running query has to keep open a resource--referred to as a snapshot in the current Postgres implementation--to prevent the database from removing data the query needs to operate. For example, just because another client has deleted a row and committed, if an already running query needs that row to complete you can't actually wipe the physical disk blocks related to that row out just yet. You have to wait until no open queries that expect that row to be visible are still around.
If you have a long-running query you want Hot Standby to execute, there are a couple of types of bad things that can happen when the recovery process is applying updates. These are described in detail in the Hot Standby Documentation. Some of these bad things will cause queries running on the standby to be canceled for reasons that might not be intuitively obvious:
The lock situation is difficult to deal with, but not very likely to happen in practice for all that long if you're just running read-only queries on the standby, because those will be isolated via MVCC. The other two are not hard to run into. The basic thing to understand is that any UPDATE or DELETE on the master can lead to interrupting any query on the standby; doesn't matter if the changes even relate to what the query is doing.
Essentially, there are three things people might want to prioritize:
In any situation with Hot Standby, it's literally impossible to have all three at once. You can only pick your trade-off. The tunable parameters available already let you optimize a couple of ways:
The only one of these you can really do something about cleanly is tightening up and improving the UI for the master limiting. That turns this into the traditional problem already present in the database: a long-running query holds open a snapshot (or at least limits the advance of visibility related transaction IDs) on the master, preventing the master from removing things needed for that query to complete. You might alternately think of this as an auto-tuning vacuum_defer_cleanup_age.
The question is how to make the primary respect the needs of long running queries on the standby. This might be possible if more information about the transaction visibility requirements of the standby were shared with the master. Doing that sort of exchange would really be something more appropriate for the new Streaming Replication implementation to share. The way a simple Hot Standby server is provisioned does not provide any feedback toward the master suitable for this data to be exchanged, besides approaches like the already mentioned dblink hack.
With PostgreSQL 9.0 just reaching a fourth alpha release, there may still be time to see some improvements in this area yet before the 9.0 release. It would be nice to see Hot Standby and Streaming Replication really integrated together in a way that accomplishes things that neither is fully capable of doing on their own before coding on this release completely freezes.
]]>The problem with checkpoints is that they can be very intensive, because to complete one requires writing every single bit of changed data in the database's buffer cache out to disk. There were a number of features added to PostgreSQL 8.3 that allow you to better monitor the checkpoint overhead, and to lower it by spreading the activity over a longer period of time. I wrote a long article about those changes called Checkpoints and the Background Writer that goes over what changed, but it's pretty dry reading.
What you probably want to know is how to monitor checkpoints on your production system, and how to tell if they're happening too often. Even though things have improved, "checkpoint spikes" where disk I/O becomes really heavy are still possible even in current PostgreSQL versions. And it doesn't help that the default configuration is tuned for very low disk space and fast crash recovery rather than performance. The checkpoint_segments parameter that's one input on how often a checkpoint happens defaults to 3, which forces a checkpoint after only 48MB of writes.
You can find out checkpoint frequency two ways. You can turn on log_checkpoints and watch what happens in the logs. You can also use the pg_stat_bgwriter view, which gives a count of each of the two sources for checkpoints (time passing and writes occurring) as well as statistics about how much work they did.
The main problem with making that easier to do is that until recently, it's been impossible to reset the counters inside of pg_stat_bgwriter. That means you have to take a snapshot with a timestamp on it, wait a while, take another snapshot, then subtract all the values to derive any useful statistics from the data. That's a pain.
Enough of a pain that I wrote a patch to make it easier. With the current development version of the database, you can now call pg_stat_reset_shared('bgwriter') and pop all these values back to 0 again. This allows following a practice that used to be common on PostgreSQL. Before 8.3, there was a parameter named stats_reset_on_server_start you could turn on. That reset all of the server's internal statistics each time you started it. That meant that you could call the handy pg_postmaster_start_time() function, compare with the current time, and always have an accurate count in terms of operations/second of any statistic available on the system.
It's still not automatic, but now that resetting these shared pieces is possible you can do it yourself. The first key is to integrate statistics clearing into your server startup sequence. A script like this will work:
pg_ctl start -l $PGLOG -w
psql -c "select pg_stat_reset();"
psql -c "select pg_stat_reset_shared('bgwriter');"
Note the "-w" on the start command there--that will make pg_ctl wait until the server is finished starting before it returns, which is vital if you want to immediately execute a statement against it.
If you've done that, and your server start time is essentially the same as when the background writer stats started collection, you can now use this fun query:
SELECT
total_checkpoints,
seconds_since_start / total_checkpoints / 60 AS minutes_between_checkpoints
FROM
(SELECT
EXTRACT(EPOCH FROM (now() - pg_postmaster_start_time())) AS seconds_since_start
(checkpoints_timed+checkpoints_req) AS total_checkpoints
FROM pg_stat_bgwriter
) AS sub;
And get a simple report of exactly how often checkpoints are happening on your system. The output looks like this:
total_checkpoints | 9
minutes_between_checkpoints | 3.82999310740741
What you do with this information is stare at the average time interval and see if it seems too fast. Normally, you'd want a checkpoint to happen no more than every five minutes, and on a busy system you might need to push it to ten minutes or more to have a hope of keeping up. With this example, every 3.8 minutes is probably too fast--this is a system that needs checkpoint_segments to be higher.
Using this technique to measure the checkpoint interval lets you know if you need to increase the checkpoint_segments and checkpoint_timeout parameters in order to achieve that goal. You can compute the numbers manually right now, and once 9.0 ships it's something you can consider making completely automatic--so long as you don't mind your stats going away each time the server restarts.
There are some other interesting ways to analyze the data the background writer provides for you in pg_stat_bgwriter, but I'm not going to give away all of my tricks today.
]]>