2ndQuadrant, Professional PostgreSQL - Greg's PlanetPostgreSQL

Intel SSDs: Lifetime and the 320 vs. 710 Series

2012-03-23T22:03:25Z

This week I've been digging deep into PostgreSQL storage hardware again. Since I'm giving a conference talk on database storage in Austin and in the DC area next week, it seems like a good time for me to actually know the material. One of the most common questions here is "what's the cheapest SSD I can put my database on?", with the implied hope "...without losing it all the time". Last year the first inexpensive answer to that appeared on the market, and I suggested people take a look at Intel's 320 series drives. With 217 days of runtime on my first 320 drive here, and Intel's 3rd generation storage line filled out with the more enterprise oriented 710 Series now, it's worth reviewing how that turned out.

It wasn't long after the 320 series drives were introduced that people started reporting a firmware problem with the drive, where it did things like report a capacity of 8MB after a restart along with "BAD_CTX 0000013x" errors. A firmware update to fix that was released. There's still some claims of continued problems floating around. You have to expect some percentage of any product are going to be bad, and the later production of this drive (after the big bug was fixed) don't seem above the usual risk level in hard drives to me. With the warranty here extended to 5 years (unless you're using it at 'enterprise usage levels'), I think that Intel would be getting killed if the reliability on these was as bad as some people claim.

The reason behind the usage level caveat is the main thing worth talking about here. The long version of the warranty suggests "The media wear-out indicator reports a normalized value of 100 (when the SSD is brand new out of the factory) and declines to a minimum value of 1. When the value reads 1, this indicates that the SSD is reaching the wear-out limit". Here's what my first 320 looks like so far:


[root@toy ~]# smartctl -a /dev/sdc
=== START OF INFORMATION SECTION ===
Device Model:     INTEL SSDSA2CW120G3
...
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  3 Spin_Up_Time            0x0020   100   100   000    Old_age   Offline      -       0
  4 Start_Stop_Count        0x0030   100   100   000    Old_age   Offline      -       0
  5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       5225
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       58
170 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       34
225 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       201389
226 Load-in_Time            0x0032   100   100   000    Old_age   Always       -       2687040
227 Torq-amp_Count          0x0032   100   100   000    Old_age   Always       -       0
228 Power-off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       314526
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   099   099   000    Old_age   Always       -       0
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       201389
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       133021

Not all of these attributes are labeled correctly, and some are "Unknown"; all the gory details are in the product specifications. You can see the Media Wearout above; here's the raw values for other interesting ones, formatted so they're more blog-friendly:


ID# ATTRIBUTE_NAME          RAW_VALUE
  9 Power_On_Hours          5225
 12 Power_Cycle_Count       58
192 Power-Off_Retract_Count 34
225 Load_Cycle_Count        201389
241 Total_LBAs_Written      201389
242 Total_LBAs_Read         133021

192 (hex C0) is "Power-Off Retract Count". That's how many unsafe shutdowns the drive has been through, which are the situations where the battery backed cache in the drive has been triggered. With 34 of them here, you can see I've tried to get this drive to die that way.

The first interesting wear figure is 225 (hex code E1) which Intel's documentation describes as "Host Writes". The units for that are 32MB. If you look carefully, you'll see that's the same value given in 241 "Total LBAs Written". That suggests the LBA unit for the drive is also 32MB, which I double-checked last year. At 32MB each, my write value of 201389 means I've written 6.15TB to this drive.

Now, computing the true lifetime of an SSD depends on a couple of magic values, like the "write amplification" of your workload. That suggests how often your workload forces small bits of data out to flash, using up some of the NAND cell lifetime faster than it might otherwise last. These numbers are really hard to estimate. The most realistic way is figure this out is to run a workload simulation after resetting the drive's internal counters, then see just how much you burned through. The process is walked through with an example at "Measuring How Long Your Intel SSD Will Last", and it's not too hard to translate that example (which uses Intel's SSD Toolbox software) into a set of of smartctl commands if you're on Linux--the article even uses smartctl for the counter reset part.

The official documentation is this is Intel's Enterprise Server addendum, and here we finally find some hard numbers about the expected life of these drives. My 120GB drive is said to have a "write endurance" of 15TB. A pessimistic look at my sample drive here would check total writes and say that, having written over 6TB, I've gone through 40% of the drive lifetime. But write endurance doesn't work that way; the firmware is constantly doing tricks to extend the life of the drive. Intel's official number they sometimes tie the warranty to, the Media Wearout, is showing 99% left! If that's true--I've only used 1% of the drive's lifespan--then I might manage 600TB of writes before this one really dies on me.

So what's the story with the true Enterprise lifetime 710 Series drives? Those almost the same drives as the 320 series ones, with three significant changes. First, they're said to use higher quality flash, probably with the same sort of "put the best tested chips first in the expensive models" approach Intel is said to use on their CPU production--what's sometimes called binning. Second, the drives are overprovisioned with a lot more unused flash compared to the 320 series models, and unused flash really helps extend longevity. Finally, they don't claim the capabilities to be quite as good. Random write IOPS numbers on the 710 series drives are lower; my 120GB 320 series drive is specified at 14K write IOPS, while the 100GB 710 series only aims for 2700. The drive doesn't claim to support lots of tiny writes and still last for years, which means it's aimed at a different set of write amplification expectations. Similarly, the 710 series drives don't refresh the stored cells in the same way. The downside there is that 710 models are only specified to retain their data for 3 months. That's probably fine for data center use, but that wouldn't be very acceptable to the more consumer oriented market the 320 series is sold to.

The end result of that, and how the 710 compares to the 320 series drives, is nicely summarized in the "Write Endurance" table in the Tom's Hardware Review. Instead of the 15TB endurance number my 320 drive specifies, the similar 100GB 710 series model aims for 500TB. That's just over 30X as long. In the real world, there may not be that big of a difference, as shown by the projected 600TB figure I'm seeing out of my 320 drive so far. But Intel's aiming at conservative engineering lifetimes on the specification sheets, and by that measure the storage cells 710 will last longer; the 320 models only may last longer. And an expected lifetime 30X as long is something some people are surely willing to pay the 710's price premium for.

Using the PostgreSQL System Columns

2012-01-31T21:53:43Z

There are a few parts of the PostgreSQL internals that poke out usefully if you look in the right place for them. One useful set to know about are the System Columns, which you can explicitly request but don't see by default. For example:

psql -x -c "SELECT oid,* FROM pg_class LIMIT 1"

There is no column named oid in the pg_class table, but it's there if you ask for it. The oid used to be relied on more heavily in PostgreSQL as a way to identify rows. That's not true for regular tables anymore, and you really don't want to start doing that for your own tables. OIDs are mainly useful now when joining parts of the System Catalog together. A good example is the Disk Usage query. If you want to find the namespace a table is in, you need to know you can ask for its OID. It's possible to get some of this data out of more portable views like information_schema.tables. But many of the useful things in this area are PostgreSQL specific. Sometimes I see people starting with the information_schema views and joining against other tables using its text name fields, such as the listed table_name. That approach has several edge cases that don't work out correctly; not handling TOAST columns is a common example. That makes them more prone to breaking on you later, probably after your system has gone into production, than an OID based join.

There is also a tableoid system column. As described in the documentation, its main use case is identifying which partition a row come from. That's not a great thing to be driving application logic from, but it can be useful for monitoring or troubleshooting purposes. For example, if you SELECT rows from the parent table in a partitioning inheritance scheme, it's normally expected that no rows will actually be stored there. Checking the tableoid is one way to confirm that. You might confirm that your INSERT/UPDATE trigger is moving rows to the right place using tableoid as well. It's possible to do that for each individual partition section, but running a query against the parent will make sure you hit every row in the table.

Another internal column related to uniquely identifying rows is the ctid. The ctid is a direct pointer to the physical block (using PostgreSQL's 8K page size) and position of a row. ctids are a pair of numbers, and the first row will be (0,1). While this is the fastest way to find a row more than once in the same transaction block, these numbers are not stable in the long term. Any UPDATE and some maintenance operations will change them. One thing you can use these for is finding duplicate data in a table. Let's say you're trying to add a unique constraint, but one row in the table is duplicated 3 times, which blocks the unique index from being created. When rows are identical in every column, you can't write any simple SELECT statement to uniquely identify them. That means deleting all of them but one copy requires some annoying and fragile SQL code, combining DELETE with LIMIT and/or OFFSET--which is always scary. If you use the ctid instead, the implementation will be PostgreSQL specific, but it will also be faster and cleaner. See Deleting Duplicate Records in a Table for an example of how that can be done.

The other system columns all relate to transaction visibility: xmin, cmin, xmax, cmax. When you delete a row in PostgreSQL, it isn't eliminated from disk immediately. It's possible that some other query that's executing at the same time will still need to see that row, and the transaction isolation in PostgreSQL worries about such things. If you ever want to learn how that isolation works, the way the Multiversion Concurrency Control (MVCC) implementation is handled, you can watch parts of it happen. Just open transactions in two different sessions, UPDATE/DELETE in one of them, and then look at those rows in the other. You can still see them in the session where they weren't touched, but they'll be marked to expire in the future via their xmax being set. To really pull that all together, you also need to know about some of the System Information Functions. txid_current() is the most useful for this sort of learning experience, it provides a reference point for the always increasing system transaction ID. You can find a more detailed exploration of using these functions and system columns in Bruce's MVCC Unmasked talk. The "Routine Maintenance" chapter of my book also shows examples how how MVCC works through the perspective of the system columns.

Global trends in deploying PostgreSQL

2011-11-14T18:08:39Z

This year's conference lineup led me all over the world, a giant rectangle triangle going from the west coast of the US, north to Canada, east to the UK and Amsterdam, then ending south in Brazil. I've now locked myself in to focus on the 3rd CommitFest for PostgreSQL 9.2, which began a few days ago. Check out the 2011-11 section of the CommitFest tracker to see what changes have been submitted, we're always looking for new volunteer patch reviewers.

Talking to people deploying PostgreSQL in several countries during a short span of time has given me some interesting perspective on where the project is at. I follow a lot of adoption trends in the US, and some of those I assume are quirks in how business is done in this country. But when I hear the same sort of feedback from people in all four of the countries I've been to this year, too, it's clear this is a larger issue.

The first thing I'm seeing a surprising amount of is satisfaction with the feature set in PostgreSQL. A few years ago, conversations about what you could and couldn't do with PostgreSQL usually stalled on one of a few common requests. There's a good survey of PostgreSQL feature feedback at User Voice. 13 important features originally on that list have been closed already, with Index Only scans as the next expected to fall in the upcoming 9.2. PostgreSQL now includes regular and synchronous replication as of 9.1. pg_upgrade has been getting an increasing amount of testing that proves it works for many in-place upgrade scenarios. Extensions are dramatically easier to use now.

It seems the total feature set has crossed the threshold where PostgreSQL is good enough for a whole lot more deployment situations than it used to be. What I'm hearing from people all over the world now is that the basic feature set and performance of PostgreSQL isn't failing the "checkbox test" so often anymore, where business people require certain things before they'll even consider a database. There are some major wants that are some distance off, such as materialized views and better OLAP support (cube/rollup/etc.) And using partitioning for bigger data sets is harder than people would like. But these are all things that are only needed for larger deployments, and some workarounds exist if you're willing to work at them.

If the feature set isn't holding back as many deployments now, what is? Well, the next thing I've been hearing everywhere is on that survey list too: better administration and monitoring tools. You really need a whole open-source stack to monitor PostgreSQL right now, from OS+database trending to query log analysis. It's fine for these tools to live outside the database core, but some changes are clearly needed to make such tools easier to write. For example, the one built-in tool that allows query monitoring is pg_stat_statements, and the limitations preventing it from being useful to most people are so obvious we've gotten two submissions to improve it in the last month.

There are a few projects that aim at the monitoring/administration problem. EnterpriseDB's PostgreSQL Enterprise Manager, Cybertec's pgwatch, OmniTI's Reconnoiter, the suite of smaller tools from End Point, and even the text UI of pg_statsinfo all hit the edges of this problem. What I hear when I have my advocacy hat on is that the community needs a major open-source project bigger than any of these to make database monitoring easier. That's now one of the major distinguishing features the commercial competition has. Getting enough of the people developing in this area all pointing in the same direction and working together is a big challenge though.

On a related note, now that the underlying features are there, it seem making replication easier to monitor and setup is a major issue too. There are so many choices in replication technologies available for PostgreSQL it's easy for new people to get overwhelmed by them all. And the documentation guides around this area are still filled with a lot of complications that aren't even really necessary to get started at this point. It's easy for newcomers get dragged into details like how old style archiving works as a precusor to setting up even basic replication, despite that they're using the easier features in the current PostgreSQL instead. This area still has some work in the core database happening in 9.2, and it will be important for the community to create replication guides that include current information covering both 9.1 and that release. What I'm hear from every country I visit now is "I need material to help me compete against the idea of using Oracle RAC".

The last of the global trends that have really jumped out at me is how companies everywhere are reinventing the development process around database applications. In some places, mostly bigger companies and government installations in particular, the expected staff "stack" is business as usual; it hasn't changed in a long time. New applications go from Developer to DBA to systems administrator. Management ideas like DevOps are catching on to improvement interface between these roles, but not really upset its basic structure. Everywhere I go now, I'm seeing everything but the developer role being squeezed out. ORM-driven development is eliminating the DBA's role in database design. Managed application hosting platforms are wiping out the systems administrator role. Startups with an idea for a web application go right from developer to deployment, and happily this is increasingly happening with a PostgreSQL backend in the database role. There isn't even the perception that DBA-like help might be needed until the application grows quite a bit. I'm seeing the need for better database specific optimization skills than a typical developer has being deferred until the application has tens of gigabytes of data to sling around.

Being able to deploy small PostgreSQL installs and grow them to a reasonable size without specialized DBA knowledge is a great thing as far as I'm concerned. The exact advances in things like ORMs that have allowed reaching this point across the world are a topic that deserves its own long discussion. I'm going to cut this off here and return to that later. In this country, there's some concrete work around the 9.2 release that needs to get done this month.

Apple's Lossless Audio Codec and Software Patents

2011-10-28T12:07:45Z

Today my mailbox was crowded with some Apple news. The source code to the Apple Lossless Audio Codec, encoders and decoders, was released to the world. A few open-source projects such as FFmpeg and FAAC has already had support for the resulting .m4a file; this gives them an official release to validate against. As a well documented audio snob, I'm known to be a fan of lossless audio, with a few hundred CDs here ripped into FLAC. Unfortunately, this source code drop doesn't change anything for me, and that's all because of software patents. Right now I consider such patents to be the greatest risk to software I work on like PostgreSQL, so I spent some time looking into just what Apple has done here, as a data point on that topic.

If your baseline is MP3 files, Apple Lossless appears to be a step forward as far as licensing goes. All MP3 playback requires a patent license, while playback of Apple's format does not--only the encoding side need be licensed. That's not my baseline though. FLAC has specifically avoided using any known patented technology, so it's the clear winner in being clean of patent issues.

You can't even read the just released source code without being shown the door that leads to Apple's patents. ALACEncoder.cpp pushes you that way when you read it, saying "The relevance of the ALAC coefficients is explained in detail in patent documents." So you can't fully understand what this code does unless you dip into the patent description. That's a big sign of trouble. I'm not sure exactly which patent they are referring to; it may be Determining scale factor values in encoding audio data with AAC.

I'm not sure because I don't read patent descriptions if I can avoid it, due to how damages are tripled with willful infringement. That rule puts open-source developers in a weird place. The act of researching which patents your free implementation might infringe on can have a wildly negative return on investment. If it's impossible to implement the idea without infringing in the patents you find, which can easily be the case given the ridiculously low bar for grating such patents, if a lawsuit does happen you'd be better off not knowing about that risk when it starts. On the PostgreSQL mailing lists, mentioning how a patented implementation of something works is one of the few things that will get you a warning and potentially blocked from the lists. Knowingly implementing a patented idea can be a very expensive mistake for the project to make.

Now, you can claim this problem has gone away for this bit of software due to how Apple has released this code under the Apache 2.0 license. If you build something using the Apple Lossless code, and you distribute the result under the Apache 2.0 license, you get the protection of its Grant of Patent License. You'll only lose it if you sue someone in a way that involves this code+patent combination. That doesn't necessarily protect manufacturers of hardware who want to include this capability, as they may not want to license the result that way. I don't really care about them though. People who want to take advantage of open-source code but not contribute back to can worry about their own legal issues, they're not my concern.

My real issue with using this format is that it implicitly approves of business practices around software patents I find hostile to my own work, and that leads me back to the patent-less FLAC again. Apple used to more often be the victim of valid and frivolous patent claims, but increasingly they've become the originator of them instead. In mid 2006, Apple was sued over patent issues by Creative Technology; they settled paying Creative $100M dollars. Since then, patent issues have increasingly been part of their well exercised legal arm.

Starting last year, Apple has seriously turned around in how it handles patents--it's now aggressively enforcing its patents rather than just being sued for violations. Steve Jobs issued a warning shot about that, saying "competitors should create their own original technology, not steal ours". Considering it had only been four years before then that Apple was sued--and paid out in a big way--for stealing other people's patented ideas, that came off as rather hypocritical to me.

The company has done well staying ahead of its competitors by production innovation, rather than court fights. iPod competitors failed to gain traction because their products weren't as good, not because they couldn't steal the design. Apple's iPad should have the same property; the knock-offs are not selling because they're not as good. The only real threat to their product line right now are how Android phones are displacing the iPhone. Dumping so much money into offensive lawsuits is burning up money that could be used for real product advances there instead. It's a shame they've resorted to this tactic.

At this point, a cautious person would avoid using technology encumbered by Apple's patents, as they clearly have the means and intent to sue for violations. And someone who values open source projects should avoid patented approaches even when they are freely licensed. Whether or not individuals using Apple Lossless is particular are exposed to problems here is missing the big picture.

As an advocate for free software, I reflexively pick the less patent encumbered approach to any problem, using that as the tie breaker for decisions that are otherwise even. Encoding into and helping popularize Apple Lossless may be legal now. But I'll keep encoding into FLAC, copying the result onto my Sansa Fuse player, and avoiding their entire music ecosystem. Software patents are too dangerous to implicitly endorse them if it can be avoided, and here they easily can.

Sync rep scaling

2011-10-26T16:31:04Z

I'm almost done with this year's crazy conference season schedule, just Brazil's PG.BR next week left. All of my recent presentations are now available at our talks page. You'll also find many of the talks from our CHAR(11) conference this summer there for the first time. Those were unavailable for a while due to an unfortunately timed web site change.

I like to do some original research for my talks, and this year that included a look into Synchronous Replication and Durability Tuning in PostgreSQL 9.1, specifically the performance side. At last week's PG.EU I gave an updated version of this talk (with Simon Riggs), including a bit more info than I had available during the Postgres Open presentation on the same topic.

The good news/bad news performance results are nicely summarized by the "Group commit magic" graph on slide 17. There I was replicating across the Atlantic Ocean, crossing from my home here in Baltimore over to the conference site in Amsterdam. When doing synchronous replication, the speed of light determines how fast any single client can commit. You can only reach about 50% of that round trip time given current (and expected future) technology here. The efficiency rate of the fiber-optic cables used is not perfect, and tasks like routing add some overhead too.

With that sort of distance, the round trip time is at least 100 milliseconds. What this means is that no one client can commit more than 10 times per second. This shows just how difficult sync rep is to run well, the maximum rate drops fast if you expect to leave your local data center to commit. Light turns out to be really slow compared with what many people expect.

The great thing I proved in that slide is that the efficiency when multiple clients are committing at the same time scales almost linearly. If you want 1000 transactions/second over this sort of distance, you can get that--but you'll need just over 125 clients going at once to do it. Each one of those clients will be seeing 10 commits/second and 100 millisecond latencies (and higher for a small percentage, peak latency in the test was closer to 600ms). But each commit reply will be acknowledging a pile of clients at once, just by sending a small packet with an updated "committed up to this point" response.

When the speed of light turns out to be your bottleneck, there's not much you can do about that attacking directly. Some people who want to reach higher rates might architect their systems with many smaller clients, as I've shown in this example. I was able to reach my goal of 2000 INSERT commits/second here, but it took 275 clients to get there.

The other great thing about how PostgreSQL implements sync rep is that it's controlled per transaction. Once you see how expensive the commits are, if that's too high for some of your data, you can always tweak that for higher performance just by disabling sync rep for some transactions. Having such fine-grained control over synchronous commits is a unique feature to PostgreSQL, allowing something like a Quality of Service suggestion to the database. The PostgreSQL code as of 9.1 really has an unprecedented range of trade-offs here. You can go for faster but not very durable at all (with unlogged tables), locally durable but not guaranteed to a remote data center at a medium speed, all the way up to fully synchronous and very expensive to commit. It's possible to argue that other database choices are better at one end of this range. You might use MongoDB for higher speeds at the low durability range, and Oracle for their better tested sync rep capabilities (I saw better tested simply because the PostgreSQL 9.1 code is very new relative to Oracle's implementation).

PostgreSQL started in the middle here, and with 9.1 it's expanded nicely toward both ends of the spectrum at once. It's now providing options for higher and lower durability at the same time, in one database, and with the speed/durability trade-off adjustable for every transaction. Building one size fits all software is really hard, and the new features in 9.1 nicely push out capabilities here for several popular use cases, all at the same time, and only when you want to pay for them.

Limitations removed in PostgreSQL 9.1

2011-09-14T12:58:50Z

The release of PostgreSQL 9.1 lining up with this week's Postgres Open conference has resulted in my getting asked the same question by several people now: "how long should I wait before deploying 9.1?" It's an interesting question that highlights some of the benefits and hazards of the PostgreSQL release schedule, and one I have a very strong opinion on: you should consider 9.1 for new projects instead of 9.0 starting right now. Read on and I'll detail why I feel that way.

Most changes to a software project are sorted into two main groups: bug fixes and features. PostgreSQL is very good about backporting bug fixes to all supported versions of the program, when practical to do so. But feature changes aren't done in older releases. Stability is highly prioritized in the stable versions--sometimes to the point where some very annoying things that could be fixed aren't.

The last release, PostgreSQL 9.0, started its life as version 8.5. When it became obvious several big, disruptive features were going to be included, the version number was bumped in part to make people a little more wary of the release. The bug count over the last year hasn't been too terrible, but there have been plenty of serious bugs. April's 9.0.4 corrected both a nasty data loss bug (for users of pg_upgrade) and a case where new database crash recovery code could introduce a problem even when not using the feature it was modified to support.

There is a third category of changes here though: removing limitations. Sometimes the new "feature" added by a new version is that a part of the database code is made to support new cases, because they didn't do everything people really wanted . These are considered new features and not backported, even though it's often the case that people are running into the limitation on older versions.

While all the genuine new features are great, too, what makes 9.1 a really compelling upgrade from my perspective is how many of these limitations are blown away by it. Here's my short list of limitations removed that I've run into on multiple installations of 9.0 and earlier versions now:

Queries using MIN or MAX against a partitioned table can now use an index to return that value. This is a showstopper for sites that run into it, one that normally doesn't show up until after you've got a large amount of data into PostgreSQL; then it hits you by surprise. The most common way I've run into this are sites that monitor the last time/entry inserted into some busy table. SELECT now() - max(timestampfield) FROM t is a good way to tell how long it's been since the last record was inserted into your database, to set an alarm if that grows too big. On a busy site that means something has probably broken and stopped importing records. But if you use this approach, and you switch to partitioned tables...you're toast. Until 9.1, you had to rewrite the query to fix this. And in some places, application changes like that take a huge amount of QA work to do. Now it just works.
Replication lag time can easily be monitored using pg_stat_replication. All of our 9.0 deployments integrate our repmgr tool in order to provide this capability. In 9.1, you get that part in the core server. (We're adding some new stuff to repmgr to keep it relevant--it's always going to be where between release features we're working on can incubate at)
Systems using Hot Standby to run long reports against standby systems can now set a large value for max_standby_archive_delay and max_standby_streaming_delay. Changing the mechanics of these two parameters was one of the very last code changes in 9.0 before its feature code freeze. And subsequent testing missed the fact that the maximum value you can set these two only allows 35 minutes of delay until it was too late to fix in 9.0.
Long reports on the standby can use the new hot_standby_feedback mechanism to keep the data they are looking at from being removed from the master. This is really the right way to handle running most longer reports on the standby. This limitation was recognized before 9.0 shipped, with one fix proposed and a prototype built. It just wasn't possible to get the code into 9.0 before its feature freeze.
There is a maximum replication_time setting. It's far too easy to have connections to a system that isn't working get stuck in TCP/IP retries for several minutes in 9.0.
Checkpoints on busy servers are much less likely to get stuck when the background writer's internal fsync aborb queue fills. The worst case of this problem can cause your checkpoints for a long time--I've seen what should have been a 5 minute checkpoint take six hours now. And even mild cases you might not normally notice can slow your system down.

When I talk with people who are looking into a 9.0 upgrade from an earlier version, it's often so that they can take advantage of all the benefits Hot Standby allows: read scaling by adding standby servers, reporting servers, live monitoring of the standby. If you do this on 9.0, you are likely to run into one of these limitations. And as more sites grow their PostgreSQL installations, and new sites are converting increasingly big databases over to it, I'm seeing a lot more of them run into either the partitioning MIN/MAX issue for the checkpoint one.
Normally, people don't like to consider the latest version of a database program because of the perception that it's more likely to have bugs than an older one. Well, in this case, the dynamic is a little different. If you deploy PostgreSQL 9.0, and you run into one of the bugs described above, there is little you can do about them other than upgrading to the newer version. Some workarounds exist, and plugging holes like that between version releases is part of why consultants and support companies have something to sell anyway when the software is free.

If you deploy 9.1 instead, you will not run into these problems. The question you should be asking is not whether there are any unknown bugs to be scared of in 9.1; there surely are. Given the sort of code changes that happened, I expect there to be a steady stream of bug reports against the new features introduced. But there weren't that many things changed that will impact users who don't use those features.

But if you deploy on 9.0, the odds are decent you will run into one of these known limitations of that version. Which would you rather have: the possibility of running into a bug, or the certainty you will run into a known limitation? Many people choose wrong and cling to older versions longer than they should, only to waste a lot of resources fighting known limitations and bugs that are already fixed in the newer release. You don't want to be one of those people, and now that 9.1 has shipped you should seriously consider deploying it instead of 9.0.

Making a VACUUM suck...wait, is going faster more or less here?

2011-08-18T05:55:49Z

One downside to having a specialty you're know for is that the people with the hardest problems in that area tend to find you. For me this is good in that it keeps me working, but it can easily turn out bad if the difficult is not distinguished from the impossible correctly. As someone who's done a lot of work on tuning PostgreSQL for heavy write volumes, the workloads I see are skewed closer than I'd like them to be some days to the impossible side. One of the areas that's really snuck up on me recently is just how hard it is to tune autovacuum for a database that's constantly written to.

The way PostgreSQL's transaction visibility system works, every database page you write will be written at least one more time, to freeze its transactions ids. And you may get yet another write on top of that, to reclaim free space on deleted items. Hint bits are another nuisance. And that's just the average case; pages can be overwritten many times if the deletion happens in chunks.

The big problem happens if you build a system that can just barely keep up the incoming write volume at its beginning. That system will then be crushed by its workload once the overhead of this background maintenance kicks in. Here's how it will happen: early performance tests say the server meets specifications. It survives initial rollout. But after a few months go by, the archive of data gets bigger, and next thing you know autovacuum needs to run 25 hours a day to keep up. (Hint: this doesn't work. That consulting advice, you get for free, as in beer. Which you'll likely end up drinking more of once this happens.)

What doesn't help matters is how arcane the parameters for tuning autovacuum are. Yesterday I submitted a patch to add the first readout to help with this problem I've been able to put together. Developed with my new co-worker Noah Misch, the patch itself is pretty simple. It takes the information autovacuum makes it decisions about and exposes it, in real-time, via the process's command line. Similarly to how you can track progress of things like the archiver this way, sampling this data turns out to be very useful for predicting how long things are going to take to run. There's also a report in the log file at the end. It looks like this:

LOG: automatic vacuum of table "pgbench.public.pgbench_accounts": index scans: 1

    pages: 0 removed, 819673 remain
    tuples: 19999999 removed, 30000022 remain
    buffer usage: 809537 hits, 749340 misses, 686660 dirtied
    system usage: CPU 5.70s/19.73u sec elapsed 2211.60 sec

The "buffer usage" line is the new one here. This is units of buffer pages, which are the 8192 byte chunks of memory. So seeing 686660 of them dirtied means this autovacuum wrote 8192 * 686660 bytes=5364MB in 2212 seconds. That makes for an average write rate of 2.43MB/s. If you've every tried to tweak autovacuum before, you'll know that guessing the rate at which it's going to write to disk is the hard part. This doesn't solve that problem completely, but it does let you trivially figure out something that is enormously useful: how fast things are writing given the current parameter set. And even that used to be really painful to figure out.

The patch is really small and could easily be applied with minimal risk to a production server. I expect it to be the next thing I end up backporting heavily onto troublesome customer systems. Next time I'll write a bit more about how the information gathered by this patch has given me new insight into vacuum tuning, some observations that can help you even if you're not running a server with it installed.

Day of the Dead Software

2011-08-13T04:16:25Z

Procrastination can be a virtue, when it saves you from having to do ultimately doomed work altogether. I like writing about topics like lazy disk writing because I personally identify with them. But sometimes putting things off ends up costing you, and it can kill you too. We're now reaching that point with the fall PostgreSQL conferences. The costing you part that is.

If you haven't booked a hotel yet for Postgres Open, during mid September in Chicago...starting on Monday the conference rate is gone. You already missed early bird registration, but if you poke around, you might find some discounts still available. Like several of the conferences this fall, there are some great bargains to be had in both the sessions and the training classes at this conference. Compared with the typical commercial database conference and training, this stuff is quite cheap. Even if you ignore me, wait until the last minute, and have to pay full price.

#PGWest is at the end of September in San Jose. That you still have some time left for; schedule isn't up yet.

Surge is in Baltimore that same week...early bird is gone for that already too, but you might still get the hotel discount. Maybe; get on that already if you want to go. Surge is near where I live, yet it's the only conference I'm mentioning here I can't make it to. It just wouldn't be right for me to have an conference that's easy to attend I guess, I need to do my talks completely frantic after various degrees of airport mayhem. Baltimore is also famous for being a place people go to die. We have a Poe house and museum, most notable because it was here old Edgar passed away at mysteriously. This tradition continues to current day television, with TV shows like "Homicide: Life in the Streets" and "The Wire" all being set here. The area where Surge is being held at? Nice part of town; you'll have to wander away from there if you want the true Baltimore experience.

Speaking of places filled with mayhem, the PostgreSQL Conference Europe this year is in Amsterdam. You still have nearly a month left to get the early rates for that conference, but who knows what will happen to travel prices by then. This would be a good one to book early too.

The last of my conference visits this year is PGBR, held in early November in São Paulo, Brazil. I'm excited to be seeing a whole new group of people at this conference, while making my first visit to South America. I was also excited to see the conference is the day after Brazil's Dia de Finados, "Day of the Dead". In addition to reminding me of Baltimore and being an awesome horror movie, this gives me a chance while I'm there to celebrate the memory of things I managed to procrastinate working on until they were dead, nicely avoiding them. I intend to pay my tribute to Sun and Solaris this year, which are of course dead now that Oracle has ruined them. MySQL is still flopping around, but the migrations off it are accelerating under its new malevolent overlords; not dead yet, but getting there. I know they're not as into the whole skull mask thing in Brazil like they are for this holiday in Mexico, but I'm going to keep a hopeful eye out for one that reminds me of Larry Ellison. Trying to kill off Oracle is one thing I never procrastinate.

Day of the Dead Mask photo from Scott Beale / Laughing Squid

Progress, SQL, and SQL Progress

2011-07-28T18:33:02Z

My work the last few years with PostgreSQL is actually my second career as a database person mainly focused on performance tuning. My first was in the mid to early 90's, working with the Progress 4GL. If you want to see just how similar that time and learning arc was to what I'm doing recently, check out my "Progress Performance FAQ" on my Progress Page. Anyone who's looked at my PostgreSQL performance book should see the many similarities. Wanting to know enough to duplicate everything covered in that 47 page document, but in a PostgreSQL content, was basically how I got started in this second database community for me; had that game plan from day one.

The similarities go so deep I can't help but laugh when I look at them again. Progress users gather on a mailing list named PEG to discuss everything about the product (my PostgreSQL tool peg was named with a nod toward that, retroactive acronym style). I settled into pgsql-performance in exactly the same way I used to spend time on PEG. We had a Tom, a Bruce, and a Peter as part of the core group of regular posters on the list. No Joshes, but Progress expert Geoff Crawford and PostgreSQL's Josh Berkus are so similar they each could play the other in a movie (checking up on Geoff, I'm unsurprised to find that just like Josh he's active at Toolbox for IT). I was of course the second Greg then too, often confused with the other one who had been there much longer than me. We the users sort of puttered around some of the time, but periodically we got input from Gus Bjorklund, the main database engine designer at Progress and a seriously deep technical guy. If I showed you some e-mail I exchanged with Gus in 1995 about details of the database's "after image writer", they'd be barely distinguishable from some messages I exchanged with Tom Lane on the PostgreSQL "background writer" in 2007.

And there's another big similarity on the technology side: in 1995 when I was deepest into this work, Progress didn't use SQL, just like early Postgres. It used a 4th Generation Language (4GL) that was much more procedural in feel. So instead of:

SELECT * FROM customer;

In the Progress 4/GL (now called 'OpenEdge Advanced Business Language') you'd do:

FOR EACH customer NO-LOCK:
DISPLAY customer.
END.

As a traditional programmer coming to databases just to pull stuff out of them for later processing, I was perfectly happy with this model. And I started out my work with PostgreSQL ambivilant about needing to improve my SQL skills to change that. Like a lot of non-database oriented developers, I wrote my share of procedural code that could have been wedged into a complicated query instead, rather than get too fancy with the query language. But you don't scale into the terabyte range of data sets that way, and the last few years have compelled me to really come to grips with with addressing problems in a way that turns into efficient SQL.

It was just today that I realized that mental transformation has really completed now: I'm now just as likely to think in SQL terms as I am procedural ones. I have some data to analyze that's sitting in some CSV files. I imported them into a spreadsheet; that made some types of analysis easier. I write a little Python script to loop over them and compute some things; that got me some of what I needed. But as I stared at this grid of data for a bit, in my head the data I wanted to extract kept decomposing into SQL. This part, SUM() and AVG() over a GROUP BY(). Running total, SQL window function. And these two parts that fit together...there's not a natural foreign key there, but I can derive one that will work to join them together.

The idea I was about to start heading toward--writing a Python program to compute the statistics needed--seems like a ridiculous one now that I consider the alternative. There's a lot of "buzz" lately against using SQL for things. But when you come across the sort of problem SQL and SQL-oriented database are good as solving, and you understand how to use SQL well enough to leverage it, you'd be crazy to solve them any other way.

Pushing all-in with PostgreSQL and Texas Hold'Em

2011-07-20T04:59:02Z

Just before I started working with PostgreSQL as my main occupation, there was a period in my life where I was a degenerate gambler. Manipulating people just to pull a few dollars out the transaction, giant cons designed to fleece the marks, everybody involved was dirty. But then I decided to take a break from trading stocks, and instead explore the comparatively clean and honest world of Texas Hold'Em Poker for a bit.

A question today about optimizing hardware for a poker hand database on one of the PostgreSQL lists got me digging out my old play record--which is in a PostgreSQL database. How the Poker industry ended up adopting PostgreSQL as its standard database of choice for high-performance analyzing of hands and play is kind of an interesting story. The great article Stranger than Fiction: The Story of PokerTracker, Online Poker, and PostgreSQL covers a lot of the background here. I thought it would be fun to revisit my own time in that land, and show just how that actually all works at the database query level.

In April of this year the FBI did a massive crackdown that has almost killed on-line poker in this country, at least in its original form. I still have $35 that's stuck in an account I can neither play with nor withdraw from after, perhaps my final "sin tax" on that era. But in the heady days circa 2006, where seemingly everyone had a not quite legal Hold'Em account they'd play from, I went through a poker apprenticeship that included around 60,000 hands worth of play.

The first six months were frustrating and I slowly lost most of the money in my account. But before wiping out I staged a comeback, won my losses back, and kept going up from there. By the end of my serious playing time, "greg1104" was the terror of the no-limit penny tables at Ultimate Bet, a tight playing grinder who'd wait hours for just the right moment to turn hyper aggressive. On a good night I'd wipe out a half dozen players in a row by forcing them all-in to chase their speculative hands. If you have a small "edge"--you're slightly more likely to win a hand than your opponent--and a large enough bankroll to support some unfair losses (players call them "bad beats"), eventually the worse player will end up broke.

To understand why this becomes a database overflowing with info eventually, you need to know a bit about on-line poker. You sit at a table that typically holds 6 to 10 people. Hands worth playing are rare enough (some players will only intentionally ante to play a hand less than 20% of the time) that any decent player will play at least 2 or 3 of them at once on-line. The speed of a table is measured in hands/hour, with a relatively fast typically going from 50 to 100. So even a journeyman player will have a hand possibility pass in front of them every 30 seconds; you have to make a quick decision about which to play and which to avoid. It doesn't take long until you have thousands of hands of history to use to make that decision.

As a database and math oriented guy, I quickly turned learning proper play into a query problem. What hands did I play well? Which did I botch? And, statistically, what types of players did I play well against? Around this same period, software aimed specifically at this problem started appearing. Each hand of Hold'Em you played could be exported into a "Hand History" file that detailed exactly what happened. Import those into a database, analyze information about each player, and you could put numbers on things you only had a gut feel for--and sometimes learn, the hard way, that your gut feel was wrong. PokerTracker was the software of choice, and it moved from Access to PostgreSQL around the same time I was playing regularly.

I don't have all my hand histories, but I saved enough of them (and kept moving them forward into new versions of PostgreSQL/PokerTracker) to have a clear record of my play during a period after I'd gotten good at the game. Here's the high level view of that play history, as a query against a PokerTracker database:

SELECT count(*),sum(amt_won) FROM player p
JOIN holdem_hand_player_statistics s USING (id_player)
WHERE p.player_name='greg1104';

count,sum
33687,68.99

So 33K hands winning $68.99. Not exactly World Series of Poker, but clearly I was a winning player. If I racked up 33K hands worth of history data in this fairly brief period, you can imagine how many a really serious player would have. The person asking about this today had 5M of them to query.

This is an example of the a first-level statistic you'd want about the other players at your table. You should react one way to a raise from a player with a winning track record over a long period, and a very different one from the guy who's been bleeding money the whole time he's been playing. The total is one thing; what percentage of the time did I win?

SELECT flg_won_hand,count(*),sum(amt_won)
FROM player p
JOIN holdem_hand_player_statistics s USING (id_player)
WHERE p.player_name='greg1104'
GROUP BY flg_won_hand;

flg_won_hand,count,sum
f,30432,-1547.84
t,3255,1616.83

Typically I was playing at 10 player tables, so winning around 10% of the time would be normal. And that's the case here. You can imagine how having these sort of statistics on every player--and a much more complicated set, like the percentage of time a player paid money to see the "flop" and play a hand--accumulated over time as you played would be a huge advantag.

What happens at a table is that good players quickly recognize the other sharks who've come to snack, and you respectfully play against them while at the same time gobbling down each new "fish" (that's their nickname) who appears. If you're running PokerTracker with its heads-up display, the easiest type of player to take money from gets a little icon over their head showing a picture of a fish. There they are, labeled for you whether or not you were even paying attention to them, all based on some database driven statistics.

Not only does having stats on the other players help, having your own lets you tune your play around your personal strengths and weaknesses. In a Hold'Em hand, the two cards you're dealt before each hand begins are your "hole cards". If you're any good, you should make a lot of your money when your hole cards are the known set where your odds are good to win. That was certainly the case for me:

SELECT hole_cards,count(*),sum(amt_won)
FROM player p
JOIN holdem_hand_player_statistics s USING (id_player)
JOIN lookup_hole_cards USING (id_holecard)
WHERE p.player_name='greg1104'
GROUP BY hole_cards
ORDER by sum(amt_won)
LIMIT 10;

hole_cards,count,sum
AA,139,51.16
55,142,25.71
AKs,95,21.44
QQ,148,20.49
AKo,313,16.82
KK,153,14.82
TT,159,13.45
JJ,166,12.95
A8s,112,11.60
T9o,343,11.10

All the usual suspects: AA, KK, QQ, AKs (suited, meaning they have the same suit, so likely to form a flush), AKo (off-suit), JJ, TT. One of my specialties was playing pocket pairs that turned into trio after the "flop" of 3 initial cards aggressively, particularly against players who didn't raise quite enough to keep me out of the hand. This is how I ended up winning so much with the slightly odd "55" combination. I was quite good at spotting when someone else had AA, KK, or QQ before the flop from their betting pattern. If they raised enough that I spotted them, but not so much it wasn't worth my trouble, I'll call and see what happened. If I got a matching third 5 on the flop (around 10% of the time it happens), I'd act like I was bluffing to steal the hand, get them committed and betting aggressively at me, and then raise to all-in. If they called, the hand was usually mine. You have to fine-tune that strategy based on how aggressive the other player is, and I'd be doing it with a stack of heads-up data showing various numeric measurements of that aspect of their play.

If I reverse the order here to see my worst hands, that's where you can find even more valuable information:

A4o,296,-13.83
QTo,300,-12.27
54o,294,-10.31
Q9s,113,-7.13
43o,294,-6.82
66,157,-6.52
J3o,294,-6.49
42s,114,-6.32
KJo,300,-6.02
A7o,348,-5.40

Apparently I got tripped up at least once using the same technique on "pocket sixes" that made me so much money on the "55" combination; that's just poker. But the fact that I lost so often with QT, Q9, and KJ suggests my play around handling those good but not great cards wasn't quite right. And the fact that I lost quite a bit, relatively, on 54 and 43 off-suit tells me I was overvaluing the possibility of a low-end straight considerably. This is very different fine-tuning from when I first started playing, where I'd lose lots of money even playing strong cards. You only have to lose a flush to a full house so many times before you start considering that possibility for a second when you decide how to bet.

By the time I gave up on playing online poker, I was just starting to get to where I had my own custom queries to run. A "win % adjusted aggression factor" was the one I was working on at the end, basically separating out people who were aggressive betters all the time from those who only did that when they actually had a hand--what I did. With a few million hands of data to research against, like the person asking on the list today has, fine tuning the statistics you collect and want to overlay (in real-time, updating after each hand) on top of the table heads-up display turns into a pretty difficult query processing challenge. Given that the lack of an early Windows version of PostgreSQL is often cited as one reason it didn't get better adoption, it's kind of amusing to see it's the database that completely owns this space--which, like PokerTracker itself and all the gaming client software--is a Windows only one.

When I was playing Texas Hold'Em, I never thought I'd be offering advice on database tuning suggestions as a way to improve play. But that really is where we're at now. It's a shame there's no good place for me to play on-line legally in the US anymore, because it forces to fold the cards I've scribbled notes on outlining a book I'd name "High-Speed Poker with PostgreSQL 9". If that ever gets sorted out, and I find myself with time to play again and a legal on-line venue to do it at here, you can bet I'm all-in on that idea.

Planetary alignment

2011-06-30T06:59:33Z

Picking back up this week's theme of where you can publicize your PostgreSQL related project at, you're probably reading this blog entry because it appeared on the Planet PostgreSQL blog aggregator. There are "Planet" feeds around many open-source projects. The Debian and GNOME ones spawned off the Planet software, which now powers a ton of other blogs such as the the well regarded Planet Python. Occasionally you'll find general open-source database news posted both here and on Planet MySQL. And I used to read Planet CentOS back when I used to care if they'd ever release CentOS 6.

Planet PostgreSQL has been around since about seven years ago, when Devrim Gündüz first made the service available on one of his servers. Like many good open-source projects, it has some history by now: it learned valuable infrastructure lessons from a tech disaster, has gone through being forked, and now it's settled into its current role as a community PostgreSQL project, run by many of the same people handling the rest of the postgresql.org web infrastructure.

Now, as I was talking about at length on my last blog entry, each of the community PostgreSQL announcement venues has its own rules for what's appropriate. But the Planet PostgreSQL policy is a lot less clear about what is and isn't appropriate content than, say, the extremely detailed News & Events Policy. You can publish a lot of things in the blog format, and whether they violate the rules laid down there is not always an easy things to decide.

While I can't personally make that call--does a Planet PostgreSQL post violate its policy or not?--I've recently become someone who gets a vote on that. And there's been a fair amount of feedback from people who are unhappy enough with recent entries to share what they don't like, and that opinion has been pretty consistent too. I apply a few basic tests to help decide whether I feel a blog entry fits the rules and the spirit this Planet embodies:

Is it interesting information about PostgreSQL? Both technical and non-technical content is encouraged here, but it should share something that leaves the reader with either some education or something to think about.
Does the blog mainly reproduce information also published to pgsql-announce or the News/Events/Training sections of the postgresql.org site? If so, that's a major point against it being appropriate content for Planet PostgreSQL in my mind, even if though that alone doesn't qualify as being against policy. People don't read the Planet PostgreSQL blog feed because they want a duplicate of the material published in those places. If they want those messages, they read those lists and feeds. I will sometimes use a blog entry here on the Planet to expand upon something that's mentioned in a terse announcement I made as news or an announcement. But you'll never find me just reposting the same sort of simple message format encouraged for pgsql-announce to a blog entry here on the Planet. That is information of sorts, but you're reaching the same audience with both routes; you shouldn't be duplicating the message too. The intended readers of Planet PostgreSQL are looking for more "meat on the bones", something readers can get involved in.
If you mention a commercial product, particularly one that people pay for, is it really clear to readers why that product is worthwhile? And did they learn something useful even if they decided that product isn't interesting to them? The policy here says "Publishing of advertising" is prohibited. Advertising in this context implies that the main purpose of your writing is to convince someone to purchase your product. So if you want to avoid that, don't make that convincing your sole message. Make the technical or business information you have a story about the focus, and wrap mentions of a product around that independently useful frame. There's a long history of writing in this format described by the term advertorial, and I think that sort of content is more likely to be useful and therefore acceptable here even though there is that touch of an ad in there.

Let's consider Blog A, which talks about a problem is hard to solve in PostgreSQL. It mentions some of the common ways that people do solve it: the sort of interesting information I've said I look for. If you then follow with a mention of a commercial product you make that happens to make that problem easier to solve, my response to this blog post is likely to be favorable. Some might call that an ad, and by a very strict definition it is. But if I've walked away from your blog entry having learned something useful, regardless of whether I buy your product or not, I'm inclined to feel that was a worthwhile contribution. And, if I run into that problem, I'm quite likely to consider your product if it really did sound like a better solution. This style of writing is not only more useful and less controversial, it's actually a more powerful ad for the people who really need your product.

Now consider Blog B. It opens by talking about a new product released by the company the blogger works for. We learn what it can be used for, and why those things are good. This is clearly an ad, even though the general content might be very similar to what A covered. The subtle distinction to me is that I haven't learned any useful information if I'm not interested in that commercial product. The sole purpose of the blog entry is to convince people to use the product, and if they aren't interested in it you've wasted their time. That's the very definition of advertising. And that's the sort of thing that will get readers of Planet PostgreSQL to complain to the site administrators, and ultimately for your blog to be suspended. For me, the distinction isn't even that much around whether the product is free or costs money. You might put free things on your site just to attract people toward the paid ones, making them just an indirect form of ad anyway. It's more about whether the blog entry stands on its own, as something useful if you just skip the parts mentioning the product altogether.

As a member of a community that will spend hours arguing about the finer shades of meaning in the word "free" when it comes to software, you're not going to find any universal answer to what an ad is either. But if you focus on making your blog useful first, then layer some commercial aspect on top, that's less likely to get you into trouble. And you may be surprised how working on that sort of writing style ultimately leads you to writing better real ads, too.

The cost of free PostgreSQL publicity

2011-06-27T20:01:04Z

The software license PostgreSQL is released under makes it extremely friendly to businesses who would like to use the database in commercial products. Partly as a result of this, a significant amount of PostgreSQL development is donated by companies who sell products derived from the database (even entire forks of the source code). Normally this feedback loop works well: companies are able to take the free community PostgreSQL, sell it along with services, customization, or add-ons, and some portion of those sales ends up driving development that flows back improving PostgreSQL itself.

Because of this synergy, the PostgreSQL community tries to be friendly to software and businesses built around the database, the entirety of which is sometimes referred to as the "PostgreSQL ecosystem". Businesses who offer PostgreSQL related services are provided several free venues for talking about their products, including:

E-mail promotion sent to the popular pgsql-announce mailing list
Mentions in the PostgreSQL Weekly News, which is syndicated to pgsql-announce and to Planet PostgreSQL.
Product and business news posted to the front of the postgresql.org site
Announcements of both events and training classes to the PostgreSQL site
Listing in the directories for Professional Services, Hosting, and the Software Catalogue
Syndication of your PostgreSQL related blog onto the Planet PostgreSQL blog aggregation site

As a community project, the work around keeping all this infrastructure going is done by a group of volunteers, who are often themselves funded by commercial companies. In addition to things like keeping the servers running, each of these resources has a set of moderators who help decide whether submitted content meets the requirements and standards for its inclusion. Moderation is a pretty thankless job where there's little positive feedback, just complaints if people don't agree with you. There are two main sides to how moderation is kept fair and even.

The policies are written down and continuously improved so they are clear to submitters and moderators. News and Events Approval Policy covers the things that go onto the main postgresql.org page, as well as pgsql-announce. Professional Services and Hosting Services has its own slightly different policy. And Planet PostgreSQL has its own policy.

The other side here is that the nature of the moderation work tends to attract people who have interest in making sure it's done fairly to their own businesses. Accordingly, the "voting" that may occur around policy decisions (it's rarely that formal) is done by a mixed group of people who want to make sure the PostgreSQL community plays well with commercial companies. You'll find myself and other members of 2ndQuadrant doing moderation duties, Dave Page and other EnterpriseDB staff handling both moderation and infrastructure duties, and a long list of similarly well known PostgreSQL contributors--a set cutting across companies all over the world. And all people who are well aware it's important to keep a balance between the open-source community at the center of PostgreSQL and its surrounding commercial ecosystem.

The end result is that when someone does something that's against one of the community policies, the normal reaction is agreement from all involved about at least the general way to handle it. Spammers and similar plagues upon the world are banned outright. Recognized community contributors who have run afoul of one policy are normally suspended from access to that resource for a period of time based on the severity of their offense. The exact response varies on a case by case basis. But you can be sure that the decisions are being made by a jury of your peers: contributors to the PostgreSQL community who also understand the value of businesses to that community, and who are trying to keep both sides of that relationship as positive as possible.

Patchy cleaning recommendations

2011-06-11T06:25:13Z

One of the under-appreciated aspects to software development is the ability to produce a good patch. Not just the coding--making sure the actual patch itself is of good quality. The backporting theme I've been discussing recently here all starts with the assumption that you can move changes from a newer version of software into an older one. That invariably involves some conflicts, but one of the best ways to avoid them is to make sure each of the individual commits--and accordingly the patches they represent--are as clean as possible. Figuring out just how to extract a good backport from a change to a newer feature is an art form, and it's easier if the patches you start with are well constructed.

Periodically newcomers to PostgreSQL submission get roasted for not following the project's coding guidelines. Most of these situations could have been resolved by the submitter, without that embarrassment/harassment, had they looked at the patch at all with the intent of cleaning it up before submission. The last time this popped up in our community, I got lumped in as one of the "formatting curmudgeons", a title I was proud to wear. It took me a while to really appreciate how important patch diff elegance is to building large pieces of software reliably, and to develop my own good techniques for doing so.

I'm not a fan of making things more difficult than they have to be for submitters to projects I work on though, so the one part I realized I could do better is helping others learn this material. While there are plenty of tutorials on how to use tools like git, I don't know that I've ever seen one that talks about the minutia of using diff to make good patches. We curmudgeons are quick to say things like "blend in with the surrounding code" or "remove spurious whitespace". But no one even talks about how you do that, exactly, in a way I'd expect newcomers to understand. That's why I just wrote a new tutorial called Creating Clean Patches, intended as supplemental reading to Submitting a Patch. Once you see what a bad patch looks like, compared to a clean one that would be reviewed only on its code quality merits, some of these fuzzy guidelines are much easier to understand. I think that article might even be useful outside of the PostgreSQL community. Plenty of other open-source projects struggle with the same issue.

MAX Partitioning with MIN Pain in PostgreSQL

2011-06-08T21:10:46Z

One of the things the PostgreSQL community is rightly proud of is the quality of the database's query optimizer. People like to squabble about whether hints or necessary, the user interface to query tuning is a bit strange, and there are some things that the database just doesn't know how to do yet that limit its ability to compete against commercial offerings with some types queries. The lack of index-only scans is a good example. But the fact that a fully open-source optimizer has managed to get in the right neighborhood, where for many applications it does a perfectly good job of optimizing even complicated queries, is pretty remarkable. For most of computing history, getting a database with a optimizer as good as the free one in PostgreSQL cost more than a little money.

One area where there are still some serious problems relative to more popular commercial products is partitioning. Table partitioning is an essential component for building large databases, both due to how indexes work and to how some maintenance tasks must happen. There is support for partitioning in PostgreSQL, but it's harder to setup and use than everyone would like it to be. There are plans around improving that, but they haven't converged to anything useful yet.

And the optimizer hasn't traditionally been too smart about what it can do with partitioned queries. The technique it implements is referred to as "constraint exclusion". The idea is that if you specifically compare a column against a fixed value, and there are constraints on the sub-partitions, the optimizer knows how to prove that some partitions cannot contain that value. They are then removed from being considered by the query. So if you set your partitions up right, and you do your comparison against a value such that optimizer can prove a partition is excluded, it will be. That's it though, and there are some common situations where the limitations of that are not obvious.

One of the nastiest of them is fixed in PostgreSQL 9.1, but lately I've been seeing a whole lot more people run into this in earlier versions and get blind-sided by it. The issue is with how aggregates like MIN() and MAX() are implemented. When you use those against a normal table that has an index on it, you expect that the index will be used as a way to get a sorted copy of the data, and then the first/last row will be popped off of there.

Here's the problem: that doesn't map directly into a constraint exclusion context. I first ran into this a few years ago, and my co-worker at the time Alan Li submitted a test case and patch to work around the problem. Here's what a test case based on that looks like:

CREATE TABLE partition_test (a int, b timestamp);
CREATE INDEX partition_test_idx on partition_test(b);
CREATE TABLE partition_test_2007_01_01 (CONSTRAINT partition_test_2007_01_01_b_check CHECK (((b >= '2007-01-01'::date) AND (b < '2007-01-02'::date)))) INHERITS (partition_test);
CREATE INDEX partition_test_2007_01_01_idx ON partition_test_2007_01_01 USING btree (b);
CREATE TABLE partition_test_2007_01_02 (CONSTRAINT partition_test_2007_01_02_b_check CHECK (((b >= '2007-01-02'::date) AND (b <'2007-01-03'::date)))) INHERITS (partition_test);
CREATE INDEX partition_test_2007_01_02_idx ON partition_test_2007_01_02 USING btree (b);
CREATE TABLE partition_test_2007_01_03 (CONSTRAINT
partition_test_2007_01_03_b_check CHECK (((b >= '2007-01-03'::date) AND (b < '2007-01-04'::date)))) INHERITS (partition_test);
CREATE INDEX partition_test_2007_01_03_idx ON partition_test_2007_01_03 USING btree (b);
EXPLAIN SELECT max(b) from partition_test;

The query plan you get out of this in PostgreSQL 9.0 and earlier is not so nice:

Aggregate (cost=137.00..137.01 rows=1 width=8)
   -> Append (cost=0.00..117.60 rows=7760 width=8)
         -> Seq Scan on partition_test (cost=0.00..29.40 rows=1940 width=8)
         -> Seq Scan on partition_test_2007_01_01 partition_test (cost=0.00..29.40 rows=1940 width=8)
         -> Seq Scan on partition_test_2007_01_02 partition_test (cost=0.00..29.40 rows=1940 width=8)
         -> Seq Scan on partition_test_2007_01_03 partition_test (cost=0.00..29.40 rows=1940 width=8)

The optimizer neither excludes any partitions, nor does it use an index even when scanning each of them! This means that a MIN() or MAX() against a giant partitioned table is guaranteed to scan every record in the table. And if you went to the trouble of partitioning, that's probably a ton of data. This sort of thing is very popular when people want to write queries like "when was the last time data was inserted into this table?" You might put SELECT now() - max(ts) FROM t into a monitoring system, then trigger an alarm if it gets too high suggesting there's a problem. But you just can't do it on a large partitioned table here. There are plenty of other use cases that need to pull the first or last value from a partitioned table too.

Unfortunately, Alan's fix for this was a bit too specialized to really solve all the problems like this that are around, so the problem has been sitting around for a while unresolved. It was finally fixed after a whole development arc ending with work from Tom Lane during the now beta PostgreSQL 9.1; the commit talks a bit about why the final approach used is the right one to have settled on. With 9.1, you get the query plan you probably expected in the first place:

Result (cost=0.10..0.11 rows=1 width=0)
   InitPlan 1 (returns $0)
     -> Limit (cost=0.04..0.10 rows=1 width=8)
           -> Merge Append (cost=0.04..485.84 rows=7720 width=8)
                 Sort Key: public.partition_test.b
                 -> Index Scan Backward using partition_test_idx on partition_test (cost=0.00..78.03 rows=1930 width=8)
                       Index Cond: (b IS NOT NULL)
                 -> Index Scan Backward using partition_test_2007_01_01_idx on partition_test_2007_01_01 partition_test (cost=0.00..78.03 rows=1930 width=8)
                       Index Cond: (b IS NOT NULL)
                 -> Index Scan Backward using partition_test_2007_01_02_idx on partition_test_2007_01_02 partition_test (cost=0.00..78.03 rows=1930 width=8)
                       Index Cond: (b IS NOT NULL)
                 -> Index Scan Backward using partition_test_2007_01_03_idx on partition_test_2007_01_03 partition_test (cost=0.00..78.03 rows=1930 width=8)
                       Index Cond: (b IS NOT NULL)

Where the index is used to get the individual MIN/MAX values, then they are combined and the true first or last value returned.

So what do you do if you need to deploy PostgreSQL with partitioning, and you need MIN/MAX to work? Well, if you deployed a new version of PostgreSQL while it was still in beta, you wouldn't be the first. At this point, you should be able to do that now, and then upgrade to the final binaries once released without changing the database itself. A major problem would have to occur before binary compatibility from beta to release would be broken at this late stage, it's avoided if at all possible.

The other possibility is to follow what I was talking about here last time, and create a backport of just this feature to your own custom PostgreSQL 9.0. It's possible to build "9.0 plus 'Reimplement planner's handling of MIN/MAX aggregate optimization'", and off you go. In theory at least. I mentioned this was the end of a development arc, it's not a single commit that adds the capability. This feature builds on top of the addition of MergeAppend plans, which is another major 9.1 change. Turns out there are at least six commits that touched this area of the code during 9.1 you need to grapple with, either taking their changes or resolving the merge conflicts in later ones. It's not really a feature backport for the squeamish.

But it's possible. I've done it, and so has at least one other person for different purposes. When my customer ran into this issue, it was months into a PostgreSQL 9.0 migration with partitioning, and the use of MAX() was not optional in their application. Without the source code and the ability to backport a fix, this migration would have failed; with it and some sweat, it succeeded. The PostgreSQL project itself doesn't do backports for new features, only bugs. But sometimes a feature backport is the least risky way to improve something in an otherwise stable product, one that prefers to avoid change whenever possible. Kernel backporting has been a major component to the success of RedHat Linux in corporate environments. A good PostgreSQL support company should be able to do the same for you, and there are several places you can purchases that quality level of work from.

Once a business has seen this sort of thing in action, I've found it makes them a whole lot more comfortable with using open-source software. When you're having a problem, being able to have some control over the process the whole way helps. And it can be hard to get that out of the large proprietary database vendors. For now I'm maintaining a backport, my customer is happy that they got a quick fix here. And everyone is happy that once 9.1 is released, this long standing issue with partitioning and these extremely common aggregates is resolved in completely standard PostgreSQL.

Backporting and checkpoint troubleshooting

2011-06-03T23:22:01Z

One of the nice parts of open-source development is that with enough motivation, you can decide exactly what features you do and don't want to use at a very fine level. This is particularly nice for what are called backports. Backports are when you see a single feature you want from a newer PostgreSQL version, but for some reason don't want all of that new version just yet. For example, the newer version may not be released yet, but it could have a fix for a problem you're having. Backporting fixes from the next version of a piece of software is a time honored approach for fixing bugs in open-source software, without going through the potential breakage a full version upgrade all most certainly will involve. Many of the things that are fixed in minor version upgrades to PostgreSQL start as patches to the latest version, are backported to earlier versions, then committed as "backpatch" bug fixes. You normally get a stack of these on during each minor version upgrade.

We do backport work as part of all our support services too, sometimes for things that make functional changes beyond what PostgreSQL does in their stable releases. I've ran into enough need for recent fixes to long-standing issues in PostgreSQL that I have two backport sets floating around I'm maintaining. Both take a small number of 9.1 features and apply them in 9.0, in a way that you can transparently switch your database to use them without having to change anything else.

When I did my The Write Stuff talk recently, several people asked me if they could get those features for testing their own 9.0 database. If you turn on log_checkpoints on your database, and it sometimes shows extremely large times (more than a few seconds) for the "sync phase" timing, you may have the problem these patches try to log and potentially fix. The PostgreSQL community would prefer that you test 9.1 directly instead if you can replicate the problem there. But if you have a production 9.0 server that's running into the sort of checkpoint issues I've been working on, going to 9.1 is probably more risk than you want to deal with right now. Backporting just those features, which are a pretty unobtrusive pair, may be more reasonable to you. And getting feedback on how that worked for you is still getting us some useful feedback that might be relevant for 9.1.

You can find my checkpoint backport work for 9.0 on our projects page. I could probably backport as far as V8.3 if motivated to do so, just haven't needed that for anyone yet. You need to know how to apply source code patches to use these, and you may need to learn how to rebuild your operating system's PostgreSQL packages. That's the sort of thing a consulting company might help with if you're not comfortable doing yourself, and any of the major ones will be happy to do this sort of thing. That's a nice switch from many commercial databases, where the only source for this sort of change (sometimes called a "hot fix") is the vendor. With PostgreSQL, you have multiple options for doing this sort of work, from doing it yourself to any number of people you can hire to help, too.

The other patch set? That revolves around partitioning, and that really deserves its own blog post. Have to save something for next time!