[root@toy ~]# smartctl -a /dev/sdc
=== START OF INFORMATION SECTION ===
Device Model:     INTEL SSDSA2CW120G3
...
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
3 Spin_Up_Time            0x0020   100   100   000    Old_age   Offline      -       0
4 Start_Stop_Count        0x0030   100   100   000    Old_age   Offline      -       0
5 Reallocated_Sector_Ct   0x0032   100   100   000    Old_age   Always       -       0
9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       5225
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       58
170 Unknown_Attribute       0x0033   100   100   010    Pre-fail  Always       -       0
171 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
172 Unknown_Attribute       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   090    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       34
225 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       201389
226 Load-in_Time            0x0032   100   100   000    Old_age   Always       -       2687040
227 Torq-amp_Count          0x0032   100   100   000    Old_age   Always       -       0
228 Power-off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       314526
232 Available_Reservd_Space 0x0033   100   100   010    Pre-fail  Always       -       0
233 Media_Wearout_Indicator 0x0032   099   099   000    Old_age   Always       -       0
241 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       201389
242 Total_LBAs_Read         0x0032   100   100   000    Old_age   Always       -       133021

Not all of these attributes are labeled correctly, and some are “Unknown”; all the gory details are in the product specifications. You can see the Media Wearout above; here’s the raw values for other interesting ones, formatted so they’re more blog-friendly:

ID# ATTRIBUTE_NAME          RAW_VALUE
9 Power_On_Hours          5225
12 Power_Cycle_Count       58
192 Power-Off_Retract_Count 34
225 Load_Cycle_Count        201389
241 Total_LBAs_Written      201389
242 Total_LBAs_Read         133021

192 (hex C0) is “Power-Off Retract Count”.  That’s how many unsafe shutdowns the drive has been through, which are the situations where the battery backed cache in the drive has been triggered.  With 34 of them here, you can see I’ve tried to get this drive to die that way.

There are a few parts of the PostgreSQL internals that poke out usefully if you look in the right place for them.  One useful set to know about are the System Columns, which you can explicitly request but don’t see by default.  For example:

psql -x -c “SELECT oid,* FROM pg_class LIMIT 1″

There is no column named oid in the pg_class table, but it’s there if you ask for it.  The oid used to be relied on more heavily in PostgreSQL as a way to identify rows.  That’s not true for regular tables anymore, and you really don’t want to start doing that for your own tables.  OIDs are mainly useful now when joining parts of the System Catalog together.  A good example is the Disk Usage query.  If you want to find the namespace a table is in, you need to know you can ask for its OID.  It’s possible to get some of this data out of more portable views like information_schema.tables.  But many of the useful things in this area are PostgreSQL specific.  Sometimes I see people starting with the information_schema views and joining against other tables using its text name fields, such as the listed table_name.  That approach has several edge cases that don’t work out correctly; not handling TOAST columns is a common example.  That makes them more prone to breaking on you later, probably after your system has gone into production, than an OID based join.

There is also a tableoid system column.  As described in the documentation, its main use case is identifying which partition a row come from.  That’s not a great thing to be driving application logic from, but it can be useful for monitoring or troubleshooting purposes.  For example, if you SELECT rows from the parent table in a partitioning inheritance scheme, it’s normally expected that no rows will actually be stored there.  Checking the tableoid is one way to confirm that.  You might confirm that your INSERT/UPDATE trigger is moving rows to the right place using tableoid as well.  It’s possible to do that for each individual partition section, but running a query against the parent will make sure you hit every row in the table.

Another internal column related to uniquely identifying rows is the ctid.  The ctid is a direct pointer to the physical block (using PostgreSQL’s 8K page size) and position of a row.  ctids are a pair of numbers, and the first row will be (0,1).  While this is the fastest way to find a row more than once in the same transaction block, these numbers are not stable in the long term.  Any UPDATE and some maintenance operations will change them.  One thing you can use these for is finding duplicate data in a table.  Let’s say you’re trying to add a unique constraint, but one row in the table is duplicated 3 times, which blocks the unique index from being created.  When rows are identical in every column, you can’t write any simple SELECT statement to uniquely identify them.  That means deleting all of them but one copy requires some annoying and fragile SQL code, combining DELETE with LIMIT and/or OFFSET–which is always scary.  If you use the ctid instead, the implementation will be PostgreSQL specific, but it will also be faster and cleaner.  See Deleting Duplicate Records in a Table for an example of how that can be done.

The other system columns all relate to transaction visibility:  xmin, cmin, xmax, cmax.  When you delete a row in PostgreSQL, it isn’t eliminated from disk immediately.  It’s possible that some other query that’s executing at the same time will still need to see that row, and the transaction isolation in PostgreSQL worries about such things.  If you ever want to learn how that isolation works, the way the Multiversion Concurrency Control (MVCC) implementation is handled, you can watch parts of it happen.  Just open transactions in two different sessions, UPDATE/DELETE in one of them, and then look at those rows in the other.  You can still see them in the session where they weren’t touched, but they’ll be marked to expire in the future via their xmax being set.  To really pull that all together, you also need to know about some of the System Information Functions.  txid_current() is the most useful for this sort of learning experience, it provides a reference point for the always increasing system transaction ID.  You can find a more detailed exploration of using these functions and system columns in Bruce’s MVCC Unmasked talk.  The “Routine Maintenance” chapter of my book also shows examples how how MVCC works through the perspective of the system columns.

Talking to people deploying PostgreSQL in several countries during a short span of time has given me some interesting perspective on where the project is at.  I follow a lot of adoption trends in the US, and some of those I assume are quirks in how business is done in this country.  But when I hear the same sort of feedback from people in all four of the countries I’ve been to this year, too, it’s clear this is a larger issue.

The first thing I’m seeing a surprising amount of is satisfaction with the feature set in PostgreSQL.  A few years ago, conversations about what you could and couldn’t do with PostgreSQL usually stalled on one of a few common requests.  There’s a good survey of PostgreSQL feature feedback at User Voice.  13 important features originally on that list have been closed already, with Index Only scans as the next expected to fall in the upcoming 9.2.  PostgreSQL now includes regular and synchronous replication as of 9.1.  pg_upgrade has been getting an increasing amount of testing that proves it works for many in-place upgrade scenarios.  Extensions are dramatically easier to use now.

It seems the total feature set has crossed the threshold where PostgreSQL is good enough for a whole lot more deployment situations than it used to be.  What I’m hearing from people all over the world now is that the basic feature set and performance of PostgreSQL isn’t failing the “checkbox test” so often anymore, where business people require certain things before they’ll even consider a database.  There are some major wants that are some distance off, such as materialized views and better OLAP support (cube/rollup/etc.)  And using partitioning for bigger data sets is harder than people would like.  But these are all things that are only needed for larger deployments, and some workarounds exist if you’re willing to work at them.

If the feature set isn’t holding back as many deployments now, what is?  Well, the next thing I’ve been hearing everywhere is on that survey list too:  better administration and monitoring tools.  You really need a whole open-source stack to monitor PostgreSQL right now, from OS+database trending to query log analysis.  It’s fine for these tools to live outside the database core, but some changes are clearly needed to make such tools easier to write.  For example, the one built-in tool that allows query monitoring is pg_stat_statements, and the limitations preventing it from being useful to most people are so obvious we’ve gotten two submissions to improve it in the last month.

There are a few projects that aim at the monitoring/administration problem.  EnterpriseDB’s PostgreSQL Enterprise Manager, Cybertec’s pgwatch, OmniTI’s Reconnoiter, the suite of smaller tools from End Point, and even the text UI of pg_statsinfo all hit the edges of this problem.  What I hear when I have my advocacy hat on is that the community needs a major open-source project bigger than any of these to make database monitoring easier.  That’s now one of the major distinguishing features the commercial competition has.  Getting enough of the people developing in this area all pointing in the same direction and working together is a big challenge though.

On a related note, now that the underlying features are there, it seem making replication easier to monitor and setup is a major issue too.  There are so many choices in replication technologies available for PostgreSQL it’s easy for new people to get overwhelmed by them all.  And the documentation guides around this area are still filled with a lot of complications that aren’t even really necessary to get started at this point.  It’s easy for newcomers get dragged into details like how old style archiving works as a precusor to setting up even basic replication, despite that they’re using the easier features in the current PostgreSQL instead.  This area still has some work in the core database happening in 9.2, and it will be important for the community to create replication guides that include current information covering both 9.1 and that release.  What I’m hear from every country I visit now is “I need material to help me compete against the idea of using Oracle RAC”.

The last of the global trends that have really jumped out at me is how companies everywhere are reinventing the development process around database applications.  In some places, mostly bigger companies and government installations in particular, the expected staff “stack” is business as usual; it hasn’t changed in a long time.  New applications go from Developer to DBA to systems administrator.  Management ideas like DevOps are catching on to improvement interface between these roles, but not really upset its basic structure.  Everywhere I go now, I’m seeing everything but the developer role being squeezed out.  ORM-driven development is eliminating the DBA’s role in database design.  Managed application hosting platforms are wiping out the systems administrator role.  Startups with an idea for a web application go right from developer to deployment, and happily this is increasingly happening with a PostgreSQL backend in the database role.  There isn’t even the perception that DBA-like help might be needed until the application grows quite a bit.  I’m seeing the need for better database specific optimization skills than a typical developer has being deferred until the application has tens of gigabytes of data to sling around.

Being able to deploy small PostgreSQL installs and grow them to a reasonable size without specialized DBA knowledge is a great thing as far as I’m concerned.  The exact advances in things like ORMs that have allowed reaching this point across the world are a topic that deserves its own long discussion.  I’m going to cut this off here and return to that later.  In this country, there’s some concrete work around the 9.2 release that needs to get done this month.

Today my mailbox was crowded with some
Apple news. The source code to the Apple Lossless Audio Codec, encoders and decoders, was released to
the world. A few open-source projects such as FFmpeg and FAAC has
already had support for the resulting .m4a file; this gives them an
official release to validate against. As a well documented audio
snob, I’m known to be a fan of lossless audio, with a few hundred CDs
here ripped into FLAC. Unfortunately, this source code drop doesn’t
change anything for me, and that’s all because of software patents.
Right now I consider such patents to be the greatest risk to software
I work on like PostgreSQL, so I spent some time looking into just
what Apple has done here, as a data point on that topic.

If your baseline is MP3 files, Apple
Lossless appears to be a step forward as far as licensing goes. All
MP3 playback requires a patent license, while playback of Apple’s
format does not–only the encoding side need be licensed. That’s
not my baseline though. FLAC has specifically avoided using any
known patented technology, so it’s the clear winner in being clean of
patent issues.

You can’t even read the just released source code without being shown the door that leads to Apple’s patents.  ALACEncoder.cpp pushes you that way when you read it, saying “The
relevance of the ALAC coefficients is explained in detail in patent
documents.” So you can’t fully understand what this code does
unless you dip into the patent description. That’s a big sign of
trouble.  I’m not sure exactly which patent they are referring to; it may be Determining scale factor values in encoding audio data with AAC.

I’m not sure because I don’t read patent descriptions if I
can avoid it, due to how damages are tripled with willful infringement. That rule puts
open-source developers in a weird place. The act of researching
which patents your free implementation might infringe on can have a
wildly negative return on investment. If it’s impossible to
implement the idea without infringing in the patents you find, which
can easily be the case given the ridiculously low bar for grating
such patents, if a lawsuit does happen you’d be better off not
knowing about that risk when it starts. On the PostgreSQL mailing
lists, mentioning how a patented implementation of something works is
one of the few things that will get you a warning and potentially
blocked from the lists. Knowingly implementing a patented idea can
be a very expensive mistake for the project to make.

Now, you can claim this problem has
gone away for this bit of software due to how Apple has released this
code under the Apache 2.0 license. If you build something using the Apple Lossless
code, and you distribute the result under the Apache 2.0 license, you
get the protection of its Grant of Patent License. You’ll only lose
it if you sue someone in a way that involves this code+patent
combination. That doesn’t necessarily protect manufacturers of
hardware who want to include this capability, as they may not want to
license the result that way. I don’t really care about them though.
People who want to take advantage of open-source code but not
contribute back to can worry about their own legal issues, they’re
not my concern.

My real issue with using this format is
that it implicitly approves of business practices around software
patents I find hostile to my own work, and that leads me back to the
patent-less FLAC again. Apple used to more often be the victim of
valid and frivolous patent claims, but increasingly they’ve become
the originator of them instead. In mid 2006, Apple was sued over
patent issues by Creative Technology; they settled paying Creative
$100M dollars. Since then, patent issues have increasingly been part
of their well exercised legal arm.

Starting last year, Apple has seriously
turned around in how it handles patents–it’s now aggressively
enforcing its patents rather than just being sued for violations. Steve
Jobs issued a warning shot about that, saying “competitors should
create their own original technology, not steal ours”.
Considering it had only been four years before then that Apple was
sued–and paid out in a big way–for stealing other people’s
patented ideas, that came off as rather hypocritical to me.

The company has done well staying ahead
of its competitors by production innovation, rather than court
fights. iPod competitors failed to gain traction because their
products weren’t as good, not because they couldn’t steal the design.
Apple’s iPad should have the same property; the knock-offs are not
selling because they’re not as good. The only real threat to their
product line right now are how Android phones are displacing the
iPhone. Dumping so much money into offensive lawsuits is burning up
money that could be used for real product advances there instead.
It’s a shame they’ve resorted to this tactic.

At this point, a cautious person would
avoid using technology encumbered by Apple’s patents, as they clearly
have the means and intent to sue for violations. And someone who
values open source projects should avoid patented approaches even
when they are freely licensed. Whether or not individuals using
Apple Lossless is particular are exposed to problems here is missing
the big picture.

As an advocate for free software, I
reflexively pick the less patent encumbered approach to any problem,
using that as the tie breaker for decisions that are otherwise even.
Encoding into and helping popularize Apple Lossless may be legal now.
But I’ll keep encoding into FLAC, copying the result onto my Sansa
Fuse player, and avoiding their entire music ecosystem. Software
patents are too dangerous to implicitly endorse them if it can be
avoided, and here they easily can.

I like to do some original research for my talks, and this year that included a look into Synchronous Replication and Durability Tuning in PostgreSQL 9.1, specifically the performance side.  At last week’s PG.EU I gave an updated version of this talk (with Simon Riggs), including a bit more info than I had available during the Postgres Open presentation on the same topic.

The good news/bad news performance results are nicely summarized by the “Group commit magic” graph on slide 17.  There I was replicating across the Atlantic Ocean, crossing from my home here in Baltimore over to the conference site in Amsterdam.  When doing synchronous replication, the speed of light determines how fast any single client can commit.  You can only reach about 50% of that round trip time given current (and expected future) technology here.  The efficiency rate of the fiber-optic cables used is not perfect, and tasks like routing add some overhead too.

 
With that sort of distance, the round trip time is at least 100 milliseconds.  What this means is that no one client can commit more than 10 times per second.  This shows just how difficult sync rep is to run well, the maximum rate drops fast if you expect to leave your local data center to commit.  Light turns out to be really slow compared with what many people expect.

 
The great thing I proved in that slide is that the efficiency when multiple clients are committing at the same time scales almost linearly.  If you want 1000 transactions/second over this sort of distance, you can get that–but you’ll need just over 125 clients going at once to do it.  Each one of those clients will be seeing 10 commits/second and 100 millisecond latencies (and higher for a small percentage, peak latency in the test was closer to 600ms).  But each commit reply will be acknowledging a pile of clients at once, just by sending a small packet with an updated “committed up to this point” response.

 
When the speed of light turns out to be your bottleneck, there’s not much you can do about that attacking directly.  Some people who want to reach higher rates might architect their systems with many smaller clients, as I’ve shown in this example.  I was able to reach my goal of 2000 INSERT commits/second here, but it took 275 clients to get there.

 
The other great thing about how PostgreSQL implements sync rep is that it’s controlled per transaction.  Once you see how expensive the commits are, if that’s too high for some of your data, you can always tweak that for higher performance just by disabling sync rep for some transactions.  Having such fine-grained control over synchronous commits is a unique feature to PostgreSQL, allowing something like a Quality of Service suggestion to the database.  The PostgreSQL code as of 9.1 really has an unprecedented range of trade-offs here.  You can go for faster but not very durable at all (with unlogged tables), locally durable but not guaranteed to a remote data center at a medium speed, all the way up to fully synchronous and very expensive to commit.  It’s possible to argue that other database choices are better at one end of this range.  You might use MongoDB for higher speeds at the low durability range, and Oracle for their better tested sync rep capabilities (I saw better tested simply because the PostgreSQL 9.1 code is very new relative to Oracle’s implementation).

PostgreSQL started in the middle here, and with 9.1 it’s expanded nicely toward both ends of the spectrum at once.  It’s now providing options for higher and lower durability at the same time, in one database, and with the speed/durability trade-off adjustable for every transaction.  Building one size fits all software is really hard, and the new features in 9.1 nicely push out capabilities here for several popular use cases, all at the same time, and only when you want to pay for them.

Read on and I’ll detail why I feel that way.

Most changes to a software project are sorted into two main groups: bug fixes and features. PostgreSQL is very good about backporting bug fixes to all supported versions of the program, when practical to do so. But feature changes aren’t done in older releases. Stability is highly prioritized in the stable versions–sometimes to the point where some very annoying things that could be fixed aren’t.

The last release, PostgreSQL 9.0, started its life as version 8.5. When it became obvious several big, disruptive features were going to be included, the version number was bumped in part to make people a little more wary of the release. The bug count over the last year hasn’t been too terrible, but there have been plenty of serious bugs. April’s 9.0.4 corrected both a nasty data loss bug (for users of pg_upgrade) and a case where new database crash recovery code could introduce a problem even when not using the feature it was modified to support.

There is a third category of changes here though: removing limitations. Sometimes the new “feature” added by a new version is that a part of the database code is made to support new cases, because they didn’t do everything people really wanted . These are considered new features and not backported, even though it’s often the case that people are running into the limitation on older versions.

While all the genuine new features are great, too, what makes 9.1 a really compelling upgrade from my perspective is how many of these limitations are blown away by it.

Here’s my short list of limitations removed that I’ve run into on multiple installations of 9.0 and earlier versions now:

• Queries using MIN or MAX against a partitioned table can now use an index to return that value. This is a showstopper for sites that run into it, one that normally doesn’t show up until after you’ve got a large amount of data into PostgreSQL; then it hits you by surprise. The most common way I’ve run into this are sites that monitor the last time/entry inserted into some busy table. SELECT now() – max(timestampfield) FROM t is a good way to tell how long it’s  been since the last record was inserted into your database, to set an alarm if that grows too big. On a busy site that means something has probably broken and stopped importing records. But if you use this approach, and you switch to partitioned tables…you’re toast. Until 9.1, you had to rewrite the query to fix this. And in some places, application changes like that take a huge amount of QA work to do. Now it just works.
• Replication lag time can easily be monitored using pg_stat_replication. All of our 9.0 deployments integrate our repmgr tool in order to provide this capability. In 9.1, you get that part in the core server.  (We’re adding some new stuff to repmgr to keep it relevant–it’s always going to be where between release features we’re working on can incubate at)
• Systems using Hot Standby to run long reports against standby systems can now set a large value for max_standby_archive_delay and max_standby_streaming_delay. Changing the mechanics of these two parameters was one of the very last code changes in 9.0 before its feature code freeze. And subsequent testing missed the fact that the maximum value you can set these two only allows 35 minutes of delay until it was too late to fix in 9.0.
• Long reports on the standby can use the new hot_standby_feedback mechanism to keep the data they are looking at from being removed from the master. This is really the right way to handle running most longer reports on the standby. This limitation was recognized before 9.0 shipped, with one fix proposed and a prototype built. It just wasn’t possible to get the code into 9.0 before its feature freeze.
• There is a maximum replication_time setting. It’s far too easy to have connections to a system that isn’t working get stuck in TCP/IP retries for several minutes in 9.0.
• Checkpoints on busy servers are much less likely to get stuck when the background writer’s internal fsync aborb queue fills. The worst case of this problem can cause your checkpoints for a long time–I’ve seen what should have been a 5
  minute checkpoint take six hours now. And even mild cases you might not normally notice can slow your system down.

When I talk with people who are looking into a 9.0 upgrade from an earlier version, it’s often so that they can take advantage of all the benefits Hot Standby allows: read scaling by adding standby servers, reporting servers, live monitoring of the standby. If you do this on 9.0, you are likely to run into one of these limitations. And as more sites grow their PostgreSQL installations, and new sites are converting increasingly big databases over to it, I’m seeing a lot more of them run into either the partitioning MIN/MAX issue for the checkpoint one.

Normally, people don’t like to consider the latest version of a database program because of the perception that it’s more likely to have bugs than an older one. Well, in this case, the dynamic is a little different. If you deploy PostgreSQL 9.0, and you run into one of the bugs described above, there is little you can do about them other than upgrading to the newer version.

Some workarounds exist, and plugging holes like that between version releases is part of why consultants and support companies have something to sell anyway when the software is free.

If you deploy 9.1 instead, you will not run into these problems. The question you should be asking is not whether there are any unknown bugs to be scared of in 9.1; there surely are. Given the sort of code changes that happened, I expect there to be a steady stream of bug reports against the new features introduced. But there weren’t that many things changed that will impact users who don’t use those features.

But if you deploy on 9.0, the odds are decent you will run into one of these known limitations of that version. Which would you rather have: the possibility of running into a bug, or the certainty you will run into a known limitation?

Many people choose wrong and cling to older versions longer than they should, only to waste a lot of resources fighting known limitations and bugs that are already fixed in the newer release. You don’t want to be one of those people, and now that 9.1 has shipped you should seriously consider deploying it instead of 9.0.

One downside to having a specialty
you’re know for is that the people with the hardest problems in that
area tend to find you. For me this is good in that it keeps me
working, but it can easily turn out bad if the difficult is not
distinguished from the impossible correctly. As someone who’s done a
lot of work on tuning PostgreSQL for heavy write volumes, the
workloads I see are skewed closer than I’d like them to be some days
to the impossible side. One of the areas that’s really snuck up on
me recently is just how hard it is to tune autovacuum for a database
that’s constantly written to.

The way PostgreSQL’s transaction
visibility system works, every database page you write will be
written at least one more time, to freeze its transactions ids. And
you may get yet another write on top of that, to reclaim free space
on deleted items. Hint bits
are another nuisance. And that’s just the average case; pages can be
overwritten many times if the deletion happens in chunks.

The big problem happens if you build a
system that can just barely keep up the incoming write volume at its
beginning. That system will then be crushed by its workload once the
overhead of this background maintenance kicks in. Here’s how it will
happen: early performance tests say the server meets specifications.
It survives initial rollout. But after a few months go by, the
archive of data gets bigger, and next thing you know autovacuum needs
to run 25 hours a day to keep up. (Hint: this doesn’t work. That
consulting advice, you get for free, as in beer. Which you’ll likely
end up drinking more of once this happens.)

What doesn’t help matters is how arcane
the parameters for tuning autovacuum are. Yesterday I submitted a
patch to add the first readout to help with this problem I’ve been able to
put together. Developed with my new co-worker Noah Misch, the patch
itself is pretty simple. It takes the information autovacuum makes
it decisions about and exposes it, in real-time, via the process’s
command line. Similarly to how you can track progress of things like
the archiver this way, sampling this data turns out to be very useful
for predicting how long things are going to take to run. There’s
also a report in the log file at the end. It looks like this:

LOG: automatic vacuum of table
"pgbench.public.pgbench_accounts": index scans: 1

    pages: 0 removed, 819673 remain
tuples: 19999999 removed, 30000022 remain
buffer usage: 809537 hits, 749340 misses, 686660 dirtied
system usage: CPU 5.70s/19.73u sec elapsed 2211.60 sec

The “buffer usage” line is the new
one here. This is units of buffer pages, which are the 8192 byte
chunks of memory. So seeing 686660 of them dirtied means this
autovacuum wrote 8192 * 686660 bytes=5364MB in 2212 seconds. That
makes for an average write rate of 2.43MB/s. If you’ve every tried
to tweak autovacuum before, you’ll know that guessing the rate at
which it’s going to write to disk is the hard part. This doesn’t
solve that problem completely, but it does let you trivially figure
out something that is enormously useful: how fast things are writing
given the current parameter set. And even that used to be really
painful to figure out.

The patch is really small and could
easily be applied with minimal risk to a production server. I expect
it to be the next thing I end up backporting heavily onto troublesome customer systems. Next time I’ll write a
bit more about how the information gathered by this patch has given
me new insight into vacuum tuning, some observations that can help
you even if you’re not running a server with it installed.

#PGWest is at the end of September in San Jose.  That you still have some time left for; schedule isn’t up yet.

Surge is in Baltimore that same week…early bird is gone for that already too, but you might still get the hotel discount.  Maybe; get on that already if you want to go.  Surge is near where I live, yet it’s the only conference I’m mentioning here I can’t make it to.  It just wouldn’t be right for me to have an conference that’s easy to attend I guess, I need to do my talks completely frantic after various degrees of airport mayhem.  Baltimore is also famous for being a place people go to die.  We have a Poe house and museum, most notable because it was here old Edgar passed away at mysteriously.  This tradition continues to current day television, with TV shows like “Homicide:  Life in the Streets” and “The Wire” all being set here.  The area where Surge is being held at?  Nice part of town; you’ll have to wander away from there if you want the true Baltimore experience.

Speaking of places filled with mayhem, the PostgreSQL Conference Europe this year is in Amsterdam.  You still have nearly a month left to get the early rates for that conference, but who knows what will happen to travel prices by then.  This would be a good one to book early too.

The last of my conference visits this year is PGBR, held in early November in São Paulo, Brazil.  I’m excited to be seeing a whole new group of people at this conference, while making my first visit to South America.  I was also excited to see the conference is the day after Brazil’s Dia de Finados, “Day of the Dead”.  In addition to reminding me of Baltimore and being an awesome horror movie, this gives me a chance while I’m there to celebrate the memory of things I managed to procrastinate working on until they were dead, nicely avoiding them.  I intend to pay my tribute to Sun and Solaris this year, which are of course dead now that Oracle has ruined them.  MySQL is still flopping around, but the migrations off it are accelerating under its new malevolent overlords; not dead yet, but getting there.  I know they’re not as into the whole skull mask thing in Brazil like they are for this holiday in Mexico, but I’m going to keep a hopeful eye out for one that reminds me of Larry Ellison.  Trying to kill off Oracle is one thing I never procrastinate.

My work the last few years with
PostgreSQL is actually my second career as a database person mainly
focused on performance tuning. My first was in the mid to early
90′s, working with the Progress 4GL. If you want to see just how
similar that time and learning arc was to what I’m doing recently,
check out my “Progress Performance FAQ” on my Progress Page.
Anyone who’s looked at my PostgreSQL performance book should see the
many similarities. Wanting to know enough to duplicate everything
covered in that 47 page document, but in a PostgreSQL content, was
basically how I got started in this second database community for me;
had that game plan from day one.

The similarities go so deep I can’t
help but laugh when I look at them again. Progress users gather on
a mailing list named PEG to discuss everything about the product (my
PostgreSQL tool peg was named with
a nod toward that, retroactive acronym style). I settled into pgsql-performance
in exactly the same way I used to spend time on PEG. We had a Tom, a
Bruce, and a Peter as part of the core group of regular posters on
the list. No Joshes, but Progress expert Geoff Crawford and
PostgreSQL’s Josh Berkus are so similar they each could play the
other in a movie (checking up on Geoff, I’m unsurprised to find that just like Josh he’s active at Toolbox for IT). I was of course the second Greg then too, often
confused with the other one who had been there much longer than me.
We the users sort of puttered around some of the time, but
periodically we got input from Gus Bjorklund, the main database
engine designer at Progress and a seriously deep technical guy. If I
showed you some e-mail I exchanged with Gus in 1995 about details of
the database’s “after image writer”, they’d be barely
distinguishable from some messages I exchanged with Tom Lane on the
PostgreSQL “background writer” in 2007.

And there’s another big similarity on
the technology side: in 1995 when I was deepest into this work,
Progress didn’t use SQL, just like early Postgres. It used a 4th
Generation Language (4GL) that was much more procedural in feel. So
instead of:

SELECT * FROM customer;

In the Progress 4/GL (now called
‘OpenEdge Advanced Business Language’) you’d do:

FOR EACH customer NO-LOCK:

DISPLAY customer.

END.

As a traditional programmer coming to
databases just to pull stuff out of them for later processing, I was
perfectly happy with this model. And I started out my work with
PostgreSQL ambivilant about needing to improve my SQL skills to
change that. Like a lot of non-database oriented developers, I wrote
my share of procedural code that could have been wedged into a
complicated query instead, rather than get too fancy with the query
language. But you don’t scale into the terabyte range of data sets
that way, and the last few years have compelled me to really come to
grips with with addressing problems in a way that turns into
efficient SQL.

It was just today that I realized that
mental transformation has really completed now: I’m now just as
likely to think in SQL terms as I am procedural ones. I have some
data to analyze that’s sitting in some CSV files. I imported them
into a spreadsheet; that made some types of analysis easier. I write
a little Python script to loop over them and compute some things;
that got me some of what I needed. But as I stared at this grid of
data for a bit, in my head the data I wanted to extract kept
decomposing into SQL. This part, SUM() and AVG() over a GROUP BY().
Running total, SQL window function. And these two parts that fit
together…there’s not a natural foreign key there, but I can derive
one that will work to join them together.

The idea I was about to start heading
toward–writing a Python program to compute the statistics
needed–seems like a ridiculous one now that I consider the
alternative. There’s a lot of “buzz” lately against using
SQL for things. But when you come across the sort of problem SQL and
SQL-oriented database are good as solving, and you understand how to
use SQL well enough to leverage it, you’d be crazy to solve them any
other way.

]]> http://blog.2ndquadrant.com/progress_sql_and_sql_progress/feed/ 0 http://blog.2ndquadrant.com/pushing_allin_with_postgresql/?utm_source=rss&utm_medium=rss&utm_campaign=pushing_allin_with_postgresql http://blog.2ndquadrant.com/pushing_allin_with_postgresql/#comments Wed, 20 Jul 2011 05:59:02 +0000 greg.smith http://blog.2ndquadrant.com/?p=58

Just before I started working with
PostgreSQL as my main occupation, there was a period in my life where
I was a degenerate gambler. Manipulating people just to pull a few
dollars out the transaction, giant cons designed to fleece the marks, everybody involved was dirty. But then I decided to take a break
from trading stocks, and instead explore the comparatively clean and
honest world of Texas Hold’Em Poker for a bit.

A question today about optimizing
hardware for a poker hand database on one of the PostgreSQL lists got
me digging out my old play record–which is in a PostgreSQL database.
How the Poker industry ended up adopting PostgreSQL as its standard
database of choice for high-performance analyzing of hands and play
is kind of an interesting story. The great article Stranger than Fiction: The Story
of PokerTracker, Online Poker, and PostgreSQL covers a lot of the background here. I thought it would be fun to
revisit my own time in that land, and show just how that actually all
works at the database query level.

In April of this year the FBI did a
massive crackdown that has almost killed on-line poker in this
country, at least in its original form. I still have $35 that’s
stuck in an account I can neither play with nor withdraw from after,
perhaps my final “sin tax” on that era. But in the heady days
circa 2006, where seemingly everyone had a not quite legal Hold’Em
account they’d play from, I went through a poker apprenticeship that
included around 60,000 hands worth of play.

The first six months were frustrating and I slowly lost most of the money in my account.  But before wiping out I staged a comeback, won my losses back, and kept going up from there.  By the end of my serious playing time, “greg1104″
was the terror of the no-limit penny tables at Ultimate Bet, a tight
playing grinder who’d wait hours for just the right moment to turn
hyper aggressive. On a good night I’d wipe out a half dozen players
in a row by forcing them all-in to chase their speculative hands. If
you have a small “edge”–you’re slightly more likely to win a
hand than your opponent–and a large enough bankroll to support some
unfair losses (players call them “bad beats”), eventually the
worse player will end up broke.

To understand why this becomes a
database overflowing with info eventually, you need to know a bit about on-line poker. You
sit at a table that typically holds 6 to 10 people. Hands worth
playing are rare enough (some players will only intentionally ante to play a
hand less than 20% of the time) that any decent player will play at
least 2 or 3 of them at once on-line. The speed of a table is
measured in hands/hour, with a relatively fast typically going from
50 to 100. So even a journeyman player will have a hand possibility
pass in front of them every 30 seconds; you have to make a quick
decision about which to play and which to avoid. It doesn’t take
long until you have thousands of hands of history to use to make that
decision.

As a database and math oriented guy, I
quickly turned learning proper play into a query problem. What hands
did I play well? Which did I botch? And, statistically, what types
of players did I play well against? Around this same period,
software aimed specifically at this problem started appearing. Each
hand of Hold’Em you played could be exported into a “Hand History”
file that detailed exactly what happened. Import those into a
database, analyze information about each player, and you could put
numbers on things you only had a gut feel for–and sometimes learn,
the hard way, that your gut feel was wrong. PokerTracker was the
software of choice, and it moved from Access to PostgreSQL around the
same time I was playing regularly.

I don’t have all my hand histories, but
I saved enough of them (and kept moving them forward into new
versions of PostgreSQL/PokerTracker) to have a clear record of my
play during a period after I’d gotten good at the game. Here’s the
high level view of that play history, as a query against a
PokerTracker database:

SELECT count(*),sum(amt_won) FROM
player p

JOIN holdem_hand_player_statistics s
USING (id_player)

WHERE p.player_name=’greg1104′;

count,sum

33687,68.99

So 33K hands winning $68.99. Not
exactly World Series of Poker, but clearly I was a winning player.
If I racked up 33K hands worth of history data in this fairly brief period, you
can imagine how many a really serious player would have. The person
asking about this today had 5M of them to query.

This is an example of the a first-level
statistic you’d want about the other players at your table. You
should react one way to a raise from a player with a winning track
record over a long period, and a very different one from the guy
who’s been bleeding money the whole time he’s been playing. The
total is one thing; what percentage of the time did I win?

SELECT
flg_won_hand,count(*),sum(amt_won)

FROM player p

JOIN holdem_hand_player_statistics s
USING (id_player)

WHERE p.player_name=’greg1104′

GROUP BY flg_won_hand;

flg_won_hand,count,sum

f,30432,-1547.84

t,3255,1616.83

Typically I was playing at 10 player
tables, so winning around 10% of the time would be normal. And
that’s the case here. You can imagine how having these sort of
statistics on every player–and a much more complicated set, like the
percentage of time a player paid money to see the “flop” and play
a hand–accumulated over time as you played would be a huge
advantag.

What happens at a table is that good players quickly recognize
the other sharks who’ve come to snack, and you respectfully play
against them while at the same time gobbling down each new “fish”
(that’s their nickname) who appears. If you’re running
PokerTracker with its heads-up display, the easiest type of player to
take money from gets a little icon over their head showing a picture
of a fish.  There they are, labeled for you whether or not you were even paying attention to them, all based on some database driven statistics.

Not only does having stats on the other
players help, having your own lets you tune your play around your
personal strengths and weaknesses. In a Hold’Em hand, the two cards
you’re dealt before each hand begins are your “hole cards”. If
you’re any good, you should make a lot of your money when your hole
cards are the known set where your odds are good to win. That was certainly the
case for me:

SELECT hole_cards,count(*),sum(amt_won)

FROM player p

JOIN holdem_hand_player_statistics s
USING (id_player)

JOIN lookup_hole_cards USING
(id_holecard)

WHERE p.player_name=’greg1104′

GROUP BY hole_cards

ORDER by sum(amt_won)

LIMIT 10;

hole_cards,count,sum

AA,139,51.16

55,142,25.71

AKs,95,21.44

QQ,148,20.49

AKo,313,16.82

KK,153,14.82

TT,159,13.45

JJ,166,12.95

A8s,112,11.60

T9o,343,11.10

All the usual suspects:  AA, KK, QQ, AKs (suited, meaning they have the same suit, so likely to form a flush), AKo (off-suit), JJ, TT.  One of my specialties was playing
pocket pairs that turned into trio after the “flop” of 3 initial
cards aggressively, particularly against players who didn’t raise
quite enough to keep me out of the hand. This is how I ended up
winning so much with the slightly odd “55″ combination.  I was
quite good at spotting when someone else had AA, KK, or QQ before the
flop from their betting pattern.  If they raised enough that I spotted them, but not so much it wasn’t worth my trouble, I’ll call and see what happened.  If I got a matching third 5 on the flop (around 10% of the time it happens), I’d act like I was
bluffing to steal the hand, get them committed and betting aggressively at me, and then raise to all-in.  If they called, the hand was usually mine.  You have to fine-tune that strategy based on how aggressive the other player is, and I’d be doing it with a stack of heads-up data showing various numeric measurements of that aspect of their play.

If I reverse the order here to see my
worst hands, that’s where you can find even more valuable
information:

A4o,296,-13.83

QTo,300,-12.27

54o,294,-10.31

Q9s,113,-7.13

43o,294,-6.82

66,157,-6.52

J3o,294,-6.49

42s,114,-6.32

KJo,300,-6.02

A7o,348,-5.40

Apparently I got tripped up at least
once using the same technique on “pocket sixes” that made me so
much money on the “55″ combination; that’s just poker. But the
fact that I lost so often with QT, Q9, and KJ suggests my play around
handling those good but not great cards wasn’t quite right. And the
fact that I lost quite a bit, relatively, on 54 and 43 off-suit tells
me I was overvaluing the possibility of a low-end straight
considerably. This is very different fine-tuning from when I first started
playing, where I’d lose lots of money even playing strong cards. You
only have to lose a flush to a full house so many times before you
start considering that possibility for a second when you decide how
to bet.

By the time I gave up on playing online
poker, I was just starting to get to where I had my own custom
queries to run. A “win % adjusted aggression factor” was the one
I was working on at the end, basically separating out people who were
aggressive betters all the time from those who only did that when
they actually had a hand–what I did. With a few million hands of
data to research against, like the person asking on the list today has, fine tuning the statistics you collect and
want to overlay (in real-time, updating after each hand) on top of
the table heads-up display turns into a pretty difficult query
processing challenge. Given that the lack of an early Windows
version of PostgreSQL is often cited as one reason it didn’t get
better adoption, it’s kind of amusing to see it’s the database that
completely owns this space–which, like PokerTracker itself and all the gaming client software–is a
Windows only one.

When I was playing Texas Hold’Em, I
never thought I’d be offering advice on database tuning suggestions
as a way to improve play. But that really is where we’re at now.
It’s a shame there’s no good place for me to play on-line legally in the US
anymore, because it forces to fold the cards I’ve scribbled notes on outlining a book I’d name “High-Speed
Poker with PostgreSQL 9″. If that ever gets sorted out, and I find
myself with time to play again and a legal on-line venue to do it at
here, you can bet I’m all-in on that idea.