Some of this evaluation requires a lot of expertise in the PostgreSQL core code, but most of it requires little development experience at all. The more initial checking and review gets done before patches get evaluated by the experts, the less work those experts have to do. So: please step up and review a patch. There are patch review guidelines on the wiki, and it’s quite an accessible process. Step right up if you want to contribute a little back to the software you use and rely on, or if there’s a particular enhancement you want to make sure goes into 9.3.

If you don’t feel up to reviewing the feature you’re interested in, you can still help by testing and initially reviewing something else, so others have more time to examine the complicated stuff. You can still help out.

I’ll be posting a series on patches proposed for 9.3, starting with today’s patch, Peter Eisentraut’s addition of a ALTER ROLE extension to change PostgreSQL settings (like work_mem, search_path, etc) for all users on all databases. The initial post explaining the patch is here and this is the commitfest entry.

To get involved, visit the current commitfest page, sign in with your PostgreSQL community login, and have a look at the list of patches needing review. If there’s one there that doesn’t have an existing reviewer listed, or that interests you, please get involved! Grab the patch, read the reviewer guidelines, and help make PostgreSQL better.

If you use PostgreSQL on Cygwin, please try out this build fix, verifying that it works on Cygwin, and that it doesn’t break the Linux/BSD builds or the MinGW Windows builds. Your help would be appreciated in ensuring that Cygwin remains a supported platform into the future.

Interested users are needed to try out the following patches and report back with their experiences if you want to see these changes in 9.3:

The patches are:

Add a new event type of object_access_hook named OAT_POST_ALTER. This allows extensions to catch controls just after system catalogs are updated. Patch also adds sepgsql permission check capability on some ALTER commands, but not all.

https://commitfest.postgresql.org/action/patch_view?id=1003

This patch adds sepgsql support for permission checks equivalent
to the existing SCHEMA USE privilege:
https://commitfest.postgresql.org/action/patch_view?id=1065

This patch adds sepgsql support for permission checks almost
equivalent to the existing FUNCTION EXECUTE privilege:
https://commitfest.postgresql.org/action/patch_view?id=1066

This patch adds sepgsql the feature of name qualified creation label:
https://commitfest.postgresql.org/action/patch_view?id=1064

If you’re interested in SELinux, please glance at the discussion linked to in those patch entries, then grab a patch and try it out as per the reviewer guidelines:

http://wiki.postgresql.org/wiki/Reviewing_a_Patch

It appears that – at least on my build machine, a Windows 7 SP1 x64 box with Windows SDK 7.1, Visual Studio 2010 Express SP1 and Visual Studio Express 2012 on it – vcregress check will hang indefinitely with a postgres.exe process sitting at 100% cpu if the regression tests are run in a path that is too deep. This seems to happen with both x64 and x86 builds.

git.exe seems to have a similar problem, where a git clean -fdx in a deep directory tree will sit at 100% cpu forever, making no progress.

In both git.exe‘s and postgres.exe's cases, Process Monitor shows a steam of QueryNameInformationFile events with result BUFFER OVERFLOW. QueryNameInformationFile is the IRP_MJ_QUERY_INFORMATION operation of ZwQueryInformationFile as documented in MSDN here. It’s a kernel-level operation.

I’m yet to determine the root cause of the issue. To work around the problem, build in a shallower directory tree.

I’ve included a bunch of details after the cut, primarily to help anyone else with this problem find this post.

This works:

	cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\BT\release\SL_OS\windows\TA\x86\TO\2008\src\tools\msvc
	"perl" vcregress.pl check
============== creating temporary installation        ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 57532 with PID 100
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test tablespace               ... ok

This, with the exact same code, fails, hanging indefinitely on test tablespace:

	cd C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\tools\msvc
	"perl" vcregress.pl check
============== creating temporary installation        ==============
============== initializing database system           ==============
============== starting postmaster                    ==============
running on port 57532 with PID 128
============== creating database "regression"         ==============
CREATE DATABASE
ALTER DATABASE
============== running regression test queries        ==============
test tablespace               ...

The postgres.exe backend the test is running on sits at 100% cpu (or rather, 50% because it’s 100% of one core on a dual core box). It cannot be terminated via Task Manager or Process Explorer – End Task appears to succeed without error, but doesn’t actually kill the process. Neither does taskkill.exe even with the /F flag. Only a reboot seems to get rid of it. Sometimes subsequent attempts to kill it fail with a “permission denied” error.

When they’re hung, a backtrace from Process Explorer shows, for a 32-bit Pg running on a 64-bit host:

ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntdll.dll!NtCreateFile+0xa
wow64.dll!Wow64EmulateAtlThunk+0xe697
wow64.dll!Wow64SystemServiceEx+0xd7
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x2d
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x429
ntdll.dll!RtlUniform+0x6e6
ntdll.dll!RtlCreateTagHeap+0xa7
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!NtCreateFile+0x12
kernel32.dll!CreateFileW+0x4a
kernel32.dll!CreateFileA+0x36
postgres.exe!pgwin32_open+0xbc
postgres.exe!BasicOpenFile+0x1e
postgres.exe!GetNewRelFileNode+0xfa
postgres.exe!heap_create_with_catalog+0x1df
postgres.exe!DefineRelation+0x44e
postgres.exe!standard_ProcessUtility+0x425
postgres.exe!PortalRunUtility+0xa2
postgres.exe!PortalRunMulti+0x11b
postgres.exe!PortalRun+0x176
postgres.exe!exec_simple_query+0x3d1
postgres.exe!PostgresMain+0x5b5
postgres.exe!BackendRun+0x179
postgres.exe!SubPostmasterMain+0x31d
postgres.exe!main+0x1f4
postgres.exe!__tmainCRTStartup+0x122
kernel32.dll!BaseThreadInitThunk+0x12
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36

In Process Monitor, you see a long stream of QueryNameInformationFile events:

Date & Time:    16/01/2013 9:31:17 PM
Event Class:    File System
Operation:      QueryNameInformationFile
Result: BUFFER OVERFLOW
Path:   C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\data
TID:    4908
Duration:       0.0000015
Name:   \jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\da

with stack traces like this, again from a 32-bit Pg on a 64-bit host:

0              0xfffff88001420067      0xfffff88001420067
1              0xfffff88001421329      0xfffff88001421329
2              0xfffff8800141f6c7      0xfffff8800141f6c7
3              0xfffff80002589636      0xfffff80002589636
4              0xfffff800026b2bbe      0xfffff800026b2bbe
5              0xfffff800026b2f2f      0xfffff800026b2f2f
6              0xfffff800026b3947      0xfffff800026b3947
7              0xfffff800026b3a4b      0xfffff800026b3a4b
8              0xfffff800025fc3ef      0xfffff800025fc3ef
9              0xfffff800025d8162      0xfffff800025d8162
10             0xfffff800025d95f6      0xfffff800025d95f6
11             0xfffff800025daefc      0xfffff800025daefc
12             0xfffff800025e5b54      0xfffff800025e5b54
13             0xfffff800022e1253      0xfffff800022e1253
14      ntdll.dll       ZwCreateFile + 0xa      0x773c186a      C:\Windows\SYSTEM32\ntdll.dll
15      wow64.dll       whNtCreateFile + 0x10f  0x7388bff7      C:\Windows\SYSTEM32\wow64.dll
16      wow64.dll       Wow64SystemServiceEx + 0xd7     0x7387cf87      C:\Windows\SYSTEM32\wow64.dll
17      wow64cpu.dll    TurboDispatchJumpAddressEnd + 0x2d      0x73802776      C:\Windows\SYSTEM32\wow64cpu.dll
18      wow64.dll       RunCpuSimulation + 0xa  0x7387d07e      C:\Windows\SYSTEM32\wow64.dll
19      wow64.dll       Wow64LdrpInitialize + 0x429     0x7387c549      C:\Windows\SYSTEM32\wow64.dll
20      ntdll.dll       LdrpInitializeProcess + 0x17e4  0x773b4956      C:\Windows\SYSTEM32\ntdll.dll
21      ntdll.dll        ?? ::FNODOBFM::`string' + 0x29220      0x773b1a17      C:\Windows\SYSTEM32\ntdll.dll
22      ntdll.dll       LdrInitializeThunk + 0xe        0x7739c32e      C:\Windows\SYSTEM32\ntdll.dll
23      ntdll.dll       ZwCreateFile + 0x12     0x775700a6      C:\Windows\SysWOW64\ntdll.dll
24      KERNELBASE.dll  CreateFileW + 0x35e     0x7628c5ef      C:\Windows\syswow64\KERNELBASE.dll
25      kernel32.dll    CreateFileWImplementation + 0x69        0x76a83f86      C:\Windows\syswow64\kernel32.dll
26      kernel32.dll    CreateFileA + 0x37      0x76a853e4      C:\Windows\syswow64\kernel32.dll
27      postgres.exe    pgwin32_open + 0xbc, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp
\src\port\open.c(77)     0x13eba5c       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src
\test\regress\tmp_check\install\bin\postgres.exe
28      postgres.exe    BasicOpenFile + 0x1e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\x
p\src\backend\storage\file\fd.c(560)     0x12eec6e       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_
TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
29      postgres.exe    GetNewRelFileNode + 0xfa, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_
os\xp\src\backend\catalog\catalog.c(578) 0x11a836a       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_
TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
30      postgres.exe    heap_create_with_catalog + 0x1df, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg
_target_os\xp\src\backend\catalog\heap.c(1073)   0x11adf8f       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH
\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
31      postgres.exe    DefineRelation + 0x44e, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\commands\tablecmds.c(636)        0x11fb98e       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
32      postgres.exe    standard_ProcessUtility + 0x425, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\utility.c(535)     0x130f425       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
33      postgres.exe    PortalRunUtility + 0xa2, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1191)     0x130d022       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
34      postgres.exe    PortalRunMulti + 0x11b, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(1320)      0x130d16b       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
35      postgres.exe    PortalRun + 0x176, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\pquery.c(815)    0x130da06       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
36      postgres.exe    exec_simple_query + 0x3d1, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(1053) 0x130ba61       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
37      postgres.exe    PostgresMain + 0x5b5, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\tcop\postgres.c(3969)      0x130c345       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
38      postgres.exe    BackendRun + 0x179, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(3987)        0x12c9459       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
39      postgres.exe    SubPostmasterMain + 0x31d, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\postmaster\postmaster.c(4491) 0x12cd08d       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
40      postgres.exe    main + 0x1f4, c:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\pg_build_type\release\pg_host_os\windows\pg_target_arch\x86\pg_target_os\xp\src\backend\main\main.c(175)   0x123fc24       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
41      postgres.exe    __tmainCRTStartup + 0x122, f:\dd\vctools\crt_bld\self_x86\crt\src\crtexe.c(554) 0x13f7366       C:\jenkins\workspace\2ndquadrant-postgresql-qa-windows\PG_BUILD_TYPE\release\PG_HOST_OS\windows\PG_TARGET_ARCH\x86\PG_TARGET_OS\xp\src\test\regress\tmp_check\install\bin\postgres.exe
42      kernel32.dll    BaseThreadInitThunk + 0xe       0x76a833aa      C:\Windows\syswow64\kernel32.dll
43      ntdll.dll       __RtlUserThreadStart + 0x70     0x77589ef2      C:\Windows\SysWOW64\ntdll.dll
44      ntdll.dll       _RtlUserThreadStart + 0x1b      0x77589ec5      C:\Windows\SysWOW64\ntdll.dll

PostgreSQL its self is pretty easy to compile on Windows. You download and install ActiveState Perl and Visual Studio or the Microsoft Windows SDK, unpack a PostgreSQL source tree, copy config_default.pl to src\tools\msvc\config.pl, edit it to reflect your environment, open an SDK command prompt for your Windows SDK version / Visual Studio version, cd to the PostgreSQL directory, and run src\tools\msvc\build.pl. Not too bad.

The trouble is getting the dependencies built, and that’s what I’m working on improving. The scripts I’ve published at http://github.com/2ndQuadrant/pg_build_win are a step toward that. I’ve written some NMake files and a wrapper Perl script that will download dependencies, compile them, and create a config.pl that points the PostgreSQL build at them. It can build a source tree you’ve checked out yourself, or it can automatically handle checking your specified PostgreSQL version(s) out from git.

At present the scripts are known to work with the Microsoft Windows SDK 7.1, Microsoft’s free stand-alone SDK. They do not yet work with Visual Studio, but that should be along shortly.

Currently the tools document the unattended installation procedure for the required 3rd party tools and libraries (Perl, Python, TCL, Windows SDK, etc) and will download and compile zlib for you. I want to add support for more of PostgreSQL’s optional dependencies, automating their installation to prevent everyone from having to suffer through the “fun” of compiling open source libraries like gettext on Windows. It’s just a matter of having the time. The mismash of precompiled binaries recommended by the documentation for compiling on Windows won’t work for x64 builds and isn’t very safe anyway; ideally all the libraries should be compiled with the same compiler and linked to the same runtime.

These scripts are primarily intended to satisfy internal requirements for compiling a variety of PostgreSQL versions and configurations under a Jenkins CI server, but I’d value any use reports or feedback.

Oh: Why perl? Because the PostgreSQL build scripts are written in Perl, so it’s the obvious first choice. I would’ve preferred Python or Powershell, but this way there isn’t an additional dependency above what PostgreSQL its self requires. NMake alone doesn’t offer the text processing features I needed to (for example) extract the Windows SDK version from the output of cl.exe.

To try the scripts out, see the the README in the repository. I won’t repeat that documentation here.

The installer already contains checks for several kinds of broken Windows install. For example, it tests to make sure that %TEMP% is writeable, and to makes sure the vbscript interpreter actually works. These were both the cause of frequent problem reports to the mailing lists in the past.

This is a less common issue, though it’s clearly turned up in the wild before, as shown by this report and this one. It turns out that some – probably rare – Windows installs have an incorrect %COMSPEC% environment variable. This causes popen to fail with the wonderfully useful error message: "No error" when initdb tries to execute the PostgreSQL backend. The message displayed to the user is:

Problem running post-install step. Installation may not complete correctly. The database cluster initialisation failed.

… which can be caused by several different issues, of which this is only one.

If this is your issue, when you examine the installer log file, %TEMP%\postgresql-installer.log, you’ll find that it contains:

Executing cscript //NoLogo "C:\Program Files\PostgreSQL\9.2/installer/server/initcluster.vbs" "NT AUTHORITY\NetworkService" "postgres" "****" "C:\Program Files\PostgreSQL\9.2" "C:\Program Files\PostgreSQL\9.2\data" 5432 "DEFAULT"
Script exit code: 1

then a bit further down in the log you’ll see:

creating template1 database in C:/Program Files/PostgreSQL/9.2/data/base/1 ... initdb: could not execute command ""C:/Program Files/PostgreSQL/9.2/bin/postgres.exe" --boot -x1 -F ": No error

If so, check to make sure your ComSpec environment variable is valid. Open a command prompt and run:

"%COMSPEC%" /C "echo test ok"

If it prints “test ok” then it’s likely fine and the issue is probably something else. If it fails, you’ve found the problem. Either way, check the value with:

echo %COMSPEC%

It should print something like:

C:\Windows\system32\cmd.exe

where C:\Windows is the location of your Windows install as shown by echo %WINDIR% or echo %SYSTEMROOT%.

In the case of the issue I was investigating it was instead:

%SystemRoot%\system32\cmd.exe;

It’s valid, though not standard, to have %SYSTEMROOT% instead of C:\Windows, so that wasn’t the issue. The problem is the trailing semicolon (‘;’). This single extra character was causing initdb to fail to launch postgres.exe during database cluster creation because the popen(...) call was failing. The popen call requires that ComSpec point to a valid shell.

The error was fixed by correcting the COMSPEC environment variable. We opened the System control panel, opened Advanced system settings, and in the Environment Variables section under System variables edited ComSpec to remove the trailing semicolon.

One character.

I had the chance to meet with a high level AWS support engineer last week. It’s prompted me to write up the basics of configuring EC2 instances for decent PostgreSQL performance. I haven’t had the chance to back the following advice with hard numbers and benchmarks yet, so remember: Always test everything with a simulation of your workload.

Before I can get into the configuration details, I need to outline how EC2 storage works.

EC2 storage types

EC2 instances have two very different storage options. These are explained quite well in the storage documentation, so I won’t repeat the details. It’s sufficient to say that the Instance Store is local to the VM and is unrecoverably lost if the VM is stopped. It is backed by local hard drive or SSD storage. EBS by contrast is durable and isn’t lost when the VM is stopped. It is more like NBD or iSCSI than local storage; it’s a network block device protocol with the corresponding latency and throughput issues that entails.

If you’ve been facing performance issues, you might’ve seen the High I/O instance types and thought “That sounds ideal for my database workload”. I thought so myself – but they won’t actually make any difference if your database storage is on EBS volumes. The High I/O instances have fast instance store storage, but aren’t any different in terms of EBS. So if you’re been using a High I/O instance with a database that’s on EBS volumes you’re not gaining any benefit from the enhanced instance store I/O you’re paying for, and are better off on an EBS-optimized large instance.

Durable databases

For a durable database where you care about your data, what you want instead of a high I/O instance is an EBS Optimized instance, which has guaranteed network bandwidth to the EBS storage servers. Use EBS volumes with provisioned IOPs and, for best results, stripe a group of EBS volumes into a RAID10 array. See increasing EBS performance.

You might also want to consider putting your temporary tablespaces on the instance store, rather than EBS. That way things like on-disk sorts won’t compete with other data for EBS bandwidth, and will get to use probably-faster local storage. (See also tablespaces).

As always, if you care about your data, back it up and use replication and/or PITR.

This is just the basics – there’s a lot more tuning, testing and performance work that can be done from here, not to mention work on HA, automated backup, and other data protection measures. Just using the right instance type will get you to vaguely reasonable performance; you won’t get the most out of your (virtual) iron without a lot more work.

Non-durable instances

If you have data you can re-generate, or you’re running streaming replicas from a master that you can just re-clone from the master, you don’t necessarily need durable storage.

In this case, consider using the instance store, which is available on every instance type except Micro.

Since you have no hope of recovering the database if the instance terminates or stops anyway, you might as well use fsync=off and full_page_writes=off in postgresql.conf, saving a lot of I/O and synchronization. This is one of the few situations where turning fsync off is acceptable; never do it for data you expect to preserve, as you’re effectively giving the database permission to corrupt your data if the power fails or the host OS crashes.

Since you’re using the instance store, you can also potentially benefit from one of the high I/O instance types, using an array of SSDs for extremely fast seeks and high IOPS.

Do not use the instance store for data you want to preserve!. Use it only for analytics work on data sets you can re-generate, or for replicas of data where the master lives elsewhere.

A non-durable storage based setup needs extensive testing and is likely to need some different tuning and configuration to a normal PostgreSQL install. Don’t expect amazing performance out of the box, you’ll need to do more than just fire up a High I/O instance and set up a default Pg install on it.

Future work

The next step, time permitting, is to quantify the above information with hard numbers. How does a High I/O instance with the DB on instance store perform compared to an EBS optimized x.large with a RAID10 striped set of provisioned I/O EBS volumes? Or the ubiquitous micro instance?

I’d like to look at integrating S3 and Glacier support into barman and repmgr down the track, as it’d be really interesting to have basebackups and WAL automatically stored in S3, archived to Glacier, used to provision new instance-store based replica servers, and more.

I’ll be doing more with EC2 in the coming months, so I expect to be able to offer more insight into future performance issues.