For the sake of simplicity, we assume a typical scenario which involves taking full backups once a week through the “barman backup” command. Suppose you want to automatically keep the latest 4 backups and let Barman automatically delete the old ones (obsolete).

The main configuration option for retention policies in Barman is “retention_policy” which can be defined both at global or server level. If you want all your servers by default to keep the last 4 periodical backups, you need to add in the general section of Barman’s configuration file the following line:

[barman]
... // General settings
retention_policy: REDUNDANCY 4

When the next “barman cron” command is executed (every minute if you installed Barman using RPMs or Debian/Ubuntu packages), Barman checks for the number of available full periodical backups for every server, order them in descending chronological order (from the most recent to the oldest one) and deletes backups from the 5th position onwards.

In case you have several servers backed up on the same Barman host and you want to differentiate the retention policy for a specific server, you can simply edit that server configuration section (or file, see “Managing the backup of several PostgreSQL servers with Barman“) and define a different setting:

[malcolm]
description = Malcolm Rocks
ssh_command = ssh malcolm
conninfo = host=malcolm port=5432 user=postgres dbname=postgres
retention_policy: REDUNDANCY 8

However, Barman allows systems administrators to manage retention policies based on time, in terms of recovery window and point of recoverability. For example, you can set another server to allow to recover at any point in time in the last 3 months:

[angus]
description = Angus Rocks
ssh_command = ssh angus
conninfo = host=angus port=5432 user=postgres dbname=postgres
retention_policy: RECOVERY WINDOW OF 3 MONTHS

Make sure you have enough space on the disk to store all the WAL files for every server you back up, and always monitor “barman check” through your alerting tools (such as Nagios/Icinga/Zabbix/etc.).

Current implementation of retention policies in Barman has some limitations: retention policies are managed only automatically (not manually – this would require to create a “barman delete –obsolete” command, for example) and there is no decoupling yet between full backups and WAL archive transactional logs (we have already thought of the “wal_retention_policy” option, but at the moment it is not handled).

More detailed information on retention policies can be found on Barman’s documentation website.

Barman 1.2.0 introduces automated management of backup retention policies of PostgreSQL servers.

Retention policies were one of the midterm goals that we had given ourselves when the whole Barman idea began to take shape. I am glad that, thanks to the vision of a French company (which explicitly requested to remain anonymous), we were able to undertake their open-source development.

What are retention policies?

A retention policy is a user-defined set of guidelines and principles that determines how long backups and their related archive logs need to be retained for recovery procedures. In a PostgreSQL database scenario:

• with “backup” we refer to a full physical backup, performed periodically, when the database is online (hence the term “hot” very often used in this context);
• with “archive logs” we refer to Write Ahead Log (WAL) files, responsible for implementing differential backup by continuously archiving every change made to the data files of the database server.

Base backups and archive logs form the so-called “backup catalogue” and allow database administrators to perform Point-In-Time recovery operations.

Further information on PostgreSQL physical backup and continuous archiving can be found in the Postgres documentation.

Why are retention policies so important?

In some countries and environments, it is the law that requires ICT departments to enforce them for data security and protection reasons. In Italy, for instance, the “Codice dell’Amministrazione Digitale” (CAD) requires that public organisations report retention periods in official documents such as their Disaster Recovery Plan.

From a more practical point of view, retention policies improve automation of a backup solution, while reducing the management and configuration complexity.

How are they implemented in Barman?

Barman at any time retains:

• the periodical backups required to satisfy the current retention policy for a given PostgreSQL server;
• the archived WAL files required for the complete recovery of those backups.

On the same topic, you may wish to read one of my previous blog posts about the backup catalogue and the WAL archive in Barman.

Barman users can define a retention policy in terms of:

• backup redundancy (how many periodical backups), or
• a recovery window (how long).

In case of a retention policy based on redundancy, the administrator decides how many periodical backups to keep.

On the other hand, a retention policy based on recovery window allows the DBA to specify a period of time (recovery window). Barman ensures retention of backups and/or archived WAL files required for point-in-time recovery to any time during that recovery window.

Retention policies are managed in Barman by the configuration option ‘retention_policy’ and the ‘barman cron’ command (responsible for maintenance operations). In the next article we will go through the configuration of retention policies with some simple examples.

Where can I find Barman?

Barman is an open source application for disaster recovery developed and maintained by 2ndQuadrant. It is written in Python and it can be installed through sources or PyPI. However, RPM packages and Debian packages are available for installation on RHEL/CentOS 5/6, Debian or Ubuntu 12.04 LTS.

More information on Barman can be found on the website (www.pgbarman.org) and the available documentation.

The event took place in Melbourne, at the offices of Experian Hitwise in St Kilda Rd. I must confess that, going back to the same building where I had worked in 2005/2006, was a strange but positive feeling. Hitwise is a long time Postgres user and it would appear that their cluster of PostgreSQL databases has grown to become one of the largest in the world. So, a double-special occasion for me!

After the opening session introduced by Jason, Josh Berkus gave an overview of PostgreSQL 9.2 with his “Full throttle” presentation.
I followed with an introduction to PostgreSQL, more focused on business reasons for its adoption in business contexts. My talk, entitled “Why use PostgreSQL? 10 reasons for using it”, is the result of several years of experience with Postgres, both as a member of the community and a consultant for 2ndQuadrant.

Over the years I have been repeatedly asked questions from both existing and potential new users. From these questions I have tried to form topics and have come up with 10 broad reasons.

The last talk of the morning was about the very interesting MADLib library for database analytics.

In the afternoon, following Josh’s talk on performance, I brought Barman DownUnder and introduced the audience to Disaster Recovery concepts and methodology using our open source tool developed at 2ndQuadrant. It was a good occasion to promote the new 1.2.0 version which introduces retention policy management.

Then it was over to Jason Godden who gave an extended review and coverage of replication tools with PostgreSQL. For some of them he did provide a demo session as well.

We were very pleased with the number of participants and overall turnout, considering it was the first event of its kind. At any one time there was always a minimum of 40 participants, while in some cases there were over 60 people present (including of course staff from Hitwise).

Finally, many thanks to Jason Godden of the Melbourne PostgreSQL Users Group, who organised the event at the local level and who arranged to have Experian Hitwise host the conference and offer food and beverages throughout the day.

I am confident that some PostgreSQL users and fans that were here today in Melbourne can form an active group, led by Jason, and make PGDay a recurring national event.

The momentum continues as PGDay national events now expand to include the Asia Pacific region. Australia joins the long list of countries which have embraced the Community spirit that is the fundamental principle of the PostgreSQL project: Brasil, China, Argentina, Ecuador, Canada, Italy, France, Germany, UK, USA, Japan, Holland, Czech Republic, …

I chose an AC/DC song title because not only they are an Australian band, one of the greatest of all times in fact, but they also named a lane way in Melbourne after them.

The event has been organised thanks to Jason Godden from the Melbourne PostgreSQL Users Group and will be hosted by Experian Hitwise in St Kilda Road. It will be a special location for me as I have had the pleasure to work for Hitwise.

I will be presenting two talks regarding PostgreSQL. Even though the schedule has not been decided yet, my intention is to have a general talk about the PostgreSQL Project in the morning and another one about disaster recovery and high availability of Postgres databases in business critical environments.

The event is free of charge but requires registration.

Cheers, mate. See you there!

With Barman you can use the backup host as a simple backup server, and simply store backup data regarding different PostgreSQL version servers. Or plan to use it for recovery as well (local recovery). In this case you need to install every PostgreSQL version for which you need to perform local recovery.

In these first 5 months of Barman’s life, we have received a few requests from users (mainly system administrators) around the world asking to simplify the management of backups for several databases. My personal opinion is that this use case is typical of hosting companies or big organisations.

The upcoming 1.1.2 release of Barman will include the so called configuration files directory setting, which allows system administrators and DBAs to specify a directory that will contain server directives. For example, you can define in Barman’s global configuration section the following option:

configuration_files_directory = /etc/barman.d

This option will make Barman look for *.conf files in that directory and evaluate them on the fly. Suppose you have to manage the backup of 6 or more PostgreSQL instances, you can specify a different file for each of those instances and name it in a meaningful way:

|- /etc
  |- barman.d
    |- 01-angus.conf
    |- 02-malcolm.conf
    |- 03-bon.conf.disabled
    |- 04-brian.conf
    |- 05-phil.conf
    |- 06-chris.conf
    |- ...

Then specify a server configuration in /etc/barman.d/01-angus.conf:

[angus]
description = Angus Rocks
ssh_command = ssh angus
conninfo = host=angus port=5432 user=postgres dbname=postgres

Adding a new server or disabling/removing a server becomes as trivial as adding a new file or renaming it.

If you type “barman list-server” you get:

angus - Angus Rocks
brian - Brian screams
chris - Chris on the bass
...

This behaviour is quite spread on Linux applications and allows to improve the integration of Barman with configuration managers such as Puppet.

More information on this feature is available in the online documentation of Barman, in the basic configuration section.

Another interesting feature added in version 1.1.2 is the shortcut ID for backups. Barman now manages shortcut aliases for backup identifiers such as latest/last or oldest/first.

This means that you can now type “barman delete SERVER_ID oldest” and easily delete the oldest backup available for that server, or type “barman show-backup SERVER_ID latest” to get information about the latest backup available for that server. Cool, isn’t it? ;)

So, stay tuned for version 1.1.2! Ciao!

As president of the Italian PostgreSQL Users Group (most likely for the last time) and host of Prato, I had the pleasure to open the event and welcome all the attendees – coming from 11 regions of Italy. 79 real attendees (PostgreSQL users and potential ones), plus 16 among speakers and staf, for a total of 95 participants. It is a remarkable result for such a specific open source advocacy event!

95% of the attendees were from the central and northern parts of the peninsula, 4% of them from the southern part.

Organising a PGDay is always a tiring activity. However, it can be very rewarding too. Especially if you see that every year, more and more people are getting closer to PostgreSQL.
Our role as PostgreSQL advocates, is to let them know what Postgres is already capable of doing so that they can evaluate its usage from a strategic point of view in their organisation.
I was particularly happy with my talks on migrations from Oracle (a lot of public administrations in Italy are forced to take this into consideration and need to be reassured, not only from a technical point of view), very large databases and business continuity.
The latter one is probably my favourite topic of the moment, especially due to the role that Barman is slowly acquiring in the Postgres eco-system.

I’d like to thank everyone at ITPUG, all the volunteers that gave a bit of their time to assist visitors and to help them before, during and after the event.

A few things I’d like to see more next year:

• more international speakers (even though this proves that the level of Italian speakers is high);
• more tutorials and introductory talks – maybe over two days;
• more social events, including a concert: the Florentine steak was awesome (ask Simon, even though he may have preferred it “more” cooked), with almost 30 attendees, as well as the beers offered by our sponsor Interlogica.

Finally, I’d like to thank the Monash University Prato Centre, for their services and the location – which seems to follow PostgreSQL’s steps and improve every year.

So, I hope to see you next year!

I was not able to attend the event in Stuttgart and Amsterdam, but I was active in Prato (2007 and 2008) and present in Paris (2009). However, I must admit that this year conference is by far the PostgreSQL conference I have enjoyed the most. Great city, very good location with excellent facilities.

I expect the positive trend I have experienced so far will be met next year as well, whichever location will be picked.

Finally I also had the chance to fully enjoy an event as attendee, not having been involved in the organisation or not having to present any talk. I attended 15 talks (ranging from average to very high quality), plus keynotes. It was also nice to catch up with international 2ndQuadrant colleagues, meet again long time members of the project and – most notably – make acquaintance with new members.

My first thought goes to all the volunteers that made this possible (in particular local ones), PostgreSQL Europe and the sponsors. Thank you for bringing the conference to such a high quality level.

Then follows my “top 5″ list for PGConf.eu 2012 in Prague.

Number 1: the cruise on the Vltava river, sponsored by Heroku, which was a great opportunity for community members to get to know each other. I had also the chance to speak about Barman with Gianni, Mladen, Lou, Tony, Palle, et al.. It was such a pleasure, and once again the feedback we received on Barman was very positive. The picture on the right shows myself (actually my hand) explaining the WAL archive concept in Barman, with periodical backups, differential backup, incoming directories, compression, etc. using pistachio shells and toothpicks (thanks to Mladen for sharing the picture).

Number 2:  the inspiring talk by Simon Riggs and Andres Freund regarding future directions for the development of multi-master replication and polymorphic logical replication through WAL decoding: I could not stop thinking about those features (actually desiring them), which will definitely bring PostgreSQL in a leading position in the market – in absolute terms.

Number 3: Jehan-Guillaume de Rorthais’ talk on the integration of PostgreSQL with a cluster resource manager such as Pacemaker for automatic failover in environment with high requirements of availability and business continuity. Even though I am a fan manual failover, I believe the perfect integration of Postgres with this kind of tools is an important path to be taken.

Number 4: the work my friend Peter is doing with Greg on in-database logging will draw future monitoring and log analysis tools. We do not yet see the benefits from this work, but we will as this kind of data will be collected and analysed in order to produce useful reports for DBAs. Their talk was one of the best, in terms of both content and delivery.

Number 5: the talks by Vincent Picavet on PostGIS, pgRouting and related technologies. GIS are so interesting!

In my reserve list: Gianni’s pgChess (great educational project!), Harald’s closing keynote (and his meteorite slide), Magnus excellent talk on backup strategies (the only slide I think his talk was missing was an overview of the existing tools for disaster recovery with PostgreSQL – in particular Barman, WAL-E and OmniPITR just to name a few), Jonathan Katz’s (always nice to see him communicate and interact with the audience), Dimitri’s on High Availability (the part I was able to attend was great).

I excuse myself with the other half of the speakers which I was not able to attend due the impossibility of real replication/mirroring of my body.

So, a big “thank you” to all that made this possible and congratulations.

Next year, I would like to candidate Italy again, with two options: Prato and Florence. I will be happy to provide concrete proposals for both if the community is interested. Right now, there is the Italian PGDay to think about, in just 3 weeks. :)

On the dawn of the release of version 1.1.0 of Barman, we are proud to publish our first video/spot for the promotion of our open source tool. The goal of the video is to emphasize the importance of physical hot backups and disaster recovery solutions for PostgreSQL databases operating in business critical environments.

One of the most important news for version 1.1.0 is the availability of Debian and Ubuntu packages (Barman will be officially shipped with Ubuntu 12.10), thanks to our Debian Developer Marco Nenciarini.

Even though version 1.0 came out last July, Barman will be officially presented next Monday, October 15 in Prato (Italy), during the first workshop on disaster recovery of PostgreSQL databases with it.

Enjoy this video.

International speakers are more than welcome to submit their proposal through the “International call for papers” section of the website. In the submission, we suggest international speakers to specify their travel expenses – as organisers we will evaluate them according to the budget.

Important: you are not required to present your talk in Italian. :)

Italian PGDay will be held in Prato, at the Monash University Centre, on November 23.

As European citizen, I have always believed that public administrations throughout Europe should use open source software and promote its development in ICT contexts, favouring the growth of local enterprises and know-how/education over licensing costs.

September 19 2012 could be an important date for the PostgreSQL community as well as the open-source/free software communities in general. The relevance of this letter from the French Prime Minister, Jean-Marc Ayrault, is extremely high from an advocacy point of view.

Maybe I am just dreaming or simply call me a fool, but I wish this could be a small “French Revolution” for the ICT sector, that, starting from Paris, spreads all over Europe.

As Italian citizen, I hope the Italian Government piggy-backs the French government on this, starting from the unconcluded works of the “Commissione Nazionale per il Software Open Source nella PA” (National Commission for Open Source Software in the Public Administration).

In Macro-Economy terms, this could be an important step towards the reduction of public expenditures and – consequently – debt, as well as an incentive for local entrepreneurship. I just hope that politicians and public managers can see that.

Can you imagine if only a tiny bit of proprietary database licenses costs could be diverted to open source investements and developments of new features of PostgreSQL core?

Finally, I would like to thank all the French and French speaking contributors (including official documentation translators) that, throughout the years, have made this possible, and I hope we could all follow your example.