Tag: postgres-xl

Jul52017 by Pavan Deolasee6 Comments

News and Roadmap from Postgres-XL 10

We expect Postgres-XL 10 to be released in the next few months, with the new features of partitioning and logical replication. You’ll be able to load Postgres-XL directly from PostgreSQL. For the first time, you’ll be able to run massively parallel queries both across the datanodes and within the datanodes to give huge performance gains. Earlier today, we updated the master branch of the Postgres-XL repository to include all commits from PostgreSQL's master branch up to June 26. That means the XL project is now fully to date with the PostgreSQL source code, meaning there is now only minimal lag between PostgreSQL and Postgres-XL. At this point the code is only in Development/Alpha, though we expect to produce Beta1 soon after PostgreSQL Beta 3 in August. Support from the (more…)

May262017 by Tomas Vondra4 Comments

Future of Postgres-XL

PostgreSQL, Tomas' PlanetPostgreSQL

You probably know that Postgres-XL is a distributed database based on PostgreSQL. A few days ago we pushed the XL 9.6 code into the public git repository. Additional details about the new stuff available in Postgres-XL 9.6 are available here. The topic of this blog post is quite different, though. I'd like to discuss some changes to the project management and development practices, and why (and how) we plan to tweak it. (more…)

May122017 by Tomas Vondra3 Comments

What’s new in Postgres-XL 9.6

2ndQuadrant, PostgreSQL, Tomas' PlanetPostgreSQL

For the last few months, we at 2ndQuadrant have been working on merging PostgreSQL 9.6 into Postgres-XL, which turned out to be quite challenging for various reasons, and took more time than initially planned due to several invasive upstream changes. If you’re interested, look at the official repository here (look at the “master” branch for now). There’s still quite a bit of work to be done - merging a few remaining bits from upstream, fixing known bugs and regression failures, testing, etc. If you’re considering contributing to Postgres-XL, this is an ideal opportunity (send me an e-mail and I’ll help you with the first steps). But overall, Postgres-XL 9.6 is clearly a major step forward in a number of important areas. (more…)

Oct242016 by Pavan Deolasee2 Comments

Postgres-XL Scalability for Loading Data

Pavan's PlanetPostgreSQL

In my last blog, we looked at the benchmark results from bulk load test for a Postgres-XL database cluster. Using a 16-datanode, 2-coordinator cluster, running on EC2 instances, we could easily clock 9M rows/sec or 3TB/hr of ingestion rate. That’s a significant number in itself. In this blog, we’ll see if the ingestion rate is scalable in Postgres-XL. In particular, we’ll try to answer if adding more nodes to the cluster can result in a linear increase in performance. Let’s use the same line item table from the TPC-H benchmark for these tests. We'll increase the cluster size from 16 datanodes to 20 datanodes and then further to 24 datanodes. We'll also repeat the tests with 1, 2 and 3 coordinators respectively. For all these tests, we are using i2.xlarge EC2 instance for a (more…)

Oct132016 by Pavan Deolasee6 Comments

Load data in Postgres-XL at over 9M rows/sec

2ndQuadrant, Pavan's PlanetPostgreSQL

We are faced with this question: “What’s the ingestion rate of Postgres-XL?”, and I realised I don’t have a very good answer to that. Since recently we made some good improvements in this area, I was curious to know too. Well, I decided to benchmark. Hardware and Software For the tests, I used a Postgres-XL cluster running on EC2 instances. Since COPY has to go through the coordinator, it seemed reasonable to use a compute-optimised c3.8xlarge instance for running coordinator. Similarly, for datanodes, storage-optimised i2.xlarge instances are more appropriate. Both these instances have attached SSD disks, though i2.xlarge instance has more storage than the c3.8xlarge instance. So the next question was how to generate data for the test? We’d used TPC-H benchmark for (more…)

May122016 by Pavan DeolaseeNo Comments

Postgres-XL 9.5r1.1 released

2ndQuadrant, Pavan's PlanetPostgreSQL

The Postgres-XL community has released a new version of the software, tagged as 9.5r1.1. This is a minor update to earlier 9.5r1 release, but contains some very important bug fixes. I would encourage everyone to immediately upgrade to this latest release. Being a minor release, you only need to download the latest sources from , compile, install new binaries and restart your servers. Major Improvements This release is fully caught up with the latest PostgreSQL 9.5 minor release i.e. it includes all bug fixes from PostgreSQL 9.5.3 version, also released today. List of important bug fixes in PostgreSQL 9.5.3 can be found here. Apart from that, it contains following important bug fixes: Fix a nasty bug in CLOG and Subtrans Log management which may lead to severe data corruption. Add a (more…)

Apr182016 by Pavan Deolasee2 Comments

Postgres-XL 9.5r1 has finally arrived!

2ndQuadrant, Pavan's PlanetPostgreSQL

After months of efforts, I'm pleased that Postgres-XL 9.5r1 is seeing the daylight. It has been tremendous collective efforts by many, both inside and outside 2ndQuadrant. Often it's not visible via commit history or mailing list communications, but I must admit that many folks have contributed in making this grand release. Contributors who wrote code and sent ideas, those who reviewed the code, QA team for testing and enhancing regression coverage, those who did benchmarks to show how we are doing, those who wrote automation to deploy the product and of course the user community for testing the product and providing invaluable feedback. So thank you folks! Over the year, I've periodically wrote about various enhancements we made to the product. But let me summarise some of them here (more…)

Mar302016 by Pavan DeolaseeNo Comments

Postgres-XL 9.5R1Beta2 Released!

Pavan's PlanetPostgreSQL

The Postgres-XL 9.5R1Beta2 release went out yesterday. It's another step forward to have a stable 9.5 release sometime very soon. A few key enhancements from the last beta release are captured in this blog. For the full list, I would recommend to read the release notes. Support for binary data transfer for JDBC and libpq If you'd trouble receiving data in binary format via JDBC or libpq, this release should fix those issues for you. The coordinator now have intelligence to figure out if the client is requesting data in binary format and handle those cases correctly. Pushdown of Append and MergeAppend plans One of the problems reported with the beta1 release was that for inherited tables, coordinator fails to pushdown plans to the remote nodes even when its possible. A simple example (more…)

Mar152016 by Pavan Deolasee4 Comments

Sharding: Bringing back Postgres-XL technology into core PostgreSQL

Pavan's PlanetPostgreSQL

Sharding or horizontal scalability is a popular topic, discussed widely on PostgreSQL mailing lists these days. When we started the Postgres-XC project back in 2010, not everyone was convinced that we need a multi-node PostgreSQL cluster that scales with increasing demand. Even more likely, we, the PostgreSQL community, were skeptical about whether we have enough resources to design and implement a complex sharding architecture. Also, there was so much work that we could have done for vertical scalability and the obvious choice for many was to focus on the vertical scalability aspects of performance. With great work that Robert Haas, Andres Freund and others have done in the last few releases, law of marginal utility is now catching up with the vertical scalability enhancements. Now, (more…)

Feb182016 by Pavan Deolasee1 Comment

Postgres-XL 9.5R1Beta1 is here!

2ndQuadrant, Pavan's PlanetPostgreSQL

I'm extremely pleased to see that after months of efforts and contributions by different people around the world, Postgres-XL 9.5 R1 Beta1 has finally arrived. This release is significantly better, in all respects such as performance, stability and high availability, as compared to the past release. Enormous amount of work has gone into PostgreSQL in the last few years and majority of that is now available in this release of Postgres-XL 9.5R1 which is based on the latest PostgreSQL release. If you're looking for a PostgreSQL based technology that can support terabytes of data, massively parallel processing, supports OLAP and OLTP workloads, offers a consistent view of the entire cluster and is compliant with PostgreSQL as far as client APIs are concerned, have a look at the latest (more…)