The rise and fall of CentOS
If it weren’t for RedHat Linux, I might not have ever found PostgreSQL. In early 2001, I was running the tech side of a company with an ISP component to it. After starting with RedHat 4.2 in 1997, we’d standardized all the servers by then on RedHat 7.0, refusing to upgrade from its 2.2 kernel even when new versions with 2.4 appeared. (We needed VPN Masquerade and it didn’t work on those crazy bleeding edge 2.4 kernels) Into the business came a new product that needed a database embedded into it. As a former Progress DBA of some reknown (seriously!), I got the job of figuring out which to use. After poking around at the open-source database options that were a simple rpm install away, I found PostgreSQL 7.0. It seemed completely sensible for a small but important database. BerkleyDB was too simple for the queries I wanted to write, MySQL 3.23 was frighteningly loose with its data to me, and I was still pissed at Oracle for killing off my previous database career–even if I had wanted to budget for it. I got a copy of Bruce’s PostgreSQL book, hot off the presses, built an app with RedHat 7.0 plus PostgreSQL 7.0 plus Perl-DBI, and the database part of it worked great.
When RedHat switched to a modified business model as part of their rebranding for RedHat Enterprise Linux (RHEL) in 2002, I parted ways with using their distribution at home. Still recommended it for businesses, and made part of my living as a 2.1 AS, RHEL3, and RHEL4 consultant. Then in 2004 I found White Box Linux, which let me play with a RHEL-like server at home again. Easy transition from RHEL4, and I’ve owned some sort of RHEL 4 or 5 derived server here ever since for tinkering purposes.
Shortly afterwards, CentOS Linux became available as another RHEL-derived option. The glory days for CentOS were the spring of 2007. RHEL5 shipped on March 14th that year, and CentOS followed with their CentOS5 release less than a month later. With some older competition like White Box failing to ever deliver a RHEL5 based release, CentOS has been the distribution to beat ever since, if you want a free-as-in-beer Linux that’s as similar as possible to RedHat’s Enterprise product. That’s made it easy for me set up CentOS systems whenever necessary at home, while also having RedHat’s commercial product available to recommend too.
The ugly memories of watching White Box shrivel up and die after failing to deliver a RHEL5 distribution have been coming back lately. RHEL6 was released in November of 2010, tomorrow it will be exactly six months old. There is no sign of a CentOS6 yet. I realized there was trouble brewing after two months when I still hadn’t seen a CentOS6, and instead saw on Planet CentOS that the developers seemed to be fighting over resources with the next RHEL5 update. It seemed like there were serious and growing issues with the CentOS development team, and I’ve been keeping a closer eye on this area ever since.
Shortly afterwards, I started seeing a lot more information about Scientific Linux. They were on track to a reasonable release date, with SL6 betas earlier in the year. Their SL6 came out in March. At this point, two months later, I’m hearing about a happy CentOS5 -> SL6 conversion every few weeks now. The similarities to how CentOS5 rose to prominence in the first place, by shipping their RHEL derived version before any other credible source, are rather obvious.
If I needed a sign that it was time for me to start installing SL6 myself, this week I found it. Any long-time user of RPM-based distributions has probably used a package from a Dag Wieers repo at some point. Dag has saved my ass by providing some tricky to compile from source package in RPM form on a regular basis for a long time now, seemingly as long as I’ve been using RedHat Linux seriously in some form. When I saw he was working seriously with the CentOS developers some time ago, that made me even more comfortable with their distribution.
Well, that party is over. Last week Dag publicly announced he was resigning from CentOS development work, seemingly over development team communication issues. In the comments there, Dag specifically suggests Scientific Linux as the right distribution to move to now, saying “their process is more open and the people are actually friendly to feedback.” If a tight development group has enough resources to keep its users happy, so long as the end result is open-source I’m not going to knock the process that got there. But when your release is at least four months later than it was expected by most people, and you’re causing major community contributors to abandon your project in a bad way, I don’t have any choice but to start looking into more open projects.
I’ve seen this movie before, and I didn’t like the ending the last time. Thanks to the CentOS development team for helping provide me with an excellent business-oriented Linux distribution I been able to use for free over the last four years. But I’m writing this as my first Scientific Linux install executes, and I have my first SL6 work for a customer also fed up with waiting on CentOS6 to start on when it’s finished. CentOS isn’t the first group to run an open-source development project without what I’d consider a really open development community, they’re just the latest to show how dangerous that is.
Welcome to the club.
Greg Smith has given up on CentOS, just like I did a couple of months ago. I agree with his conclusions 100%.
One way or another my ScientificLinux server now hosts all my development work, and five of my seven buildfarm members. (The exceptions are th
Dag failed to devulge what exactly happend. He sent private e-mail with harsh words to dev mailing list and then he said he is is sorry for a mistake, that he only uses harsh words behind their backs (in private conversation/mails):
I am sorry for this. Because this mail arrived in my inbox, I was confident Jean-Marc was mailing me privately. So my response is how I reply privately. I did not intend to send this to the list.
Then he did it again and unsubscribed from list:
Ok, I seem to be unable to use a mail-client properly and send another mail to the list that I did not want to bring up here. So let me make this easy on everyone. I will unsubscribe from this list.
Also, most of the problem was that he has different view of how to do necessary work, and my personal view of the debate is that he has problem with grassping that CentOS dev do NOT want to change SRPMS just to compile the binary. I would also call him as stubborn as CentOS devs.
His assessment of how SL devs are ahead of CentOS devs is also misleading. SL5.6 is STILL not OUT, 4 months late, because they worked on 6.0 first. When you combine time for all 3 releases, 4.9, 5.6 and 6.0, both SL and CentOS will spend similar/same amount of time on it, they just did it in different order.
CentOS 6.0 is scheduled for end of May, it’s in QA now, so it is not dead, and I intend to use it for many years. And CentOS devs showed greater worry about 4.x and 5.x servers then SL by working on updates before the mayor release. Just because We, satisfied owners of over a million of CentOS servers, do not praise CentOS devs all the time, does not mean the few of complainers should be seen as majority.
Ljubomir
I’ve gotten some similar words privately from commenter, pointing out that “Dag has left CentOS development team quite a while ago — almost two years to be more precise — http://dag.wieers.com/blog/leaving-centos-team-not-centos-community The post which you have referred to just mentions that he won’t follow centos-devel mailing list anymore. Also, SL has not yet published 5.6…” I’m sure I didn’t fully capture the exact detail of all the dag drama.
But this all kind of misses the point I was making here. I know what an open, healthy development community behind a growing open-source project looks like. And CentOS isn’t it anymore. They have a huge lead in installed base, and should have been able to turn that into volunteer resources. Yet they still aren’t coming even close to keeping up with upstream releases anymore. Even if SL6 has reached parity with them already, just with different priorities to work on things, I’m still going to prefer them given that their community is still growing. Who is ahead at this very moment is extremely close, so the tie breaker for me is who’s likely to pull ahead based on growth rate in the future.
RHEL 5.6 wasn’t released until January 13, 2011. At that point RHEL6 had already been released for two months. The idea that they dumped both releases onto downstream packagers at the same time is silly–for a project with as many users as CentOS, RHEL6 should have been able to find the resources to repackage and get it and out the door before 5.6 was even announced. Also, there’s little compelling in 5.6; my customers are perfectly happy waiting for that major update, and many aren’t even considering an upgrade from 5.5 yet. But the delay between RHEL5 and RHEL6 has been so long that getting evaluation work going on that new version, particularly the new kernel, is absolutely vital to business planning.
I am holding my breath for CentOS6 since first RHEL6 Beta, almost a year now. I put everything on hold for that one. Even my move to nearby town was held for few months so I can reinstall some overdue servers (they also have to double for Virtually based Windows, etc..). So I would think similarly to you if I was not following how things develop and as a developer and re-packager made an effort to ask and understand the problems CentOS devs face. Here is a snippet from one of the devs:
“First off, we can not use any of the existing infrastructure to build on because we can not build on a CentOS 4 or CentOS 5 machine because of the changing of MD5SUM in the RPMs themselves.
Secondly, the distribution will not build on the Beta (much like the 3.x release and UNLIKE the 4.0 and 5.0 releases). Not only that, but upstream used many “non released” packages to build on … packages we can not see or get.”
To only recompile Fedora packages to CentOS is excruciatingly slow proces when you have to find a version of related packages that can be also recompiled. Sometimes it took me days of stubborn attempts to find out that no newer version of some package can be compiled without changing base of CentOS. Trying to find out against what combination of packages some problematic package is compiled against is slow and monotonous process, and it takes time to make it right, if at all possible.
And Red Hat intentionally complicated thing by compiling some of its packages against unreleased versions of packages in attempt to delay Oracle in their RHEL clone, because Oracle started aggressive campaign to steal customers from Red Hat.
As for 5.6 updates, there was sort of the vote on one of the CentOS lists where devs asked userbase what to do first, updates or 6.0. Large majority of users who replied voted for updates, and some even was revolted that 5.6 was not fast enough because there are several security patches in 5.6. As you can see, CentOS users are diverse and with different needs. Most of the users however need fast updates, to keep their servers secure.
If less demanding users want to go to SL that is perfectly alright, but then they should do it without a hassle and doomsday words, that is all I am saying. Instead of that, I was exposed to 20-30-50 e-mails a day in endless debate (CentOS devs and supporters vs. max 10 CentOS users) how CentOS devs are basically useless and SL devs are great, and how those complainers are going to move to SL. And even after 100th time one of the devs telling them “so go then” they stayed on continuing to harass all of us with endless mails of complaints and debates.
As for Dag, he has not left CentOS users mailing list, and is currently in debate with defenders of CentOS over the same stuff they fought over in CentOS-devel mailing list. I commented on his blog yesterday just before I commented on yours, but it has not been authorized yet, and I wonder if ever will.
Yes, building packages is hard, and RedHat has made it harder recently. I regularly rebuild the full set of packages for PostgreSQL and some around it for my customers. While that’s only a subset of the total, it gives me some hands-on feel for what problems the CentOS developers face. I’m not one of the people on your list harassing you without any knowledge of the problems here.
But the CentOS user base is filled with competent people who could be organized to help, which is the way other successful open-source projects scale to support harder goals than they started with. And in cases where Scientific Linux has prioritized differently, you can use their work as a roadmap to save time on your own research. Instead I keep hearing blame-shifting and complaints about the users instead of someone leading toward healthy expansion. The way your wording here is suggesting I’m a “less demanding user” is the sort of thing I’m talking about.
And, yes, the 5.6 release was too slow from everyone, too, and it sucks that SL5 dropped that, and we don’t know yet who will catch up to being current on all releases at once. If I were running CentOS, I’d have figured out how to cooperate with the SL development team months ago, so that each release gets out more quickly and you don’t all solve the same set of problems twice. Surely their research into package versioning is useful to you and vice-versa.
It’s not so much about communities as it is about money, time and business strategy. If you need RHEL, *buy* RHEL. I don’t think you have the “right” to expect RHEL quality for free.
Scientific Linux is partially supported by government labs, and funding priorities for government labs are determined by elected officials with agendas, not by users or “distro community members”. They may be in better shape than CentOS now, but that can change.
I too have seen this movie before. The people responsible for the management of the Gentoo Foundation dropped the ball – they didn’t file their paperwork on time. Funding was a problem. There were lots of disagreements in the community as well, but ultimately developers need to get *paid*.
First of all, i must appologise to Dag. My post was published and responded to. Obvioslly both CentOS devs and Dag (and probably others on “his” side) are much more tactfull off-list then we can read from their clashes on-list.
I know your name from mailing list, and do not think tht you are incompetent. “Less demanding users” (Less in Less then…) are assesed from practical standpoint. Most demanding users run RHEL so they can receive fast support.
Important point is that 99,9% of CentOS users would and do not care about being part of the dev team, and you can not relly on them. And how about you accept obligation to fill me in on your development process so I can help you with your next Postgress release and then I skip one release and help you with third from now because I am too busy and/or not too busy because I will have what I need for time being. Would you be happy with that development? And accepting obligation to be fully transparent would mean devs would be obligated to also explain every detail of their build process to every airhead that thought he can do better then seasoned veterans. Every question would have to be answered in detail or devs would be accused of non-transparency. Who in their right mind whould willlingly accept something like that???
Those who trully wish to be part of the team can start with QA team and slowly progress and show that they are here to stay. And I for one would not allow unproven people messing with something I use for business.
Your hypothetical impossible to support development model is exactly how the PostgreSQL development community already works. The whole development process is documented and transparent, see http://wiki.postgresql.org/wiki/Development_information and the pgsql-hackers mailing list. People show up and contribute or not at random, and contributions are reviewed for their merit without a requirement like “start with QA”. We recommend that people start with patch review, because it’s the easiest way to get started, but that’s certainly not a requirement. Doing good QA requires a mindset that’s not the same one necessary to innovate on things like software build process. The whole idea of being forced to start with an entry-level boring job, regardless of one’s background or inclination, is the sort of things stagnant traditional software companies do.
And if every aspect of the build process for a piece of open source software isn’t explained in detail so that even “airheads” can take a look at it, I get nervous about that project. Just because a percentage of suggested changes will be rejected as not really improvements doesn’t mean you should be hostile to new contributors. If your project management problems are going to scale to support more work as popularity expands, you have to figure out how to add people. Or eventually you will stop keeping up with the needs of your users, and get replaced with another project that does.
I agree with the mayority of what you said, but it does not transate well to reverse engeneering where you have no idea what you need to do until you actualy do it, and where everything is dependant one on another. Just imagine having to recompile 1000+ pacakages (and an amount of time it needs for a system to do it and then to for you/devs to check reported differencies) in order to find out that something in build system (and not the packages themself) was not set right so you change one parameter in build system and then run the build again and wait for it to finish so you can work on it again. When one (or many) thing(s) depend on one tiny change, there is no parallel process.
I do agree that changes to development proces need to happen, and even devs accept that. In case you missed his e-mail, Karanbir said at least a month ago that AFTER CentOS 6.0 (now it should read CentOS 6.1) is out they will publish and explain everything. But argument, or shell we say flame war continues on and on.
sixgun’s status on Monday, 23-May-11 08:24:23 UTC
“The rise and fall of CentOS – 2ndQuadrant, Professional PostgreSQL” http://ur1.ca/454jd #centos #development #lxnews #linuxoutlaws
Ljubomir,
You do realise that you just described both the Debian and Ubuntu projects?
Right. CentOS has no problems that aren’t already solved by many other open source projects, including every other Linux distribution. And the way they are solved is by having a process that encourages volunteers to do useful work, at whatever level they are capable of. What I said “open development community”, that’s what I meant. The way this is supposed to work as that as the project grows, more experienced people move to higher level work that includes integrating and testing other people’s changes. That’s how you scale development upwards. And it never seems to have happened here.
The recent rush of upstream releases should have been handled like this: the most senior CentOS developers stay focused on the 6.0 development, while less experienced project members get practice doing the 5.6 update; checking their work should take much less time than doing it. Karanbir needed to realize long ago that every piece of the project must be public for this to happen, and that’s ultimately why CentOS is on its way out of so many places now.
The death of CentOS has been greatly exaggerated :)
We are still the 2nd most used Linux distribution on the internet and we are supported on all major public clouds.
A search of the Mars Project at Arizona State University, the Pisgah Astronomical Research Institute and the Human Genome Project show that major projects are still using CentOS.
Do a search for CentOS super computer on google to see the Universities that have built their HPC machines on CentOS.
CentOS is growing strong in 2013 … maybe the next end of the world will get us.