CentOS Project Leader Karanbir Singh Opens Up on Red Hat Deal

In the 10 years since the CentOS project was launched there has been no board of directors, or legal team, or commercial backing. The developers who labored to build the community-led version of Red Hat Enterprise Linux (RHEL) worked largely unpaid (though some took a few consulting gigs on the side.) They had a few hundred dollars in their bank account to pay for event t-shirts and that was it. And the project’s direction was decided based on the developers’ immediate needs, not a grand vision of future technology.

That’s all changing with the news last week that CentOS will join Red Hat, said Karanbir Singh, CentOS project leader and one of four CentOS developers going to work for Red Hat as part of the new collaboration. For the first time, developers will work on CentOS professionally and that’s created a “paradigm shift” for the project and its contributors, he said.

Red Hat will bring resources and structure to the project. And in return, CentOS will provide a stable testing ground for new technologies – for RHEL and many other projects as well, Singh said.

“We’ll be working on infrastructure other projects need to be successful with us,” Singh said. “We won’t be delivering the features. We’ll make it as trivial as possible to come in and do the build you need, without needing to learn everything about building CentOS images.”

Here Singh discusses how CentOS and Red Hat joined up; what to expect from the project now; Red Hat’s role in the community; opportunities for more partnerships; and his own feelings on working on CentOS professionally for Red Hat.

How did this partnership come about? Did you really just get a call from Red Hat lawyers one day (as you joked on Twitter)?

The big challenge was when Red Hat called us and said, “who are your lawyers? Who do we speak to?” We don’t have a commercial interest or a legal team, so there’s no entity to partner with. Red Hat is coming in to incubate the project, provide some of the resources and help us meet goals. So it’s more accurate to say we’re joining forces. It’s a bit of legalese, but it’s a better description because some of the things they came up with we thought were too corporate.

@markimbriaco you can imagine what i had to say when I picked up the phone call to find RH lawyers on the other side…

— Karanbir Singh (@kbsingh) January 7, 2014

What enabled you to “join forces?” Has your relationship with Red Hat improved?

A lot of people seem to think the relationship with Red Hat is brand spanking new but it’s not, we’ve always used them as upstream. We were taking what they were putting out and making it available for everybody.

But one of the challenges we have was isolating people from legal issues that might come up. Say someone forgot to remove the appropriate trademark somewhere. By law (Red Hat) is required to protect trademarks. The only way we could protect that is by having structure, working as an organized group privately. One of my biggest gripes was CentOS couldn’t be worked on out in the public because if we did it opened us up to possible legal action.

As part of the relation with Red Hat now we can work with them to make sure that (trademark) work is done upstream. That allows us to be a lot more open.

How will CentOS be more open?

In the past the way to get into the CentOS build process was to be nominated out of the community into the QA team. You had to have a few years of history, lots of forum posts, and not have a commercial interest, then you could come into the QA. Then those were elevated into privileged positions with access to servers.

That entire infrastructure just goes away. We go from a build team of 6 to15 people, with 60 to 70 people in the QA team, and the big win – what I’m most excited about – is other people in other projects trying to do their own thing can come and work with us to do what they need to do.

How can other projects start working with CentOS?

We’re still working out the details but we want to call them CentOS variants. They can add layers and components and configure the system to make their own whether they want to do VOIP, hosting, cloud, storage, (etc.)

When we were talking to Red Hat originally we thought it would be ideal to have one project that steps forward and model the variant projects on that relationship. We wanted six projects to come through in first 6 months. So far, we’ve had seven come through in the first 24 hours and three more since then.

One requirement, though, is every project needs a nominee from an existing development team to make sure we set the right expectations. There are a bunch of conversations going on.

What are you hearing from projects that want to be involved?

Going back 4-5 years, a lot of people found CentOS to be a stable OS they could plan against, but because there’s no SLA, it’s a best-effort set up. They’d take it back to their own ecosystem and do their own customizations in-house. They have to own every bug, feature and patch.

The variant process will allow them to come within the CentOS project. They only have to maintain the stuff they care about. It’s like if 40 different projects are using Apache to do something, only one needs to maintain Apache, the others can focus on their own bit.

How will CentOS benefit from the relationship?

People can come in and we can expand the ecosystem together. There’s so much stuff we’ve had to be very pessimistic about by keeping things private, by not allowing people we didn’t personally know to get involved. Those barriers are coming down.

How do you feel about being in a structured set up?

We have had levels of structure, but not to the extent of having a proper board. The great thing about Red Hat people is they understand how some of these communities work. They meet everybody on IRC every day, for example. We will firm up processes and procedures, like what infrastructure we’ll use, how code will flow, over the next few months and everybody can pitch in and make the decisions together.

We haven’t been a feature-led community, we’ve been a problem-led community. Eighty-five percent of the time, it’s the same thing. It’s that 15 percent when it’s not the same thing that a developer-led community is important.

The new things we’ve brought into the project weren’t because someone had a big idea of what’s coming in the future. It’s been to address problems we already have. For example, everyone out there is trying to build cloud instances to adapt for what’s coming down the road. Our approach to that right now is if they’re smart enough to install it, they don’t need our help anyway.

The CentOS project as it was is going to stay intact, but we’ll be working on infrastructure other projects need to be successful with us. We won’t be delivering the features. We’ll make it as trivial as possible to come in and do the build you need, without needing to learn everything about building CentOS images. People will come together to work on the same problems.

Will CentOS become a proving ground for new technologies in RHEL as some commenters have speculated?

I think it should. But it would be wrong to say it will be a proving ground for things into RHEL. It’s more appropriate to say it’s a proving ground for anything. Because the platform is resilient you can build on CentOS without things caving in. There’s nothing in what we’re doing that limits ourselves to what Red Hat is doing. Our biggest (potential) variant is Xen Project and Red Hat isn’t involved in that at all.

We do expect our work to feed back to RHEL. But we’re pretty serious that the platform we’re building isn’t specific to RHEL.

CentOS 6.5 was released in December – what’s on tap for the next release? How will the new partnership affect that?

In spite of the fact that I work for Red Hat now, I’m not privy to their release schedules. So over the next few months the board will publish our own road map. The mainline, the core, will stay as it is. But the variants will be able to set up their own road maps (based on their own release schedule).

The big thing we’re trying to get to is to have an installer for CentOS users to select what variants they want. But that’s a wish-list item that needs to be specced out before it’s developed and designed.

You are personally going to Red Hat – what do you think you’ll be able to do in house that you wouldn’t be able to do otherwise?

Ten years ago when some of us were getting together to start the project, the aim was to get 300 people to use it, that was fantastic. From our perspective it’s been fairly successful. How we define success is to build something we would use and that comes back to the user-driven approach. We cared about how things worked, where they worked, and overall it worked out well having that user perspective.

I’ve never worked for a big open source company before but I hope to bring that user perspective to Red Hat and what I’ll take a way is a large approach to user communities and hopefully manage that better.

Otherwise, not much has changed. They sent me a phone and a laptop and that’s how it’s going to go. I feel quite privileged to have this opportunity to focus on the CentOS larger ecosystem side of things.

Will you be able to relax a little now? Maybe enjoy your work more?

There’s been no money involved in the project. We have a bank account that’s never had more than a couple hundred dollars for printing t-shirts for events. This is the first time there’s a group of people professionally working on CentOS as a platform.

How CentOS used to happen was some of us would go to work and then work another 40 hours a week on CentOS. You can’t sustain 80 hours a week. The reason I did it wasn’t for compensation, it was because I wanted to.

You wrote on Twitter, “I think this is also a great opportunity for RH to prove it cares about community in an open, user focused, transparent way.” How can they do that?

That’s a hard question. Here you have a total community-led project suddenly going from a place of not accepting financial donation to being backed. We have access to certain resources we can rely on – people available during working hours and machines available, for example. There’s a paradigm shift in the project, but we want to keep the project as it is.

There’s nothing we’re doing right now that should make you feel it will change. If there is make sure you make those feelings public and judge us in six months. The onus for proving that, the entity to make that point and make it visible, is Red Hat.

@jmccrohan I think this is also a great opportunity for RH to prove it cares about community in an open, user focused, transparent way.

— Karanbir Singh (@kbsingh) January 8, 2014

What do you think this partnership will do for Red Hat’s image in the Linux community?

I think it should be good. They already contribute so much. The second day there I made the mistake of looking through the address book of people I could call and I was intimidated. These are the people I’ve looked up to and suddenly he’s two extensions away from me. But having them on board now it just expands what they’re doing. I think it should get them some good will.

A lot of feedback I get from people at Red Hat, from senior managers to junior developers, is that they appreciate what we do, and that ethos of being open and a community is quite strong within Red Hat itself. People outside the developer world don’t really get how engrained they are in the open source community mindset.

Anything else you’d like to share with the Linux.com community?

Come join us.