Greg Kroah-Hartman on Contributing to the Kernel, Life as a Maintainer, Beer, and More

A Reddit AMA last week with Greg Kroah-Hartman, Linux kernel developer and Linux Foundation Fellow, went beyond the usual questions about his workstation setup and job description. Much of that was preempted by Kroah-Hartman’s lengthy list of resources where the un-initiated can find his previous writing and presentations on those topics. Instead, he was able to answer more timely and specific questions which ranged from his thoughts on specific kernel patches, to the overall development process, to personal questions about his family, work habits and favorite beer. Below is an edited digest of some of the best responses. Visit the r/linux subreddit for the full AMA. 

On being a subsystem maintainer

How much of your time do you spend programming and how much is spent communicating?

David Miller said it best years ago, “Kernel subsystem maintainers are like editors. We take work from other people, review it, suggest changes, work with them on it, and eventually accept the best submissions. Every once in a while, because we used to also be programmers, we have a side project implementing something on our own, to keep ourselves sane.”

I currently have a few side-projects, that I work on to keep me sane, but the majority of my time spent on the kernel is communicating with others about their patches.

What’s your favorite side project at the moment?

I’m working on a new hardware “bus” for a new type of device, allowing me to help define the application protocol being used to talk on the bus. The team of developers I’m working with is world-class and I’m having a lot of fun collaborating with them.

Hopefully the hardware designs come together properly and work well and then the code can be merged into the kernel tree, as the companies involved all understand how the kernel development model works.

Do you ever sleep? Are you even human? How do you so many things? You even answer questions on IRC, how?

No, yes, skill, typing.

Have you ever been angry over poor quality patches? You know….like how Linus gets angry and starts cursing over poor code?

I get grumpy and mad and frustrated reading poor quality patches all the time. Just did so a few days ago and threatened to delete a huge driver from the kernel tree unless the developers got their act together. If you didn’t get angry and mad and frustrated, that means you don’t care about the end result, and are doing something wrong.

What would you say has been the most rewarding part of working on the kernel?

The most rewarding thing personally is seeing other people get involved and having it change their life for the better by getting jobs that provide the ability to allow them to change.

If you could go back a few years and make one big architectural change to the kernel, what would it be?

Why would I have to go back in time? If I thought there was a big change needed, I could do it now, just like anyone else could.

What would make you even more happy with Linux?

If you contribute to it.

On the Linux kernel development process

Everyone admits that the amount of traffic on LKML is simply unmanageable, but LKML is still a required step to getting your patches into mainline. Do you see this changing at all in the future?

All kernel subsystems have their own mailing lists, which has quite reasonable traffic loads, so there really isn’t a problem. You never just post patches to lkml and hope someone will pick them up, you use the tools we have to identify the correct maintainer and subsystem mailing list and send them to that list (scripts/get_maintainer.pl in the kernel source tree).

Everyone filters lkml based on the topics they are interested in if they want to subscribe to that huge volume, so they can pick out the bits they care about.

Some people view the kernel’s everything-via-email development model as quaint and antiquated. Do you have a good answer why tools like Github, Gerrit, Gitorious (and the like) will not work for a project like Linux?

There is NO way the github/gerrit/gitorious model would work at all for the kernel. The scale at which we work is a totally different level than could be handled by those tools.

In fact, a number of “popular” projects are hitting the “github scaling wall” and are working with Linux kernel developers to learn how they can scale their projects like we do…

We have over 3400 developers contributing last year from over 450 different companies. Our rate of change is on average 7.8 changes accepted per hour, 24 hours a day, and constantly going up almost every release (the 3.16 kernel was 9.5 changes an hour.) We have over 18 million lines of code and have been increasing at a constant rate of 1-2% for the past decade, only going down in size for 2 different kernel releases (the 3.17 release being one of them.) Nothing else comes close in size or scope that I am aware of…

We evaluate our development process all the time, and talk about it, in person, at least once a year to try to see if we are doing things wrong, and what we can do better. We tweak and change things constantly based on responses and what we think might or might not work well, and change based on feedback. If someone shows up with a tool that will work better for us, great, we’ll be glad to look at it, but that is usually quite rare, we end up writing our own tools for our work (git, kernel.org, etc.) as what we are doing is, again, unlike anything else out there.

Is the complexity of the kernel and its rate-of-change a problem or will it become a problem in the future? If so, is there a plan to address it and what is the plan?

The rate-of-change number is something that I have been tracking for almost 10 years now. And it’s something that every year I say, “there’s no possible way we can go faster.” And every year I am wrong.

So don’t listen to me, I don’t know what I’m talking about 🙂

On software and hardware

What is it about Arch Linux that impresses you?

It works really well for me. It has constantly updated packages based as closely as possible to upstream. It’s a model that I think is the correct way to do a Linux distro, and I’ve helped work on at lest 6 different distros over the years.

How does gentoo compare, if you’ve tried it?

You do know I’m also a Gentoo developer, right? 🙂 I have not had any problems with anything on Arch, and their wiki is by far the best resource out there that I have ever seen. Yes, the command line options for pacman are a bit odd, but read the documentation, or use a different package manager, and you should be fine.

Are there any applications that you use that run natively on Windows that you run under wine?

I don’t use Wine, or any native Windows applications, sorry.

Do you still use Terminology?

I still use Terminology every day as my main terminal program. It’s a great terminal, and the developers behind it are very active and responsive to bug reports.

What is your opinion on X11, do you agree it’s holding desktop Linux back significantly? Do you think Wayland will improve the situation?

I have no opinion on X11, other than liking it a lot, and I’ve gotten a few patches into the project many years ago. I also like Wayland, they are doing great work there.

X11 and/or Wayland has nothing to do with “holding desktop Linux back” at all, that is not what is preventing Linux from succeeding in that specific, tiny, market.

Do you have any strong preference for specific compilers, e.g. GCC versus clang?

Having the kernel be able to be built with llvm is great to have, competition is wonderful. We do a lot of gcc-specific things in the kernel, mostly because we had to, and gcc provided us a way to do those things.

Compile speeds are an issue to me, but runtime speeds is what really matters to everyone, for that, I don’t think clang is there just yet compared to gcc, but it is getting closer.

How do you feel about major Linux/OSS commercial companies and the distributions they support like RHEL and SLES?

RHEL and SLES provide a real solution to a lot of companies and use cases out there, I think they are great products and am happy to see them succeed.

Both Red Hat and SUSE contribute a lot back to Linux kernel development as well, so that makes me really happy. Internally the companies work very differently, but the end result for users is much the same, so they are an interesting “business study” if you like those types of things.

That being said, the old “enterprise” model doesn’t work for everyone, lots of companies do their own thing, and base their internal Linux use on quickly-moving community-based distros. This is how groups like NASDAQ works (Gentoo based), and many others. Which is why I am very happy to see CoreOS out there doing really well (disclaimer, I’m an adviser to them, as the founders are good friends of mine.) The “constantly updated with the latest stable version” is a very good solution to a constantly changing world with different requirements every day.

What makes me happy is that Linux works for so many different use cases, be it the stodgy-old-enterprise-install-once-and-never-touch-it-for-a-decade use, or the “I have to run the latest version of Ruby today!” use case, or the “power up 10000 containers on one kernel image” use case. That’s why Linux has succeeded so well over other operating systems, you can turn it into anything you want, as everyone wants something different.

What is your preferred way of updating kernels? Is Ksplice worth trying?

rebuild / reboot, why would you ever do anything else? 🙂

As for ksplice, if you have the time and effort to keep those patch files up to date, great, use it, otherwise, just reboot.

On contributing to the kernel 

I have contributed smaller patches to the kernel and really like it, but I do not know where to find something to really work on. Any suggestions, aside from watching the subsystem lists?

That’s a question I get asked a lot, and honestly I don’t have a good answer. I usually say, yes, just read the subsystem mailing lists that you are interested in, and pick up things to do from there. But that doesn’t work for everyone.

We have a lot of very talented developers who know how to contribute, but don’t know what to do. They can’t do it full-time as they have other jobs or responsibilities, but they want to help out in some manner. We need to come up with some type of project to work on that allows people to drop in and quickly help out for a short amount of time.

This is something that I’ve been thinking about for a while now, and have some half-baked ideas, and hope to do something about it next year, if at all possible.

You often encourage us newbs to make patches to correct superficial things like whitespace, comments, etc., but it seems that lots of other maintainer around the web discourage and ridicule these patches. Do you stand by doing this?

Other subsystem maintainers consider whitespace and spelling fixes to be a waste of their time, and it is, as they don’t want to deal with that type of stuff.

So don’t do whitespace fixes for their subsystems, do it in the areas of the kernel where it is encouraged and common. A specific example of that is the drivers/staging/ area of the kernel, I maintain that part and want you to send in whitespace fixes as I know it is a way to get people involved and that is what I want to encourage and do.

How would you suggest a developer with only a little bit of experience in C, and no experience with kernel programming could head towards kernel development?

Don’t. Seriously, why do kernel development if you don’t know C? Learn C first, really really well before ever thinking you want to be a kernel developer as you will need to know it and live it for a long time if you want to do this type of work.

And the kernel is not the place to learn C development, do something else, like write a ssl library or fork udev, both projects were started by people wanting to learn the C language…

What advice do you have for someone studying computer science? What topics or subjects do you deem as essential?

Get a degree, and while you have all of this free time in college (seriously, you do), work on open source projects so that you will have a much easier time to get a job out of college. As for what topics and subjects, most degrees force you to learn about lots of different things (databases, operating systems, design methodologies, etc.) so enjoy the high-level overview of it all, it will be useful to you someday.

On a personal note

Are you aware that if you do a google search for your name, google presents a photo of you flipping the bird?

Very aware, my kids love to make fun of me for that.

What activities do you do outside of linux? What’s your second favorite hobby/thing to do in life besides linux?

What is there to do besides Linux? 🙂 My favorite thing is to do things with my wife and kids, they are what keeps me sane in this crazy world.

As for a hobby, it’s a real problem, my hobby became my job so then what do I do for a hobby? For many years I remodeled my house, adding on rooms and spaces, but we have since moved from there to an island. So I built a wooden kayak, which took me much longer than anticipated (3 years.) That is now finished so I need another hobby now as it’s too cold out to go ride the kayak this winter…

What’s your favorite kind of beer?

I do like a good pilsner. Due to me traveling a lot, I get to try lots of different, local, beers, and have liked them all. There are just so many out there, start sampling and enjoying them, don’t ask others, make up your own mind.