Linux Kernel Development Report 2008

Rate of Change

When preparing work for submission to the Linux kernel, developers break their changes down into small, individual units, called patches. These patches usually do only one thing to the source code; they are built on top of each other, modifying the source code by changing, adding, or removing lines of code. Each patch should, when applied, yield a kernel which still builds and works properly.

This discipline forces kernel developers to break their changes down into small, logical pieces; as a result, each change can be reviewed for code quality and correctness. One other result is that the number of individual changes that go into each kernel release is very large, as can be seen in Figure 2.

Figure 2 – Changes per kernel release

By taking into account the amount of time required for each kernel release, one can arrive at the number of changes accepted into the kernel per hour. The results can be seen in Figure 3.

Figure 3 – Changes per hour by kernel release

So, from the 2.6.11 to the 2.6.24 kernel release (a total of 1140 days), there were, on average, 2.83 patches applied to the kernel tree per hour. And that is only the patches that were accepted. The ability to sustain this rate of change for years is unprecedented in any previous public software project.

Kernel Source Size

The Linux kernel keeps growing in size over time as more hardware is supported and new features are added. For the following numbers, we have counted everything in the released Linux source package as “source code” even though a small percentage of the total are the scripts used to configure and build the kernel, as well as a minor amount of documentation. Those files, too, are part of the larger work, and thus merit being counted.

The information in Figure 4 show the number of files and lines in each kernel version.

Figure 4 – Size per kernel release

Over these releases, the kernel team has a very constant growth rate of about 10% per year, a very impressive number given the size of the code tree. But the kernel is not just growing. With every change that is made to the kernel source tree, lines are added, modified, and deleted in order to accomplish the needed changes. Looking at these numbers, broken down by days, shows how quickly the kernel source tree is being worked on over time. This can be seen in Figure 5.

Figure 5 – Rate of change by kernel release

Summing up these numbers, it comes to an impressive 3,621 lines added, 1,550 lines removed, and 1,425 lines changed every day for the past 2 1/2 years. That rate of change is larger than any other public software project of any size.

Who is Doing the Work

The number of different developers who are doing Linux kernel development and the identifiable companies1 who are sponsoring this work, have been increasing over the different kernel versions, as can be seen in Table 2.

Table 2 – Number of individual developers and employers

In fact, the individual development community has doubled in the last three years.

Despite the large number of individual developers, there is still a relatively small number who are doing the majority of the work. Over the past three years, the top 10 individual developers have contributed almost 15 percent of the number of changes and the top 30 developers have contributed 30 percent. The list of individual developers, the number of changes they have contributed, and the percentage of the overall total can be seen in Table 3.

Table 3 – Individual kernel contributors

Who is Sponsoring the Work

The Linux kernel is a resource which is used by a large variety of companies. Many of those companies never participate in the development of the kernel; they are content with the software as it is and do not feel the need to help drive its development in any particular direction. But, as can be seen in Table 4, an increasing number of companies are working toward the improvement of the kernel.

Table 4 – Companies working toward the improvement of the kernel

Below we look more closely at the companies which are employing kernel developers. For each developer, corporate affiliation was obtained through one or more of the following: (1) the use of company email addresses, (2) sponsorship information included in the code they submit, or (3) simply asking the developers directly. The numbers presented are necessarily approximate; developers occasionally change employers, and they may do personal work out of the office. But they will be close enough to support a number of conclusions.

There are a number of developers for whom we were unable to determine a corporate affiliation; those are grouped under “unknown” in Table 4. With few exceptions, all of the people in this category have contributed 10 or fewer changes to the kernel over the past three years, yet the large number of these developers causes their total contribution to be quite high.

The category “None,” instead, represents developers who are known to be doing this work on their own, with no financial contribution happening from any company.

The top 10 contributors, including the groups “unknown” and “none” make up over 75% of the total contributions to the kernel. It is worth noting that, even if one assumes that all of the “unknown” contributors were working on their own time, over 70% of all kernel development is demonstrably done by developers who are being paid for their work.

What we see here is that a small number of companies are responsible for a large portion of the total changes to the kernel. But there is a “long tail” of companies which have made significant changes. There may be no other examples of such a large, common resource being supported by such a large group of independent actors in such a collaborative way.

Why Companies Support Kernel Development

The list of companies participating in Linux kernel development includes many of the most successful technology firms in existence. None of these companies are supporting Linux development as an act of charity; in each case, these companies find that improving the kernel helps them to be more competitive in their markets. Some examples:

• Companies like IBM, Intel, SGI, MIPS, Freescale, HP, etc. are all working to ensure that Linux runs well on their hardware. That, in turn, makes their offerings more attractive to Linux users, resulting in increased sales.
• Distributors like Red Hat, Novell, and MontaVista have a clear interest in making Linux as capable as it can be. Though these firms compete strongly with each other for customers, they all work together to make the Linux kernel better.
• Companies like Sony, Nokia, and Samsung ship Linux as a component of products like video cameras, television sets, and mobile telephones. Working with the development process helps these companies ensure that Linux will continue to be a solid base for their products in the future.
• Companies which are not in the information technology business can still find working with Linux beneficial. The 2.6.25 kernel will include an implementation of the PF_CAN network protocol which was contributed by Volkswagen. PF_CAN allows for reliable communications between components in an interference-prone environment – such as that found in an automobile. Linux gave Volkswagen a platform upon which it could build its networking code; the company then found it worthwhile to contribute the code back so that it could be maintained with the rest of the kernel. http://lwn.net/Articles/253425/ for more information on this work.

There are a number of good reasons for companies to support the Linux kernel. As a result, Linux has a broad base of support which is not dependent on any single company. Even if the largest contributor were to cease participation tomorrow, the Linux kernel would remain on a solid footing with a large and active development community.

Conclusion

The Linux kernel is one of the largest and most successful open source projects that has ever come about. The huge rate of change and number of individual contributors shows that it has a vibrant and active community, constantly causing the evolution of the kernel in response to a number of different environments it is used in. There are enough companies participating to fund the bulk of the development effort, even if many companies which could benefit from contributing to Linux have, thus far, chosen not to. With the current expansion of Linux in the server, desktop and embedded markets, it’s reasonable to expect this number of contributing companies – and individual developers – will continue to increase.

Thanks

The authors would like to thank the thousands of individual kernel contributors; without them, papers like this would not be interesting to anyone.

Resources

We would also like to acknowledge Jonathan Corbet’s gitdm tool that was used to create a large number of these different statistics. The information for this paper was retrieved directly from the Linux kernel releases as found at the kernel.org web site and from the git kernel repository. Some of the logs from the git repository were cleaned up by hand due to email addresses changing over time, and minor typos in authorship information. A spreadsheet was used to compute a number of the statistics. All of the logs, scripts, and spreadsheets can be found at http://www.kernel.org/pub/linux/kernel/people/gregkh/kernel_history.