Linux Foundation study goes behind the scenes of kernel development

Published today by the Linux Foundation, the study entitled Linux Kernel Development provides some background information on the development process for the Linux kernel and shows which kernel hackers and companies do the most development work. The authors include Amanda McPherson (Marketing Director at the Linux Foundation), kernel developer Greg Kroah-Hartmann (who has made a name for himself in part for his criticism of proprietary drivers) and Jonathan Corbet, founder and Executive Editor of LWN.net (Linux Weekly News). The latter two develpers are also two of the authors behind the book Linux Device Drivers. With the assistance of Kroah-Hartmann, Corbet has already published articles with similar titles as those in the study at LWN.net. The current study, however, provides an updated overview that is more comprehensive and is more attractive with the addition of a number of diagrams.

For instance, the study explains the most important kernel series and the development cycle of the kernel within Linux series 2.6, which has been continuously developed for some time. The study shows that Linux version 2.6.24 published at the end of January not only contains by far the largest number of changes in the past three years, but also had the longest development timeframe. Between the publication of 2.6.11 and 2.6.24, the study found that the developers integrated some 2.83 patches per hour in the main development line – a total of 3,621 new lines, 1,550 deleted lines, and 1,425 modified lines in the source code archive per day. The kernel hackers release a new version in the main development line roughly every 80 days.

The authors also took a look at which developers and companies are behind the changes. They found that the number of developers involved in the most recent kernels is almost 3 times greater than it was three years ago with Linux 2.6.11. Al Viro, David S. Miller and Adrian Bunk authored most of the patches; Andrew Morton came in fifth. Linus Torvalds, the creator of Linux, is found far down on the list. He probably spends a large part of his time delegating and coordinating a large number of developers and their work; in addition, during the period under investigation he launched the Git source code management system used for kernel development -- and probably did a considerable part of the programming himself.

13.9 percent of the developers work on the kernel without receiving any compensation from a specific company. 12.9 percent of the time, the company's identity is not known. 11.2 percent of kernel developers are funded by Red Hat, the largest company share just ahead of Novell at 8.9 percent and IBM at 8.3 percent. At some distance behind them come Intel, the Linux Foundation, and a number of other well known companies from the Linux and hardware world.

As with any other study, one should be prudent when interpreting the figures and data provided. The patches provided by Al Viro and Adrian Bunk are generally minor modifications (often single lines) and work done to tidy up various parts of the kernel, such as by removing old drivers or infrastructure. The work they do is certainly crucial, for the patches they provide correct potential or existing flaws and generally keep the Linux kernel in working order for further maintenance or operation – a task that many other developers would probably find too boring. Because the study lists the authors based on the number of patches provided, these two developers land at the top thanks to the large number of minor modifications they produce.

Had the study been based on the number of lines of source code modified, other developers would probably have topped the list; and if the criterion had been "new features", the list would have looked at different once again. Another way of ranking of the companies behind the developers would be based not on the number of changes, but on their scope. Corbert produced similar rankings in previous analyses published at LWN.net (1, 2, 3, 4, 5).

The company does not pay much attention to the tasks that the kernel hackers perform aside from programming. For instance, David S. Miller came in second in terms of the number of patches integrated. But for kernel development, his work as subsystem maintainer for the kernel's network infrastructure is probably much more important; he reviews patches produced by other developers and collects and shares changes approved for the various kernel development branches. In the current development cycle, some 13 percent of the patches that made their way into the main developer branch under the direction of Linus Torvalds passed through Miller's hands; this is the branch that is to be released as Linux 2.6.25 in the next few weeks. Miller is therefore one of the most important gateways and hurdles for patches intended for integration in the official kernel. To perform this task, Miller relies on additional maintainers, to whom he delegates the oversight of the parts of the network infrastructure, which he nonetheless keeps an eye on.

(trk)