Linux-Kongress 2009: New filesystems, optimised programming
Prominent open source personalities presented a host of information about current and future developments in Linux at the Linux-Kongress 2009 conference. This time, filesystems were discussed in several presentations. Among the topics were the advantages of Ext4 and Btrfs, fast Ext4 filesystem checks, and kernel and userspace tricks for minimising the time-consuming movements of read/write heads, all of which can speed up some tasks by several orders of magnitude!
The Linux-Kongress 2009 conference was held in Dresden last week. The long-established conference hosted by the GUUG (German Unix Users Group) attracted its usual large number of prominent Linux developers both in the audience and as presenters. Its two-day schedule included speakers like kernel developer Theodore "tytso" Ts'o, who is known for his work on the Ext filesystems, Andi Kleen, Jan Kára, Jan Blunck, Heinz Mauelshagen, and Samba developer Volker Lendecke. Most of the presentations were of a high standard and offered a good overview of the current and future developments in the Linux kernel area.
IBM developer Theodore Ts'o, who has been "outsourced" to the Linux Foundation as a CTO (Chief Technology Officer) for the time being, presented the conference's opening keynote. In his presentation "Linux and Open Source in 2010 and Beyond" (slides of the presentation), he identified mobile computing and cloud computing as the two IT sectors where companies are currently showing a particular interest in Linux, replacing proprietary solutions with Linux-based technologies.
According to Ts'o, both areas also have a high degree of "shared innovation" – which means that companies co-develop basic technologies with their competitors in open source fashion. This not only keeps down development costs but it also creates better products due to the shared use of improvements – while Ts'o elaborated particularly on the former aspect, he also emphasised that companies should not, or at least not immediately, share their most outstanding products, their "unique product value", which distinguishes them from their competitors. To illustrate the places where innovation is happening, he presented some figures from a study updated last August which quantifies the input of the various companies involved in the kernel's development.
Theodore Ts'o said that it is hard to predict which trends will emerge after mobile and cloud computing in two or three years – but that those who can answer this question might become rich. 2010 will (not) be the year of the Linux desktop just like the years before it, said the developer. Ts'o considers developing Linux for desktop PCs a difficult task – while the required improvements to the software stack are in progress, they aren't happening as fast as many people would like, he explained.
Filesystems for today and tomorrow
In "Ext4, Btrfs and the others" (presentation slides and proceedings), SUSE Labs developer Jan Kára presented the two new Linux filesystems likely to become most prominent in the near future. Kára opened his presentation by illustrating various problem fields in the filesystem area: for instance, the storage capacity of hard disks is rapidly increasing while their throughput and especially their access times remain virtually unchanged, he said. Filesystems need to be adjusted to address these trends, said the programmer.
Kára explained some of the most important features as well as structural and functional aspects of Ext4 and Btrfs. The developer rated Ext4 mature enough for desktop systems, adding that although this filesystem is not quite as stable and tested as Ext3 yet, he anticipates that it will soon be ready for production use in corporate environments. According to Kára, the Btrfs copy-on-write filesystem has not yet reached maturity but is already quite presentable and would "probably not eat your data"; the programmer estimated that it will be ready for use on desktop systems in a few months. He highlighted the fast snapshot capabilities of Btrfs as well as its integrated checksum and RAID features and its SSD mode.
At the bottom of each of the presentation documents there are some benchmark results for various filesystems – Btrfs and Ext4, for instance, copy unpacked kernel sources considerably faster than Ext3, and much faster than XFS. The latter is slightly faster than Btrfs when reading large directories.
Fast checking with Ext4
In his second presentation entitled "Speeding up file system checks in ext4" (presentation slides), Theodore Ts'o described further aspects of Ext4 and discussed various important differences between Ext4 and Ext3. The developer said that some of the differences – for example extent-based data storage as well as several program optimisation measures for checking Ext filesystems – make checking disks much faster in Ext4 than it was in Ext3. Details of these optimisation measures as well as numerous test results to prove this can be found in the presentation's PDF file.
Ted Ts'o said that fast checking is especially important with very large disks because the resulting server down-time while checking could become so long that it might be quicker to just restore the backup data and lose the data that was not included in the backup. Filesystem checks are essential because we don't live in a perfect world and it is impossible to rule out faults in the kernel, the storage media or in RAM, added the programmer. He recommended that users run a script that creates an LVM snapshot and checks it in the background on a weekly basis – while no faults are found, this does not create any down-time, he said.
Faster data throughputs and overlays
Håvard Espeland of Oslo University presented "Improving Disk I/O Performance on Linux" (presentation slides and proceedings) – an abridged version of a master thesis by Carl Henrik Lunde. Both developers describe an optimised variant of the Linux CFQ (Completely Fair Queuing) I/O scheduler they developed together with a team. By craftily reorganising its processing order, the optimised scheduler tries to reduce the movements of the hard disk's read/write heads – which minimises a hard disk's average access time and increases its throughput.
Since the kernel can only reorganise and optimise the requests it has in its queue, the team around Carl Henrik Lunde has also taken a closer look at the userspace and developed an optimised version of the "tar" archiver. In recent kernels, this version obtains the on-disk data position from the Ext3 or Ext4 filesystem and can reorganise requests to avoid unnecessary movements of the read/write heads from the start – in an artificially aged filesystem this allows a kernel source tree containing 22,500 files to be archived in less than 20 seconds instead of about 80.
Novell developer Jan Blunck presented the much discussed problem of union mounts, which has yet to be solved at the kernel level, in his presentation "State of the Union (Mount)" (presentation slides). The developer attempted to list the most important pros and cons of the various current ways of connecting one mount point to another – this is, for example, interesting for live media, where write-protected data from a CD or DVD is integrated as a root filesystem, while a writeable overlay filesystem provides data from, and writes data to, a different disk. The "writeable overlays (union mounts)" solution mentioned in the presentation and masterminded by Jan Blunck and Valerie Aurora was only recently submitted by Aurora to the LKML for review.
Further information and outlook
Rather than filesystems, veteran kernel developer and current Intel employee Andi Kleen talked about kernel programming techniques for optimum scalability in multi-processor systems in "Linux multi-core scalability" – this presentation is available both as slides and in a text version that includes many detailed explanations. Felix (Fefe) von Leitner's presentation "Compiler Optimization Survey" discussed the programming aspects of modern compilers and related opportunities for optimisation – the relevant PDF file comprises 72 pages of presentation slides and offers many examples and micro benchmark results.
Samba icon Volker Lendecke presented "Samba Status Update" (presentation slides), an overview of the current developments around the free CIFS server. For more than a year, the Samba team has been discussing the potential future directions of the Samba 3.x and Samba 4 development lines. Now, the developers have agreed on a release strategy: the strategy involves combining the two branches into one package side by side, a solution which has been discussed under the code name Franky. While the Samba 3.x part will act as a file server and print server, Samba 4 will mainly handle active directory and domain controller tasks. The Samba team is currently adjusting the two components for this purpose, and the first combined release is expected in the first quarter of 2010.
In "Fighting regressions with git bisect" (presentation slides), Christian Couder demonstrated the git bisect feature using a variety of examples. With the right scripts, this feature can be fully automated to check and compile various kernel versions as well as test them after rebooting – in recent kernel versions, this allows developers to establish which change is responsible for a previously nonexistent flaw. Couder has also covered this subject in an article published on LWN.net.
You can find PDF slides of the presentations not mentioned above on the program page of the Linux-Kongress 2009. The proceedings of some of the presentations should become available at Lehmanns' online shop soon.