"Programming X-Windows is like trying to find the square root of pi using Roman numerals" - Anon
Wayland is set to be a great leap forward for the Linux desktop, but the reason it came into existence, and the reason it has had the backing of the likes of Red Hat, Intel (which employs Høgsberg and is leading the way on Wayland development) and Canonical, is because it is a solution to the cumulative problems of X, which is nearing the end of its useful life.
The X Window System began as the one man project of Robert Scheifler at MIT in 1984. The letter X was chosen because X was derived from, and built upon, an earlier window system called W; this ran on the V microkernel operating system which had been developed at Stanford university. Scheifler's initial announcement of X, dated 19 June 1984, noted that "I stole a fair amount of code from W, surrounded it with an asynchronous rather than a synchronous interface, and called it X. Overall performance appears to be about twice that of W."
X allowed a server to use a protocol that made it possible for a single machine to generate and display windows on multiple clients. In a historically confusing twist, the X Server ran on the client where the screen, keyboard and mouse was, while X clients ran on remote systems. X was hardware independent, and the resulting network transparency was seen as X's greatest virtue. An X client on one machine could serve up windows for display on any number of remote terminals and machines.
Network transparency made commercial sense at a time when the most common requirement of a server was to generate an Xterm window on a client terminal, and graphics cards were relatively primitive. The UNIX companies saw X as the guarantor of some measure of interoperability at client level. Because the X Window System was portable and offered network transparency between different versions of UNIX and VAX, it became the default mode for graphics on UNIX, and later on Linux.
Compromise and redundancy
The X Window System was, though, only concerned with the very basic elements of rendering graphics; "mechanism not policy" was an often heard reason for why the MIT X Consortium would not define standards for buttons, scroll bars and other UI elements. Even window management was left up to vendors; UI toolkits appeared from Unix vendors trying to differentiate themselves, such as AT&T's Open Look and the OSF's Motif.
In 1993 control of X passed from MIT's X Consortium to the X Consortium Inc, an independent non-profit, which came up with a interoperability compliance standard, the ICCCM (Inter Client Communication Conventions Manual), and also took part in the development of Motif and the Common Desktop Environment (CDE), which were intended to bring a common look and feel to UNIX systems.
As Motif and CDE were not open source or free software, Linux users later came up with their own solutions: toolkits such as GTK+ and Qt and companion desktop environments such as GNOME and KDE. These have gone some way towards resolving the issue of consistency between applications and protecting applications and programmers from the worst aspects of X and the relative ugliness of Motif. But the graphics toolkits have still had to deal with X-related performance issues, specifically with 3D acceleration, despite the introduction of the GEM API in 2008 and other attempts to impose order on the system.
At the systems level, developers have had to work around an increasingly dated infrastructure. As the Wayland FAQ observes, X has "a tremendous amount of functionality that you must support to claim to speak the X protocol, yet nobody will ever use this... This includes code tables, glyph rasterization and caching, XLFDs (seriously, XLFDs!) Also, the entire core rendering API that lets you draw stippled lines, polygons, wide arcs and many more state-of-the-1980s style graphics primitives". Wayland does away entirely with the idea of graphics primitives at the server level, and lets libraries provide appropriate rendering APIs such as OpenGL.
Developers have added "extensions such as XRandR, XRender and COMPOSITE" to bring the X server up to date, but it is impossible to remove the core rendering API of X or much of the other complexity that is rarely used in a modern desktop without it stopping being X. For many operations the X server has become a redundant fixture between the kernel and the interface, adding several layers of complexity to every keystroke, and separation between the core libraries and the kernel and the GPU.
"One of the last things that X still does is input handling, but even that belongs in the compositor", says Høgsberg. "We were just never able to fix that. So with Wayland I wanted to make a display server that directly supported that model, instead of building it out of duct-tape and window properties on top of a different type of display server."