14 September 2010, 09:39

German researchers accelerate Hadoop

At the VLDB conference in Singapore, researchers from Saarland University have presented the results of the Hadoop++ project which aims to accelerate the distributed computing framework Hadoop when performing analytical queries. The technique involves plugging a kind of query planner into Hadoop using hooks provided for the purpose. The query planner evaluates map/reduce functions and injects them with its own hidden code which can deliver the same results more quickly. No changes to Hadoop's own source code or the public APIs visible to applications are required.

Project leader Professor Jens Dittrich has told The H's associates at heise online that Hadoop++ is up to eighteen times faster, "This means that one server can now do the work previously performed by eighteen". Dittrich says that he doesn't believe that the problem is inherent to the MapReduce concepts implemented by Hadoop, but "rather some implementation choices made in Hadoop". The researchers used a benchmark PDF that was published last year which measures execution times for typical map/reduce, relational and analytical queries to measure the performance boost.