Heka beta offers high performance IT logging infrastucture

The Mozilla Services team, which looks after the back-room applications and servers that power Mozilla's various platforms, has released the first beta of Heka, a tool for gathering, analysing, monitoring, and reporting on data sourced from numerous locations around a network. The system is built around a lightweight portable daemon called hekad written in Go.

Hekad is able to gather data by reading log files, testing server health, or handling network connections from syslog, statsd, http, or heka protocols. It then converts this data into a standardised representation and wraps it with appropriate metadata before handing it on to an evaluator which can decide where the message should be routed. While being routed, messages can also be processed "in-flight" to feed aggregation mechanisms, sliding window event processors, and monitors, or be subjected to further parsing from which new messages can be generated and routed. Those routing destinations can be a database, db time series, filesystem, or network services, including another hekad instance upstream.

Heka data rendered as a graph.
Source: Mozilla Services

Designed with a plugin architecture, hekad allows developers to create, in Go, any number of custom plugins. The plugins come in four types – Input, Decoder, Filter and Output – with the Filter type being designed to handle data processing. The plugin configuration is determined in a configuration file. Users can also use the Lua language to work with scripted "Sandbox Filters", which can be added or removed without editing and restarting the daemon.

The architecture provides the core infrastructure for creating a data pipeline for managing and monitoring systems. Testing has shown that the Go-based application is capable of receiving and routing 10 gigabits of data per second. Although Mozilla is using it in a few production sites, it is a new technology and is "a bit rough around the edges". The developers are looking for early adopters and other developers to help move the project forward.

Hekad binaries, source code, build environment, Python and Node.js clients are all available for download or cloning. The Heka code is licensed under the Mozilla Public License 2.0. Further details are available on the heka project and hekad documentation sites.

(djwm)