OsmocomBB is a project to create a fully featured open source GSM baseband firmware and comes from the same group that brought us OpenBSC, the open source "GSM network in a box". At present, only handsets that use the Texas Instruments Calypso baseband processor are supported, which in practice means basic and somewhat dated models, but this does have the advantage that these are available to developers at a low cost.
The TI Calypso actually contains two processors: a TMS320c5x DSP core that sits closer to the RF hardware and takes care of signal processing, and an ARM core that uses a memory-based API to interface with the DSP at the MAC layer. The code for the DSP is burnt into a ROM on the chip and while it is possible to load patches this is used as-is. And it's the ARM processor where the OsmocomBB firmware gets loaded and the GSM stack is implemented.
Building OsmocomBB is fairly straightforward but does require an ARM toolchain to be installed. Once built, loading applications onto a phone is trivial and is achieved via a serial cable plugged into the headphone socket. The simplest method involves loading these into volatile memory, but it's also possible to replace the original phone firmware that resides in flash storage.
Source: OsmocomBB Project
At the time of writing, a full implementation of the OsmocomBB GSM stack involves running layer 1 on the phone and layers 2 and 3 on a host connected via a serial cable. While clearly not ideal if you were hoping for a practical mobile phone solution, this has the benefit that it's easier to explore and debug the upper layers of the stack. And as with OpenBSC, the command line interface that's used for configuration will feel reassuringly familiar to anyone who has worked with Cisco kit.
Copious debug messages are printed to the console by both the layer1 firmware and the host-based mobile application which implements layers 2 and 3. But to get a real insight into what's happening, Wireshark can be used to decode GSMTAP messages which encapsulate the air interface, Um, passing it over UDP packets.
In addition to a functioning GSM stack that can be used to make voice calls and send and receive text messages, OsmocomBB also comes supplied with an incredibly useful network monitor firmware called simply "RSSI". This application not only allows you to monitor the received signal strength (RSSI) of base stations and mobiles, but provides cell configuration details and a spectrum analyser display which can be used to quickly ascertain which radio channels are in use.
At this point it's important to point out that the default is for transmit to be disabled in OsmocomBB firmware images. And while it's not difficult to create a firmware build with transmit enabled, it would be asking for trouble if you don't have access to your own licensed spectrum or a carefully controlled lab environment. Using a non-approved transmitter on a public GSM network could very easily cause serious problems and may bring you to the attention of the authorities.
To date the main uses of OsmocomBB have been as a tool for learning about GSM and by researchers probing the stack in order to uncover security flaws. In such cases it's typically used in conjunction with professional GSM test equipment or a low power network that has been created with OpenBSC or OpenBTS. Either a test licence will have been obtained from the communications regulator or all RF emissions will be constrained to the workbench.