05 February 2008, 10:51

Microsoft plans to improve protection from buffer overflows

Microsoft is to add an API to Windows that allows developers to turn Data Executing Prevention (DEP) on and off at runtime. DEP can prevent buffer overflows, but also stops some older dynamic code generation techniques from working. Developers often switch off DEP completely to allow their non-compliant applications to run smoothly. This new option allows them to turn it off only for non-compliant sections of code.

DEP limits the damage a buffer overflow can cause. If it is enabled, Windows terminates applications if their stack data gets corrupted, preventing the execution of injected code on the stack. Under Service Pack 1 for Vista, Service Pack 3 for Windows XP, and Windows 2008, users will in future be able to switch DEP on and off at runtime.

Programs written using versions of Visual Studio C++ before 2005 cause problems because a number of classes generate code dynamically (thunks). In his blog at Microsoft, Michael Howard says that outdated Active Template Libraries (ATLs) are the culprits. Because DEP can be triggered by calls to these libraries, developers tend to disable it in the application at link time. The entire application then runs without DEP, potentially exposing it to stack-based attacks.

New APIs will allow DEP to be enabled and disabled at runtime, though only for 32-bit applications in which protection from buffer overflows is realized by means of canaries or stack cookies. The SetProcessDEPPolicy API allows policies to be defined for runtime processes. The following parameters are available:

0x00000000: switches off DEP for this process
PROCESS_DEP_ENABLE (0x00000001): switches on DEP for this process
PROCESS_DEP_ENABLE | PROCESS_DEP_DISABLE_ATL_THUNK_EMULATION: switches DEP on, but prevents ATLs using thunks

Microsoft says the third option is the most important one. When it is invoked even ATL components that are not DEP-compatible can run although DEP is enabled for the process as a whole. The functions GetSystemDEPPolicy and GetProcessDEPPolicy are also available. In his blog, Howard provides a code fragment showing how the new API can be used on Windows. But he also points out a drawback of the API: SetPRocessDEPPolicy often provides an error message that is not very informative (5, Access Denied) and does not necessarily indicate that access is not possible. Apparently the message really indicates that the policy is not compatible with previous system settings because, for instance, the option /NXCOMPAT was used or because Windows is set up to insist on DEP regardless of application settings.

Instead of resorting to such workarounds, developers could nevertheless simply write applications using tools that are DEP-compliant. Using Visual Studio C++ 2005 or its successors should serve the purpose.