Debugging DLL’s – 3 techniques to help you get started

During some redteam engagements, we find ourselves in the need of writing DLL’s. However, debugging DLL’s is not as easy as it seems, as a DLL isn’t built to run on its own.
In this article, we will explore how you can debug a DLL effectively.

What is a DLL?

A DLL is short for a dynamically linked library. It’s a modular approach of writing code. DLL’s are written in C or C++ and, under normal circumstances, serve as a library which programs can access to execute functions out of. To some extent, you could compare a DLL with a python package, or an npm package, any library really. The following image illustrates the concept of a DLL vs an EXE quite nicely:

As already mentioned in the preamble, a DLL is not made to be executed on its own, as a result the conventional way of debugging cannot be achieved out of the box.

Debugging method 1: Using OutputDebugString

Microsoft has a function in kernel32.dll (oh look a DLL!) called OutputDebugString which will print the contents of the string into a system debugger. Visual Studio has built in capabilities to catch these outputs (more about that later). My system debugger of choice is Dbgview.exe of the sysinternals suite. To illustrate how OutputDebugString works, I have prepared a little piece of demo code:

// dllmain.cpp : Defines the entry point for the DLL application.
#include "pch.h"

void DemoFunction()
{
#ifdef _DEBUG
    OutputDebugStringA("[DBG] hi from demofunction!\n");
#endif
    MessageBoxA(NULL, "This is a demo!", "DLLDEMO", MB_OK);
}

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
#ifdef _DEBUG
        OutputDebugStringA("[DBG]hi from DllMain!");
#endif
        DemoFunction();
        break;
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;

}

As proof that a DLL cannot run by itself I attempt to start the DemoDLL using the run button in Visual Studio and get greeted by this little guy:

Luckily, Microsoft has a built-in DLL loader called rundlll32.exe, but rundll32.exe can be annoying if you don’t have any exported functions (= functions to be used by other programs) in your DLL. My colleague Didier Stevens explains DLL entrypoints quite nicely in his own blogpost. If you are interested in learning more about that, please check it out!

Because of the hassle of dealing with entry points, I have made a skeleton DLL loader program, easily modifiable to suit your own needs, it can be found here: https://github.com/NVISO-BE/DLLoader

As you can see in the Demo code, all my ouput debug strings have “[DBG]” in them, this was not done without reason.
Dbgview is a system debugger, which will make a ton of noise, depending on what you have installed on the system. However, Dbgview has filter options. Because all debug strings have “[DBG]” we can instruct Dbgview to only show us these messages like so:

Everything is setup, time to run our DLL and look at the DebugView output:

As expected, the DebugView output shows our debug strings perfectly!

Debugging method 2: using Printf

This method uses printf instead of OutputDebugString so let us adapt our code:

// dllmain.cpp : Defines the entry point for the DLL application.
#include "pch.h"
#include <stdio.h>

void DemoFunction()
{
#ifdef _DEBUG
   printf("[DBG] hi from demofunction!\r\n");
#endif
    MessageBoxA(NULL, "This is a demo!", "DLLDEMO", MB_OK);
}

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
#ifdef _DEBUG
        printf("[DBG]hi from DllMain!\r\n");
#endif
        DemoFunction();
        break;
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;

}

Using the DLL loader, we will see all output of printf written to the DLL loader program’s standard output, which in this case is our console window.

Debugging method 3: Using the DLL Loader to debug in Visual Studio

Now that we have a surrogate DLL loader mechanism, we can instruct Visual Studio to use it as a parent process, this way we can run the DLL in the Visual Studio debugging environment, and we will be able to debug DLL’s just like you would be able to debug EXE’s.

Go to project properties and choose the Debugging tab

In the Command field, specify the path to the DLL loader executable, in the Command Arguments field, specify the arguments for the DLL loader, in this case the path to the debug demo DLL, as we do not have any exported function, we are not using a second argument. In case you want to use a second argument, you can, just like you would in the console.

Now we can setup some breakpoints in Visual Studio:

Launching the DLL can now be done using the green play button in Visual studio itself, this will instruct the DLL loader to start with the arguments specified in the project properties page.
As you can see, Visual Studio will now correctly hit and handle breakpoints as usual:

As the printf statements are still there, they will still get printed to the Console as well:

Conclusion

Depending on the complexity of your DLL code, most of the time you should be able to get away with using the DLL loader in Visual Studio so you can debug like you would debug any other program. Optionally you can combine the DLL loader with printf statements so you’ll see a real-time overview of what’s happening through the console output of the DLL loader.
However, if your DLL is doing complex operations such as creating new threads or doing asynchronous calls, OutputDebugString is a better option, as the DLL loader won’t take any new threads or async calls in consideration.

This concludes our trip into the debugging world of DLL’s. I hope this blogpost was instructive and has taught you something useful!

About the author

Jean-François Maes is a red teaming and social engineering expert working in the NVISO Cyber Resilience team. When he is not working, you can probably find Jean-François in the Gym or conducting research. Apart from his work with NVISO, he is also the creator of redteamer.tips, a website dedicated to help red teamers.
He was also ranked #1 on the Belgian leaderboard of Hack The Box (a popular penetration testing platform).
You can find Jean-François on LinkedIn , Twitter , GitHub and on Hack The Box.