A wxWidgets dialog in a DLL plugin

One of my recent projects was a DLL plugin for another software. I decided to implement plugin’s configuration dialog in wxWidgets.

The plugin functions showing user interface usually get the HWND of the host window, so the plugin window can act as a child of the host window.

Though the task seemed simple and straightforward in the beginning, it turned out to comprise many problems to solve, before I got it working as it should.

DllMain

HINSTANCE g_instance = NULL;
HANDLE g_act_ctx = NULL;

BOOL WINAPI DllMain(
    _In_ HINSTANCE hinstDLL,
    _In_ DWORD fdwReason,
    _In_ LPVOID lpvReserved)
{
    UNREFERENCED_PARAMETER(lpvReserved);

    if (fdwReason == DLL_PROCESS_ATTACH) {
        // Save DLL's instance handle.
        g_instance = hinstDLL;

        // Save current activation context.
        GetCurrentActCtx(&g_act_ctx);
    } else if (fdwReason == DLL_PROCESS_DETACH) {
        if (g_act_ctx)
            ReleaseActCtx(g_act_ctx);
    }

    return TRUE;
}

The host application in my case is not wxWidgets based. Therefore, the plugin must initialize and keep its own instance of wxWidgets library.

Since, I had my share of deadlocks in DllMain(), my personal choice was to leave DllMain() as simple as possible, and do the initialization and clean-up in the plugin’s function ShowConfigUI() later.

Please note that DllMain() is the only place you can get the correct SxS activation context in your DLL. The activation context determines which version of comctl32.dll is referred when you need Windows XP Visual Styles.

Therefore, it is vital that you remember activation context in DllMain(), and switch to it each time inside your DLL exported function(s). This is important for the Windows XP Visual Styles to work properly.

Note: Make sure you apply other measures to enable Windows XP Visual Styles, as described on MSDN (ISOLATION_AWARE_ENABLED=1, comctl32.dll manifest in DLL resource Type 24/ID 2).

ShowConfigUI

DWORD ShowConfigUI(_In_ HWND hwndParent)
{
    // Restore plugin's activation context.
    actctx_activator actctx(g_act_ctx);

    // Initialize application.
    new wxApp();
    wxEntryStart(g_instance);

    int result;
    {
        // Create wxWidget-approved parent window.
        wxWindow parent;
        parent.SetHWND((WXHWND)hwndParent);
        parent.AdoptAttributesFromHWND();
        wxTopLevelWindows.Append(&parent);

        // Create and launch configuration dialog.
        wxDialog dlg(&parent);
        result = dlg.ShowModal();

        wxTopLevelWindows.DeleteObject(&parent);
        parent.SetHWND((WXHWND)NULL);
    }

    // Clean-up and return.
    wxEntryCleanup();
    return result == wxID_OK ? ERROR_SUCCESS : ERROR_CANCELLED;
}

For activation context switching I introduced a simple helper class actctx_activator (code is below).

Then comes the wxWidgets initialization.

The next trick, hard to find on the web, is the correct preparation of the parent window. Once you create wxWindow from HWND, it is important to register it in the wxTopLevelWindows collection. Not doing so will make wxDialog::ShowModal() run the dialog modeless instead of modal.

Please note additional set of parentheses: they introduce a scope to get the local variables parent and dlg destroyed before wxEntryCleanup() is called. Otherwise, wxEntryCleanup() would have called delete on them. You can omit the scope, but you have to create those variables on heap then.

actctx_activator

class actctx_activator
{
protected:
    ULONG_PTR m_cookie; // Cookie for context deactivation

public:
    // Construct the activator and activates the given activation context
    actctx_activator(_In_ HANDLE hActCtx)
    {
        if (!ActivateActCtx(hActCtx, &m_cookie))
            m_cookie = 0;
    }

    // Deactivates activation context and destructs the activator
    virtual ~actctx_activator()
    {
        if (m_cookie)
            DeactivateActCtx(0, m_cookie);
    }
};

This helper class automates the switch-back-to-host’s context when destroyed: Usually created on function’s stack and destroyed at function return. With some reader skill, it can be extended to do the wxWidgets initialization and clean-up and wxWindow parent preparations too.

Environment

The code was tested with wxWidgets 3.0.2, Microsoft Visual Studio 2010, and Windows 7.

References

1. wxDialog icon in the taskbar, post on Google Groups
2. ::GetModuleHandle(“comctl32.dll”) returns 0 when executed from a dll calling wxWidgets code on windows 7, Discussion about activation contexts in DLLs