Steven Engelhardt

Here is a block of code which correctly performs the COM self-registration process on a DLL (distilled from the source code of regsvr32.exe):

BOOL SelfRegisterDll(LPCTSTR szDllName)
{
    BOOL bRet = FALSE;
    if (SUCCEEDED(OleInitialize(NULL))) {
        HMODULE hm = LoadLibraryEx(szDllName, NULL, LOAD_WITH_ALTERED_SEARCH_PATH);
        if (hm != NULL) {
            FARPROC fp = GetProcAddress(hm, "DllRegisterServer");
            if (fp != NULL) {
                typedef HRESULT (STDAPICALLTYPE *FnDllRegisterServer)(void);
                FnDllRegisterServer fn = (FnDllRegisterServer) fp;

                HRESULT hr = (*fn)();
                if (SUCCEEDED(hr)) {
                    bRet = TRUE;
                }
            }
            FreeLibrary(hm);
        }
        OleUninitialize();
    }
    return bRet;
}

The most notable subtlety is the use of LoadLibraryEx() with the LOAD_WITH_ALTERED_SEARCH_PATH flag instead of LoadLibrary(). As the MSDN article Dynamic-Link Library Search Order explains, the LOAD_WITH_ALTERED_SEARCH_PATH tells Windows to first look for DLL dependencies in the directory which contains the loading DLL as opposed to the directory from which the application loaded.

This flag turns out to be rather important. I recently was informed of a bug in which the COM DLL self-registration part of an installer created with InstallShield 5.5 would fail, but regsvr32 on the DLL succeeded. Apparently InstallShield 5.5 uses its own self-registration code which does not use the LOAD_WITH_ALTERED_SEARCH_PATH to LoadLibraryEx. This caused the COM DLL to attempt to load an incompatible, third-party DLL dependency from the SYSTEM32 directory (which was installed by another product) rather than the correct version from the application directory¹.