I ran into an interesting problem this week trying to get a legacy application running on Windows Vista. If you ran the installer, it would fail, claiming that it was unable to register a required component. So, I grabbed the ocx it was trying to install to have a look. Taking a peek by attaching a debugger to regsvr32, I could see that the component itself was just calling MFC42!AfxOleRegisterTypeLib, so it wasn't trying to do anything strange inside of DllRegisterServer. So, what was the problem?
Well, you can take a peek at the source for MFC, and see that it just calls in to OLEAUT32!RegisterTypeLib, so I started by setting a breakpoint there. It was reading through the type library, added registry values for the typelib itself, but then choked when adding the interface registry entries. Specifically, it was failing at OLEAUT32!CTypeLib2::IsValidHinfoDef. It turns out that we changed the format for type libraries quite some time ago, and we stopped supporting the old format as of Windows XP (but it was working fine on Windows 2000). This ocx was using the old format, so it couldn't register. How can you tell if your type library is doing the same thing? Set a breakpoint on OLEAUT32!CTypeLib2::IsValidHinfoDef and keep an eye on the EAX register. If you see it change from 0 to an HRESULT value (specifically 0x80004005), then it's basically telling you that the type library is not valid.
Of course, understanding the issue is only half of the battle - if you are doing serious application compatibity work, you aren't done until you have either fixed it or proven it unfixable. I hadn't proven it unfixable yet, so I had to start thinking about how to fix it. The type library is embedded as a resource, so we could theoretically replace it, but first we have to generate a new one with the updated format.
So, I just walked over to a Windows 2000 machine and used OLEVIEW to crack the binary and extract the typelib. Conveniently, it generates IDL from the type library it finds. I then copied that IDL, stuck it on a USB drive, and brought it over to my Windows Vista box. After updating the IDL to be compliant with current formats, I ran it through MIDL.exe to generate a new type library. I then opened up the ocx in Visual Studio, which gives you a nice view of the resources in a file. I added the new tlb, deleted the old one, renumbered the tlb resource to 1, and saved. Voila - same binary, new (and valid) type library. Running regsvr32 this time succeeds - hooray! Now we just need to verify that the component we registered doesn't have other compatibility issues!