This is often a question that has been asked a lot of times, and I couldn’t find an article which outlines the steps to be followed to enable a custom ISAPI extension in IIS7 (and above). You really don’t need to do anything other than the steps below:
And, that’s it! These are the only steps needed. If you have done any other steps like adding the dll path to the ISAPI module, removing the MIME map, etc, please go ahead, and revert those changes. If you do not know how to revert those changes, it is not a bad idea to start from the scratch. Check if you have a backup available, or just re-install IIS.
I worked on a customer issue, where even after following the above steps, the ISAPI extension wasn’t working. In his case, he was getting 404.0 error for the .dll when browsed. We enabled FREB, and it was showing the ErrorCode as “The specified module could not be found. (0x8007007e)”. We opened the DLL in Dependency Walker, and figured out that he placed a few DLLs which are referenced by the custom ISAPI extension on a different folder path. In fact, he placed them under c:\windows\system32 where the Application Pool was running under 32-bit mode, and it was probing under c:\windows\syswow64 as the default path. It is always good if you run Dependency Walker against your custom applications before hosting them on the website.
Hope this helps!