It never ceases to amaze me how easily users will download and install arbitrary binaries from arbitrary sources as administrator... but the security concerns aside, I am going to address the popular question of how exactly to configure CustomAuth and how exactly does it work...


Can someone attempt to install the CustomAuth.dll from, and see if they can get it to work?  The article linking to the zip file is here:

I've tried simply taking the dll in the zip file and installing it as an ISAPI filter and wild card script map but I never get prompted.

If someone could successfully install it and then guide me through the steps, it'd be very much appreciated.



Umm... I am not going to download and install a binary from an arbitrary location that you posted, and you should not, either. Always get your binaries from a trusted, original source. In this case, CustomAuth is a sample ISAPI produced by Microsoft and can be obtained in the following official locations:

  1. IIS Resource Kit Tools - CustomAuth.dll pre-compiled and CustomAuth.ini
  2. IIS Platform SDK - CustomAuth source code is located at "%ProgramFiles%\Microsoft Platform SDK\samples\Web\iis\ISAPI_6.0\CustomAuth". I am going to assume that you can compile it Visual C++ 2005 Express Edition. It is really no different than the standard ISAPI instructions I posted in this blog entry, except you have to also:
    1. Add an additional "include" directory of "%ProgramFiles%\Microsoft Platform SDK\samples\Web\iis\ISAPI_6.0\inc"
    2. Compile "%ProgramFiles%\Microsoft Platform SDK\samples\Web\iis\ISAPI_6.0\src" into a Library named IsapiTools.lib
    3. Add the directory containng IsapiTools.lib file as an additional "library" directory
    4. Add IsapiTools.lib, crypt32.lib, and advapi32.lib as additional Link dependencies 

I did all of my experiments with the copy from IIS Resource Kit Tools. If you see a different behavior with your version, I suggest you abandon it and use the official one from Microsoft.

Configuring CustomAuth

Regarding your question on how to get CustomAuth to work - I really suggest reading and following the instructions contained within CustomAuth.ini - because if you do not follow instructions, you will not get prompted and it just looks like it is not working. Users of URLScan should be familiar with this approach; incidentally, the same author wrote both ISAPIs.

For the benefits of future readers, I am going to reproduce the setup instructions from CustomAuth.ini below and describe why each step is necessary, then show one semi-automatic way to configure CustomAuth.

; Sample CustomAuth.ini
; For CustomAuth.dll to function correctly, the following steps
; must be taken:
;   - It must be running on at least IIS 6.0
;   - CustomAuth.dll must be installed as a filter
;   - CustomAuth.dll must be installed as a wildcard script map
;   - In the wildcard script map dialog, "verify file exists" must
;     be unchecked.
;   - CustomAuth.dll must be allowed in the IIS MMC Web Service
;     Extensions folder
;   - CustomAuth.ini must exist in the same directory as CustomAuth.dll
;   - If not using the built-in logon and logoff pages, then the
;     specified pages must have ACLs set such that the Internet Guest
;     Account must have access to them
;   - For any pages that require successful logon, set an ACL that
;     denies the anonymous user account read access.

Since CustomAuth is customized authentication solution modifying IIS's anonymous authentication behavior, proper configuration is more than just a checkbox, even though using it is exactly the same as other Authentication protocols. The reasons each step is neccesary is as follows:

  • IIS 6 is required because the wildcard application mapping portion requires HSE_REQ_EXEC_URL to change the NT user token used to access the requested resource. It completes the CustomAuth protocol which uses a web page to gather NT username/password.instead of the plain user login dialog.
  • Installation as ISAPI Filter is important because it traps the 401 on denial by ACL of the anonymous user on the protected resource and initiates the CustomAuth protocol to send the login webpage which gathers NT username/password to the client.
  • Installation as wildcard application mapping is necessary because it allows CustomAuth to automatically apply to all resource extensions in the protected URL namespace.
  • "Verify file exists" must be disabled because some of the internal request traffic used by CustomAuth to complete its protocol (such as receiving the login webpage which gathers the NT username/password) uses non-existent URLs (from IIS's perspective), and we do not want IIS to prematurely interfere.
  • CustomAuth.dll must be enabled in Web Service Extensions because it has an ISAPI Extension component configured as wildcard application mapping which must function...
  • CustomAuth.ini must be located with CustomAuth.dll because that is how the ISAPI is written to look for its configuration.
  • If you customize the login/logout web page, then it must be accessible to the Anonymous user or else when CustomAuth redirects the browser to those web pages, the browser gets rejected because it is not authenticated nor is anonymous access allowed. You need to avoid a catch-22 when protecting URLs with content that is also a URL...
  • To determine which resource requires authentication via CustomAuth, you MUST disable access to it by the Anonymous user. This triggers IIS to send the 401 denied by ACL (which gets captured by the ISAPI Filter portion and kicks off CustomAuth protocol).

Sample Automation to Install CustomAuth

Here is a little batch script snippet that automates configuration according to the rules in CustomAuth.ini. You have to put in the appropriate values for the various scripts up front - such as the website to install CustomAuth on, location of the CustomAuth ISAPI DLL, location to the script tools, and the sample URL to test CustomAuth with. No, the script is not idempotent; if you run it >1 times the result may not be correct, and you may lose custom ISAPI filter configuration, but the point is to illustrate one way to configure CustomAuth.

  • ChgList.vbs can be found here (Manipulates LIST properties like an array)
  • FiltTool.js can be found here (Manipulates ISAPI  Filters, global or site)
  • documentation to built-in IIS sample administration script tools can be found here

[04/25/2006 - modified to use FiltTool.js to manipulate filters]

SET   FILE_ISAPI=%SYSTEMDRIVE%\Inetpub\CustomAuth.dll
SET  CMD_ADSUTIL=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\adsutil.vbs
SET  CMD_CHGLIST=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\chglist.vbs
SET CMD_FILTTOOL=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\filttool.js
SET FILE_PROTECT=%SYSTEMDRIVE%\Inetpub\wwwroot\pagerror.gif

%CMD_FILTTOOL% -action:add -site:%WIBSITE% -name:CustomAuth -dll:%FILE_ISAPI%
%CMD_CHGLIST% %WEBSITE%/root/scriptmaps first *,%FILE_ISAPI%,1 /insert /commit
%CMD_IISEXT% /addfile %FILE_ISAPI% 1 CustomAuth 1 "MS CustomAuth Sample"
%CMD_ADSUTIL% set %WEBSITE%/root/AuthFlags 1
CACLS %FILE_PROTECT% /P Administrators:F


Th test is based on trying to access /pagerror.gif, which will trigger CustomAuth login and display the built-in login dialogbox. If it does not display for you because https:// is not enabled, then you need to configure UseSSLForFormSubmission=0 (or you can enable SSL by using SelfSSL from the IIS Resource Kit Tools...).


Based on your description, I suspect that you either did not  force that all URLs under CustomAuth protection to be Anonymous Authentication only, or you failed to deny Anonymous user read ACL to the protected resource. In either case, the ISAPI-related configuration all looks good and CustomAuth.dll loads into memory as verified by "TASKLIST /m CustomAuth.dll", but without removing anonymous user access to protected resources to trigger 401 as well as force Anonymous authentication by clients, you will not see any prompts for credentials.