It never ceases to amaze me how easily users will download and install arbitrary binaries from arbitrary sources as administrator... but the security concerns aside, I am going to address the popular question of how exactly to configure CustomAuth and how exactly does it work...
Can someone attempt to install the CustomAuth.dll from http://www.theserverside.net/articles/content/ImplementingSSO/CustomAuth.zip, and see if they can get it to work? The article linking to the zip file is here: http://www.theserverside.net/articles/showarticle.tss?id=ImplementingSSO.
I've tried simply taking the dll in the zip file and installing it as an ISAPI filter and wild card script map but I never get prompted.
If someone could successfully install it and then guide me through the steps, it'd be very much appreciated.
Umm... I am not going to download and install a binary from an arbitrary location that you posted, and you should not, either. Always get your binaries from a trusted, original source. In this case, CustomAuth is a sample ISAPI produced by Microsoft and can be obtained in the following official locations:
I did all of my experiments with the copy from IIS Resource Kit Tools. If you see a different behavior with your version, I suggest you abandon it and use the official one from Microsoft.
Regarding your question on how to get CustomAuth to work - I really suggest reading and following the instructions contained within CustomAuth.ini - because if you do not follow instructions, you will not get prompted and it just looks like it is not working. Users of URLScan should be familiar with this approach; incidentally, the same author wrote both ISAPIs.
For the benefits of future readers, I am going to reproduce the setup instructions from CustomAuth.ini below and describe why each step is necessary, then show one semi-automatic way to configure CustomAuth.
; Sample CustomAuth.ini
; For CustomAuth.dll to function correctly, the following steps
; must be taken:
; - It must be running on at least IIS 6.0
; - CustomAuth.dll must be installed as a filter
; - CustomAuth.dll must be installed as a wildcard script map
; - In the wildcard script map dialog, "verify file exists" must
; be unchecked.
; - CustomAuth.dll must be allowed in the IIS MMC Web Service
; Extensions folder
; - CustomAuth.ini must exist in the same directory as CustomAuth.dll
; - If not using the built-in logon and logoff pages, then the
; specified pages must have ACLs set such that the Internet Guest
; Account must have access to them
; - For any pages that require successful logon, set an ACL that
; denies the anonymous user account read access.
Since CustomAuth is customized authentication solution modifying IIS's anonymous authentication behavior, proper configuration is more than just a checkbox, even though using it is exactly the same as other Authentication protocols. The reasons each step is neccesary is as follows:
Here is a little batch script snippet that automates configuration according to the rules in CustomAuth.ini. You have to put in the appropriate values for the various scripts up front - such as the website to install CustomAuth on, location of the CustomAuth ISAPI DLL, location to the script tools, and the sample URL to test CustomAuth with. No, the script is not idempotent; if you run it >1 times the result may not be correct, and you may lose custom ISAPI filter configuration, but the point is to illustrate one way to configure CustomAuth.
[04/25/2006 - modified to use FiltTool.js to manipulate filters]
SET CMD_ADSUTIL=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\adsutil.vbs
SET CMD_CHGLIST=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\chglist.vbs
SET CMD_FILTTOOL=CSCRIPT %SYSTEMDRIVE%\Inetpub\Adminscripts\filttool.js
SET CMD_IISEXT=CSCRIPT %SYSTEMROOT%\System32\iisext.vbs
%CMD_FILTTOOL% -action:add -site:%WIBSITE% -name:CustomAuth -dll:%FILE_ISAPI%
%CMD_CHGLIST% %WEBSITE%/root/scriptmaps first *,%FILE_ISAPI%,1 /insert /commit
%CMD_IISEXT% /addfile %FILE_ISAPI% 1 CustomAuth 1 "MS CustomAuth Sample"
%CMD_ADSUTIL% set %WEBSITE%/root/AuthFlags 1
CACLS %FILE_PROTECT% /P Administrators:F
Th test is based on trying to access /pagerror.gif, which will trigger CustomAuth login and display the built-in login dialogbox. If it does not display for you because https:// is not enabled, then you need to configure UseSSLForFormSubmission=0 (or you can enable SSL by using SelfSSL from the IIS Resource Kit Tools...).
Based on your description, I suspect that you either did not force that all URLs under CustomAuth protection to be Anonymous Authentication only, or you failed to deny Anonymous user read ACL to the protected resource. In either case, the ISAPI-related configuration all looks good and CustomAuth.dll loads into memory as verified by "TASKLIST /m CustomAuth.dll", but without removing anonymous user access to protected resources to trigger 401 as well as force Anonymous authentication by clients, you will not see any prompts for credentials.