Well, I am back to Client certificate again, guess the reason being a lot of support calls that we getting off late are related to any of the following four errors, especially the first two.
403.17 ( I will cover .16 and .17 very briefly since they are very self-explanatory and easy to troubleshoot)
Earlier I had discussed the setup of the client certificate with IIS and AD for authentication mapping etc.
Here I will discuss the troubleshooting strategies on client certificate related errors that are listed above.
To understand how Client certificate is used while accessing a resource on the server, you may prefer to look at this brief but quite explanatory KB by David Dietz from IIS support.
So here we go...
We see that 403.7 can be thrown by IIS when Client certificate is required and the browser is not sending the client certificate details to the web server (IIS). Either the client did not send the certificate for some reason or else the client did not have a certificate issued by a CA that was also trusted by IIS server. If the client sends a certificate which is not mutually trusted by both client and the server you may see this error.
You may get a meaningful error like this in the browser:
HTTP Error 403 403.7 Forbidden: Client certificate required This error occurs when the resource you are attempting to access requires your browser to have a client Secure Sockets Layer (SSL) certificate that the server recognizes. This is used for authenticating you as a valid user of the resource. Please contact the Web server's administrator to obtain a valid client certificate.
To start with, follow this KB http://support.microsoft.com/kb/332077/en-us
A CTL is a list of trusted certification authorities (CAs) that can be used for client authentication for a particular Web site . You can use CTLs to configure your Web server to accept certificates from a specific list of CAs, and automatically verify client certificates against this list. Only users with a client certificate that is issued by a CA in the CTL can gain access to the server. Each Web site on your server can be configured to accept certificates from a different CTL. You may want to do this if you need a different list of trusted CAs for each Web site.
If CTL is present, this is the list which is actually used to check for CA's which can issue client certificate to a user. If it is disabled then root CA store will be used for the above. Also make sure that the certificate is a valid client certificate. Make sure it is intended for user authentication.
Check the certificate for "Ensures the identity of a remote computer" and Enhanced Key usage says Client Authentication.
Also Using >Certutil -verify -urlfetch should show:
Verified Application Policies:184.108.40.206.220.127.116.11.2 Client Authentication
The problem can also be identified when the following entry is logged on the Web server. It is quite explanatory in itself.
Event Type: WarningEvent Source: SchannelEvent Category: NoneEvent ID: 36885Date: 2/9/2007Time: 9:32:44 AMUser: N/AComputer: USMASVGDOIM259Description:When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a clientcertificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificateauthorities trusted for client authentication and remove those that do not really need to be trusted.
Trusted root certificates that are required by Windows Server 2003, by Windows XP, and by Windows 2000http://support.microsoft.com/kb/293781931125 Microsoft root certificate program members (January 2007)http://support.microsoft.com/default.aspx?scid=kb;EN-US;931125
The error you may see in the browser will be as shown below:
HTTP 403.13 Forbidden: Client certificate revoked The page requires a valid client certificate
For an understanding of this error message check KB 248058.
This error message means that the client sent a certificate, but either the certificate shows up as revoked in the issuing authority's Certificate Revocation List or the server could not retrieve a CRL from the issuing authority.
IIS , by default retrieves a CRL whenever it receives a client cert to make sure that cert is not revoked as long as local cache is expired. For this it contacts the CA to get the CRL which is a list of revoked certificates and compares the list with the presented client cert. If for any reason it cannot retrieve the CRL, it will go ahead and throw error message as 403.13 even if cert is valid and not revoked. This can happen in cases where some Proxy/firewall may block access to CDP to get the CRLs.
If a CDP extension is present in a certificate that is part of the certification path, IIS must be able to download at least one of the CRLs. If IIS is unable to resolve the CRL, it returns the HTTP 403.13 error. In this case, we cannot access the above CDP so we fail. Prior to MS04-011 Win2k did not limit validation based on this. However, we now require that the CDP be reachable when validating a certificate chain.To work around this we must either use a reachable CDP in the client certificate or disable CertCheckMode on the IIS server, thus preventing it from doing any revocation checking.
So, if we are getting Client certificate revoked errors, then check to see if the server can get to the CRL distribution point specified in the client certificate and if it can and is still giving this error, then download the Root and Subordinate CA CRLs and install them on the IIS server so that it can get to it locally.
Also there is a metabase key in IIS called certcheckmode, which if disabled will stop IIS from trying to retrieve CRL checking. In such a case client cert will be accepted even if the cert is revoked. Disabling CRL checking is a quick way to test the cause.
The CertCheckMode property enables or disables Certificate Revocation List (CRL) checking. When CertCheckMode is set to a value greater than 0 (CertCheckMode>0), the CRL does not search for certificates that have been revoked. When CertCheckMode is equal to 0 (CertCheckMode=0), the CRL searches for certificates that have been revoked.
With CertCheckMode disabled, IIS will no longer try to verify revocation of incoming client certificate requests. The client certificates will still need to be within their valid dates and still must be trusted by the IIS server (the IIS server must trust the issuing CA).
We disable the Certcheckmode key by setting it to 1.
>C:\Inetpub\Adminscript\cscript.exe adsutil.vbs Set W3SVC/<Website identifier>/CertCheckMode 1
I had seen an interesting case where 2 of the websites were accepting the same client cert whereas another one was not accepting it on the same web server.
Checking the metabase.xml for the server showed this:
<IIsWebServer Location ="/LM/W3SVC/690402"
<IIsWebServer Location ="/LM/W3SVC/90326589"
Look at the difference between them. You see CertCheckMode is absent in the Non-working site, and absence of this key is equivalent to it being enabled. So once we put the CertCheckMode set to "1" for non-working site we should be able to resolve the issue. But this means that CRL chekcing is disabled. You may downlaod the CRL on to the server or else open up the relevant ports in order to allow CRL to be retrieved.
Check the KB 294305.
You may also check KB 841632 if IIS 5.0 is in picture.
There was an interesting case, where users were getting 403.13 even when client cert was not revoked and we were able to access the get the CRL from the CDP for the client cert by accessing it through a browser. Yet after a lot of tracing and monitoring we found that there was a 4-level hierarchy in the certificate chain, with let's say Root CA1 ->Subordinate Root CA2->Subordinate Root CA3 -> Client certificate and one of the subordinate root CA's crl was not accessible. There are tools like certutil or SSLspy that can come handy. We ran certutil.exe -verify -urlfetch <location of the client cert.cer> on the IIS server and found that CRL retrieval for Subordinate Root CA2 was failing, and hence the issue.
So remember that we need to make sure that the CDPs for all the subordinate CAs certifcates in the chain should also be reachable.
Let's say for my client certificate, the Certification path shows:
Microsoft Corporate Root CA
|--> Microsoft Corp Enterprise CA 2
|--> Saurabh Singh
Here is the information for certificate "Saurabh singh"
CRL Distribution Points (Under Details->Field) shows:
CRL Distribution PointDistribution Point Name:Full Name:URL=ldap:///CN=Microsoft%20Corp%20Enterprise%20CA%202(4),CN=CRL,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=microsoft,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPointURL=http://corppki/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crlURL=http://crl.microsoft.com/pki/mscorp/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crl
Authority Information Access shows:
Authority Info AccessAccess Method=Certification Authority Issuer (18.104.22.168.22.214.171.124.2)Alternative Name:URL=http://corppki/aia/Microsoft%20Corp%20Enterprise%20CA%202(4).crtAuthority Info AccessAccess Method=Certification Authority Issuer (126.96.36.199.188.8.131.52.2)Alternative Name:URL=http://www.microsoft.com/pki/mscorp/Microsoft%20Corp%20Enterprise%20CA%202(4).crt
Here is the information for certificate "Microsoft Corp Enterprise CA 2":
CRL Distribution PointDistribution Point Name:Full Name:URL=http://corppki/crl/mscrca.crlURL=http://crl.microsoft.com/pki/mscorp/crl/mscrca.crl
Authority Info AccessAccess Method=Certification Authority Issuer (184.108.40.206.220.127.116.11.2)Alternative Name:URL=http://corppki/aia/mscrca.crtAuthority Info AccessAccess Method=Certification Authority Issuer (18.104.22.168.22.214.171.124.2)Alternative Name:URL=http://www.microsoft.com/pki/mscorp/mscrca.crt
Now runnning the Certutil.exe as shown below:
cmd prompt> certutil.exe -verify -urlfetch cert.cer <-------the client certificate
Here is the output:
Issuer:CN=Microsoft Corp Enterprise CA 2Subject:CN=Saurabh SinghCert Serial Number: 258a555c0004008b1c42
dwFlags = CA_VERIFY_FLAGS_DUMP_CHAIN (0x40000000)ChainFlags = CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT (0x40000000)HCCE_LOCAL_MACHINECERT_CHAIN_POLICY_BASE-------- CERT_CHAIN_CONTEXT --------ChainContext.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)ChainContext.dwRevocationFreshnessTime: 176 Days, 6 Hours, 5 Minutes, 17 Seconds
SimpleChain.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)SimpleChain.dwRevocationFreshnessTime: 176 Days, 6 Hours, 5 Minutes, 17 Seconds
CertContext: dwInfoStatus=102 dwErrorStatus=0Issuer: CN=Microsoft Corp Enterprise CA 2Subject: CN=Saurabh SinghSerial: 258a555c0004008b1c42Template: AutoEnrolled Client Auth48 b7 48 da 00 51 21 77 b3 e1 3a ce 98 7d 35 2f b7 e8 0c 1cElement.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)---------------- Certificate AIA ----------------Verified "Certificate (0)" Time: 1[0.0] http://corppki/aia/Microsoft%20Corp%20Enterprise%20CA%202(4).crt
Verified "Certificate (0)" Time: 1[1.0] http://www.microsoft.com/pki/mscorp/Microsoft%20Corp%20Enterprise%20CA%202(4).crt
---------------- Certificate CDP ----------------Verified "Base CRL (821)" Time: 0[0.0] ldap:///CN=Microsoft%20Corp%20Enterprise%20CA%202(4),CN=CRL,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=microsoft,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint
Verified "Base CRL (821)" Time: 0[1.0] http://corppki/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crl
Verified "Base CRL (821)" Time: 1[2.0] http://crl.microsoft.com/pki/mscorp/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crl
---------------- Base CRL CDP ----------------No URLs "None" Time: 0--------------------------------CRL 821:Issuer: CN=Microsoft Corp Enterprise CA 2fd 19 3c 2f 0c 24 ea 1c 4a 5d df c4 26 2a b0 1b 98 48 ef 99Application = 126.96.36.199.188.8.131.52.2 Client Authentication
CertContext: dwInfoStatus=102 dwErrorStatus=0Issuer: CN=Microsoft Corporate Root CA, O=Microsoft CorporationSubject: CN=Microsoft Corp Enterprise CA 2Serial: 610d1de0000000000019Template: SubCA17 0a 7b 9d 52 85 07 7e 74 1a f5 a0 6b db 05 78 9e bc f1 8dElement.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)---------------- Certificate AIA ----------------No CRL "Certificate (0)" Time: 0[0.0] http://corppki/aia/mscrca.crt
No CRL "Certificate (0)" Time: 1[1.0] http://www.microsoft.com/pki/mscorp/mscrca.crt
---------------- Certificate CDP ----------------Verified "Base CRL (18)" Time: 0[0.0] http://corppki/crl/mscrca.crl
Verified "Base CRL (18)" Time: 0[1.0] http://crl.microsoft.com/pki/mscorp/crl/mscrca.crl
---------------- Base CRL CDP ----------------No URLs "None" Time: 0--------------------------------CRL 18:Issuer: CN=Microsoft Corporate Root CA, O=Microsoft Corporation0e 70 65 69 a7 4c f9 7d 9f 50 7b db 9c e1 b8 27 9e 53 ba f4
CertContext: dwInfoStatus=10c dwErrorStatus=0Issuer: CN=Microsoft Corporate Root CA, O=Microsoft CorporationSubject: CN=Microsoft Corporate Root CA, O=Microsoft CorporationSerial: 443c2a54b59cd69d4c09b18a9b02eb55d2 d3 8e ba 60 ca a1 c1 20 55 a2 e1 c8 3b 15 ad 45 01 10 c2Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4)Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)---------------- Certificate AIA ----------------No URLs "None" Time: 0---------------- Certificate CDP ----------------No URLs "None" Time: 0--------------------------------
Exclude leaf cert:8a cf e9 23 e2 d7 cd d1 f0 bb 05 6e 63 b5 31 95 6e 46 0d adFull chain:5b fa 04 32 34 21 49 11 92 56 b3 ee 41 94 b4 b8 f3 f6 44 f2------------------------------------Verified Issuance Policies: NoneVerified Application Policies:184.108.40.206.220.127.116.11.2 Client AuthenticationLeaf certificate revocation check passedCertUtil: -verify command completed successfully.
If you notice the Certutil.exe tries to check the CRL accessibility by accessing the CRL Distribution points. The above command ouptput should give you an idea regarding the cause. You may see an error in accessing the CRL in the output above in cases where you get the above errors.
Here is something similar when you get an error:
---------------- Certificate CDP ----------------
Failed "CDP" Time: 0
Error retrieving URL: The specified network resource or device is no longer available. 0x80070037 (WIN32: 55)
ldap:///CN=CRL1, CN=xxxx, OU=xxxx, OU=xxxx. by ref. (limits liab.), O=xxxx, C=US?certificateRevocationList;binary,authorityRevocationList;binary,deltaRevocationList;binary
Error retrieving URL: The server name or address could not be resolved 0x80072ee7 (WIN32: 12007)
Application = 18.104.22.168.22.214.171.124.4 Secure Email
Application = 126.96.36.199.188.8.131.52.1 Server Authentication
Application = 184.108.40.206.220.127.116.11.2 Client Authentication
Application = 18.104.22.168.22.214.171.124.3 Code Signing
Exclude leaf cert:
14 4c 46 42 11 66 a4 a9 42 70 ad b6 e0 1e 23 ca d4 9b 24 0e
fe 37 4a cf 76 3e 01 14 21 a6 c7 25 35 14 97 e5 91 87 e3 b7
Issuer: CN=A company, OU=PKI, DC=company, DC=com
Subject: OID.0.9.2342.19200300.100.1.1=ZALDI001, OU=People, OU=SAP, DC=company, DC=com
6e 33 5f 13 e1 67 ad 41 71 02 96 17 c7 57 c9 91 ea cb 1d 24
The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613)
Revocation check skipped -- server offline
Cert is an End Entity certificate
ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613)
CertUtil: The revocation function was unable to check revocation because the revocation server was offline.
Ensure that the necessary firewall/network configuration changes to allow the IIS server to access ALL of the external CDP’s listed in the client cert’s revocation chain, or download the CRL(s) to the IIS server manually and set CertCheckMode to MD_CERT_CACHE_RETRIEVAL_ONLY (see this link http://technet2.microsoft.com/windowsserver/en/library/173427fd-eb90-44ef-8a9c-d7bb4ff41ab81033.mspx?mfr=true ). That will tell IIS to look at the CRL in its local store and not try to attempt to access the CRL via the CDP entries specified in the client cert.
One more confusing point that should be clarified here:
If you have a certificate chain, let's say: Root CA -> Intermediate CA1 -> Intermediate CA2 ->..... -><Your Client ceritficate>, then CRL checking will be done for all the Certificates in the hierarchy except the Root CA.
If you are really interested to dig further as to how Certificate Revocation etc. works at a lower level, here is a real exhaustive link to check it out.
"Choose a digital certificate" popup window in Internet Explorer is blank when attempting to use client certificates to authenticate against IIS.
The total size of the certificates in the Trusted Root Certification Authorities store on the IIS server was too large to send to the client. The list was truncated as a result.
The following event was written to the System log:Event Type: Warning Event Source: Schannel Event ID: 36885 Description: When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.
Resolution would be to Remove unused certificates from the Trusted Root Certification Authorities store on the IIS server, reducing the number of certificates.
Also this should be of help at times http://support.microsoft.com/kb/285069/
Refer to one of our finest Escalation engineer (Andreas Klein)'s blog which talks about limiting the list of CA's allowed for Client authentication, without deleting the CAs from the store.
Also while you may have the certificate in your personal store (using the mmc snap-in it shows up properly), you may not see it in the IE browser. If you go through Internet Options->Content and click Certificates, it doesn’t show up at all. Open the certificate MMC and check whether the cert has a Private key or not.
If the General tab on the cert properties does not say at the bottom that you have a Private Key corresponding to this cert then you don’t, and this may lead to the above problem.
403.16 - Client certificate is untrusted or invalid.
This error message is primarily generated when the certificate that the client provided is improperly formed. It can also be generated if there are intermediate certification authorities in the certificate chain that are not trusted by the Web server.
403.17 - Client certificate has expired or is not yet valid
This error message is fairly self-explanatory. It means that the current date on the server is not within the valid date ranges that are presented in the client certificate. You may also want to ensure that the client certificate and its issuing CAs (including Intermediate CAs) are not expired or invalid.
This is a great post. I learned many of the tips prior to reading this post through agonizing trial and error, but this is a great summary.
One quick question. I have disabled CRL checking by setting certCheckMode to 1 but the initial SSL connection still takes a very long time when using client certificates. Do you know why this is? It seems slow on IIS. Any tips to optomize performance?
Hi Yogesh, Does the issue happen even without SSL in picture. I mean does it take same amount of time when using HTTP instead of HTTPS for the website?
Do you see any spike in CPU utilization or high memory on the server during the period when the issue occurs. Remember that SSL handshake is a CPU intensive process, so it will have an impact on the performance but not to a great extent. Also do we have any proxy/firewall/Accelerator coming into picture.
Thanks for this article, was a huge help debugging why client certs stopped working!
Glad this article saved people precious time and energy. I will continuously be refining this as and when i get new issues.
Hi, great post. Is there any way I can disable the "403.17 - Client certificate has expired or is not yet valid" error checking in the IIS? The reason for this is that we have a dedicated software to handle all client certificates (Nexus MultiID Server) so we just want the IIS to get the cert but not do any revocation or exipration checking. The revocation checking we have disable fine, but I have not been able to find out how to disable the expiration date check. Any ideas?
I had sent you an email for this earlier...here it is for wider audience.
IIS doesn’t offer a way to disable Client certificate expiration date checking. Only the Revocation checking can be disabled.
Hope this helps.
Hi, great post. The Verisign Trial Secure Server Root CA certificate is not being sent in the list of trusted certificate authorities to my client. I know that it is not being truncated because there is no error in the system log. I also tried putting it in a CTL but no luck. Is there something different about trial root ca certs?
To my knowledge Verisign Trial Secure Server Root CA certificate is used for testing purpose. It should not be used for production or public site. You may be seeing the above behavior because Trial certificate has to be manually installed in your client browser (This requirement is to prevent fraudulent use of test certificates).
You may check the FAQs for furter information at http://www.verisign.com.sg/support/ssl/knowledgebase/trialfaq.shtml
Great post, helped me understand some fundamentals regarding the IIS Metabase. Cheers.
I am having a real problem getting Client Certificate Mapping working. I have it working on my development system (2003 SP2/IIS 6/no domain). But when I try to set it up on my test system (2003 SP2/IIS 6/domain member) it does not work. All it get is a 500 0 0 error in the IIS logs. I have the server cert set up properly since I can hit the site using https without client cert required. I can also log on as the mapped user then I have windows auth check and client mapping off. I have searched high and low on the web but no one has any suggestions. I have turned on IIS tracing and I have used IIS Diag, all with no helpful information.
It looks like from your problem description that client certificate mapping is not done correctly. Have you checked my other post on client certificate mapping
If this doesn't help please let me know.
Yes I have followed your other guide without success on my test system or production server. I have successfully set it up on my development server and on and XP system. I have not been able to get it working on my test 2003 server or on my production 2003 server. They are both domain members where as my development and XP boxes are not. On my test server I ran the SSL Diagnostics tool and it shows my IIsCerMapper client map to a user and it shows a successful log. How can I get more information about a IIS 500 error?
IIS 500 is a generic error message and there could be N number of reasons. I will do some research to find out what reasons could cause it to throw 500 error in your case. In the meanwhile can you ensure that we uncheck "Show Friendly HTTP error messages" from IE->Tools->Internet Options->Advanced->Browsing and then try browsing to the site and see if it gives you some meaningful error message in the browser. Also ensure you are browsing to a test page (like html/asp) in the site which doesn't have any complex functionality in it. A sample test.asp which does response.write or something similar.
Are you mapping client cert to a local user account or a domain account? Are you using Active Directory mapping or 1-to-1 or Many-to-1 mapping? Ensure you do not have both AD and 1-to-1/Many-to--1 mapping enabled. Do you have any ISAPI filter loaded in IIS at global or website level?
I have already unchecked "Show Friendly HTTP error messages" on my client and i don't see anything. The page always returns blank with no content. I am hosting a web service. I am just trying to browse to the web service and view the WSDL so it should not be too complicated. I have also put your logoninfo.asp page out there and I get the same results, 500 error in IIS log and blank content, or should I say no content since IIS is failing. I don't see any error messages any where else, nothing in the eventlogs.
I am mapping to a local user right now. I think I tried a domain account before (I have been working on this for several weeks). I will try a domain account again and let you know. I am using 1-to-1 mapping and would prefer that. I don't what to have domain mapping if I don't need it. I have not tried domain mapping yet, but I will and see if that helps.
I don't have any special ISAPI filters in place. I am just using the standard IIS 6 setup. I have created a new web site. I am using host headers for the site and I have the cert/host header matching each other. I also ran adsutil to set the SecureBindings SSL host header. The host header are properly set since I can browse to the web service with both http/https when I don't have reqiures certificate checked.
I am stumped and very frustrated with this. It should be a piece of cake to setup, but for some reason it does not want to work.
Really useful post and a good insight into IIS and client certs with some really great tips. Amazingly, it actually helped me solve a System Center Configuration Manager problem I was having, so hurrah!!! :-)