Well, I am back to Client certificate again, guess the reason being a lot of support calls that we getting off late are related to any of the following four errors, especially the first two.
403.17 ( I will cover .16 and .17 very briefly since they are very self-explanatory and easy to troubleshoot)
Earlier I had discussed the setup of the client certificate with IIS and AD for authentication mapping etc.
Here I will discuss the troubleshooting strategies on client certificate related errors that are listed above.
To understand how Client certificate is used while accessing a resource on the server, you may prefer to look at this brief but quite explanatory KB by David Dietz from IIS support.
So here we go...
We see that 403.7 can be thrown by IIS when Client certificate is required and the browser is not sending the client certificate details to the web server (IIS). Either the client did not send the certificate for some reason or else the client did not have a certificate issued by a CA that was also trusted by IIS server. If the client sends a certificate which is not mutually trusted by both client and the server you may see this error.
You may get a meaningful error like this in the browser:
HTTP Error 403 403.7 Forbidden: Client certificate required This error occurs when the resource you are attempting to access requires your browser to have a client Secure Sockets Layer (SSL) certificate that the server recognizes. This is used for authenticating you as a valid user of the resource. Please contact the Web server's administrator to obtain a valid client certificate.
To start with, follow this KB http://support.microsoft.com/kb/332077/en-us
A CTL is a list of trusted certification authorities (CAs) that can be used for client authentication for a particular Web site . You can use CTLs to configure your Web server to accept certificates from a specific list of CAs, and automatically verify client certificates against this list. Only users with a client certificate that is issued by a CA in the CTL can gain access to the server. Each Web site on your server can be configured to accept certificates from a different CTL. You may want to do this if you need a different list of trusted CAs for each Web site.
If CTL is present, this is the list which is actually used to check for CA's which can issue client certificate to a user. If it is disabled then root CA store will be used for the above. Also make sure that the certificate is a valid client certificate. Make sure it is intended for user authentication.
Check the certificate for "Ensures the identity of a remote computer" and Enhanced Key usage says Client Authentication.
Also Using >Certutil -verify -urlfetch should show:
Verified Application Policies:220.127.116.11.18.104.22.168.2 Client Authentication
The problem can also be identified when the following entry is logged on the Web server. It is quite explanatory in itself.
Event Type: WarningEvent Source: SchannelEvent Category: NoneEvent ID: 36885Date: 2/9/2007Time: 9:32:44 AMUser: N/AComputer: USMASVGDOIM259Description:When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a clientcertificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificateauthorities trusted for client authentication and remove those that do not really need to be trusted.
Trusted root certificates that are required by Windows Server 2003, by Windows XP, and by Windows 2000http://support.microsoft.com/kb/293781931125 Microsoft root certificate program members (January 2007)http://support.microsoft.com/default.aspx?scid=kb;EN-US;931125
The error you may see in the browser will be as shown below:
HTTP 403.13 Forbidden: Client certificate revoked The page requires a valid client certificate
For an understanding of this error message check KB 248058.
This error message means that the client sent a certificate, but either the certificate shows up as revoked in the issuing authority's Certificate Revocation List or the server could not retrieve a CRL from the issuing authority.
IIS , by default retrieves a CRL whenever it receives a client cert to make sure that cert is not revoked as long as local cache is expired. For this it contacts the CA to get the CRL which is a list of revoked certificates and compares the list with the presented client cert. If for any reason it cannot retrieve the CRL, it will go ahead and throw error message as 403.13 even if cert is valid and not revoked. This can happen in cases where some Proxy/firewall may block access to CDP to get the CRLs.
If a CDP extension is present in a certificate that is part of the certification path, IIS must be able to download at least one of the CRLs. If IIS is unable to resolve the CRL, it returns the HTTP 403.13 error. In this case, we cannot access the above CDP so we fail. Prior to MS04-011 Win2k did not limit validation based on this. However, we now require that the CDP be reachable when validating a certificate chain.To work around this we must either use a reachable CDP in the client certificate or disable CertCheckMode on the IIS server, thus preventing it from doing any revocation checking.
So, if we are getting Client certificate revoked errors, then check to see if the server can get to the CRL distribution point specified in the client certificate and if it can and is still giving this error, then download the Root and Subordinate CA CRLs and install them on the IIS server so that it can get to it locally.
Also there is a metabase key in IIS called certcheckmode, which if disabled will stop IIS from trying to retrieve CRL checking. In such a case client cert will be accepted even if the cert is revoked. Disabling CRL checking is a quick way to test the cause.
The CertCheckMode property enables or disables Certificate Revocation List (CRL) checking. When CertCheckMode is set to a value greater than 0 (CertCheckMode>0), the CRL does not search for certificates that have been revoked. When CertCheckMode is equal to 0 (CertCheckMode=0), the CRL searches for certificates that have been revoked.
With CertCheckMode disabled, IIS will no longer try to verify revocation of incoming client certificate requests. The client certificates will still need to be within their valid dates and still must be trusted by the IIS server (the IIS server must trust the issuing CA).
We disable the Certcheckmode key by setting it to 1.
>C:\Inetpub\Adminscript\cscript.exe adsutil.vbs Set W3SVC/<Website identifier>/CertCheckMode 1
I had seen an interesting case where 2 of the websites were accepting the same client cert whereas another one was not accepting it on the same web server.
Checking the metabase.xml for the server showed this:
<IIsWebServer Location ="/LM/W3SVC/690402"
<IIsWebServer Location ="/LM/W3SVC/90326589"
Look at the difference between them. You see CertCheckMode is absent in the Non-working site, and absence of this key is equivalent to it being enabled. So once we put the CertCheckMode set to "1" for non-working site we should be able to resolve the issue. But this means that CRL chekcing is disabled. You may downlaod the CRL on to the server or else open up the relevant ports in order to allow CRL to be retrieved.
Check the KB 294305.
You may also check KB 841632 if IIS 5.0 is in picture.
There was an interesting case, where users were getting 403.13 even when client cert was not revoked and we were able to access the get the CRL from the CDP for the client cert by accessing it through a browser. Yet after a lot of tracing and monitoring we found that there was a 4-level hierarchy in the certificate chain, with let's say Root CA1 ->Subordinate Root CA2->Subordinate Root CA3 -> Client certificate and one of the subordinate root CA's crl was not accessible. There are tools like certutil or SSLspy that can come handy. We ran certutil.exe -verify -urlfetch <location of the client cert.cer> on the IIS server and found that CRL retrieval for Subordinate Root CA2 was failing, and hence the issue.
So remember that we need to make sure that the CDPs for all the subordinate CAs certifcates in the chain should also be reachable.
Let's say for my client certificate, the Certification path shows:
Microsoft Corporate Root CA
|--> Microsoft Corp Enterprise CA 2
|--> Saurabh Singh
Here is the information for certificate "Saurabh singh"
CRL Distribution Points (Under Details->Field) shows:
CRL Distribution PointDistribution Point Name:Full Name:URL=ldap:///CN=Microsoft%20Corp%20Enterprise%20CA%202(4),CN=CRL,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=microsoft,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPointURL=http://corppki/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crlURL=http://crl.microsoft.com/pki/mscorp/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crl
Authority Information Access shows:
Authority Info AccessAccess Method=Certification Authority Issuer (22.214.171.124.126.96.36.199.2)Alternative Name:URL=http://corppki/aia/Microsoft%20Corp%20Enterprise%20CA%202(4).crtAuthority Info AccessAccess Method=Certification Authority Issuer (188.8.131.52.184.108.40.206.2)Alternative Name:URL=http://www.microsoft.com/pki/mscorp/Microsoft%20Corp%20Enterprise%20CA%202(4).crt
Here is the information for certificate "Microsoft Corp Enterprise CA 2":
CRL Distribution PointDistribution Point Name:Full Name:URL=http://corppki/crl/mscrca.crlURL=http://crl.microsoft.com/pki/mscorp/crl/mscrca.crl
Authority Info AccessAccess Method=Certification Authority Issuer (220.127.116.11.18.104.22.168.2)Alternative Name:URL=http://corppki/aia/mscrca.crtAuthority Info AccessAccess Method=Certification Authority Issuer (22.214.171.124.126.96.36.199.2)Alternative Name:URL=http://www.microsoft.com/pki/mscorp/mscrca.crt
Now runnning the Certutil.exe as shown below:
cmd prompt> certutil.exe -verify -urlfetch cert.cer <-------the client certificate
Here is the output:
Issuer:CN=Microsoft Corp Enterprise CA 2Subject:CN=Saurabh SinghCert Serial Number: 258a555c0004008b1c42
dwFlags = CA_VERIFY_FLAGS_DUMP_CHAIN (0x40000000)ChainFlags = CERT_CHAIN_REVOCATION_CHECK_CHAIN_EXCLUDE_ROOT (0x40000000)HCCE_LOCAL_MACHINECERT_CHAIN_POLICY_BASE-------- CERT_CHAIN_CONTEXT --------ChainContext.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)ChainContext.dwRevocationFreshnessTime: 176 Days, 6 Hours, 5 Minutes, 17 Seconds
SimpleChain.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)SimpleChain.dwRevocationFreshnessTime: 176 Days, 6 Hours, 5 Minutes, 17 Seconds
CertContext: dwInfoStatus=102 dwErrorStatus=0Issuer: CN=Microsoft Corp Enterprise CA 2Subject: CN=Saurabh SinghSerial: 258a555c0004008b1c42Template: AutoEnrolled Client Auth48 b7 48 da 00 51 21 77 b3 e1 3a ce 98 7d 35 2f b7 e8 0c 1cElement.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)---------------- Certificate AIA ----------------Verified "Certificate (0)" Time: 1[0.0] http://corppki/aia/Microsoft%20Corp%20Enterprise%20CA%202(4).crt
Verified "Certificate (0)" Time: 1[1.0] http://www.microsoft.com/pki/mscorp/Microsoft%20Corp%20Enterprise%20CA%202(4).crt
---------------- Certificate CDP ----------------Verified "Base CRL (821)" Time: 0[0.0] ldap:///CN=Microsoft%20Corp%20Enterprise%20CA%202(4),CN=CRL,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=corp,DC=microsoft,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint
Verified "Base CRL (821)" Time: 0[1.0] http://corppki/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crl
Verified "Base CRL (821)" Time: 1[2.0] http://crl.microsoft.com/pki/mscorp/crl/Microsoft%20Corp%20Enterprise%20CA%202(4).crl
---------------- Base CRL CDP ----------------No URLs "None" Time: 0--------------------------------CRL 821:Issuer: CN=Microsoft Corp Enterprise CA 2fd 19 3c 2f 0c 24 ea 1c 4a 5d df c4 26 2a b0 1b 98 48 ef 99Application = 188.8.131.52.184.108.40.206.2 Client Authentication
CertContext: dwInfoStatus=102 dwErrorStatus=0Issuer: CN=Microsoft Corporate Root CA, O=Microsoft CorporationSubject: CN=Microsoft Corp Enterprise CA 2Serial: 610d1de0000000000019Template: SubCA17 0a 7b 9d 52 85 07 7e 74 1a f5 a0 6b db 05 78 9e bc f1 8dElement.dwInfoStatus = CERT_TRUST_HAS_KEY_MATCH_ISSUER (0x2)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)---------------- Certificate AIA ----------------No CRL "Certificate (0)" Time: 0[0.0] http://corppki/aia/mscrca.crt
No CRL "Certificate (0)" Time: 1[1.0] http://www.microsoft.com/pki/mscorp/mscrca.crt
---------------- Certificate CDP ----------------Verified "Base CRL (18)" Time: 0[0.0] http://corppki/crl/mscrca.crl
Verified "Base CRL (18)" Time: 0[1.0] http://crl.microsoft.com/pki/mscorp/crl/mscrca.crl
---------------- Base CRL CDP ----------------No URLs "None" Time: 0--------------------------------CRL 18:Issuer: CN=Microsoft Corporate Root CA, O=Microsoft Corporation0e 70 65 69 a7 4c f9 7d 9f 50 7b db 9c e1 b8 27 9e 53 ba f4
CertContext: dwInfoStatus=10c dwErrorStatus=0Issuer: CN=Microsoft Corporate Root CA, O=Microsoft CorporationSubject: CN=Microsoft Corporate Root CA, O=Microsoft CorporationSerial: 443c2a54b59cd69d4c09b18a9b02eb55d2 d3 8e ba 60 ca a1 c1 20 55 a2 e1 c8 3b 15 ad 45 01 10 c2Element.dwInfoStatus = CERT_TRUST_HAS_NAME_MATCH_ISSUER (0x4)Element.dwInfoStatus = CERT_TRUST_IS_SELF_SIGNED (0x8)Element.dwInfoStatus = CERT_TRUST_HAS_PREFERRED_ISSUER (0x100)---------------- Certificate AIA ----------------No URLs "None" Time: 0---------------- Certificate CDP ----------------No URLs "None" Time: 0--------------------------------
Exclude leaf cert:8a cf e9 23 e2 d7 cd d1 f0 bb 05 6e 63 b5 31 95 6e 46 0d adFull chain:5b fa 04 32 34 21 49 11 92 56 b3 ee 41 94 b4 b8 f3 f6 44 f2------------------------------------Verified Issuance Policies: NoneVerified Application Policies:220.127.116.11.18.104.22.168.2 Client AuthenticationLeaf certificate revocation check passedCertUtil: -verify command completed successfully.
If you notice the Certutil.exe tries to check the CRL accessibility by accessing the CRL Distribution points. The above command ouptput should give you an idea regarding the cause. You may see an error in accessing the CRL in the output above in cases where you get the above errors.
Here is something similar when you get an error:
---------------- Certificate CDP ----------------
Failed "CDP" Time: 0
Error retrieving URL: The specified network resource or device is no longer available. 0x80070037 (WIN32: 55)
ldap:///CN=CRL1, CN=xxxx, OU=xxxx, OU=xxxx. by ref. (limits liab.), O=xxxx, C=US?certificateRevocationList;binary,authorityRevocationList;binary,deltaRevocationList;binary
Error retrieving URL: The server name or address could not be resolved 0x80072ee7 (WIN32: 12007)
Application = 22.214.171.124.126.96.36.199.4 Secure Email
Application = 188.8.131.52.184.108.40.206.1 Server Authentication
Application = 220.127.116.11.18.104.22.168.2 Client Authentication
Application = 22.214.171.124.126.96.36.199.3 Code Signing
Exclude leaf cert:
14 4c 46 42 11 66 a4 a9 42 70 ad b6 e0 1e 23 ca d4 9b 24 0e
fe 37 4a cf 76 3e 01 14 21 a6 c7 25 35 14 97 e5 91 87 e3 b7
Issuer: CN=A company, OU=PKI, DC=company, DC=com
Subject: OID.0.9.2342.19200300.100.1.1=ZALDI001, OU=People, OU=SAP, DC=company, DC=com
6e 33 5f 13 e1 67 ad 41 71 02 96 17 c7 57 c9 91 ea cb 1d 24
The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613)
Revocation check skipped -- server offline
Cert is an End Entity certificate
ERROR: Verifying leaf certificate revocation status returned The revocation function was unable to check revocation because the revocation server was offline. 0x80092013 (-2146885613)
CertUtil: The revocation function was unable to check revocation because the revocation server was offline.
Ensure that the necessary firewall/network configuration changes to allow the IIS server to access ALL of the external CDP’s listed in the client cert’s revocation chain, or download the CRL(s) to the IIS server manually and set CertCheckMode to MD_CERT_CACHE_RETRIEVAL_ONLY (see this link http://technet2.microsoft.com/windowsserver/en/library/173427fd-eb90-44ef-8a9c-d7bb4ff41ab81033.mspx?mfr=true ). That will tell IIS to look at the CRL in its local store and not try to attempt to access the CRL via the CDP entries specified in the client cert.
One more confusing point that should be clarified here:
If you have a certificate chain, let's say: Root CA -> Intermediate CA1 -> Intermediate CA2 ->..... -><Your Client ceritficate>, then CRL checking will be done for all the Certificates in the hierarchy except the Root CA.
If you are really interested to dig further as to how Certificate Revocation etc. works at a lower level, here is a real exhaustive link to check it out.
"Choose a digital certificate" popup window in Internet Explorer is blank when attempting to use client certificates to authenticate against IIS.
The total size of the certificates in the Trusted Root Certification Authorities store on the IIS server was too large to send to the client. The list was truncated as a result.
The following event was written to the System log:Event Type: Warning Event Source: Schannel Event ID: 36885 Description: When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.
Resolution would be to Remove unused certificates from the Trusted Root Certification Authorities store on the IIS server, reducing the number of certificates.
Also this should be of help at times http://support.microsoft.com/kb/285069/
Refer to one of our finest Escalation engineer (Andreas Klein)'s blog which talks about limiting the list of CA's allowed for Client authentication, without deleting the CAs from the store.
Also while you may have the certificate in your personal store (using the mmc snap-in it shows up properly), you may not see it in the IE browser. If you go through Internet Options->Content and click Certificates, it doesn’t show up at all. Open the certificate MMC and check whether the cert has a Private key or not.
If the General tab on the cert properties does not say at the bottom that you have a Private Key corresponding to this cert then you don’t, and this may lead to the above problem.
403.16 - Client certificate is untrusted or invalid.
This error message is primarily generated when the certificate that the client provided is improperly formed. It can also be generated if there are intermediate certification authorities in the certificate chain that are not trusted by the Web server.
403.17 - Client certificate has expired or is not yet valid
This error message is fairly self-explanatory. It means that the current date on the server is not within the valid date ranges that are presented in the client certificate. You may also want to ensure that the client certificate and its issuing CAs (including Intermediate CAs) are not expired or invalid.
Thank you very much for this sir.... you have helped immensely in solving a Java client <-> IIS webservice cert exchange issue I was having. Wonderful!
I was wondering how you can differentiate between the 403 error you get when a HttpWebRequest fails to authenticate with a Proxy and the 403 error (403.7 which is thrown when a SSL certificate is required.
Thanks and Regards,
I am facing the 403.17 issue during client authentication. The funny thing is that the regular CRL check (when CertCheckMode =0) works fine, but when I set CertCheckMode =4 (CRL Refresh), the client authentication fails. I have run the certutil on the client cert and it completes successfully without error. I could see the network traffic to the CA, but the still get the 403.17 error. Any thoughts?
Since you are seeing 403.17, i would suggest to cross check all the certificates in the entire chain for expiry/validity.
--> Intermediate CA1
---> Intermediate CA1
Your Client cert
Do you see any of the intermediate CAs in your Cert chain, if yes can you cross check whether they have expired or not. You may also want to check the same in the Intermediate CA store on the machine on both client and server.
Also can you try replacing the current cert with a new one, just to see if that works for you or not.
Let me know...
Thanks for your reply. All the CAs are current and able to validate the cert and I have tried this with multiple certs. When I set the Certcheckmode=4 and CRLRefreshtime to 120 seconds, the IIS is unable to authenticate the cert. I was reading somewhere that, if IIS in unable to download these CRLs immediately, it creates a background process to download the CRLs and continues with client authentication which in turn fails due to lack of CRLs. Is this correct?
I know this is an old post so I'm hoping someone is still following.
So what if you get authenticated and within a couple (roughly 2) minutes you get a 403.13 when you navigate to a different page on the same server? Is the authentication cached and that cache checked for each page?
This is the MOST comprehensive article on certificates issues I have ever seen.
It has helped me to complete a 3 day torture with mission critical production web server.
Thanks a lot, keep up the good work!
p.s. For everyone's future reference and feel free to include this in the blog, the SOURCE of my problem was IIS 7.0 which hosted CRL lists and did not allow + (plus) sign in the filename to be downloaded, which delta CRL list HAS to have.
Solved this matter by allowing "Request filtering -> Allow double escaping" feature on IIS 7!