Troubleshooting SSL related issues with IIS

Troubleshooting SSL related issues with IIS

  • Comments 39

In this post I am going to discuss troubleshooting SSL server certificate related issues in IIS. Please remember I am no SSL expert...I am an IIS Asp.Net guy and will discuss issues with regard to SSL with IIS.

We have seen a lot many times that the issues could have been resolved by our customers if they were aware of some targeted troubleshooting techniques.

Here I will be walking through the steps which you can follow yourself before requiring to call Microsoft PSS.

Problem Description

The most common error associated with SSL not working for an IIS website is "The page cannot be displayed" or "Cannot find server or DNS Error".

Troubleshooting steps to be followed:

First thing first: You notice the above error when you browse to your website over SSL (HTTPS). Do you also see the same behavior when you access the website over HTTP? If yes then there is a different problem you need to address first. SSL doesn't come into picture when we access a site over HTTP.
So if you are facing the problem in both the scenarios ensure you first troubleshoot why the request is not reaching the IIS server...maybe a DNS configuration etc.

My topic today is SSL with IIS so I won't be focusing on HTTP.

We have the SSL port listed in IIS mmc as shown below. By default it's 443. You can configure it to listen on a different port.

image

Now, also check whether the issue is happening for all the users, i.e. External and internal users to the network. If the issue is happening only for let's say external users check whether the requests are coming over a firewall like ISA etc. At times Firewalls may block the SSL port. You may want to check the firewall settings to unblock the SSL port. Also firewalls like ISA requires certificate to be published on it as well apart from IIS server. Ensure that the certificate that we have installed on ISA is valid. In case of ISA, if you have renewed the cert on IIS server but not on ISA users will still see the old certificate and not the new certificate. Ensure that you replace the old certificate on ISA with the new certificate. Also you may want to check if the security gateway (if any) such as NFUSE (Citrix) controllers is still referencing the old certificate.

Also at times if you see a wrong certificate being displayed for your Web Site, ensure that you do not have kernel-mode SSL enabled on your web server.Although this improves the performance but I recommend to have it disabled when you have multiple SSL enabled-websites on your server. Kernel-mode SSL is only recommended if you have just one site, mostly static content and no client certificates. In all other cases you should use User-mode SSL.

Please find this link to enable/disable kernel-mode SSL. If it is set to 1 it means Kernel-mode SSL is enabled, if absent or set to 0 it means disabled and user-mode SSL will be used. Ensure that you stop and restart http service as shown below after doing the changes. No reboot is required for the above change.

>net stop http

>net start http

>net start w3svc

[There has been few concerns from people as to why this is suggested. I do not have a proper answer to it at this point, but I have seen such issues in some support incidents where disabling Kernel mode SSL had resolved the above problem]

------------------------------------------------------------------------------------------------------------------------------------------------------------------

Now once you have isolated the issue to be only with HTTPS and all users let's proceed with the following steps:

Run netstat -an (or fport.exe) for IIS 5 (and netstat -ano for IIS 6) and verify whether IIS is listening on the SSL port.

When you start Internet Information Services (IIS) 6.0 on Microsoft Windows Server 2003, IIS binds to all IP addresses on the server, not just the IP addresses that are assigned to Web sites. This is because of socket pooling enabled for IIS by default.

On IIS 6.0, if the website is listening on "All unassigned" IP address or some specific IP address (like 192.134.123.209 etc) and SSL port 443, verify that Local address entry is 0.0.0.0:443 in netstat output.

> netstat -ano

Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    0.0.0.0:21             0.0.0.0:0              LISTENING       2396
  TCP    0.0.0.0:25             0.0.0.0:0              LISTENING       2396
  TCP    0.0.0.0:80           0.0.0.0:0            LISTENING       4
  TCP    0.0.0.0:81             0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:83             0.0.0.0:0              LISTENING       4
    ...................

  TCP   0.0.0.0:443          0.0.0.0:0             LISTENING       4
  TCP    0.0.0.0:445            0.0.0.0:0              LISTENING       4
  TCP    0.0.0.0:446            0.0.0.0:0              LISTENING       4

Here the PID should show you which process is listening on that port.
Now let's say I have configured my website to run on a specific SSL port and that port is being used by a different process, then website will not start up.

Also you won't see the netstat output showing anything like this depending upon the port being used:

TCP    0.0.0.0:80             0.0.0.0:0              LISTENING       4

TCP    0.0.0.0:443           0.0.0.0:0              LISTENING       4

You can run netstat -ano again and see the PID corresponding to the process which is listening on that port. Stopping or disabling that service/process should allow Websites to be up and listening on the desired port. For IIS 5.x you can run Fport.exe to find out which process is listening on a specific port.

The first thing that I check for (strictly my personal preference) when we have issues with SSL is MACHINEKEYS. Yes, this is a very important (I prefer to call strategic) part of your checklist while dealing with SSL issues. In my personal experience it has resolved a lot of issues for our customers. By default machinekeys folder should have permission for Administrator, System and Everyone according to this KB278381. Ensure that we inherit the above permission for all child objects under the machinekeys folder, that way all the machine keys have the necessary permissions.

This is done by clicking the "Advanced" button on the security settings page, Clicking the check box "Replace permissions entries on all child objects...." and
Apply. This will propagate the settings to the contents in the folder.

Another reason for the above error ('The page cannot be displayed') can be if you do not have a private key corresponding to the SSL server certificate. You can check that by opening the IIS mmc -> <<YourWebSite>> ->Properties ->Directory Security ->Secure Communications ->View certificate ->General -> "You have a private key that corresponds to this certificate" as shown below.

image

If you do not see the above then SSL won't work for your website.
You can try running the following command to recover the private key from the certificate thumbprint.

To associate the certificate with its private key, we run the following command:

> certutil -repairstore my "c9 66 67 68 49 97 88 7f 05 b8 89 b7 b3 f7 37 c8 c4 da 5f 16"

Note: "c9 66 67 68 49 97 88 7f 05 b8 89 b7 b3 f7 37 c8 c4 da 5f 16" is the Thumbprint of the missing certificate. You can see this if you double-click on the .cer file, choose the Details tab and select Thumbprint.

image

Make sure you use the 2003 version of certutil.exe (with the associated certadm.dll and certcli.dll), or you will not have access to the repairstore command.

If the above command does not get back the private key or for some reason it still does not work you may prefer to get a new certificate from the CA.

One of the best tools that Microsoft has come up with is SSL Diagnostics to troubleshoot SSL related issues. Just download it to the IIS server and run it and if there is an error due to configuration of IIS or certificate it will show up. You can also use this tool to issue a test certificate for your website to check whether the problem was occurring because of your server certificate or some other IIS configuration related issues. Things that can be shown quickly through this tool includes on "no private key", "SSL port being used by a different process", "machine keys not having enough permission", "IP address conflicts" etc.

Another scenario can be wherein if you view the certificate, it will state that "You have a private key that corresponds to this certificate," and if you run SSL Diagnostics, you'll get the error as below:

#WARNING: You have a private key that corresponds to this certificate but CryptAcquireCertificatePrivateKey failed

When you initially go through the IIS Certificate Wizard to create a new certificate, a file is added to the MachineKeys folder. This is the private key.

If you add permissions to the new file in the MachineKeys folder before you process the pending request in the IIS certificate wizard, the certificate will work properly. If you've already gone past that point and got a new certificate, then necessary permissions should be added to the MachineKeys folder and a new certificate request must again be submitted.

Now consider a scenario wherein you are able to access the site through SSL Diagnostics tool (through 'Simulate SSL Handshake') for the same certificate but not through IE. What I mean is your SSL handshake gives no error etc in SSL Diagnostics although when you access the website through IE over SSL it still fails. In that case there could be an IE setting that you may want to check for such a behavior. Check this out KB811834.

Now one last scenario. At times we see customers coming up with requests wherein they want to use the same server certificate for multiple websites.

This is not possible. You cannot use the same certificate for multiple websites on the same server. Check this KB187504.

Now there is a workaround in IIS 6.0 Windows 2003 SP1, wherein you can use Wildcard certificates to get a similar functionality. Here is the link.

If you have a Load balanced or clustered environment you can export the same certificate to all the servers and install them on the related websites. It should work fine.

Hope this helps....let me know if there are any questions around the topic.

Cheers!!!

Leave a Comment
  • Please add 3 and 3 and type the answer here:
  • Post
  • Very sweet post!

  • PingBack from http://msdnrss.thecoderblogs.com/2007/09/05/troubleshooting-ssl-related-issues-with-iis/

  • Another thing to double check is that the certificate file in the machine key folder has correctly inherited permissions. I just spent an hour and a half pouring over everything, only to find out that the system did not have full access permissions on the certificate I installed.

    This article saved my life though, I would have never found it otherwise.

  • Glad that this article saved you precious time.

    Please feel free to shoot any question if they are related to any of the topics that I work on like IIS, ASP and Asp.Net and i will try to get you an answer by some means, even if that includes me asking other people here to get your answers.

    Thanks!

  • Excellent Post.. I guess its a better then Microsoft explanition.:)

    Question:

    I am facing a similar problem, when I enter port 443 in the IIS website window, my website stop working and I get a message that "service is being used by another process". When i remove port 443 from IIS window, the website start 2 work fine. (But then I can't use it in the https mode.)

    Can I try the same solution which you have mentioned with my probelm?

    OR i would need to change some setting in the registry which is suggest by Microsoft ( support.microsoft.com/kb/890015 ).

    Best regards,

    Please reply to my email also.

    gbwebs@eim.ae

  • Hi Murtaza, thanks for your comment :-)

    Yes, you can follow the article that you have mentioned there but there are some checklist to ensure you are facing the same  issue.

    -- Do you see any of those event logs mentioned in the KB article on your machine when you try to start the website related to IP Listen-only list? If yes then you can follow the KB.

    -- How many websites do you have in the IIS server?

    -- Are you running your faulting website on some specific IP address or "All Unassigned"?

    -- If it is IIS 6.0, do you see any other process running on SSL port, by running netstat -ano?

    The above pointers will give you the right action plan.

    Feel free to contact me if there are any questions or concerns.

    -Saurabh

  • What about multiple sites on one server and the sites use host headers?  You only want the cert on a subfolder on one web site?  Two sites have IP addresses, one doesn't.  You get the DNS error.

  • Hi Heather,

    Probably my bad, but I didn't get what your question was.

    How about if you contact me directly through Email on Top of the page and we can take it from there.

    Saurabh

  • Great informational how to. This saved me a lot of time and research. I appreciate you taking the time to post this.

  • I don't understand the reasons behind recommending the disabling of Kernel Mode SSL with multiple web sites.  SSL is bound to a web site based on an IP/port combination.  The certificate should be chosen on that basis, and not based on any server-level attributes.  With IIS 7 using Kernel Mode SSL by default it seems to further confirm that this is a good thing to use.

  • thanks so much! I was killing myself trying to figure out why my site wasn't coming up in https requests until I read your firewall comment. Added port 443 and it came right up!

  • Thank you... I simply tried to re-install the cert from verisign and as a result web server 2 didn't have the private key, and would not function.  It's odd as it too the cert, and seemed like all was well but wouldn't server over 443, no errors, nothing.  I went back to web1 and exported the cert with key, did the import, and all was well.  Thank you so much for your post, you really helped me out!

  • If we have a SSL enabled website and a non-SSL website and when we try to access the non-SSL site over SSL port (let’s say 443) it may respond back with 400 Bad Request. Here you will still get a security alert from the other SSL enabled site. Reason being that SSL encryption/decryption occurs at http.sys level in the kernel mode. So the request host header cannot be determined when it reaches the IIS server (still encrypted), and so the request is routed to the SSL enabled site. This site will send the security alert for obvious reasons because of host header mismatch.

    And now since the original request header doesn’t match the host header of the SSL enabled site it will respond back with 400 Bad Request.

    ***Remember, even if we have multiple SSL enabled sites with different certificates, users will still get security alert because of host header mismatch.

    You can make the sites accessible though by providing the proper securebindings entry in the metabase but you will get wrong certificates and hence a security alert.

    If we want to make multiple SSL sites work (even if they have different certificates) without prompting for security alerts, we need to run them on different IP address/port combinations  else if we want to use same the IP/Port we need to use Wildcard certificate on Win2k3 SP1+ environment.

  • I also do not understand your reasoning for stating not to use kernel mode SSL if you have multiple sites on the same server using SSL.  Why not?

    trayharrison@gmail.com

  • I have highlighted the section which talks about disabling Kernel mode SSL. I am not sure why but this had resolved many prior issues for us under the scenario mentioned in the same section. Once I have an answer or findings for/against it i will post it here. If any one has an idea please share it with us.

    thanks.

Page 1 of 3 (39 items) 123