SSL Troubleshooting for IIS Web Sites contd...

SSL Troubleshooting for IIS Web Sites contd...

  • Comments 17

Recently a colleague of mine was working on a customer's case which was a Critical level incident. High pressure job, huhh!

The issue was with SSL not working for one of their web sites. They were seeing "Page cannot be displayed" when trying to access this site over SSL. It worked just fine over HTTP.

In the System event log we were seeing this intermittently:

Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1114
Description:
One of the IP/Port combinations for site 'NNNNN' has already been configured to be used
by another program. The other program's SSL configuration will be used.

We troubleshot on this issue for hours without luck :-(. We tried all the steps I guess as mentioned here .

Here is what all we tried:

  • Checked the Certificate properties to ensure it was a valid one. It was good.
  • Yet, replaced the current certificate with a new one, still no luck.
  • Here customer had all the sites running under different IP addresses. Rest of the other sites were working over SSL, except this one :-(.
  • We ran SSLDiag which gave a misleading error.
  • We tried running the site on a different SSL port, still no luck.
  • We setup the securebindings metabase property for the web site in question, still no luck.
  • We ran netstat -ano to check for any other process listening on this port, everything looked clean. refer this.
  • We disabled all the 3rd party non-MS services, restarted Windows Server in selective startup mode, no luck.
  • We installed Windows Server 2003 Service Pack 1 32-bit Support Tools on the server, ran the httpcfg query iplisten. It gave a clean output, no specific IP entries listed by it.
  • Restarted IIS/HTTP services umpteen number of times during the course of troubleshooting, no luck whatsoever. Even reboot was done a couple of times.

Finally after few hours of troubleshooting we decided to run this site on a different IP address (we had thought of this earlier but our customer was under a constraint) and hurray it worked this time!!!. Now everything was set but we had a lingering question in mind as to why, why, why this site did not work on that IP address we had. It had an entry in the Advanced TCP/IP Settings, was a valid one in all the sense to our best knowledge.

Finally we figured out that there was a problem with the IIS SSL listener.

To get a list of IP and port configuration binded to a certificate, run "httpcfg query ssl". Here is an excerpt from a technet article:

The HTTP API enables applications to communicate over HTTP without using Microsoft Internet Information Services (IIS). Applications can register to receive HTTP requests for particular URLs, receive HTTP requests, and send HTTP responses. The HTTP API includes SSL support so applications can also exchange data over secure HTTP connections without depending on IIS. It is also designed to work with I/O completion ports.....Such meta-information is maintained by the HTTP API in a metastore, and is used to locate certificates for certificate exchange in HTTPS sessions.

Below is a sample of a working and non-working scenario:
------------------------------------------------------------------------------

\Program Files\Support Tools> httpcfg.exe query ssl

Working scenario:

IP                      : 192.168.100.118:443
Hash                  : c96667684997887f 5b889b7b3f737c8c4da5f16
Guid                  : {4dc3e181-e14b-4a21-b022-59fc669b0914}
CertStoreName           : MY
CertCheckMode           : 0
RevocationFreshnessTime : 0
UrlRetrievalTimeout     : 0
SslCtlIdentifier        :
SslCtlStoreName         :
Flags                   : 0

Non-working scenario:

IP                     : 192.168.100.234:443
Hash                :
Guid                : {00000000-0000-0000-0000-000000000000}
CertStoreName : (null)
CertCheckMode : 0
RevocationFreshnessTime : 0
UrlRetrievalTimeout : 0
SslCtlIdentifier : (null)
SslCtlStoreName : (null)
Flags : 0

Here Hash will have the same value as the Thumbprint in your SSL certificate. You will notice that the Guid is all zero in a non-working scenario. You may see the Hash either having some value or blank. Even if we remove the certificate from the web site, and then run "httpcfg query ssl", the site with all Guid as all "0" will still be listed. If you see the GUID as "{0000...............000}, there is a problem.

We need to remove this entry by running the command "httpcfg delete ssl -i <IP:Port Number>". In the above example, we need to type "httpcfg delete ssl -i 192.168.100.234:443". Once we remove it, then we need to reinstall the certificate back on to the web site.

Also once certificate is installed, in the cmd prompt type in "httpcfg query ssl" to confirm the GUID is no longer all 0.

This fixed the issue for the web site on the failing IP address.

Hope this helps someone.

Till next time, Cheers!

 

Leave a Comment
  • Please add 7 and 8 and type the answer here:
  • Post
  • Dude, nice finding, it will really help in future, if at all I come across this type of issue. Will try it at my end.

  • Congrats for the solution, and thanks for posting it!

  • Good going. Its really helpful.

  • You're a life savior!!! Just what I needed!

  • Wow.  I just spent 4 hours trying to figure this out.  You're a lifesaver!

  • Hello,

    Not in IT just curious to how this stuff works. I am missing what appears to be a simple step/concept. Any and all clarification is welcome and appreciated!!

    General subdomain/host header question:

    My Senerio:

    Main site: test.com    HomeDir:  c:\test  IP: x.x.x.x

    subsite: driving.test.com  Site files: c:\test\driving  same IP

    Main site created in IIS (any version i guess)

    sub site setup using host headers: driving.test.com and www.driving.test.com binded to main site IP

    When "http://driving.test.com" is entered in browser how does the default page in the sub directory (c:\test\driving) get configured to load? Do you specify home dir somewhere? Please fill in the blank for me

    Thnx, vin

  • Hi Vin,

    Are you seeing any error message when you go to the above URL?

    You don;t have to configure anything else here. If you have a site with the name driving.test.com, it should take you to the correct web site, provided you have a host header entry to differentiate various sites.

  • Having the same type of problem (we see the 0 guid and want to delete), but the delete command is failing with error code 2. What else could be holding it open?

  • Robert, Error code 2 indicates "Error_File_Not_Found". The combination of IP:PORT that you are trying is it valid on this server? Do you see it as listed in there. Perhaps you can share the complete command stack that you tried.

  • Yes it was valid. We worked around by changing to a different IP address in the pool and it works. For some reason, the system would not give up that address and we do not have complete control over the machine or the network so have to call it in to get it rebooted to see if it will release...

  • We ran across this issue as well with a client site yesterday and I must say this blog entry was my savior.

    @Robert

    When working on the site that had problems I too received the error code 2 the first time I tried to delete it.  I had to remove the existing SSL config in IIS for the affected site, the the httpcfg command to delete worked like a charm.  Not sure if that's going to help you if you have limited access to the box, but that's what fixed it for me.

  • THANK YOU!

    I spent hours Friday trying to figure out why our www server was putting out an SSL certificate FROM ANOTHER MACHINE after I installed our new certificate from an authority.

    I ran through your previous post and everything checked out. A commenter mentioned the event logs... so I found the same error message you posted about here (but I don't have any other services running on that port). So I tried your httpcfg trick. In my case, I had data coming back bound to 0.0.0.0:443, but it was not the thumbprint of the new cert that I installed. So I used your command to delete it and then remove and re-assigned my new cert and BINGO!

    Thanks again very much for both of these posts!

  • I tried your solution, but I am getting HttpDeleteServiceConfiguration completed with 1019, which translates into "System could not allocate the required space in a registry log.” according to windows error messaging.

    The server is a development server, and it only had 200 megs free on the C: drive. It probably ran out of drive space at some point.

    More details are on serverfault:

    http://serverfault.com/questions/86975/problem-deleting-host-ssl-certs-in-iis-with-httpcfg-ssl-delete

  • I went through the link mentioned, please reboot the box once and try again the same steps, i don't think 200 mb is low enough to cause this command to fail.

  • We got a new ip address for the box, so that put this on the back burner. However, today I have noticed the box has been rebooted. I tried the command again and it was successful.

Page 1 of 2 (17 items) 12