When connecting to an SSL-enabled web site with Internet Explorer, the client and server must negotiate an SSL session. The server sends its list of trusted root certificates to the client in the form of a non-encrypted record. The server requires that the client have a digital certificate for authentication, the client is able to select one that corresponds to a root certificate trusted by the server.
The problem appears when the root certificate list sent by the server exceeds the 16k size limit. This is defined in RFC 2246 (http://www.ietf.org/rfc/rfc2246.txt). When the list exceeds the 16k limit, the server sends several packets. Now, Internet Explorer relies its security to the operating system via Schannel. Schannel on Windows XP / Server 2003 and Windows Vista / Server 2008 is not able to process those packets by default. So when Internet Explorer tries to connect to the SSL-enabled web site, the connection fails.
This issue won't happen on Windows 7 / Server 2008 R2, as they support record fragmentation in Schannel. They can manage the rest of the packets sent by the web server with the rest of the trusted root certificate list.
SSL/TLS Record Fragmentation Support
2215054 SChannel does not support SSL Fragmentation
2219505 Internet Explorer was failing to connect to a SSL/TLS enabled web site, Schannel issue
Fortunatelly we recently released an update to overcome this limitation for older versions of the OS:
An update that enables Internet Explorer in Windows XP, in Windows Vista, or in Windows Server 2008 to parse fragmented TLS/SSL handshake messages is available
Note this update should be available as a High Priority update in Windows Update, too.
Alex (Alejandro Campos Magencio)
And for Windows 2003?
For Windows 2003 there is no fix, and as far as I know, there will never be because this OS version is in Extended Support phase.