Some customers do have connection issues between their SAP NetWeaver Java application server and SQL Server. Especially when they are trying to update or upgrade their SAP systems because some issues only occur if you use a more recent JVM patch level.

The SUM logs contain exceptions like

The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: SQL Server returned an incomplete response. The connection has been closed.

or

The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: SQL Server did not return a response. The connection has been closed.

The reason for these issues are mostly related to SSL. We already discussed one issue that can occur when you try to connect to SQL Server using a Microsoft JDBC 1.2 driver in an earlier blog and SQL Server uses a certificate that is stored in the Windows certificate store and which is larger than 4KB.

In this blog post, we want to analyze connection issues that are caused by a misconfiguration of SCHANNEL. The configuration of SCHANNEL is described in Microsoft KB 245030.

Test setup

 

image

 

To be able to trace the communication between the JDBC driver and SQL Server, we execute the Java process on a different host.

The first thing we need to know is which SSL version the client uses and which Ciphers it offers the server. If we trace the SSL handshake, we can see that the client offers the following Ciphers (see MSDN: Cipher Suites in Schannel):

 

Name Encryption
SCHANNEL\Ciphers\
Hash
SCHANNEL\Hashes\
Exchange
SCHANNEL\
KeyExchangeAlgorithms\
Protocol
TLS_RSA_WITH_RC4_128_MD5 RC4 128/128 MD5 PKCS TLS 1.2
TLS 1.1
TLS 1.0
SSL 3.0
SSL_CK_RC4_128_WITH_MD5 RC4 128/128 MD5 PKCS SSL 2.0
TLS_RSA_WITH_RC4_128_SHA RC4 128/128 SHA PKCS TLS 1.2
TLS 1.1
TLS 1.0
SSL 3.0
TLS_RSA_WITH_AES_128_CBC_SHA AES 128/128 SHA PKCS TLS 1.2
TLS 1.1
TLS 1.0
TLS_DHE_RSA_WITH_AES_128_CBC_SHA        
TLS_DHE_DSS_WITH_AES_128_CBC_SHA AES 128/128 SHA Diffie-Hellman TLS 1.2
TLS 1.1
TLS 1.0
TLS_RSA_WITH_3DES_EDE_CBC_SHA Triple DES 168/168 SHA PKCS TLS 1.2
TLS 1.1
TLS 1.0
SSL_CK_DES_192_EDE3_CBC_WITH_MD5       SSL 2.0
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA       TLS
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA Triple DES 168/168 SHA Diffie-Hellman TLS 1.2
TLS 1.1
TLS 1.0
SSL 3.0
TLS_RSA_WITH_DES_CBC_SHA        
SSL_CK_DES_64_CBC_WITH_MD5        
TLS_DHE_RSA_WITH_DES_CBC_SHA        
TLS_DHE_DSS_WITH_DES_CBC_SHA        
TLS_RSA_EXPORT_WITH_RC4_40_MD5        
SSL_CK_RC4_128_EXPORT40_WITH_MD5        
TLS_RSA_EXPORT_WITH_DES40_CBC_SHA        
TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA        
TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA        

The server then chooses which Cipher it want to use. From the trace we can see that the SQL Server chooses TLS_RSA_WITH_AES_128_CBC_SHA. The server can only choose this cipher, if the following SCHANNEL configuration in the registry is configured like (or the keys do not exist):

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Ciphers\AES 128/128]
"enabled"=dword:0xffffffff

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Hashes\SHA]
"enabled"=dword:0xffffffff

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\PKCS]
"enabled"=dword:0xffffffff

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]
"enabled"=dword:0xffffffff

If you change the configuration for SCHANNEL, each key may disable or enable multiple ciphers – see http://support.microsoft.com/kb/245030/en-us

If we disable TLS by setting

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]
"enabled"=dword:0x00000000

all TLS related ciphers cannot be used by SQL Server and it will choose a different cipher. In our case this would be SSL_RSA_WITH_RC4_128_SHA but since this cipher is not supported by the client, the JDBC connection fails with

The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: SQL Server returned an incomplete response. The connection has been closed.

Let’s disable something different. If we enable TLS again by deleting the TLS 1.0 key and disabling the RSA key exchange by adding the following registry key:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\PKCS]
"enabled"=dword:0x00000000

we get the following error:

The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: SQL Server did not return a response. The connection has been closed.

Both errors create an error in the Windows Server system event log.

image

Conclusion

If you encounter connection issues from your Java application and SQL Server that are related to SSL, first check the Windows Server system event log if it contains SCHANNEL errors or warnings. If you see such entries you can test if SCHANNEL is misconfigured by first exporting the current registry key and then delete the complete

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL]

key. Restart the computer and check if the JDBC connection works. If the connection works, ask your IT department, if they set these keys via group policy or if they use tools that configure SCHANNEL.