Encryption is one of the new SMB 3.0 security enhancements in Windows Server 2012 RTM. It can be enabled on a per-share basis, or enforced for all shares on the server. SMB 3.0 uses AES-CCM [RFC5084] as encryption algorithm, and this also provides data integrity (signing).
This blog takes a protocol walk on the topic through [MS-SMB2], and provides a sample test vector. [MS-SMB2] describes SMB 2.x and SMB 3.0 dialects. Throughout this blog, I use the terms SMB2 and SMB3 interchangeably because both share the core SMB2 Packet format. Details on encryption configuration aspects can be found in the blogs listed in references.
When encryption is active for a given exchange, it is applied before submission to the transport. An exchange could consist of a single request or response, or a series of compounded chain operations. The sender performs encryption as an outer layer after the SMB2 messages in a given submission have been assembled.
Ko = SMB3KDF (Ki, Label, Context)
SMB3KDF() is defined as the KDF algorithm in Counter Mode, as specified in [SP800-108] section 5.1, with 'r' value of 32 and 'L' value of 128, and HMAC-SHA256 as the PRF.
Ki – Key derivation key, used as an input to the KDF. For SMB 3.0, Ki is the SessionKey.
Label – the purpose of this derived key, encoded as string and length for SMB 3.0.
Context – the context information of this derived key, encoded as string and length for SMB 3.0.
L – An integer that specifies the length of the derived keying material Ko, L is 128 bits for SMB 3.0 cryptographic keys. Note that L is a constant since all SMB 3.0 keys are 16 bytes in length (SigningKey, EncryptionKey, DecryptionKey, and ApplicationKey).
Ko – Keying material output from the KDF, a binary string of length L, where Ko is the leftmost L bits of KDF result.
EncryptionKey (Client) = DecryptionKey (Server) = ServerInKey (ClientToServer)
EncryptionKey (Server) = DecryptionKey (Client) = ServerOutKey (ServerToClient)
The server and client generate the encryption keys upon session establishment of the primary channel. If the server is configured for encryption (i.e. SmbServerConfiguration.EncryptData) and the Connection.ClientCapabilities includes the SMB2_GLOBAL_CAP_ENCRYPTION, the server generates the EncryptionKey and DecryptionKey and sets SMB2_SESSION_FLAG_ENCRYPT_DATA flag in the SessionFlags field of the SessionSetup response; the client must also generate its encryption and decryption keys.
Upon successful TreeConnect, if Session.EncryptData is FALSE and Share.EncryptData (e.g. the share is configured to require encryption), the server generates its EncryptionKey and DecryptionKey and sets the SMB2_SHAREFLAG_ENCRYPT_DATA bit in the ShareFlags field of the TreeConnect response; the client must also generate its encryption and decryption keys.
Note: These cryptographic keys are all derived from the SessionKey. As a result, SMB 3.0 signing and encryption is as secure as the session key. Not only must this key be unique and very random, but also it needs be kept secret.
A transformed message consists of a transform_header followed by its encrypted SMB2 message.
The transform_header has the following fields:
ProtocolId (4 bytes): 0xFD, 'S', 'M', and 'B' (in network order).
Signature (16 bytes): Signature of the encrypted message.
Nonce (16 bytes): An implementation-specific value that is unique for every encrypted message within a session.
OriginalMessageSize (4 bytes): The size in bytes of the SMB2 message.
Reserved (2 bytes): Set to zeros and ignored.
EncryptionAlgorithm (2 bytes): the value 0x0001 is for SMB2_ENCRYPTION_AES128_CCM.
SessionId (8 bytes): Uniquely identifies the established session for the command.
With the SMB2_ENCRYPTION_AES128_CCM algorithm, SMB 3.0 encryption implementation uses the leftmost 11 bytes of the Nonce field in the transform_header, let’s call this Aes128Nonce. Per [RFC3610], the nonce passed to AES-CCM can be between 7 and 13 bytes. Windows SMB 3.0 made the trade-off choice of 11 bytes.
Note that it would almost be impossible that all possible nonce values can be consumed on a channel, because it would take over several centuries before a sender could recycle a nonce on a given channel.
The sender builds the transformed message with these specifics. AES-CCM [RFC5084] is called with the following inputs:
- AES key: Session.EncryptionKey.
- Aes-nonce: Aes128Nonce must be used.
- Plaintext: The SMB2 message including the header and the payload.
- The optional authenticated data (AAD): The SMB2 transform_header excluding the ProtocolId and Signature fields; these are the 32 bytes starting from the Nonce field.
The AES-CCM outputs are:
- Ciphertext: the encrypted SMB2 message
- Message authentication code: the Signature field of the transform_header.
The sender appends the encrypted SMB2 message to the transform_header and sends it to the receiver.
The message is decrypted using:
- The EncryptionAlgorithm in the transform_header.
- The Session.DecryptionKey of the Session that corresponds to the SessionId in the transform_header.
- The AAD passed to AES-CCM is the transform_header excluding the ProtocolId and Signature fields.
- The nonce passed to AES-CCM is the Aes128Nonce; that is the first 11 bytes of the Nonce field.
The signature returned by the decryption algorithm is then verified against the Signature in the transform_header.
The sender encrypts the message if any of the following conditions is satisfied:
Note: TreeConnect.EncryptData is on the client side. Share.EncryptData is on the server side.
This means SESSION_SETUP for re-authentication or session binding will be encrypted.
If the client performs TREE_DISCONNECT before a LOGOFF, the logoff will not be encrypted.
It is expected that all operations in an encrypted message belong to the SessionId in the transform_header. The encrypted message is sent as a single submission to the underlying transport, there is no provision for a next transformed message in the transform_header.
In case of related compounded requests, all the requests/responses belong to the same SessionId. The sender indicates to the receiver that it is using the SessionId, TreeId, and FileId of the previous operation or resulted from the receiver processing that previous operation.
From a broader scope, the expected protocol behaviors for sending an encrypted message can be summarized as follows, be it for compounded requests/responses or not.
The sender should not send an encrypted request separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. In other words, the sender should not use a SessionId to convoy the encrypted traffic of other authenticated sessions.
The receiver should not send an encrypted response separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. Windows-based servers will respond in-kind to what the client sends. If the client violates the above statement regarding the SessionId, a Windows 8-based server will currently send the response as such.
Oplock break notifications/acknowledgments/responses must be encrypted when encryption is active. For an Oplock, the FileID is used to derive the SessionId which is set in the notification/acknowledgement/response. See more details in MS-SMB2.
Lease break notifications - sent by the server - do not have a SessionId, and as a result, are neither signed nor encrypted. Lease keys are not tied to a particular session from the client.
However, Lease break acknowledgements sent by the client - and their responses sent by the server - must be encrypted when encryption is active. The client is responsible for selecting a session associated with one of the existing opens associated with that Lease Key. The response is sent on the session that receives the acknowledgment.
This sample data should be considered “as-is”. It should also be noted that examples do not replace normative protocol specifications. The reference must be [MS-SMB2].
The test program running on a Windows 8 client negotiates SMB3 and communicates with a Windows 2012 server. It opens a file and WRITEs the following content. It then READs back the file.
This is the content written and read:
Smb3 encryption testing
These outputs show the encryption and decryption of the WRITE and READ commands.
The decrypted content is verified to be same at the end of the SMB2 READ response.
SessionId 0x8e40014000011SessionKey 0xB4546771B515F766A86735532DD6C4F0SigningKey 0xF773CD23C18FD1E08EE510CADA7CF852EncryptionKey (Client) 0x261B72350558F2E9DCF613070383EDBFDecryptionKey (Client) 0x8FE2B57EC34D2DB5B1A9727F526BBDB5ApplicationKey 0x77432F808CE99156B5BC6A3676D730D1
Header.Command 0x0009 WRITEEncryption of the request ---Key 0x261B72350558F2E9DCF613070383EDBFAES-128-CCM nonce 0x66E69A111892584FB5ED52AAD 0x66E69A111892584FB5ED524A744DA3EE87000000000001001100001400E40800SMB2 packet 0xFE534D4240000100000000000900400008000000000000000400000000000000FFFE0000010000001100001400E4080000000000000000000000000000000000310070001700000000000000000000001501000039000002010000003902000000000000000000007000000000000000536D623320656E6372797074696F6E2074657374696E67transform_header.Signature 0x81A286535415445DAE393921E44FA42Etransform_header.Nonce 0x66E69A111892584FB5ED524A744DA3EEtransform_header.OriginalMessageSize 0x87 transform_header.SessionId 0x8e40014000011 Encrypted message 0x25C8FEE16605A437832D1CD52DA9F4645333482A175FE5384563F45FCDAFAEF38BC62BA4D5C62897996625A44C29BE5658DE2E6117585779E7B59FFD971278D08580D7FA899E410E910EABF5AA1DB43050B33B49182637759AC15D84BFCDF5B6B238993C0F4CF4D6012023F6C627297075D84B7803912D0A9639634453595EF3E33FFE4E7AC2ABTransformed message 0xFD534D4281A286535415445DAE393921E44FA42E66E69A111892584FB5ED524A744DA3EE87000000000001001100001400E4080025C8FEE16605A437832D1CD52DA9F4645333482A175FE5384563F45FCDAFAEF38BC62BA4D5C62897996625A44C29BE5658DE2E6117585779E7B59FFD971278D08580D7FA899E410E910EABF5AA1DB43050B33B49182637759AC15D84BFCDF5B6B238993C0F4CF4D6012023F6C627297075D84B7803912D0A9639634453595EF3E33FFE4E7AC2ABDecryption of the response ---Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5Transformed message 0xFD534D42A6015530A18F6D9AFFE22AFAE8E6648486000000000000001100001400E4080050000000000001001100001400E40800DBF46435C5F14169293CE079E344479BF670227E49873F458672C3098DAC467DD5809F369D674091665157871483E01F7BECD02064EAC3E235F913668BBC2F097980D4B378F1993EFF6E60D177309E5Btransform_header.Signature 0xA6015530A18F6D9AFFE22AFAE8E66484transform_header.Nonce 0x86000000000000001100001400E40800transform_header.OriginalMessageSize 0x50 transform_header.SessionId 0x8e40014000011 AES-128-CCM nonce 0x8600000000000000110000 AAD 0x86000000000000001100001400E4080050000000000001001100001400E40800Decrypted SMB2 packet 0xFE534D4240000100000000000900210009000000000000000400000000000000FFFE0000010000001100001400E408000000000000000000000000000000000011000000170000000000000000000000
Header.Command 0x0008 READEncryption of the request ---Key 0x261B72350558F2E9DCF613070383EDBFAES-128-CCM nonce 0xA5123A25F983E245983F41AAD 0xA5123A25F983E245983F413B8B429AF271000000000001001100001400E40800SMB2 packet 0xFE534D4240000100000000000800400008000000000000000500000000000000FFFE0000010000001100001400E408000000000000000000000000000000000031000000170000000000000000000000150100003900000201000000390200000000000000000000000000000000000000transform_header.Signature 0xE93601498B76D6F7A72D5EF9B6C79FAFtransform_header.Nonce 0xA5123A25F983E245983F413B8B429AF2transform_header.OriginalMessageSize 0x71 transform_header.SessionId 0x8e40014000011 Encrypted message 0x9A464F709AA663F8C2FC3907D63CBF6F98B1E3DD649ED366009FD0B40A365224718E5440E053F6E01AE462FDB721BF91C3A6E52E14F9EFF005F445761289FF1272908B52754C8FCB949F228AC104A66204289A205BCBC47509D04AF9A907002B96863358B3B7CBA5E377930074FCDF3550Transformed message 0xFD534D42E93601498B76D6F7A72D5EF9B6C79FAFA5123A25F983E245983F413B8B429AF271000000000001001100001400E408009A464F709AA663F8C2FC3907D63CBF6F98B1E3DD649ED366009FD0B40A365224718E5440E053F6E01AE462FDB721BF91C3A6E52E14F9EFF005F445761289FF1272908B52754C8FCB949F228AC104A66204289A205BCBC47509D04AF9A907002B96863358B3B7CBA5E377930074FCDF3550Decryption of the response ---Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5Transformed message 0xFD534D42ABD518B68C2F04D7879F482B689EB83F87000000000000001100001400E4080067000000000001001100001400E40800493D6FE2BDBEB435CF5F546970C7BB57BF20E713C75A3D045507E0D68E5C0346659D6FFB8AC1504A786CA2BB89C9E7FE4F313E910A04180D2D0EA7DF636329E5A3285984500EF86FE9D55DA4FAB9531CFDD4C551D47F3C73124BB4590A45052B694048B991CCF5transform_header.Signature 0xABD518B68C2F04D7879F482B689EB83Ftransform_header.Nonce 0x87000000000000001100001400E40800transform_header.OriginalMessageSize 0x67 transform_header.SessionId 0x8e40014000011 AES-128-CCM nonce 0x8700000000000000110000AAD 0x87000000000000001100001400E4080067000000000001001100001400E40800 Decrypted SMB2 packet 0xFE534D4240000100000000000800210009000000000000000500000000000000FFFE0000010000001100001400E408000000000000000000000000000000000011005000170000000000000000000000536D623320656E6372797074696F6E2074657374696E67
[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3 Specification
[SP800-108] National Institute of Standards and Technology. "Special Publication 800-108, Recommendation for Key Derivation Using Pseudorandom Functions", October 2009, http://csrc.nist.gov/publications/nistpubs/800-108/sp800-108.pdf
[RFC5084] Housley, R., "Using AES-CCM and AES-GCM Authenticated Encryption in the Cryptographic Message Syntax (CMS)", RFC 5084, November 2007, http://www.ietf.org/rfc/rfc5084.txt
SMB 3 Security Enhancements in Windows Server 2012
Encryption in SMB3