Microsoft Open Specifications Support Team Blog

RDPESC parser modification

Sebastian Canevari — Thu, 30 May 2013 19:30:00 GMT

Hello world!

I’ve decided to write this entry to talk about two
intertwined subjects:

- The published RDPESC parser needs a little tweak
in order to function properly

- That tweak is a real life example of how to
modify an existing Netmon Parser

My goal is not to rewrite the [MS-RDPESC]
document in this publication so I’ll be assuming that you are familiar with it
and will touch the protocol details just tangentially.

Introduction

I’ve recently worked on a case where a customer was
observing that Netmon 3.4
was not being able to decode some of the RDPESC packets correctly. The parser
was only going as far as the TLS layer and was taking the rest of the data as a
simple Blob.

This is pretty much what it looked like:

Easy case

The first thing that came to my mind was obviously that the
parser version the customer was using was stale so, I provided the link to the
latest and greatest version of Netmon parsers: 03.04.2978.0001.

Hold your horses!

Although customer was happy with the way the newer version
of the parsers was handling the RDPESC structures… there was something not
working…

This is what the same frame looked like with the plain
vanilla new parsers:

As we can see, this is day and night compared with not
having anything parsed below TLS.

But… and there’s always a “but”, look at the RDPESC packet
itself. There’s something wrong…

This is what other RDPESC packets look like (other control
codes that is):

Don’t lose your headER!!!

Different calls and returns contain different data (duh!)… Even
different headers sometimes! (hmmm)

But, shouldn’t packets ALWAYS have a header?!?!

(What? What red pill? Oh…. Ok… lots of water? Ok, here it
goes…)

Sorry, my friend Morpheus was talking to me.

As I was saying, even though there might be protocols that
do not have a properly design header structure, RDPESC is not one of those and
its packets do indeed possess a header.

So, what could possibly go wrong with the parser that it is
treating one return call differently from another return call?

Welcome to "The Parserix"…

If we right click on the RDPESC call return and
select “Go To Data Type Definition”, the RDPESC parser file opens up and the
first thing we can see is this:

At first sight, having a “switch” clause is a good sign. It
wouldn’t be the first time someone misses a case in a switch clause… would it?

Although the above mentioned is a very common scenario, it
was not the EXACT case this time around.

Because of the way the RDPESC parser works, we need to look
at the initial call’s control code in order to figure out which of the
scenarios it is that we are dealing with.

I present you with the control code for SCARD_IOCTL_ACCESSSTARTEDEVENT:

Once the control code has been identified, we can then see
that 0x900E0 is a special case on the RDPESC parser world:

All fine and dandy but, what’s wrong?

Well, basically the document specifies in section 3.1.4
that:

“6. Otherwise,
DR_DEVICE_IOCOMPLETION.IOStatus MUST be set to 0 (STATUS_SUCCESS) and
DR_DEVICE_IOCOMPLETION.Parameters.DeviceIOControl.OutputBuffer MUST contain an
encoding of the structure (as specified in the preceding Message Processing
Events and Sequencing Rules IOCTL Table) as specified in [MS-RPCE] section 2.2.6. DR_DEVICE_IOCOMPLETION.Parameters.DeviceIOControl.OutputBufferLength
is the length of the data.”

That means that the return call is to be parsed WITH an
MSRPCHeader.

So, we need to modify the parser.

MUST… MODIFY… PARSER…

So in order to make this change to the parser, we can follow
either of the two main ways we know we can use to modify parsers:

- We can either add a personalized copy of the
parser to some “myparsers” folder and then set that folder as the first option
in the list of precedence in the current profile OR

- We can replace the existing RDPESC parser from
the Windows profile in Network Monitor with an updated version of it.

In this case, we’ll use the second approach.

These are the steps:

1) Navigate to “C:\ProgramData\Microsoft\Network
Monitor 3\NPL\NetworkMonitor Parsers\Windows”

2) Save a copy of rdpesc.npl to a safe location

3) Open the original file with a text editor

4) Replace:

// SmartCardCall W/O MSRPCHeader

case 0x000900E0:

switch(property.RDPEFSPacketId)

{

case 0x4952:

UINT32 Unused;

case 0x4943:

RDPESCLongReturn LongReturn;

}

5) With:

//

// SmartCardCall (Special case, see below)

case 0x000900E0:

switch(property.RDPEFSPacketId)

{

case 0x4952:

//The "Call" does NOT have a RPCHeader

UINT32 Unused;

case 0x4943:

// Whereas, the "Return" does (so, handle the same way as all the other
IOCTLs)

_struct RDPESCSmartCardCallWithMSRPCHeader

{

RPCECommonTypeHeader CommonTypeHeader;

RPCEPrivateTypeHeader PrivateTypeHeader;

RDPESCLongReturn LongReturn;

}

6) Save the file back to its original place

Once the modification has been made, reopen the
capture in Network Monitor and look at the results. The return call should now
look like this:

Conclusion

“It is done!”

PS: I was going to be more wordy for the conclusion of this entry but I figured out that
the post was pretty much self-explanatory.

BTW, if you want to learn some more details regarding parser modifications, you
can take a peek at my co-worker’s entry from some time ago: http://blogs.msdn.com/b/openspecification/archive/2011/08/08/customizing-in-box-netmon-parsers-how-to-edit-and-deploy-updated-netmon-parsers.aspx

PowerShell script for finding Microsoft Office legacy files

Vilmos Foltenyi MSFT — Mon, 08 Apr 2013 21:08:00 GMT

Referenced documents:
[MS-CFB]: Compound File Binary File Format
[MS-OLEPS]: Object Linking and Embedding (OLE) Property Set Data Structures
Windows PowerShell Cookbook, 3^rd edition, by Lee Holmes

NOTE: Questions and comments are welcome. However, please DO NOT post a comment using the comment tool at the end of this post. Instead, post a new thread in the Open Specifications Forum: Office File Formats at
http://social.msdn.microsoft.com/Forums/en-US/os_binaryfile/threads

#########################
# WHAT THE SCRIPT DOES
#########################

This blog is complementary to the blog “Determining Office Binary File Format Types”, by JCurry (Josh). That blog describes in details how to find PIDSI_APPNAME, i.e. the name of the application, which created the file. For the Office legacy files it can be: “Microsoft Office Word”, “Microsoft Excel”, or “Microsoft Office PowerPoint”. The PowerShell (PS) script in this blog prints more information, some information from the header, and the properties from the “Summary Information sector”. This looks adequate to meet the requests of recent cases on the Open Specifications Forum / Office File Formats, but, of course, more annotation can be added if there is demand for it.

The OfficeLegacyFilter.ps1 PS script, see attached file, starts with a comment block, which contains the disclaimer and the name, version, and usage of the script. The script has one parameter, which can be a directory or file name. If the parameter is a directory, then all the files in this directory and in the subdirectories are recursively checked.

All checks are made on the content of the file; the file name extension (sometime called file type) is not used. If a check fails the file is just skipped. Which means if no Office Legacy file is in scope, the script returns nothing. This makes easier to use the script in pipe.

The checks start with the Header Signature, Minor and Major versions, etc. until the First Directory Sector Location is reached. Currently we are interested only in one directory sector, which has the name “Summary Information”. At offset 0x74 of that directory sector is the Starting Sector Location, see [MS-CFB] v20130118 / 2.6.1 Compound File Directory Entry, from that value we can navigate to [MS-OLEPS] — v20130118 / 2.21 PropertySetStream and to 2.20 PropertySet, where you’ll find NumProperties; all the properties with their values are printed out. All these steps can be easily followed in the script.

#########################
# HOW TO RUN THE SCRIPT
#########################

If your machine is not set up for running a PowerShell script, the OfficeLegacyFilter.ps1 script will not run, because scripting support is disabled by default. To see your current execution policy setting run the:

Get-ExecutionPolicy

cmdlet. If you have some kind of restriction, it can be lifted by running the

Set-ExecutionPolicy Unrestricted

cmdlet. If you try to run the script now, probably you will be prompted for permission and the script will run.

If you want to run the script in a more secure way, set the execution policy to the desired level. If you already have a code-signing certificate, you can use it. To check whether you have a code-signing certificate do the next:
You can go to the cert: driver and look around the certificates

cd cert: # Go to the cert driver and look around

You can find all code-signing certificates

dir cert: –Recurse –CodeSigningCert

If you have at least one, use, e.g., the first one

$cert = @(dir cert: -Recurse –CodeSigningCert)[0]

Set-AuthenticodeSignature <ps1 file> $cert

At the end of the <ps1 file> you should see a comment block with the next structure:

# SIG # Begin signature block
# <64 base64 digits>
.......................
# <64 base64 digits>
# <base64 digits>
# SIG # End signature block

and running the script should not be a problem.

If you don’t have code-signing certificate, then you can make one. The steps how to create a self-signed certificate are described in many places. First you need the utility for making certificate, makecert.exe. You can read about it here:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa386968(v=vs.85).aspx

First you have to generate a local certificate authority and after to create a self-signed certificate by using the local certification authority.
You can read about this, e.g., at
Program: Create a Self-Signed Certificate
Holmes, Lee (2012-12-21). Windows PowerShell Cookbook: The Complete Guide to Scripting Microsoft's Command Shell (p. 522). O'Reilly Media. Kindle Edition.

You should be able to see the properties of the newly created signing certificate by running:

dir cert: -Recurse –CodeSigningCert | Format-List *

#########################
# EXAMPLE
#########################

After creating two of the three test files I changed their extension to txt. It can be anything, it is not used in the script.

PS C:\Projects\PS\scripts> C:\Projects\PS\scripts\OfficeLegacyFilter.ps1 'C:\Test\a'

C:\Test\a\x\Autonumbering.txt
    000000 Header Signature   . . . . : D0 CF 11 E0 A1 B1 1A E1
    000018 MinorVersion . . . . . . : 62
    00001A MajorVersion . . . . . . : 3
    009E94 01 CODEPAGE_PROPERTY_IDENTIFIER: 1252
    009E9C 02 PIDSI_TITLE . . . . . : PowerPoint Presentation
    009EBC 04 PIDSI_AUTHOR . . . . . : Vilmos Foltenyi
    009EBC 0A PIDSI_EDITTIME . . . . : 00:02:53.4790000
    009ED4 08 PIDSI_LASTAUTHOR   . . . : Vilmos Foltenyi
    009EEC 09 PIDSI_REVNUMBER . . . . : 1
    009EF8 12 PIDSI_APPNAME   . . . . : Microsoft Office PowerPoint
    009F28 0C PIDSI_CREATE_DTM   . . . : Tuesday, 12/20/2011 3:33:05 PM
    009F34 0D PIDSI_LASTSAVE_DTM . . . : Thursday, 1/24/2013 12:05:04 AM
    009F40 0F PIDSI_WORDCOUNT . . . . : 3
    009F48 11 PIDSI_THUMBNAIL size format : 57736 FFFFFFFF

C:\Test\a\x\glow test.txt
    000000 Header Signature   . . . . : D0 CF 11 E0 A1 B1 1A E1
    000018 MinorVersion . . . . . . : 62
    00001A MajorVersion . . . . . . : 3
    0118A4 01 CODEPAGE_PROPERTY_IDENTIFIER: 1252
    0118AC 04 PIDSI_AUTHOR . . . . . : Tim
    0118AC 0A PIDSI_EDITTIME . . . . : 00:00:00
    0118B8 07 PIDSI_TEMPLATE . . . . : Normal.dotm
    0118CC 08 PIDSI_LASTAUTHOR   . . . : Tim
    0118D8 09 PIDSI_REVNUMBER . . . . : 2
    0118E4 12 PIDSI_APPNAME   . . . . : Microsoft Office Word
    011910 0C PIDSI_CREATE_DTM   . . . : Wednesday, 1/16/2013 8:33:00 AM
    01191C 0D PIDSI_LASTSAVE_DTM . . . : Wednesday, 1/16/2013 8:33:00 AM
    011928 0E PIDSI_PAGECOUNT . . . . : 1
    011930 0F PIDSI_WORDCOUNT . . . . : 24
    011938 10 PIDSI_CHARCOUNT . . . . : 137
    011940 13 PIDSI_DOC_SECURITY . . . : 0

C:\Test\a\y\Acronyms.xls
    000000 Header Signature   . . . . : D0 CF 11 E0 A1 B1 1A E1
    000018 MinorVersion . . . . . . : 62
    00001A MajorVersion . . . . . . : 3
    009474 01 CODEPAGE_PROPERTY_IDENTIFIER: 1252
    00947C 04 PIDSI_AUTHOR . . . . . :
    009488 08 PIDSI_LASTAUTHOR   . . . :
    009494 12 PIDSI_APPNAME   . . . . : Microsoft Excel
    0094AC 0C PIDSI_CREATE_DTM   . . . : Friday, 9/15/2006 5:00:00 PM
    0094B8 0D PIDSI_LASTSAVE_DTM . . . : Thursday, 2/14/2013 11:49:09 AM
    0094C4 13 PIDSI_DOC_SECURITY . . . : 0

SMB 2.x and SMB 3.0 Timeouts in Windows

Edgar A Olougouna — Wed, 27 Mar 2013 20:20:00 GMT

This blog talks about common timeouts for SMB dialects 2.x and 3.0 [MS-SMB2] in Windows. It also covers continuous availability timeout, witness keep alive [MS-SWN], and some SMB-Direct timers [MS-SMBD]. The behaviors are generally version-specific and therefore may change in future Windows releases or fixes.

A previous blog discusses “CIFS and SMB Timeouts in Windows”:
http://blogs.msdn.com/b/openspecification/archive/2013/03/19/cifs-and-smb-timeouts-in-windows.aspx

NOTE: For questions on MS-SMB2, MS-SWN, MS-SMBD documents, please post in the Open Specifications Forum: Windows Protocols at http://social.msdn.microsoft.com/Forums/en-US/os_windowsprotocols.

Given a SMB2 file sharing scenario, these are frequent troubleshooting questions on timeouts:
- What timeouts are involved?
- What are the related Windows behaviors?
- What timers are configurable and what are their settings in Windows?
Just as a refresher, the following are the Windows SKUs where SMB dialects 2.x and 3.0 were introduced.
Dialect 2.002, Windows Vista and Windows Server 2008.
Dialect 2.1, Windows 7 and Windows Server 2008 R2.
Dialect 3.0, Windows 8 and Windows Server 2012.
All these SMB 2.x and 3.0 dialects share the same core SMB2 Packet format [MS-SMB2].

Request Expiration Timer [MS-SMB2]

This is the amount of time the client waits for the server to respond to an outstanding request. This timeout value can be adjusted through the following registry setting:
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
Value type: Dword
Value name: SessTimeout
Default: 60 seconds (Windows Vista)

When the client does not receive the response to a request before the Request Expiration Timer expires, it will reset the connection because the operation is considered blocked.

In Windows 8, the request expiration timer for the SMB 2 Negotiate is set to a smaller value, typically under 20 seconds, so that if a node of a continuously available (CA) cluster server is not responding, the SMB 3.0 client can expedite failover to the other node.

If a request is being processed asynchronously, i.e. the server sends an interim response with STATUS_PENDING, Windows clients extend the time-out as follows:

• If the asynchronous operation is SMB2 Directory Change Notification, the client will not enforce a timeout.
• Otherwise, if the client is running at least Windows 7 and ExtendedSessTimeout is configured, the timeout is extended to the value of ExtendedSessTimeout:
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
Value type: Dword
Value name: ExtendedSessTimeout
• Otherwise, if the client is running at least Windows 7 and ExtendedSessTimeout is not configured, the timeout is extended to four times the value of SessTimeout (4 * SessTimeout). By default, ExtendedSessTimeout is not configured.

For example, it is typical that an asynchronous write operation expires if a backend Windows 2008 R2-based storage server is taking over 4 minutes (4 * 60 sec default SessTimeout plus the scanning time to detect that the request expired) to complete the operation. Increasing SessTimeout would effectively extend the time for asynchronous operations.

The client does not enforce this timer for the following commands:
• Named Pipe Read
• Named Pipe Write
• Asynchronous Directory Change Notifications
• Blocking byte range lock requests
• FSCTLs: FSCTL_PIPE_PEEK, FSCTL_PIPE_TRANSCEIVE, FSCTL_PIPE_WAIT

Note that SessTimeout and ExtendedSessTimeout also apply to Windows-based CIFS/SMB as well, see previous blog. However, the use of ExtendedSessTimeout in SMB is controlled by client configuration of ServersWithExtendedSessTimeout rather a server response.

Session Expiration Timer [MS-SMB2]

This timer is used as a frequency to scan and mark sessions as expired when their specific expiration time is reached. This timer value is 45 seconds in Windows-based servers.
If a session is in expired state and a request is received, the server should return STATUS_NETWORK_SESSION_EXPIRED and the client must re-authenticate. However, while a session is in expired state, the server processes requests in the following cases:
- LOGOFF, CLOSE, and LOCK (unlock) which would allow to gracefully teardown.
- SESSION_SETUP for re-authentication.
- Windows releases prior to Windows 8 do not fail a signed request, i.e. the SMB2 header has SMB2_FLAGS_SIGNED set in the Flags field, and the request is not an SMB2 LOCK.
Authentication-specific expiration is driven by the authentication package. See previous blog on “CIFS and SMB Timeouts in Windows” for more details. Session.ExpirationTime is set to the value returned by SSPI AcceptSecurityContext.
Note that for a given connection object, if the SessionTable remains empty between two cycles of session expiration timer, Windows-based servers will scavenge and disconnect the connection.

Resilient Open Scavenger Timer [MS-SMB2]
This feature was introduced with SMB 2.1 in Windows 7.
This timer is started when the transport connection associated with a resilient handle is lost. It controls the amount of time the server keeps a resilient handle active after the transport connection to the client is lost.
A resilient handle/open is meant to survive temporary transport network disruption. If the client re-establishes connection in a reasonable time after the connection was lost, the client can reconnect to the handle. A client marks a handle resilient via SMB2 IOCTL with CtlCode FSCTL_LMR_REQUEST_RESILIENCY. Note that Windows does not check the negotiated dialect when processing this FSCTL.
The Open.ResiliencyTimeout is set as follows:
- Either a non-zero value is supplied in the Timeout field of the NETWORK_RESILIENCY_REQUEST request. If the requested timeout is greater than MaxResiliencyTimeout, the server returns STATUS_INVALID_PARAMETER.
- Otherwise, an implementation-specific value is used for resiliency timeout. Windows 7 and Windows Server 2008 R2 servers keep the resilient handle open indefinitely when the Timeout value (requested in NETWORK_RESILIENCY_REQUEST) is equal to zero. Windows 8 and Windows Server 2012 set a default value of 120 seconds.
The MaxResiliencyTimeout value can be configured through:
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\
Value type: Dword
Value name: ResilientTimeout
Default: 300 seconds (Windows 7, Server 2008 R2, 8, Server 2012)

Durable Open Scavenger Timer [MS-SMB2]
This feature was introduced with SMB 2.1 in Windows 7.
This timer is started when the transport connection associated with a durable handle is lost. It controls the amount of time the server keeps a durable handle active after the transport connection to the client is lost.
A durable handle/open allows the client to attempt to preserve and reestablish a file handle after a network disconnection. A client requests an open to be durable through one of the create contexts SMB2_CREATE_DURABLE_HANDLE_REQUEST or SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2 (v2 requires SMB 3.0 dialect).
The durability timeout is set as follows:
- For SMB2_CREATE_DURABLE_HANDLE_REQUEST, Windows 7 and Windows 2008 R2 set this timeout to 16 minutes, Windows 8 and Windows Server 2012 set the value to 2 minutes.
- For SMB2_CREATE_DURABLE_HANDLE_REQUEST_V2, the timeout is set in the following order:
a) A non-zero value is supplied in the Timeout field of the durable v2 create context request.
b) A non-zero value is configured on the share’s CATimeout property.
c) the server’s implementation specific value; Windows-based servers use the value of the registry setting:
     \HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\
     Value type: Dword
     Value name: DurableHandleV2TimeoutInSecond
     Default:    60 seconds (Windows 8, Windows Server 2012)
     Maximum: 300 seconds

Continuous Availability Timeout
This feature was introduced with SMB 3.0 in Windows 8.
With SMB 3.0, each share has a CATimeout property which defines the minimum time the server should hold a persistent handle on a continuously available share before closing the handle if it is un-reclaimed. By default, Windows 8 and Windows Server 2012 set CATimeout to zero.
CATimeout can be set or retrieved using PowerShell command Set-SmbShare or Get-SmbShare.
Each share’s CATimeout needs to be configured to a sensible value to enable the SMB 3 client to perform transparent file handle recovery during server failovers.
In the event of server failover, the persistent handle may have timed out before the client reconnects to the clustered server and attempts to reclaim the handle. If that occurs, the client may replay an outstanding Read, Write, or IOCTL operation by using a stale handle which no longer exists on the server side.
Ideally, if a persistent handle times out, the client should abandon the outstanding operation and return an error to the application.

Witness keep-alive interval [MS-SWN]
This functionality was introduced for SMB 3.0 in Windows 8.
The witness protocol is used to explicitly notify a client of resource changes that have occurred on a highly available cluster server. This enables faster recovery from unplanned failures, so that the client does not need to wait for TCP timeouts.
The server advertises the support of witness protocol monitoring through the SMB2 TREE_CONNECT response capability flag SMB2_SHARE_CAP_CLUSTER. The client instructs its witness client to register for asynchronous notifications for desired resources on the cluster node it is not connected to. The witness (server) service listens and reports cluster events related to the clustered file server that the client is connected to.
When the client registers (i.e. WintnessrRegister), the server assigns a registration key – a unique UID – that is used for subsequent requests on that context handle. A normal client shutdown (e.g. LanmanWorkstation) would trigger WintnessrUnregister and clear the associated state information on both sides.
However, if the client crashes or gets disconnected, the witness service gets notified by RPC runtime for the disconnection. The witness service uses a default RPC keep-alive interval that can be configured via the following registry setting:
\HKLM\SYSTEM\CurrentControlSet\Services\SMBWitness\Parameters\
Value type: Dword
Value name: KeepAliveInterval
Default: 20 minutes (Windows 8, Windows Server 2012)
Upon receipt of disconnection notification, the witness service will implicitly unregister the client.
When the client comes back online after it crashed, it will register again since it has lost its state information.
If the client simply lost the connection, and reconnected before the server noticed, the client cancels any outstanding WitnessrAsyncNotify just in case RPC runtime is still holding its state and then re-issues a new RPC call.

SMB-Direct timers [MS-SMBD]

SMB-Direct is a new transport supported in Windows 8. It is designed to carry SMB2 over Remote Direct Memory Access (RDMA) Transport Protocol.

Negotiation Timer
This timer is per-connection. It controls the amount of time to:
- Establish a connection and complete negotiation. ConnectTimeoutInMs is the deadline for the remote peer to accept the connection request and complete SMB Direct negotiation.
- Accept negotiation: The SMB Direct Negotiate request should be received before AcceptTimeoutInMs expires. The servers starts this timer as soon as it accepted the connection.
\HKLM\System\CurrentControlSet\Services\SmbDirect\Parameters
Value type: Dword
Value name: ConnectTimeoutInMs
Default: 120 seconds (Windows 8)
Value type: Dword
Value name: AcceptTimeoutInMs
Default: 5 seconds (Windows 8)

Idle Connection Timer
This timer is per-connection. It is the amount of time the connection can be idle without receiving a message from the remote peer. Before the local peer terminates the connection, it sends a keep alive request to the remote peer and applies a keep alive timer.
\HKLM\System\CurrentControlSet\Services\SmbDirect\Parameters
Value type: Dword
Value name: IdleConnectionTimeoutInMs
Default: 120 seconds (Windows 8)

Keep alive interval
This attribute is per-connection. It defines the timeout to wait for the peer response for a keep-alive message on an idle RDMA connection.
\HKLM\System\CurrentControlSet\Services\SmbDirect\Parameters
Value type: Dword
Value name: KeepaliveResponseTimeoutInMs
Default: 5 seconds (Windows 8)

Send Credit Grant Timer
This timer is per-connection. It regulates the amount of time that the local peer waits for the remote peer to grant Send credits before disconnecting the connection. This timer is started when the local peer runs out of Send credits.
\HKLM\System\CurrentControlSet\Services\SmbDirect\Parameters
Value type: Dword
Value name: CreditGrantTimeoutInMs
Default: 5 seconds (Windows 8)

If any of these SMB-Direct timers expires, the local peer terminates the connection, then signals the connection loss to the RDMA provider.

References
[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3
http://msdn.microsoft.com/en-us/library/cc246482.aspx
[MS-SWN]: Service Witness Protocol
http://msdn.microsoft.com/en-us/library/hh536748.aspx
[MS-SMBD]: SMB2 Remote Direct Memory Access (RDMA) Transport Protocol
http://msdn.microsoft.com/en-us/library/hh536346.aspx

NTLM and Channel Binding Hash (aka Extended Protection for Authentication)

Obaid Farooqi — Tue, 26 Mar 2013 23:25:00 GMT

Extended Protection for Authnetication (EPA) was introduced in Windows 7/WS2008R2 to thwart reflection attacks. This blog describes the changes in the implementation of NTLM Authentication that are needed to successfully authenticate to servers that have EPA enabled. Windows 7/WS 2008R2 and Windows 8/ WS2012 have EPA enabled out of the box.

You can read the details about EPA here http://technet.microsoft.com/en-us/security/advisory/973811

The concept in EPA is that authentication packets should be bound to the secure channel on which they are transmitted. This concept is not new and is known as channel binding (RFC 5056). RFC 5929 describes channel bindings for TLS that Winodws uses to bind the secure channel to authentication. Please note that EPA also uses Service Pricipal Name (SPN) but it is not used for TLS and we will not discuss it here.

Let's take an example of HTTPS, a protocol that uses TLS. Once a secure channel is established and cipher change has happened, HTTP traffic starts flowing. In this example, we are only considering services that require authentication. NTLM or Kerberos will be used if you are using Windows authentication. You are most likely to use NTLM since the whole point of using HTTP and TLS is to allow clients to connect over internet (In Windows 8, Kerberos can be used on the internet but we will concentrate on NTLM here). In case of Windows client, these are the steps that are taken to incorporate channel binding in the authentication process after secure channel has been established:

1. The hashing algorithm for the signature in the certificate is identified, if present.

2. SSPI calculates a hash (almost always SHA256 hash, see below for details/exceptions) of the certificate, appends other data relevent to the type of channel bindings and returns it to the application.

3. The application, at the receipt of channel bindings, calls Initialize security context (ISC) and passes channel bindings as a parameter to the method call.

4. SSPI calculates the MD5 hash of channel bindings and uses it in the calculation of NTLM version 2 response.

5. When server recives authenticate message, it queries SSPI for channel bindings. SSPI does exactly the same thing as on the client side and returns the data to the service. The service includes it in the call to method Accept Security Context (ASC)

6. In the process of verifying authenticate message, SSPI also takes into account the channel bindings. It calculates the MD5 hash of the channel bindings that were provided by the application (service) and compares it to the one sent by the client. If they match and rest of the authentication requirements are met, authentication is successful.

I'll now elaborate on each of the step listed above with a concrete example of RPC-over-HTTP traffic. The TLS network traffic is encrypted and I used Network Monitor expert Network Monitor Decryption Expert (NmDecrypt) to decrypt it. The decrypted network trace is attached to this blog.

If you open the network trace in Network Monitor, you'll see that in frame 16 server sends a certificate to client, as below (copied and pasted from the trace):

00000000`095c13e0 30 82 02 09 30 82 01 76-a0 03 02 01 02 02 10 cb 0...0..v........

00000000`095c13f0 69 79 cd 51 75 c5 b7 4b-67 30 83 6c 78 44 27 30 iy.Qu..Kg0.lxD'0

00000000`095c1400 09 06 05 2b 0e 03 02 1d-05 00 30 16 31 14 30 12 ...+......0.1.0.

00000000`095c1410 06 03 55 04 03 13 0b 44-43 2d 57 53 32 30 30 38 ..U....DC-WS2008

00000000`095c1420 52 32 30 1e 17 0d 31 32-31 31 31 37 30 30 35 39 R20...1211170059

00000000`095c1430 32 31 5a 17 0d 33 39 31-32 33 31 32 33 35 39 35 21Z..39123123595

00000000`095c1440 39 5a 30 16 31 14 30 12-06 03 55 04 03 13 0b 44 9Z0.1.0...U....D

00000000`095c1450 43 2d 57 53 32 30 30 38-52 32 30 81 9f 30 0d 06 C-WS2008R20..0..

00000000`095c1460 09 2a 86 48 86 f7 0d 01-01 01 05 00 03 81 8d 00 .*.H............

00000000`095c1470 30 81 89 02 81 81 00 9b-00 f8 1a 2d 37 c6 8d a1 0..........-7...

00000000`095c1480 39 91 46 f3 6a 1b f9 60-6c b3 6c a0 ac ed 85 e0 9.F.j..`l.l.....

00000000`095c1490 3f dc 92 86 36 bd 64 bf-36 51 db 57 3a 8a 82 6b ?...6.d.6Q.W:..k

00000000`095c14a0 d8 94 17 7b d3 91 11 98-ef 19 06 52 30 03 73 67 ...{.......R0.sg

00000000`095c14b0 c8 ed 8e fa 0b 3d 4c c9-10 63 9f cf b4 cf 39 d8 .....=L..c....9.

00000000`095c14c0 fe 99 eb 5b 11 f2 fc fa-86 24 d9 ff d9 19 f5 69 ...[.....$.....i

00000000`095c14d0 b4 df 5a 5a c4 94 b4 b0-07 25 97 13 ad 7e 38 14 ..ZZ.....%...~8.

00000000`095c14e0 fb d6 33 65 6f e6 f7 48-4b 2d b3 51 2e 6d c7 ea ..3eo..HK-.Q.m..

00000000`095c14f0 11 76 9a 2b f0 00 4d 02-03 01 00 01 a3 60 30 5e .v.+..M......`0^

00000000`095c1500 30 13 06 03 55 1d 25 04-0c 30 0a 06 08 2b 06 01 0...U.%..0...+..

00000000`095c1510 05 05 07 03 01 30 47 06-03 55 1d 01 04 40 30 3e .....0G..U...@0>

00000000`095c1520 80 10 eb 65 26 03 95 4b-d6 c0 54 75 78 7c b6 2a ...e&..K..Tux|.*

00000000`095c1530 a1 bb a1 18 30 16 31 14-30 12 06 03 55 04 03 13 ....0.1.0...U...

00000000`095c1540 0b 44 43 2d 57 53 32 30-30 38 52 32 82 10 cb 69 .DC-WS2008R2...i

00000000`095c1550 79 cd 51 75 c5 b7 4b 67-30 83 6c 78 44 27 30 09 y.Qu..Kg0.lxD'0.

00000000`095c1560 06 05 2b 0e 03 02 1d 05-00 03 81 81 00 7b fa fe ..+..........{..

00000000`095c1570 ee 74 05 ac bb 79 e9 da-ca 00 44 96 94 71 92 b1 .t...y....D..q..

00000000`095c1580 db c9 9b 71 29 c0 e4 28-5e 6a 50 99 cd a8 17 e4 ...q)..(^jP.....

00000000`095c1590 56 b9 ef 7f 02 7d 96 a3-48 14 72 75 2f b0 b5 87 V....}..H.ru/...

00000000`095c15a0 ee 55 e9 6a 6d 28 3c c1-fd 00 e4 76 e3 80 88 78 .U.jm(<....v...x

00000000`095c15b0 26 0d 6c 8c b8 64 61 63-b7 13 3a ab c7 dd 1d 0a &.l..dac..:.....

00000000`095c15c0 d7 15 45 a1 d6 d9 34 c7-21 48 fb 43 87 38 da 1f ..E...4.!H.C.8..

00000000`095c15d0 50 47 b1 a5 5c 47 ed 04-44 97 d3 ac 74 2d eb 09 PG..\G..D...t-..

00000000`095c15e0 77 59 bf a3 54 5b de 42-d5 23 5a 71 9f wY..T[.B.#Zq..

After secure channel is established and cipher change has taken place, HTTP traffic starts flowing.

In this example, HTTP is being used as a transport for RPC and RPC server requires authentication. For authentication, the client application first calculates the channel binding by using the following process(in Windows this is done by SSPI but that is not important in this discussion ). This process is based on RFC 5929.

1. The channel binding type for this example is "tls-server-end-point" since a certificate is used in handshake (RFC5929).

2. The client calculates a hash of the certificate. The hashing algorithm is SHA-256, unless all of the following conditions are met, in which case the signature algorithm in the certificate will be used.

A certificate signature algorithm exist
The algorithm is only implemented in CNG (ALG_ID is CALG_OID_INFO_CNG_ONLY)
The algorithm has a corresponding CNG algorithm identifier string (pwszCNGAlgid)
The algorithm is not SHA1
The algorithm is not MD5

3. The SHA-256 hash of the above certificate is: ea 05 fe fe cc 6b 0b d5 71 db bc 5b aa 3e d4 53 86 d0 44 68 35 f7 b7 4c 85 62 1b 99 83 47 5f 95

4. The Channel binding unique prefix (RFC5929) "tls-server-end-point" is prefixed to the hash above (with a colon), resulting in

74 6c 73 2d 73 65 72 76 65 72 2d 65 6e 64 2d 70 6f 69 6e 74 3a ea 05 fe fe cc 6b 0b d5 71 db bc 5b aa 3e d4 53 86 d0 44 68 35 f7 b7 4c 85 62 1b 99 83 47 5f 95

5. The above value is inserted as the value of application_data field of gss_channel_bindings_struct structure, as pointed out by MS-NLMP section 2.2.2.1

6. Windows always sets the other fields of gss_channel_bindings_struct as zeros (SEC_CHANNEL_BINDINGS
structure). The resulting gss_channel_bindings_struct is as follows (little endian format):

00 00 00 00 //initiator_addtype
00 00 00 00 //initiator_address length
00 00 00 00 //initiator_address pointer
00 00 00 00 //acceptor_addrtype
00 00 00 00 //acceptor_address length
00 00 00 00 //acceptor_address pointer
35 00 00 00 //application_data length (53 bytes)
20 00 00 00 //application_data pointer (32 bytes from start of this structure)
74 6c 73 2d //application data, as calculated above
73 65 72 76
65 72 2d 65
6e 64 2d 70
6f 69 6e 74
3a ea 05 fe
fe cc 6b 0b
d5 71 db bc
5b aa 3e d4
53 86 d0 44
68 35 f7 b7
4c 85 62 1b
99 83 47 5f
95

After calculating channel binding, the client application starts authentication and include channel binding as part of authentication. In case of NTLM, the gss_channel_bindings_struct is called ClientChannelBindingUnhashed (MS-NLMP section 3.1.1.2). As explained in MS-NLMP section 3.1.5.1.2, the client adds an AV_PAIR structure and set the AvId field to MsvAvChannelBindings and the Value field to MD5(ClientChannelBindingsUnhashed). The MD5 hash of the above gss_channel_bindings_struct turns out to be:

65 86 E9 9D 81 C2 FC 98 4E 47 17 2F D4 DD 03 10

This value is part of the AUTHENTICATE_MESSAGE in frame 27 in the network trace attached (in the network trace it is shown in Base64 encoding as 45 41 42 6C 68 75 6D 64 67 63 4C 38 6D 45 35 48 46 79 2F 55 33 51 4D 51 with AvLen) .

When server receives the AUTHENTICATE_MESSAGE, in addition to the regular authentication processing, it also verifies the channel binding hash by calculating it the same way the client did. If the channel binding hash does not match, the authentication will not be successful. The subsequent behavior is server dependent. In this example (IIS), the server will stop communication on unsuccessful authentication.

Please note that two step hashing is being employed here. First the application creates a hash of the certificate which becomes a part of gss_channel_bindings_struct structure. This structure is MD5 hashed again to be included in AUTHENTICATE_MESSAGE.

There are configurations on both Windows client and server side to disable the EPA. For the server side, please consult the server specific documentation. As for the server in this example, IIS, please consult http://www.iis.net/configreference/system.webserver/security/authentication/windowsauthentication/extendedprotection.

On the client side, there is a registry setting that is described in KB976918 (http://support.microsoft.com/kb/976918) that can be used to configure EPA.

CIFS and SMB Timeouts in Windows

Edgar A Olougouna — Tue, 19 Mar 2013 19:36:00 GMT

This blog gives a consolidated overview of the most common SMB timeouts in Windows and their behaviors. Some of these legacy timeouts or timers are optional, implementation specific, not defined or not required by the protocol specifications. Let’s recall that MS-CIFS documents the protocol implemented in Windows NT and Windows 98, whereas MS-SMB describes the extensions added in Windows 2000 and beyond.

A subsequent blog discusses “SMB 2.x and SMB 3.0 Timeouts in Windows”:
http://blogs.msdn.com/b/openspecification/archive/2013/03/27/smb-2-x-and-smb-3-0-timeouts-in-windows.aspx

NOTE: For questions on MS-CIFS and MS-SMB documents, please post in the Open Specifications Forum: Windows Protocols at http://social.msdn.microsoft.com/Forums/en-US/os_windowsprotocols.

SMB exchange timeout

Here, I am using the term “exchange timeout”. There is no such definition in the specifications.
The exchange timeout can be defined as the maximum amount of time that the client redirector waits for a server to respond to a timed SMB message. There are three variants of SMB exchange timeout that are observed commonly in Windows: client session timeout, extended session timeout, offline file timeout.

Client session timeout
It is the most common exchange timeout. This is defined in [MS-CIFS] as a system-wide parameter Client.SessionTimeoutValue. This value can be configured through the SessTimeout registry key [KB102067].
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
Value type: Dword
Value name: SessTimeout
Default: 45 seconds (Windows NT)
Default: 60 seconds (Windows 2000)

Extended session timeout
This optional timeout may be useful in high latency networks. It applies to a timed exchange with a server listed in ServersWithExtendedSessTimeout.
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
Value type: Dword
Value name: ExtendedSessTimeout
Default: 1000 (seconds)
Value type: Multi-string
Value name: ServersWithExtendedSessTimeout
Default: <null>
These were introduced in Windows XP.

Offline file timeout
The client uses this optional timeout for a timed operation on an offline file, normally indicated by the FILE_ATTRIBUTE_OFFLINE attribute. This timeout can be configured with the registry parameter
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
Value type: Dword
Value name: OffLineFileTimeoutIntervalInSeconds
Default: 1000
This was introduced in Windows 2000.

Request Expiration Timer

This optional timer controls the scanning interval of outstanding commands. It is used as a scavenger cycle for outstanding commands that met their exchange timeout deadlines. Windows NT and Windows 98 CIFS clients implement a default value of 30 seconds for the Request Expiration Timer.
When an outstanding command has passed the exchange timeout, the client should close the connection if this remains idle. The time the client effectively closes the connection will vary, but bounded to a minimum of Client.SessionTimeoutValue.
For example, let’s assume that at a 30 seconds scanning interval, an outstanding command has a 60 seconds session timeout due to expire in 5 seconds, the redirector will close the connection at the next scavenger cycle. The effective connection closing time would be around:
Session timeout (60 sec) – time due at the scavenger timer (5 sec) + next scavenger time (30 sec) = 85 sec.
Commands exempt from the Request Expiration Timer
The list of exempt commands is defined in MS-CIFS 3.2.6.1 Request Expiration Timer Event.
• The NT_TRANSACT_NOTIFY_CHANGE subcommand
• Read and write commands issued on an Open to a named pipe via the following commands:
• SMB_COM_READ
• SMB_COM_WRITE
• SMB_COM_READ_ANDX
• SMB_COM_WRITE_ANDX
• SMB_COM_WRITE_AND_CLOSE
• TRANS_READ_NMPIPE
• TRANS_WRITE_NMPIPE
• TRANS_RAW_READ_NMPIPE
• TRANS_TRANSACT_NMPIPE
• TRANS_RAW_WRITE_NMPIPE
• TRANS_CALL_NMPIPE
• TRANS_WAIT_NMPIPE
• SMB_COM_LOCKING_ANDX Request with the Timeout field set to a nonzero value

The Echo behavior

An Echo request is sent to determine whether or not an idle connection has been lost. For an outstanding timed network operation, the connection would be closed upon the expiration of the exchange timer. The Echo mechanism does not prevent the connection from being closed if there is no response to any outstanding command beyond the exchange timeout. This way, the client avoids continuing Echo-probing a known dead connection. If the server is responding to Echo requests, the client will wait until the exchange times out, and the client will not send any interim response to the calling application.

OpLock Break Acknowledgment Timer

This timer controls the time the server waits for a client to acknowledge an OpLock break request, if an acknowledgement is required. If an OpLock break acknowledgment is not received before the timer elapses, the server locally breaks the lock. This is controlled through the OplockBreakWait registry [KB129202].
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\
Value type: Dword
Value name: OplockBreakWait
Default: 35 (seconds)

Idle Connection Timer

This is the amount of time that a connection can be idle before being disconnected. An idle connection is defined as a connection which has no existing open handles (no open files, directories, search contexts, etc.), and no pending operation. The Idle Connection Timer is implementation-specific.
When the server receives a message, Server.Connection.IdleTime is set to the current time plus Server.AutoDisconnectTimeout [MS-CIFS].
On Windows servers it can be configured through the Autodisconnect registry key [KB297684].
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\
Value type: Dword
Value name: Autodisconnect
Default: 15 (minutes)
The Autodisconnect can also be configured through group policy:
Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options
"Microsoft network server: Amount of idle time required before suspending session"

In older Windows clients (e.g. Windows 2000, Windows 2003), the client closes an idle connection based on the KeepConn workstation parameter [KB819108].
HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\parameters\
Value type: Dword
Value name: KeepConn
Default: 600 (seconds)
KeepConn is no longer used, except in Installable File System Kit (IFSKIT).

Unused Open Search Timer

This optional timer that controls the amount of time that an open search can stay unused before the server closes the search context. It is used as maximum age to scavenge open searches, see Server.SrvSearchMaxTimeout [MS-CIFS].
On Windows, the unused search timer can be configured as
\HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\
Value type: Dword
Value name: MaxKeepSearch
Default: 3600 (seconds)
Range: 10 – 10,000

This legacy timer was for MS DOS applications which did not have the capability to issue a “close search” primitive, such as Win32 FindClose, to explicitly close a search.

Authentication Expiration Timer

This timer was introduced in MS-SMB. It is used to periodically scan and mark authenticated sessions as expired when their specific expiration time is reached. If a session is expired, SMB operations will receive STATUS_NETWORK_SESSION_EXPIRED and the client must re-authenticate. Note that the server does not enforce session expiration on the following commands while the authentication state of a session is expired:
- SMB_COM_SESSION_SETUP_ANDX for session renewal,
- SMB_COM_CLOSE,
- SMB_COM_LOGOFF_ANDX,
- SMB_COM_FLUSH (see KB 943459) ,
- SMB_COM_LOCKING_ANDX as an oplock break acknowledgement (see KB 943459),
- SMB_COM_TREE_DISCONNECT.

Note that KB 943459 does not apply to SMB 2. The fix was done on srv.sys (SMB server) and mitigates older SMB clients that are not capable of re-authenticating upon receiving a STATUS_SESSION_EXPIRED error in response to an oplock break acknowledgment.
More details are discussed later on session expiration and dynamic re-authentication.

Authentication-specific expiration is driven by the authentication package. For instance, NTLM authentication has no explicit expiration time; authentications done with NTLM do not explicitly expire.
For Kerberos authentication package, the service ticket’s endtime is derived from the MaxServiceTicketAge when the ticket is issued, as documented in [MS-KILE]. The default MaxServiceTicketAge is 10 hours and can be adjusted though Group Policy Management Console under Default Domain Policy/ Account Policies / Kerberos Policy.
In addition, even if a session did not explicitly expire, the server may forcibly disconnect the user SMB session based upon a group policy setting and user’s logonHours [MS-ADA1].
The group policy setting is:
Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options
"Microsoft network server: Disconnect clients when logon hours expire"
This policy sets the registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\LanmanServer\Parameters\enableforcedlogoff
When this policy is enabled, the SMB session is disconnected when the user's logon hours expire.
When this policy is disabled, the session remains valid as long as it stays connected, even after logon hours expire, provided the session did not expire.
This policy may be configured on the Default Domain Policy and propagated through group policy to the computers.
As an example, let’s assume contoso\user1 has logonHours configured to “Logon Denied” between 5:00 PM and 9:00 AM. An SMB share \\server2008r2\Share is accessed from a Windows XP SP3 client at 4:59:30 PM where enableforcedlogoff is enabled. Once the clock ticks 5:00 PM, the client falls in logonHours expiry and the session is dropped. At 5:05 PM, when user1 attempts to access the share, a similar error is observed:

\\server2008r2\Share is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.
Logon failure: account logon time restriction violation.

Session expiration and dynamic re-authentication
In Windows implementation, SMB session expiration is enforced based upon the client’s support of dynamic re-authentication capability [MS-SMB].
If the client enables the CAP_DYNAMIC_REAUTH capability bit, the server will enforce session expiration. If a client does not set CAP_DYNAMIC_REAUTH, the Windows server does not return STATUS_NETWORK_SESSION_EXPIRED.
The SMB dynamic re-authentication feature was introduced in Windows XP. From there, Windows-based clients set the CAP_DYNAMIC_REAUTH capability bit to indicate to the server that the client supports re-authentication when the Kerberos service ticket for the session expires.
Windows servers do check CAP_DYNAMIC_REAUTH:
• If clientCapabilities sets CAP_DYNAMIC_REAUTH, the server will set Server.Session.AuthenticationExpirationTime to the expiry time returned by AcceptSecuirtyContext.
• If clientCapabilities does not set CAP_DYNAMIC_REAUTH, the server will not set Server.Session.AuthenticationExpirationTime, basically a CAP_DYNAMIC_REAUTH capability bit not set by the client means the session will not expire on the server side.

References:

[MS-CIFS]: Common Internet File System (CIFS) Protocol
http://msdn.microsoft.com/en-us/library/ee442092.aspx
[MS-SMB]: Server Message Block (SMB) Protocol
http://msdn.microsoft.com/en-us/library/cc246231.aspx
[MS-WKST]: Workstation Service Remote Protocol
http://msdn.microsoft.com/en-us/library/cc250262.aspx
When the Kerberos ticket expires for a Kerberos-authenticated SMB connection that is created to a Windows Server 2003-based server, the oplock on a file cannot be broken in a timely manner
http://support.microsoft.com/kb/943459
LanmanWorkstation Parameters: SessTimeout
http://technet.microsoft.com/en-us/library/cc938292.aspx
LanmanServer Parameters: MaxKeepSearch
http://technet.microsoft.com/en-us/library/cc957456.aspx
Remote Storage and Windows 2000
http://technet.microsoft.com/en-us/library/cc938445.aspx
How to Troubleshoot Event ID 2009 Errors
http://support.microsoft.com/kb/165815
“Microsoft network server: Disconnect clients when logon hours expire”
http://technet.microsoft.com/en-us/library/cc758192.aspx
HOW TO: Limit User Logon Time in a Domain in Windows Server 2003
http://support.microsoft.com/kb/816666

Rich Text Format (RTF) and Watermarks

Mark Miller_DSC — Mon, 04 Feb 2013 20:19:00 GMT

Seldom is the question asked, "Is there an RTF directive that can be used to add watermarks in RTF documents?"

One day recently this question found me, and after delving into the world of the Rich Text Format (RTF ) specification you may in turn be interested in what I found, which includes a new appreciation for RTF. At risk of repeating what is stated in many blogs in many similar ways, the RTF format affords fluidity of fidelity over time. What this essentially means is that RTF format provides a significant hedge against ever changing specifications. Thus, RTF is kind of a time capsule document format that can be opened in the future, with the same fidelity potential encoded at the time it was created.

Of course, RTF implementations do differ from application to application. However, since applications SHOULD ignore control words they do not implement they will only lack the ability to render the RTF at the same fidelity encoded in the original document. And, since RTF is a Rich "Text" Format, the encoding is in plain text and not encoded in a more abstruse Binary File Format. So reading the format and implementing the format is far more accessible by comparison.

To delve the RTF specification, refer to this blog, which references the Word 2007: Rich Text Format (RTF) Specification, version 1.9.1 (the latest version at the time of this writing). This is the essential reference for implementing the RTF specification.

Now that I’ve provided a little background and the necessary references, I’ll restate the purpose of this blog, which is to answer the question, "Is there an RTF directive that can be used to add watermarks in RTF documents?"

The answer is derived from the RTF specification.

First, nowhere in the RFT specification will you find the word "Watermark" or an explanation for it. That’s because there is no Watermark control code or object in the RTF specification per-se. Watermarking is implemented in RFT by various control codes that compose a Watermark, and in this case also includes a special Word Watermark object that of course is not part of the RTF specification (see below).

An easy way to discover the control codes for implementing a Watermark in RTF (among other things) is to create a Watermark in Word and save the file as an RTF document. A Watermark in Word is a Shape (i.e. a Drawing Object). In RTF the shape that composes the Watermark begins with the control word "\shp".

In the RTF specification reference, refer to the section titled, "Word 97 Through Word 2007 RTF for Drawing Objects (Shapes)", which details the meaning of the highlighted keywords below as they relate to "\shp". The first destination (\shp) is always present. This control word groups everything related to a shape together. Following the destination change is basic information regarding the shape. The following keywords with values can appear in any order after the "{\shp" control word.

The basic syntax for drawing objects in RTF is as follows:

<shape>	'{' \shp <shpinfo> <shpinst> <shprslt> '}'
<shpinfo>	\shpleftN? \shptopN? \shpbottomN? \shprightN? \shplidN? \shpzN? \shpfhdrN? \shpbxpage ? \shpbxmargin ? \shpbxcolumn? \shpbxignore? \shpbypage ? \shpbymargin ? \shpbypara? \shpbyignore? \shpwrN? \shpwrkN? \shpfblwtxtN? \shplockanchor? \shptxt?
<shpinst>	'{\' \shpinst* <sp>+ '}'
<sp>	'{' \sp <sn> <sv> <hsv>? '}'
<sn>	'{' \sn ... '}'
<sv>	'{' \sv ... '}'
<shprslt>	'{\' \shprslt* ... '}'
<hsv>	'{\' \hsv* <accent> & \ctintN & \cshadeN '}'
<accent>	\caccentone\| \caccenttwo \| \caccentthree \| \caccentfour \| \caccentfive \| \caccentsix

I won’t go into every control word definition, you can surely follow every control word definition in the RTF specification, but I will point out the control words germane to answering the question at hand. The following is a snippet from a test RTF document I used by saving a Microsoft Word document with a Watermark as RTF.

{\shp{\*\shpinst\shpleft0\shptop0\shpright10557\shpbottom2639\shpfhdr0\shpbxmargin\shpbxignore\shpbymargin\shpbyignore\shpwr3\shpwrk0\shpfblwtxt1\shpz0\shplid2049{\sp{\snshapeType}{\sv 136}}{\sp{\sn fFlipH}{\sv 0}}{\sp{\sn fFlipV}{\sv 0}}

{\sp{\sn gtextUNICODE}{\sv CONFIDENTIAL}}{\sp{\sn gtextSize}{\sv 65536}}{\sp{\sn gtextFont}{\sv Calibri}}{\sp{\sn fGtext}{\sv 1}}{\sp{\sn gtextFStretch}{\sv 1}}{\sp{\snfillColor}{\sv 12632256}}

{\sp{\snfillOpacity}{\sv 32768}}{\sp{\sn fLine}{\sv 0}}{\sp{\sn wzName}{\svPowerPlusWaterMarkObject357476642}}{\sp{\sn posh}{\sv 2}}{\sp{\snposrelh}{\sv 0}}{\sp{\sn posv}{\sv 2}}{\sp{\sn posrelv}{\sv 0}}{\sp{\sn dhgt}{\sv 251659264}}…

The first thing to point out is the control symbol "\*". I’ve alluded to this before, and per the RTF specification, "Destinations added after the 1987 RTF Specification may be preceded by the control symbol \* (backslash asterisk).This control symbol identifies destinations whose related text should be ignored if the RTF reader does not recognize the destination control word." This means the shape will be ignored by an RTF reader that does not implement the control word. You can test out a quick example of this by creating an RTF document in Word that contains a Watermark then open the document in WordPad and you’ll notice the Watermark will not be rendered since WordPad does not support Watermarks. Open the same RTF file in Word and of course you see the Watermark.

In my example file above, I used a "CONFIDENTIAL" Watermark which refers to the Shape (control word "\shp") and the Drawing Object "PowerPlusWaterMarkObject"). The Drawing Object is a special object identifier specifically for Word. You’ll find references to this object in API calls, macros, etc. (i.e. wherever you are dealing with a Word Watermark).

You can refer to the RTF specification and note the definitions for the shape (position, text font/size, left, top, right, bottom, etc). Just follow the nested open/close braces "{ }" (the Group symbol), which indicates the beginning and end of each Group (and Groups can and will be nested). Essentially, the entire RTF file is in a Group.

An RTF file has the following syntax:

<File>

'{' <header> <document> '}'

Note, the important takeaway point in this example is that there is no "Watermark" per-se in RTF, but rather the "\shp" control words compose the Watermark feature for the RTF reader.

If there is another feature you’d like to examine to see how it is composed/represented in RTF, just open Word and save your example as RTF and open in your editor of choice. It’s an easy way to understand RTF and how various features are implemented in the RTF format.

NOTE: Questions and comments are welcome. However, please DO NOT post a comment using the comment tool at the end of this post. Instead, post a new thread in the Open Specifications Forum at http://social.msdn.microsoft.com/Forums/en-US/

How to manually decode an ActiveSync WBXML stream

JCurry — Mon, 04 Feb 2013 18:14:00 GMT

Overview
ActiveSync requests and responses are sent as HTTP messages. In order to reduce the size of the messages, the body is encoded in a format known as WAP Binary XML. The information about Microsoft's implementation of this protocol that is used by ActiveSync is detailed in the MS-ASWBXML document which is based in the WAP Binary XML Content Format version 1.2.

Resources:
- MS-ASWBXML
http://msdn.microsoft.com/en-us/library/dd299442(EXCHG.80).aspx

- WAP Binary XML Content Format version 1.2
http://www.w3.org/1999/06/NOTE-wbxml-19990624/

   - IANA Character Sets
         http://www.iana.org/assignments/character-sets/character-sets.xml

Why do you need to know how to do this?
If you are troubleshooting an application that uses ActiveSync you will probably collect some type of network trace that contains request and/or response messages. However, when you try to look at the message contents it will just appear as a stream of bytes. Unless you have a utility that can interpret these bytes for you, you will need to know how to decode them manually. Once you know how to do this you might decide to write a utility that can do the work for you next time.

How does it work?
WBXML is NOT an encryption method, it is an encoding method. The XML elements in a message are replaced with single byte tokens. MS-ASWBXML contains a list of these tokens and maps them to an XML element. Simple enough right?

Lets get started!
The following streams of bytes are a request and response from an ActiveSync mobile device to an Exchange Server.

Request:
03 01 6A 00 45 5C 4F 4B 03 30 00 01 52 03 32 00
01 57 00 11 45 46 03 31 00 01 47 03 33 32 37 36
38 00 01 01 01 01 01 01

Response:
03 01 6A 00 45 5C 4F 4B 03 31 39 39 37 34 33 33
30 37 34 00 01 52 03 32 00 01 4E 03 31 00 01 01
01 01

Each message is divided into 3 parts:
   - Header
   - String Table
   - Token List

Header
The Header consists of the first 3 bytes of the stream and contains the Version Number, the Document Public Identifier, and the Character Set. Lets take a look at these.

03 01 6A

The first byte represents the Version Number and tells us that the message is encoded using the WBXML 1.3 format. You may have noticed that MS-ASWBXML references the WBXML 1.2 format. However, we will not encounter any issues because the message is encoded with the 1.3 format. The second byte represents the Document Public Identifier. In this case it means that the public identifier is unknown or missing. The last byte specifies the Character Set that is being used. The possible values that you might find here are defined in the IANA Character Sets specification. In this case, 6A (106) is UTF-8. The first line of the request might look something like this:

<?xml version="1.0" encoding="utf-8"?>

You can find more information about the Version Number, Document Public Identifier, and the Character Set in the WAP Binary XML Format version 1.2 and IANA Character Sets specifications.

String Table
The 4th byte of the message specifies the length of the String Table. In this case, the String Table is 0 bytes in length and no other work is necessary.

Token List
Following the String Table is the Token List. This is where the real work/fun starts! If you haven't already downloaded the latest version of the MS-ASWBXML document, now would be a good time. To get the most out of this blog I suggest you follow along in the document. You'll be glad you did if you ever have to do this again on your own.

45 5C 4F 4B 03 30 00 01 52 03 32 00 01 57 00 11
45 46 03 31 00 01 47 03 33 32 37 36 38 00 01 01
01 01 01 01

For the remainder of the stream it's a simple matter of looking up the token value in MS-ASWBXML and constructing the XML message. Lets take a look at the first token, 0x45. Go look it up in MS-ASWBXML and then come back here to see if you are correct.

What do you mean you can't find it? Here's a hint, it's in section 2.1.2.1.1. Go look again.

You still can't find it? OK, sorry, that was a trick question. The truth is that it really is in section 2.1.2.1.1, but you have to take a closer look at the individual bits in the token first. This is the only real hurdle in understanding WBXML encoding. Once you understand the Tag Format the rest is easy.

The Tag Format looks like this.

Bits	Description	Example
8	If set, the element contains attributes	<element attr="1"></element>
7	If set, the element contains content	<element>test</element>
6-1	The tag identity

Note: If you are using Windows 7 or newer, I have found that using the built in Calculator in Programmer mode makes this much easier. You can simply click on the individual bits in the display to toggle them on/off.

The tag 0x45 in binary is 01000101. We can clearly see that the 7th bit is set which means that the element will contain content of some sort. This could be a string or it could be other elements, we don't know yet. If we ignore the 7th bit and just examine the first 6 bits (see my note above about using the Windows Calculator for this) we can see that the actual value is 5. Now, go look at MS-ASWBXML section 2.1.2.1.1 and find the element with a token value of 0x05. See, I told you it was there, it's the <Sync> element!

More information about the Tag Format can be found in the WAP Binary XML Content Format version 1.2 in the section titled Tag Code Space.

Follow the same process to decode the next 2 tokens 5C and 4F. Each one of these also has the 7th bit set. The actual token values are 1C and 0F. The next 2 elements are <Collections> and <Collection>. So far, our completed XML looks like this.

<?xml version="1.0" encoding="utf-8"?>
<Sync xmlns="AirSync">
<Collections>
<Collection>

You might be wondering where the xmlns="AirSync" part came from. Every ActiveSync element is defined in a code page, or namespace. In order for the server to be able to interpret the XML, it needs to know what code page the elements belong to. AirSync is the default code page (or namespace) for ActiveSync requests and responses. Unless you see an explicit code page switch (see SWITCH_PAGE below) after the String Table, this namespace definition is implied even though its not encoded into the byte stream. It's not necessary to provide an alias for the namespace. Any elements that are not prefixed with an alias will be assumed to be part of the AirSync namespace.

Decode the next token in the same fashion. 0x4B is 0x0B when you flip the 7th bit. This is the <SyncKey> element.

4B
<SyncKey>

The next token is 0x03, which you won't find in any of the code pages. This is because 0x03 is a global token. Global tokens are listed in MS-ASWBXML section 1.6 and described in more detail in the WAP Binary XML Content Format version 1.2 specification. The token 0x03 represents an inline string with a terminator (0x00). In this case the string is very short and only consists of a single value, 0x30, followed by the terminator 0x00. The string is encoded using the Character Set specified in the header, which is UTF-8. UTF-8 is very similar to the ANSI character set and in this case you can look up the character value in either. 0x30 is the character '0'.

The following token, 0x01, is another global token, END. This token is generic in that it represents a closing tag for whatever the currently 'open' tag is, which is the <SyncKey> element. So, this 0x01 token represents </SyncKey>. The following is the entire sequence of bytes for the <SyncKey> element. Once you get used to looking at WBXML you'll start to notice small patterns like this.

4B 03 30 00 01
<SyncKey>0</SyncKey>

Do you see the next pattern? Bytes 9-13 look like this: 52 03 32 00 01. If you've been following along you might assume that the next chunk of XML is going to look something like this <element>string</element>, and you would be right. 0x52 is actually 0x12 when you flip the 7th bit, which is the <CollectionId> element. We know that 0x03 means the next bytes represent an inline string followed by a terminator. Again, it’s only a single character, 0x32, which is the value '2' followed by the terminator 0x00, and the END token, 0x01. These bytes represent the following XML segment.

52 03 32 00 01
<CollectionId>2</CollectionId>

You now have almost everything you need to decode the rest of the stream with 1 exception so let's keep going. The next token is 0x57, which is 0x17 without the 7th bit set, which is the <Options> element. The token right after this is 0x00. What does that mean? It's another global token, SWITCH_PAGE. You might have been wondering why all of the tokens we have looked up so far have been in section 2.1.2.1.1 "Code Page 0: AirSync". This is the default code page. When we encounter the token 0x00 when not in the context of an inline string this tells us that we are switching to a different code page and that any elements that follow this switch should reference that code page. The byte that follows the switch is 0x11. The list of Code Pages can be found in section 2.1.2.1. However, remember that the token is a hexidecimal value and the code pages are base 10 integers. This is important because the switch is telling us to look at code page 17, not code page 11.

00 11
SWITCH_PAGE to Code Page 17

The next token is 0x45, but remember that because of the code page switch that we need to look at code page 17, AirSyncBase, which can be found in section 2.1.2.1.18. Flipping the 7th bit gives us 0x05 which is <BodyPreference>.

45
<airsyncbase:BodyPreference xmlns:airsyncbase="AirSyncBase">

There it is again! I snuck in something that isn't actually in the byte stream. Here again, defining the alias for the xml namespace is implied, but not actually present in the byte stream. It actually doesn't matter if we put it here or after the AirSync namespace in the Sync element, however it MUST exist in one of these two places. Without the xmlns:airsyncbase="AirSyncBase" definition we cannot use the airsyncbase: prefix. We MUST prefix any elements contained in the AirSyncBase namespace with the alias that we provided in the definition. If you don’t, the server will look in the default namespace, which in this case is AirSync and it will think the 0x45 token is a <Sync> element and you will get an error.

The next token 0x46 would be 0x06, <Type>, which is then followed by an inline string as indicated by the 0x03 token. The string consists of a single character 0x31, '1' followed by the terminator and 0x01 token indicating the closing tag for the <Type> element.

46 03 31 00 01
<airsyncbase:Type>1</airsyncbase:Type>

The next pattern of bytes is very similar. 0x47, which is 0x07, or <TruncationSize>, followed by 0x03 indicating an inline string, "32768", the string terminator, and the closing element.

47 03 33 32 37 36 38 00 01
<airsyncbase:TruncationSize>32768</airsyncbase:TruncationSize>

Here we are at the home stretch and all we see is a few 0x01 tokens. If we examine the complete XML that we have built so far those last 0x01 tokens will make sense.

<?xml version="1.0" encoding="utf-8"?>
<Sync xmlns:"AirSync">
   <Collections>
      <Collection>
         <SyncKey>0</SyncKey>
         <CollectionID>2</CollectionID>
         <Options>
            <airsyncbase:BodyPreference xmlns:airsyncbase="AirSyncBase">
               <airsyncbase:Type>1</airsyncbase:Type>
               <airsyncbase:TruncationSize>32768</airsyncbase:TruncationSize>

All that is missing is a series of closing elements for BodyPreference, Options, Collection, Collections, and Sync. Notice that there are exactly (5) 0x01 tokens remaining in the stream. That is exactly what those are, closing tags in that order. And you're done! The complete XML request should look like this. I've tried to highlight some key areas so you can associate those with the bytes from the stream.

Decoded Request
03 01 6A 00 45 5C 4F 4B 03 30 00 01 52 03 32 00
01 57 00 11 45 46 03 31 00 01 47 03 33 32 37 36
38 00 01 01 01 01 01 01

<?xml version="1.0" encoding="utf-8"?>
<Sync xmlns:"AirSync">
   <Collections>
      <Collection>
        <SyncKey>0</SyncKey>
         <CollectionID>2</CollectionID>
         <Options>
            <airsyncbase:BodyPreference xmlns:airsyncbase="AirSyncBase">
               <airsyncbase:Type>1</airsyncbase:Type>
               <airsyncbase:TruncationSize>32768</airsyncbase:TruncationSize>
            </airsyncbase:BodyPreference>
         </Options>
      </Collection>
   </Collections>
</Sync>

You should now have enough information to be able to decode the response on your own. Try to decode it on your own. Check what you get against the answer below.

Decoded Response
03 01 6A 00 45 5C 4F 4B 03 31 39 39 37 34 33 33
30 37 34 00 01 52 03 32 00 01 4E 03 31 00 01 01
01 01

<?xml version="1.0" encoding="utf-8"?>
<Sync xmlns:"AirSync">
   <Collections>
      <Collection>
         <SyncKey>1997433074</SyncKey>
         <CollectionID>2</CollectionID>
         <Status>1</Status>
      </Collection>
   </Collections>
</Sync>

Determining Office Binary File Format Types

JCurry — Wed, 16 Jan 2013 16:36:00 GMT

Referenced Documents:

If you need to programmatically determine the office file type for a file and cannot rely on the file extension you can use the following method. These files are based on the MS-CFB specification and you will have to parse the file accordingly. Here is the method that you will need to follow to locate the name of the application that created the file. I am using a .xls file that was created with Microsoft Excel 2010 and saved in the Excel 97-2003 Workbook format for this example. The same method can be used for .doc and .ppt files.

1. Examine the file header (section 2.2) and check the values of the Major Version and the Sector Shift. These will most likely be 0x0003 and 0x0009 respectively which means that the sector size is 512 bytes. If the Major Version is 0x0004 and the Sector Shift is 0x000C the sector size will be 4096 bytes and you will need to adjust your file offset calculations accordingly. I am going to assume that the sector size is 512 bytes for the purposes of this explanation.

2. At offset 0x30 you will find the First Directory Sector Location. This value is 4 bytes in length. In my example file the value is 0x0000002B. One very important thing to understand is that this is NOT a file offset location, it's an index. In order to locate the First Directory Sector you take that value, add 1 to account for the file header, and multiple that by the sector size (see step 1). This will give you the actual file offset location of the First Directory Sector. For my example file I use the following math to find it. (0x2B + 0x01) * 0x200 (512 bytes) = 0x5800.

Figure 1: The CFB Header

3. Locate the 512 byte block at the file location from step 2. This will contain one or more Compound File Directory Entry structures (section 2.6.1), each of which is 128 bytes in length. Examine the Directory Entry Name of each entry until you find one that contains 0x0005 + "Summary Information".

4. Once you have located the Summary Information entry you need to locate and make note of the Starting Sector Location and Stream Size properties. In my test file these values are 0x0000001A and 0x0000000000001000 respectively. Similar to how you located the First Directory Sector Location (step 2), you need to find the location of the Summary Information sector using the same type of calculation. In my test file it looks like this. (0x1A + 0x01) * 0x200 (512 bytes) = 0x3600.

Figure 2: The First Directory Sector

5. Go to the location that was calculated in step 4 and read the number of bytes as specified by the Stream Size. In my case, I am going to file offset 0x3600 and reading 0x1000 (4096) bytes. The contents of that sector are contained in a PropertySetStream as specified in the MS-OLEPS specification section 2.21.

6. Locate the Offset0 value and jump to that offset in the sector. In my test file, the Offset0 value is 0x30. When I add this to the file offset of the current sector I get 0x3630. This location contains a PropertySet structure (section 2.20).

7. Skip over the next 8 bytes and you'll get to the list of PropertyIdentifierAndOffset structures (section 2.19). Each PropertyIdentifierAndOffset is 8 bytes in length and contains the PropertyIdentifier and Offset value. You want to find the one with a PropertyIdentifier of 0x00000012 and make note of it's Offset value. In my test file this is 0x00000070.

Note: Property 0x00000012 is the PIDSI_APPNAME identifier, which contains the name of the application that created the file. The complete list of possible SummaryInformation values can be found in section 2.25.1.

8. The Offset value from step 7 must be added to the Offset0 value from step 6 to correctly locate the Property. In my case, the property will be located at 0x3630 + 0x70 = 0x36A0. The contents at that location are stored in a TypedPropertyValue structure (section 2.15). The Type should be 0x001E which indicates that the Value will be stored as a CodePageString. You can ignore the Padding bytes.

9. The Value is stored as a CodePageString (section 2.5). The first 4 bytes indicate the length of the string, followed by the actual string. In my example file the Size is 0x10 (16 bytes) and the Characters value is "Microsoft Excel" + 0x00. The 0x00 at the end is the standard string terminator.

Note: For .doc files the Size should be 0x18 (24 bytes) and the Characters value should be "Microsoft Office Word"+ 0x00 plus 2 extra bytes of padding. For .ppt files the Size should be 0x1C (28 bytes) and the Characters value should be "Microsoft Office PowerPoint" + 0x00.

Figure 3: The Summary Information Sector

Unencrypted MS-EVEN6 Traffic

Tarun Chopra - MSFT — Mon, 14 Jan 2013 04:59:00 GMT

This blog entry is intended for readers interested in generating unencrypted MS-EVEN6 (http://msdn.microsoft.com/en-us/library/cc231282(v=PROT.13).aspx ) protocol traffic.

Can windows inbox tools be configured to generate unencrypted traffic?

No, Windows inbox tools eventviewer and wevtutil generates encrypted MS-EVEN6 traffic and there is NO registry setting to alter this behavior. These tools invokes RPC bind with authentication-level set to RPC_C_AUTHN_LEVEL_PKT_PRIVACY which ensures encrypting the argument value of each remote procedure call. Refer below screenshot highlighting authentication-level field.

What does MS-EVEN6 specification say about authentication-level?

MS-EVEN6 specification, http://msdn.microsoft.com/en-us/library/cc231303(v=prot.13), does not impose any such constraint on authentication-level and we confirmed that windows server does accept RPC bind with authentication-level set to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY. Benefit of using RPC_C_AUTHN_LEVEL_PKT_INTEGRITY authentication-level over RPC_C_AUTHN_LEVEL_PKT_PRIVACY --- MS-EVEN6 traffic will be unencrypted.

This is a good news for the implementers interested in analyzing unencrypted EVEN6 packet. But it requires some work as we have to write a custom client with authentication-level set to RPC_C_AUTHN_LEVEL_PKT_INTEGRITY during RPC bind and then use that binding handle to invoke EVEN6 methods

A feasible option to generate unencrypted MS-EVEN6 traffic

I recently wrote a sample application in C++ to generate unencrypted protocol traffic and detail steps are as follows:

1. Install visual studio 2008/2010 Ultimate/professional on a development machine.

2. Copy and paste the IDL given in EVEN6 [MS_EVEN6] specification to a notepad and save file as even6.idl

3. Since it’s an RPC based protocol, copy and paste IDL given in MS-RPCE specification to notepad and save file as ms-rpce.idl

4. Place both the above files in same folder and use MIDL compiler to generate stub and header files. For this, go to start, type “visual studio”, launch “Visual Studio Command Prompt” and issue below command:

midl even6.idl

5. Step 4 will generate even6.h, ms-rpce.h, even6_c.c which you need to include in your project. Choose VC++, Win32 console application as project type.

6. In the file that has “_tmain” function, include header files and create binding handle (IfHandle) that has to be passed to EVEN6 methods. Following snippet shows how to create a binding handle:

int bindStatus, bStatStringBnd;

RPC_BINDING_HANDLE IfHandle = NULL;

RPC_WSTR pszStringBinding;

TCHAR *pszEven6UUID = TEXT("F6BEAFF7-1E19-4FBB-9F8F-B89E2018337C");

TCHAR *pszEndpoint = TEXT("");

TCHAR *protoSeq = TEXT("ncacn_ip_tcp");

wchar_t exceptionString[512];

ULONG_PTR lpArguments = (ULONG_PTR)exceptionString;

pszStringBinding = (RPC_WSTR)MIDL_user_allocate(512);

bindStatus = RpcStringBindingCompose( (RPC_WSTR)pszEven6UUID, (RPC_WSTR)protoSeq, (RPC_WSTR)pszServer, NULL, NULL, (RPC_WSTR*)&pszStringBinding);

if (RPC_S_OK == bindStatus){ printf ("The call succeeded. RpcStringBindingCompose\n");

}

if (RPC_S_INVALID_STRING_UUID == bindStatus) { sprintf((char *)exceptionString, "The string representation of the UUID: , %s , is not valid.\n", pszEven6UUID);

RaiseException( bindStatus, 0, 1, &lpArguments );

}

bStatStringBnd = RpcBindingFromStringBinding( (RPC_WSTR)pszStringBinding, &IfHandle);

bStatStringBnd = RpcBindingSetAuthInfo( IfHandle, (RPC_WSTR)pszServer, RPC_C_AUTHN_LEVEL_PKT_INTEGRITY, RPC_C_AUTHN_WINNT, NULL, RPC_C_AUTHZ_NAME);

7. Pass this IfHandle to various EVEN6 calls, I tested EvtRpcOpenLogHandle, and you should be able to generate unencrypted traffic.

If (RPC_S_OK == bStatStringBnd) {

hEven6Binding = IfHandle;

RpcInfo error2;

error_status_t error = EvtRpcOpenLogHandle(L"Application", 1, (PCONTEXT_HANDLE_LOG_HANDLE*)&hEven6Binding, &error2);

}

Bind Request Packet with authentication-level as RPC_C_AUTHN_LEVEL_PKT_INTEGRITY:

Bind Response Packet with authentication-level as RPC_C_AUTHN_LEVEL_PKT_INTEGRITY:

Unencrypted EvtRpcOpenLogHandle request packet:

Unencrypted EvtRpcOpenLogHandle response packet:

References:

Open Specifications Support Team Blog: http://blogs.msdn.com/b/openspecification/

Open Specifications Forums: http://blogs.msdn.com/b/openspecification/

Open Specifications Forum/Windows Protocols: http://social.msdn.microsoft.com/Forums/en-US/os_windowsprotocols/threads

Network Monitor Blog: http://blogs.technet.com/b/netmon/

Encryption in SMB 3.0: A protocol perspective

Edgar A Olougouna — Fri, 05 Oct 2012 20:26:00 GMT

Encryption is one of the new SMB 3.0 security enhancements in Windows Server 2012 RTM. It can be enabled on a per-share basis, or enforced for all shares on the server. SMB 3.0 uses AES-CCM [RFC5084] as encryption algorithm, and this also provides data integrity (signing).

This blog takes a protocol walk on the topic through [MS-SMB2], and provides a sample test vector. [MS-SMB2] describes SMB 2.x and SMB 3.0 dialects. Throughout this blog, I use the terms SMB2 and SMB3 interchangeably because both share the core SMB2 Packet format. Details on encryption configuration aspects can be found in the blogs listed in references.

Capability activation

The client and server advertise encryption support via the SMB2_GLOBAL_CAP_ENCRYPTION capability flag during SMB 3.0 dialect negotiation. Then at least one of the following two options must be happen:
For enabling encryption for a whole session, SessionFlags of SessionSetup response includes SMB2_SESSION_FLAG_ENCRYPT_DATA. This is the case where encryption is enforced on the whole server.
For a per-share enabled encryption, ShareFlags of TreeConnect response includes SMB2_SHAREFLAG_ENCRYPT_DATA.

When encryption is active for a given exchange, it is applied before submission to the transport. An exchange could consist of a single request or response, or a series of compounded chain operations. The sender performs encryption as an outer layer after the SMB2 messages in a given submission have been assembled.

Cryptographic keys

Ko = SMB3KDF (Ki, Label, Context)

SMB3KDF() is defined as the KDF algorithm in Counter Mode, as specified in [SP800-108] section 5.1, with 'r' value of 32 and 'L' value of 128, and HMAC-SHA256 as the PRF.

Ki – Key derivation key, used as an input to the KDF. For SMB 3.0, Ki is the SessionKey.

Label – the purpose of this derived key, encoded as string and length for SMB 3.0.

Context – the context information of this derived key, encoded as string and length for SMB 3.0.

L – An integer that specifies the length of the derived keying material Ko, L is 128 bits for SMB 3.0 cryptographic keys. Note that L is a constant since all SMB 3.0 keys are 16 bytes in length (SigningKey, EncryptionKey, DecryptionKey, and ApplicationKey).

Ko – Keying material output from the KDF, a binary string of length L, where Ko is the leftmost L bits of KDF result.

ServerInKey (ClientToServer) = SMB3KDF (SessionKey, "SMB2AESCCM\0", "ServerIn \0")

EncryptionKey (Client) = DecryptionKey (Server) = ServerInKey (ClientToServer)

ServerOutKey (ServerToClient) = SMB3KDF ( SessionKey, "SMB2AESCCM\0", "ServerOut\0")

EncryptionKey (Server) = DecryptionKey (Client) = ServerOutKey (ServerToClient)

The server and client generate the encryption keys upon session establishment of the primary channel. If the server is configured for encryption (i.e. SmbServerConfiguration.EncryptData) and the Connection.ClientCapabilities includes the SMB2_GLOBAL_CAP_ENCRYPTION, the server generates the EncryptionKey and DecryptionKey and sets SMB2_SESSION_FLAG_ENCRYPT_DATA flag in the SessionFlags field of the SessionSetup response; the client must also generate its encryption and decryption keys.

Upon successful TreeConnect, if Session.EncryptData is FALSE and Share.EncryptData (e.g. the share is configured to require encryption), the server generates its EncryptionKey and DecryptionKey and sets the SMB2_SHAREFLAG_ENCRYPT_DATA bit in the ShareFlags field of the TreeConnect response; the client must also generate its encryption and decryption keys.

Note: These cryptographic keys are all derived from the SessionKey. As a result, SMB 3.0 signing and encryption is as secure as the session key. Not only must this key be unique and very random, but also it needs be kept secret.

Transformed message

A transformed message consists of a transform_header followed by its encrypted SMB2 message.

The transform_header has the following fields:

ProtocolId (4 bytes): 0xFD, 'S', 'M', and 'B' (in network order).

Signature (16 bytes): Signature of the encrypted message.

Nonce (16 bytes): An implementation-specific value that is unique for every encrypted message within a session.

OriginalMessageSize (4 bytes): The size in bytes of the SMB2 message.

Reserved (2 bytes): Set to zeros and ignored.

EncryptionAlgorithm (2 bytes): the value 0x0001 is for SMB2_ENCRYPTION_AES128_CCM.

SessionId (8 bytes): Uniquely identifies the established session for the command.

With the SMB2_ENCRYPTION_AES128_CCM algorithm, SMB 3.0 encryption implementation uses the leftmost 11 bytes of the Nonce field in the transform_header, let’s call this Aes128Nonce. Per [RFC3610], the nonce passed to AES-CCM can be between 7 and 13 bytes. Windows SMB 3.0 made the trade-off choice of 11 bytes.

Note that it would almost be impossible that all possible nonce values can be consumed on a channel, because it would take over several centuries before a sender could recycle a nonce on a given channel.

Encrypting the message

The sender builds the transformed message with these specifics. AES-CCM [RFC5084] is called with the following inputs:

- AES key: Session.EncryptionKey.

- Aes-nonce: Aes128Nonce must be used.

- Plaintext: The SMB2 message including the header and the payload.

- The optional authenticated data (AAD): The SMB2 transform_header excluding the ProtocolId and Signature fields; these are the 32 bytes starting from the Nonce field.

The AES-CCM outputs are:

- Ciphertext: the encrypted SMB2 message

- Message authentication code: the Signature field of the transform_header.

The sender appends the encrypted SMB2 message to the transform_header and sends it to the receiver.

Decrypting the message

The message is decrypted using:

- The EncryptionAlgorithm in the transform_header.

- The Session.DecryptionKey of the Session that corresponds to the SessionId in the transform_header.

- The AAD passed to AES-CCM is the transform_header excluding the ProtocolId and Signature fields.

- The nonce passed to AES-CCM is the Aes128Nonce; that is the first 11 bytes of the Nonce field.

The signature returned by the decryption algorithm is then verified against the Signature in the transform_header.

Encryption clauses

The sender encrypts the message if any of the following conditions is satisfied:

If the sender is sending a response to an encrypted request.
If Session.EncryptData is TRUE and the request or response being sent is not NEGOTIATE.
If Session.EncryptData is FALSE, the request or response being sent is not NEGOTIATE or SESSION_SETUP or TREE_CONNECT, and <TreeConnect|Share>.EncryptData is TRUE.

Note: TreeConnect.EncryptData is on the client side. Share.EncryptData is on the server side.

Review of encryption clauses

All clauses exclude any operation which does use a SessionId. A SessionId is needed to find the Session object and derive the encryption and decryption keys from its session key. For example, if the client sends a non-encrypted ECHO, Windows 8 server will respond with a non-encrypted response.
Clause “response to an encrypted request”: if the sender encrypts the request, the receiver will respond in the same way by encrypting the response. There is however a prerequisite that encryption is active, i.e. encryption keys have been generated. For example, if the client encrypts an ECHO, Windows 8 server responds in-kind by encrypting the response.
Clause "Session.EncryptData is TRUE": This excludes encryption of the initial SESSION_SETUP.

This means SESSION_SETUP for re-authentication or session binding will be encrypted.

Clause “Session.EncryptData is FALSE and <TreeConnect|Share>.EncryptData is TRUE”:

If the client performs TREE_DISCONNECT before a LOGOFF, the logoff will not be encrypted.

Compounded messages

It is expected that all operations in an encrypted message belong to the SessionId in the transform_header. The encrypted message is sent as a single submission to the underlying transport, there is no provision for a next transformed message in the transform_header.

Related compounded messages

In case of related compounded requests, all the requests/responses belong to the same SessionId. The sender indicates to the receiver that it is using the SessionId, TreeId, and FileId of the previous operation or resulted from the receiver processing that previous operation.

Unrelated compounded messages

From a broader scope, the expected protocol behaviors for sending an encrypted message can be summarized as follows, be it for compounded requests/responses or not.

The sender should not send an encrypted request separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. In other words, the sender should not use a SessionId to convoy the encrypted traffic of other authenticated sessions.

The receiver should not send an encrypted response separately or as part of a compounded chain (related or unrelated) that contains a SessionId different from the session used for encryption. Windows-based servers will respond in-kind to what the client sends. If the client violates the above statement regarding the SessionId, a Windows 8-based server will currently send the response as such.

Oplock and Lease Breaks

Oplock break notifications/acknowledgments/responses must be encrypted when encryption is active. For an Oplock, the FileID is used to derive the SessionId which is set in the notification/acknowledgement/response. See more details in MS-SMB2.

Lease break notifications - sent by the server - do not have a SessionId, and as a result, are neither signed nor encrypted. Lease keys are not tied to a particular session from the client.

However, Lease break acknowledgements sent by the client - and their responses sent by the server - must be encrypted when encryption is active. The client is responsible for selecting a session associated with one of the existing opens associated with that Lease Key. The response is sent on the session that receives the acknowledgment.

Test vector

This sample data should be considered “as-is”. It should also be noted that examples do not replace normative protocol specifications. The reference must be [MS-SMB2].

The test program running on a Windows 8 client negotiates SMB3 and communicates with a Windows 2012 server. It opens a file and WRITEs the following content. It then READs back the file.

This is the content written and read:

Smb3 encryption testing

Hex value:

536D623320656E6372797074696F6E2074657374696E67

These outputs show the encryption and decryption of the WRITE and READ commands.

The decrypted content is verified to be same at the end of the SMB2 READ response.

SessionId 0x8e40014000011
SessionKey 0xB4546771B515F766A86735532DD6C4F0
SigningKey 0xF773CD23C18FD1E08EE510CADA7CF852
EncryptionKey (Client) 0x261B72350558F2E9DCF613070383EDBF
DecryptionKey (Client) 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
ApplicationKey 0x77432F808CE99156B5BC6A3676D730D1

Header.Command 0x0009 WRITE
Encryption of the request ---
Key 0x261B72350558F2E9DCF613070383EDBF
AES-128-CCM nonce 0x66E69A111892584FB5ED52
AAD
0x
66E69A111892584FB5ED524A744DA3EE87000000000001001100001400E40800
SMB2 packet
0x
FE534D4240000100000000000900400008000000000000000400000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
3100700017000000000000000000000015010000390000020100000039020000
00000000000000007000000000000000536D623320656E6372797074696F6E20
74657374696E67
transform_header.Signature 0x81A286535415445DAE393921E44FA42E
transform_header.Nonce 0x66E69A111892584FB5ED524A744DA3EE
transform_header.OriginalMessageSize 0x87
transform_header.SessionId 0x8e40014000011
Encrypted message
0x
25C8FEE16605A437832D1CD52DA9F4645333482A175FE5384563F45FCDAFAEF3
8BC62BA4D5C62897996625A44C29BE5658DE2E6117585779E7B59FFD971278D0
8580D7FA899E410E910EABF5AA1DB43050B33B49182637759AC15D84BFCDF5B6
B238993C0F4CF4D6012023F6C627297075D84B7803912D0A9639634453595EF3
E33FFE4E7AC2AB
Transformed message
0x
FD534D4281A286535415445DAE393921E44FA42E66E69A111892584FB5ED524A
744DA3EE87000000000001001100001400E4080025C8FEE16605A437832D1CD5
2DA9F4645333482A175FE5384563F45FCDAFAEF38BC62BA4D5C62897996625A4
4C29BE5658DE2E6117585779E7B59FFD971278D08580D7FA899E410E910EABF5
AA1DB43050B33B49182637759AC15D84BFCDF5B6B238993C0F4CF4D6012023F6
C627297075D84B7803912D0A9639634453595EF3E33FFE4E7AC2AB
Decryption of the response ---
Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
Transformed message
0x
FD534D42A6015530A18F6D9AFFE22AFAE8E66484860000000000000011000014
00E4080050000000000001001100001400E40800DBF46435C5F14169293CE079
E344479BF670227E49873F458672C3098DAC467DD5809F369D67409166515787
1483E01F7BECD02064EAC3E235F913668BBC2F097980D4B378F1993EFF6E60D1
77309E5B
transform_header.Signature 0xA6015530A18F6D9AFFE22AFAE8E66484
transform_header.Nonce 0x86000000000000001100001400E40800
transform_header.OriginalMessageSize 0x50
transform_header.SessionId 0x8e40014000011
AES-128-CCM nonce 0x8600000000000000110000
AAD
0x
86000000000000001100001400E4080050000000000001001100001400E40800
Decrypted SMB2 packet
0x
FE534D4240000100000000000900210009000000000000000400000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
11000000170000000000000000000000

Header.Command 0x0008 READ
Encryption of the request ---
Key 0x261B72350558F2E9DCF613070383EDBF
AES-128-CCM nonce 0xA5123A25F983E245983F41
AAD
0x
A5123A25F983E245983F413B8B429AF271000000000001001100001400E40800
SMB2 packet
0x
FE534D4240000100000000000800400008000000000000000500000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
3100000017000000000000000000000015010000390000020100000039020000
0000000000000000000000000000000000
transform_header.Signature 0xE93601498B76D6F7A72D5EF9B6C79FAF
transform_header.Nonce 0xA5123A25F983E245983F413B8B429AF2
transform_header.OriginalMessageSize 0x71
transform_header.SessionId 0x8e40014000011
Encrypted message
0x
9A464F709AA663F8C2FC3907D63CBF6F98B1E3DD649ED366009FD0B40A365224
718E5440E053F6E01AE462FDB721BF91C3A6E52E14F9EFF005F445761289FF12
72908B52754C8FCB949F228AC104A66204289A205BCBC47509D04AF9A907002B
96863358B3B7CBA5E377930074FCDF3550
Transformed message
0x
FD534D42E93601498B76D6F7A72D5EF9B6C79FAFA5123A25F983E245983F413B
8B429AF271000000000001001100001400E408009A464F709AA663F8C2FC3907
D63CBF6F98B1E3DD649ED366009FD0B40A365224718E5440E053F6E01AE462FD
B721BF91C3A6E52E14F9EFF005F445761289FF1272908B52754C8FCB949F228A
C104A66204289A205BCBC47509D04AF9A907002B96863358B3B7CBA5E3779300
74FCDF3550
Decryption of the response ---
Key 0x8FE2B57EC34D2DB5B1A9727F526BBDB5
Transformed message
0x
FD534D42ABD518B68C2F04D7879F482B689EB83F870000000000000011000014
00E4080067000000000001001100001400E40800493D6FE2BDBEB435CF5F5469
70C7BB57BF20E713C75A3D045507E0D68E5C0346659D6FFB8AC1504A786CA2BB
89C9E7FE4F313E910A04180D2D0EA7DF636329E5A3285984500EF86FE9D55DA4
FAB9531CFDD4C551D47F3C73124BB4590A45052B694048B991CCF5
transform_header.Signature 0xABD518B68C2F04D7879F482B689EB83F
transform_header.Nonce 0x87000000000000001100001400E40800
transform_header.OriginalMessageSize 0x67
transform_header.SessionId 0x8e40014000011
AES-128-CCM nonce 0x8700000000000000110000
AAD
0x
87000000000000001100001400E4080067000000000001001100001400E40800
Decrypted SMB2 packet
0x
FE534D4240000100000000000800210009000000000000000500000000000000
FFFE0000010000001100001400E4080000000000000000000000000000000000
11005000170000000000000000000000536D623320656E6372797074696F6E20
74657374696E67

References:

[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3 Specification

[SP800-108] National Institute of Standards and Technology. "Special Publication 800-108, Recommendation for Key Derivation Using Pseudorandom Functions", October 2009, http://csrc.nist.gov/publications/nistpubs/800-108/sp800-108.pdf

[RFC5084] Housley, R., "Using AES-CCM and AES-GCM Authenticated Encryption in the Cryptographic Message Syntax (CMS)", RFC 5084, November 2007, http://www.ietf.org/rfc/rfc5084.txt

SMB 3 Security Enhancements in Windows Server 2012

http://blogs.technet.com/b/filecab/archive/2012/05/03/smb-3-security-enhancements-in-windows-server-2012.aspx

Encryption in SMB3

http://blogs.msdn.com/b/openspecification/archive/2012/06/08/encryption-in-smb3.aspx