MS-RDPEUDP is a new protocol in RDP8 and operates in 2 modes : Reliable (RDP-UDP-R) and Best Efforts “Loss” (RDP-UDP-L). RDPEUDP is preferred by default if both the endpoints are RDP8 capable, however, this can be changed through Group policy (On the client side, we have Computer Configuration, Administrative Templates, Windows Components, Remote Desktop Services, Remote Desktop Connection Client: Turn off UDP On Client.We don’t have a corresponding “Turn off TCP on Client”, so the options are TCP only or TCP and UDP. On the server-side, we have Computer Configuration, Administrative Templates, Windows Components, Remote Desktop Services, and Remote Desktop Session Host: Select RDP Transport Protocols to “Use both UDP and TCP”, “Use only TCP” and “Use Either TCP or UDP”. ).
Furthermore, minencryption level (http://technet.microsoft.com/en-us/library/cc785662(v=ws.10).aspx ) MUST be set to 3 (TS_ENCRYPTION_LEVEL_HIGH) and securitylayer to 2 (TS_SECURITY_LAYER_SSL) for RDPEUDP to flow and If it’s set to any other value then RDPEUDP will not be used .In RDP7, if minencryptionlevel is set to 1 then unencrypted RDP PDUs will flow from server to client, but this will not work for RDPEUDP.
Flow of TLS/DTLS packets over UDP is bit different from traditional TLS packets exchanged on TCP connection. Primary difference being, TLS\DTLS packets including session packets are enveloped within RDPEUDP header which was not the case with TLS over TCP. Following is the related excerpt from section 1.4 of MS-RDPEMT specification :
"The TLS or DTLS handshake, as well as the encrypted payload, are embedded in the RDPUDP_SOURCE_PAYLOAD_HEADER as defined in [MS-RDPEUDP]."
In this blog, with the help of screenshot, I will show packet layout of TLS\DTLS PDUs enveloped within RDPEUDP header.
4. If the ‘RDPEUDP.Payload’ field in first ACK and Source Packets Data’ PDU has ‘’16 03 01” or “16 03 02” as starting bytes then it’s a TLS (not DTLS) packet. Provided it’s a session setup packet, it can be decoded as per instructions in Step3.
5. Likewise, we can decode remaining TLS session setup PDUs over UDP for further analysis.
Hope this blog will help in understanding RDPEUDP flow.
For win7Sp1/Windows 2008R2Sp1 refer following articles to install\configure DTLS and RDP 8.
[MS-RDPEUDP] : http://msdn.microsoft.com/en-us/library/hh536846.aspx
[MS-RDPEMT] : http://msdn.microsoft.com/en-us/library/hh554775.aspx