Decrypt my World

CAPICOM support on Windows 7

Hi all,

CAPICOM has finally been deprecated, and it won't be supported on Windows 7. This link is up to date and proposes alternatives to CAPICOM classes by using .NET classes and CryptoAPI: Alternatives to Using CAPICOM.

The following article may help if you were using CAPICOM in your web site and you are planning on using .NET now: Writing an ActiveX Control in .NET.

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)

Working with CNG in .NET 3.5

Hi all,

Do you want to work with CNG and .NET 3.5 and don't want to P/Invoke into CryptoAPI & CNG API? Check out Security.Cryptography.dll library at CLR Security:

"
Security.Cryptography.dll provides a new set of algorithm implementations to augment the built in .NET framework supported algorithms. It also provides some APIs to extend the existing framework cryptography APIs. Within this project you will find:
§ A CNG implementation of the AES, RSA, and TripleDES encryption algorithms
§ A CNG implementation of a random number generator
§ A class that allows dynamically creating algorithms both from this library as well as all of the algorithms that ship with .NET 3.5
§ An enumerator over all of the installed CNG providers on the current machine
§ Extension methods that allow access to all of the keys installed in a CNG provider, as well as all of the algorithms the provider supports
" 

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)

CryptographicException: The data to be decrypted exceeds the maximum for this modulus of 0 bytes

Hi all,

Some time ago a customer of mine was getting the following CryptographicException when calling EncryptedXml.DecryptDocument method to decrypt an XmlDocument in their .NET app:

 [System.Security.Cryptography.CryptographicException] = {"The data to be decrypted exceeds the maximum for this modulus of 0 bytes"}
_HResult = -2146233296
_HResult = 0x80131430
No Inner Exception

This was only happening with a smart card certificate associated to a specific third-party CSP (non-MS).

In order to troubleshoot this, I used my CryptoAPI Tracer script to get some traces while reproducing the issue. Thanks to my script I could see these calls that .NET is making to the third-party CSP behind the scenes (I will omit key container and CSP names as they are not needed to understand the issue):

>>>>>>>>>>>>>>>>>>>>>>

CryptAcquireContextA (0x570)

IN
pszContainer
001bf1a0 "Cert Container Name"

pszProvider
001b4698 "Third-party CSP"

dwProvType
PROV_RSA_FULL

dwFlags
0

OUT
hProv
0x1bea28

RESULT
CryptAcquireContextA (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetUserKey (0x570)

IN
hProv
0x1bea28

dwKeySpec
AT_KEYEXCHANGE

OUT
hUserKey
0x1bf1a0

RESULT
CryptGetUserKey (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_ALGID

pbData
NULL

dwDataLen
0

dwFlags
0

OUT
dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_ALGID

pbData
0x1bd228

dwDataLen
4

dwFlags
0

OUT
bData
001bd228 0000a400

dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_KEYLEN

pbData
NULL

dwDataLen
0

dwFlags
0

OUT
dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

>>>>>>>>>>>>>>>>>>>>>>

CryptGetKeyParam (0x570)

IN
hKey
0x1bf1a0

dwParam
KP_KEYLEN

pbData
0x1bd228

dwDataLen
4

dwFlags
0

OUT
bData
001bd228 00000000

dwDataLen
4

RESULT
CryptGetKeyParam (0x570) SUCCEEDED

<<<<<<<<<<<<<<<<<<<<<<

These calls mean the following: before .NET tries to decrypt the XML, it asks the CSP for some information on the key it will use to decrypt, like its length (CryptGetKeyParam API with KP_KEYLEN flag). For some unknown reason the third-party CSP is returning a length of 0 for the key (pbData points to 0), which is invalid, thus .NET won't continue with the decryption and will raise the exception we've seen. Why is the CSP returning an invalid length? Only the CSP developers can answer that question. 

In my customer's particular case, the CSP developers quickly localized and fixed the issue.

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)

X509Certificate creates temporary files which are never deleted

Hi all,

When creating an X509Certificate object in your .NET application ("X509Certificate cert = new X509Certificate(certBytes);") you may see that two 0kb files are created in the Temp folder of your Windows Server 2003, and are never deleted. After creating many of those objects the Temp folder fills up and the app won't function properly.

The cause of this issue is the following bug:

On a Windows Server 2003-based client computer, the system does not delete a temporary file that is created when an application calls the "CryptQueryObject" function

X509Certificate class uses CryptQueryObject API behind the scenes. Fortunatelly there is a fix for this referenced on the link above.

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)

How to timestamp Authenticode signatures when our proxy requires authentication

Hi all,

The other day a customer of mine was trying to add an Authenticode signature with timestamp to their PowerShell scripts with PowerShell itself and its Set-AuthenticodeSignature cmdlet. He was able to sign the scripts just fine, but when using the timestamp option the cmdlet was not honoring Internet Explorer Proxy settings and the timestamping process failed.

If you have never seen how to add such a signature with PowerShell, check the following sample:

First we can create a test certificate in i.e. a cmd.exe:

c:\>makecert -n "CN=PowerShell Local Certificate Root" -a sha1 -eku 1.3.6.1.5.5.7.3.3 -r -sv root.pvk root.cer -ss Root -sr localMachine
c:\>makecert -pe -n "CN=Powershell User" -ss MY -a sha1 -eku 1.3.6.1.5.5.7.3.3 -iv root.pvk -ic root.cer

Then we can create a signature with timestamp in PowerShell:

C:\PS>$cert=Get-ChildItem -Path cert:\CurrentUser\my -CodeSigningCert
C:\PS>Set-AuthenticodeSignature -filepath c:\notepad.exe -certificate $cert -IncludeChain All -TimeStampServer "http://timestamp.globalsign.com/scripts/timstamp.dll" 

Additionally, remember that we have several other ways to do Authenticode signing as I already commented here:

How to sign EXE files with an Authenticode certificate (VB.NET)

How to sign EXE files with an Authenticode certificate (part 2)

So my customer tried to use signtool.exe instead, with the same results: IE proxy settings are not getting used at all and they get an error like the following:

C:\Program Files\Microsoft SDKs\Windows\v6.1\Bin>signtool timestamp /v /t http://timestamp.globalsign.com/scripts/timstamp.dll c:\notepad.exe

Timestamping: c:\notepad.exe
SignTool Error: The specified timestamp server could not be reached.
SignTool Error: An error occurred while attempting to timestamp: c:\notepad.exe

Number of files successfully timestamped: 0
Number of errors: 1

Note that if we open Internet Explorer and enter http://timestamp.globalsign.com/scripts/timstamp.dll into the address bar, IE presents the credentials dialog box to send to the proxy and we can access the timestamp server without problems.

The cause of this issue is the following: 

All signing methods we saw in my previous posts (SignTool, CAPICOM, etc.) end up using SignerSignEx API to sign, and SignerTimeStampEx API to timestamp the signature. The same applies to PowerShell's Set-AuthenticodeSignature cmdlet.

SignerTimeStampEx will connect to the remote timestamp server in a very simple way, without using any proxy info or user credentials. And this is not configurable in any way.

So I'm afraid all signing methods we have at our disposal suffer from this limitation in the API.

In order to timestamp code, it is necessary to have unrestricted access to the Internet.

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Locks

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 2.

MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 2

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 1.

SubAuthentication package gets Zero LogonId

Hi all,

The other day I worked on an issue related to a SubAuthentication package that one of my customers had developed. One of the things that package was doing was getting the LogonId field of the LogonInformation struct being passed to Msv1_0SubAuthenticationFilter method by the Authentication package.

This SubAuth package had been working fine on i.e. Windows XP, but when doing interactive logon on a Windows Server 2003 Domain Controller, LogonId field was always set to Zero. But doing i.e. Network logon on the same DC returned a valid LogonId value. Why?

I debugged the issue and I found out what was going on.

SubAuth packages can only get called by the following Authentication packages: Kerberos or MSV1_0/NTLM.

MSV1_0 (MICROSOFT_AUTHENTICATION_PACKAGE_V1_0) package will provide our SubAuth dll with LogonId info, while Kerberos package won’t. Kerberos ignores LogonId on purpose. This is by design and happens in all versions of Windows.

So every time we get a valid LogonId, it is because logon is taking place through MSV1_0 Auth package. Note that LogonId is just a local ID on the machine and it's not of actual use for SubAuth packages. The purpose of SubAuth packages is not for capturing or tracking logon activities.

SubAuth package supplements part of the authentication and validation criteria used by the main Auth package. It can enforce additional restrictions to the authentication sequence when the DLL is registered under Auth0 registry value for MSV1_0 or Kerberos.

SubAuth package (Msv1_0SubAuthenticationFilter) will get called only after the corresponding Windows logon authentication is successful. The built-in Auth packages get first chance at logon authentication, before SubAuth DLL code gets invoked.

For domain user logon authentication, SubAuth DLLs installed on the DCs of that domain will be called. For local user logon authentication, SubAuth DLLs installed on that member machine will be called.

If you are simply tracking logon events, you would need to enable Auditing Account Logon and look for events logged in the Security Event Log. Audit logon events and Audit account logon events provide tracking of logons at workstations, servers, and DCs (see Audit Account Logon Events for details).

Then an application may get those events programmatically with i.e. WMI and its Win32_NTLogEvent class. I’ve seen people using this idea, for instance here.

There is one thing left to explain. Why is Kerberos always getting used on the DC when doing Interactive logon? We've already seen the Auth packages that can get used, and according to MSDN:

Kerberos
"
The Kerberos authentication package is used when logging on to a network; local logons are handled by MSV1_0.
"

MSV1_0
"
MSV1_0 also supports domain logons. MSV1_0 processes domain logons using pass-through authentication
"

Summing up, MSV1_0 deals with local logons and both MSV1_0 and Kerberos with network logons.

Now, whether Negotiate/Kerberos or NTLM/MSV1_0 is used for network logon depends on the client machine and the component that is doing the authentication. Windows logon components will always attempt to use Kerberos if they can, if there is an SPN (Service Principal Name) for the target name specified. For example, if you are accessing a server say MYSERVER that requires windows authentication and you want to use Kerberos, you have to supply a name that matches an SPN defined on the server account such as MYSERVER.MYDOMAIN.COM. Just specifying MYSERVER may use NTLM if there is no matching SPN. If there is no matching SPN, Kerberos will fall-back to NTLM.

So in my customer's environment we ended up using NTLM/MSV1_0 with network logons and most of local logons, thus getting a valid LogonId thanks to MSV1_0 Auth package.

Now, the exception for local logons are the DCs. Local logon is for local accounts only and not for AD (Active Directory) accounts. Doing an Interactive logon on a DC is not considered a local logon. Because of that we try to use Negotiate/Kerberos first as expected, and we just succeed. And we don't get LogonId because Kerberos Auth package won't provide us with it.

I hope this helps.
Regards,

Alex (Alejandro Campos Magencio)

MANAGED DEBUGGING with WINDBG. Breaking on an Exception. Part 1

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 5.

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 5

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 4.

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 4

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 3.

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 3

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 2.

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 2

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Managed Heap. Part 1.

MANAGED DEBUGGING with WINDBG. Managed Heap. Part 1

Hi all,

This post is a continuation of MANAGED DEBUGGING with WINDBG. Thread Stacks. Part 2.

Controls won't get resized once the nesting hierarchy of windows exceeds a certain depth (x64)

Hi all, welcome back,

I've been working on an issue where WM_SIZE events are not properly generated once the nesting hierarchy of windows exceeds a certain depth. This issue only occurs on current x64 Windows: like XP, Server 2008 or the latest Windows 7.

For ilustration purposes, let's imagine we have a C# application which creates a hierarchy of nested panels. Panels' OnSize handler resize their child panel so that it has the same size as the parent panel minus a border frame:

protected override void OnSizeChanged(System.EventArgs ea)
{
  if (childPanel != null)
  {
    childPanel.Size = new Size(
    this.ClientSize.Width - 2 * childPanel.Left,
    this.ClientSize.Height - 2 * childPanel.Top);
  }

  base.OnResize(ea);
}

We have around 30 nested panels. If we resize the main dialog, only the first 12-15 panels will get resized along with it. The OnSizeChanged method of the smaller panels is not getting called at all. On x86, all panels get resized, though.

Well, this behavior that we are experiencing is a design limitation on Windows kernel. Let's see this in greater detail.

In this sample application, this is how the call stack looks after several calls to WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged:

0:000> kL100
Child-SP RetAddr Call Site
00000000`002589f8 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)
00000000`00258a00 00000642`7605e77c System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds(Int32, Int32, Int32, Int32, Int32, Int32)+0x10b
00000000`00258a80 00000642`7606fcbc System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds()+0x2bc
00000000`00258ba0 00000642`7605d44b System_Windows_Forms_ni!System.Windows.Forms.Control.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x3c
00000000`00258c10 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x27b
00000000`00258dc0 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`00258e10 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`00258ec0 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`00258f00 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`00258f90 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`00259050 00000000`76f73789 USER32!DispatchClientMessage+0xc3
00000000`002590b0 00000000`770759a6 USER32!__fnINLPWINDOWPOS+0x2d
00000000`00259110 00000000`76f73a5a ntdll!KiUserCallbackDispatcherContinue
00000000`002591b8 00000642`7f67b167 USER32!ZwUserSetWindowPos+0xa
00000000`002591c0 00000642`7606fbd3 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
00000000`00259280 00000642`76067ae1 System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, Int32, Int32, Int32, Int32, Int32)+0xa3
00000000`00259360 00000642`7606764e System_Windows_Forms_ni!System.Windows.Forms.Control.SetBoundsCore(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0x251
00000000`00259470 00000642`76067423 System_Windows_Forms_ni!System.Windows.Forms.Control.SetBounds(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0xae
00000000`002594e0 00000642`80150b74 System_Windows_Forms_ni!System.Windows.Forms.Control.set_Size(System.Drawing.Size)+0x33
00000000`00259520 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)+0x1d4
00000000`002595e0 00000642`7605e77c System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds(Int32, Int32, Int32, Int32, Int32, Int32)+0x10b
00000000`00259660 00000642`7606fcbc System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds()+0x2bc
00000000`00259780 00000642`7605d44b System_Windows_Forms_ni!System.Windows.Forms.Control.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x3c
00000000`002597f0 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x27b
00000000`002599a0 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`002599f0 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`00259aa0 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`00259ae0 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`00259b70 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`00259c30 00000000`76f73789 USER32!DispatchClientMessage+0xc3
00000000`00259c90 00000000`770759a6 USER32!__fnINLPWINDOWPOS+0x2d
00000000`00259cf0 00000000`76f73a5a ntdll!KiUserCallbackDispatcherContinue
00000000`00259d98 00000642`7f67b167 USER32!ZwUserSetWindowPos+0xa
00000000`00259da0 00000642`7606fbd3 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
00000000`00259e60 00000642`76067ae1 System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, Int32, Int32, Int32, Int32, Int32)+0xa3
00000000`00259f40 00000642`7606764e System_Windows_Forms_ni!System.Windows.Forms.Control.SetBoundsCore(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0x251
00000000`0025a050 00000642`76067423 System_Windows_Forms_ni!System.Windows.Forms.Control.SetBounds(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0xae
00000000`0025a0c0 00000642`80150b74 System_Windows_Forms_ni!System.Windows.Forms.Control.set_Size(System.Drawing.Size)+0x33
00000000`0025a100 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)+0x1d4
00000000`0025a1c0 00000642`7605e77c System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds(Int32, Int32, Int32, Int32, Int32, Int32)+0x10b
00000000`0025a240 00000642`7606fcbc System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds()+0x2bc
00000000`0025a360 00000642`7605d44b System_Windows_Forms_ni!System.Windows.Forms.Control.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x3c
00000000`0025a3d0 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x27b
00000000`0025a580 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`0025a5d0 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`0025a680 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`0025a6c0 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`0025a750 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`0025a810 00000000`76f73789 USER32!DispatchClientMessage+0xc3
00000000`0025a870 00000000`770759a6 USER32!__fnINLPWINDOWPOS+0x2d
00000000`0025a8d0 00000000`76f73a5a ntdll!KiUserCallbackDispatcherContinue
00000000`0025a978 00000642`7f67b167 USER32!ZwUserSetWindowPos+0xa
00000000`0025a980 00000642`7606fbd3 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
00000000`0025aa40 00000642`76067ae1 System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, Int32, Int32, Int32, Int32, Int32)+0xa3
00000000`0025ab20 00000642`7606764e System_Windows_Forms_ni!System.Windows.Forms.Control.SetBoundsCore(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0x251
00000000`0025ac30 00000642`76067423 System_Windows_Forms_ni!System.Windows.Forms.Control.SetBounds(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0xae
00000000`0025aca0 00000642`80150b74 System_Windows_Forms_ni!System.Windows.Forms.Control.set_Size(System.Drawing.Size)+0x33
00000000`0025ace0 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)+0x1d4
00000000`0025ada0 00000642`7605e77c System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds(Int32, Int32, Int32, Int32, Int32, Int32)+0x10b
00000000`0025ae20 00000642`7606fcbc System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds()+0x2bc
00000000`0025af40 00000642`7605d44b System_Windows_Forms_ni!System.Windows.Forms.Control.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x3c
00000000`0025afb0 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x27b
00000000`0025b160 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`0025b1b0 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`0025b260 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`0025b2a0 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`0025b330 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`0025b3f0 00000000`76f73789 USER32!DispatchClientMessage+0xc3
00000000`0025b450 00000000`770759a6 USER32!__fnINLPWINDOWPOS+0x2d
00000000`0025b4b0 00000000`76f73a5a ntdll!KiUserCallbackDispatcherContinue
00000000`0025b558 00000642`7f67b167 USER32!ZwUserSetWindowPos+0xa
00000000`0025b560 00000642`7606fbd3 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
00000000`0025b620 00000642`76067ae1 System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, Int32, Int32, Int32, Int32, Int32)+0xa3
00000000`0025b700 00000642`7606764e System_Windows_Forms_ni!System.Windows.Forms.Control.SetBoundsCore(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0x251
00000000`0025b810 00000642`76067423 System_Windows_Forms_ni!System.Windows.Forms.Control.SetBounds(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0xae
00000000`0025b880 00000642`80150b74 System_Windows_Forms_ni!System.Windows.Forms.Control.set_Size(System.Drawing.Size)+0x33
00000000`0025b8c0 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)+0x1d4
00000000`0025b980 00000642`7605e77c System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds(Int32, Int32, Int32, Int32, Int32, Int32)+0x10b
00000000`0025ba00 00000642`7606fcbc System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds()+0x2bc
00000000`0025bb20 00000642`7605d44b System_Windows_Forms_ni!System.Windows.Forms.Control.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x3c
00000000`0025bb90 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x27b
00000000`0025bd40 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`0025bd90 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`0025be40 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`0025be80 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`0025bf10 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`0025bfd0 00000000`76f73789 USER32!DispatchClientMessage+0xc3
00000000`0025c030 00000000`770759a6 USER32!__fnINLPWINDOWPOS+0x2d
00000000`0025c090 00000000`76f73a5a ntdll!KiUserCallbackDispatcherContinue
00000000`0025c138 00000642`7f67b167 USER32!ZwUserSetWindowPos+0xa
00000000`0025c140 00000642`7606fbd3 mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
00000000`0025c200 00000642`76067ae1 System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(System.Runtime.InteropServices.HandleRef, System.Runtime.InteropServices.HandleRef, Int32, Int32, Int32, Int32, Int32)+0xa3
00000000`0025c2e0 00000642`7604fb99 System_Windows_Forms_ni!System.Windows.Forms.Control.SetBoundsCore(Int32, Int32, Int32, Int32, System.Windows.Forms.BoundsSpecified)+0x251
00000000`0025c3f0 00000642`7606d3bf System_Windows_Forms_ni!System.Windows.Forms.Control.System.Windows.Forms.Layout.IArrangedElement.SetBounds(System.Drawing.Rectangle, System.Windows.Forms.BoundsSpecified)+0x249
00000000`0025c490 00000642`76069f84 System_Windows_Forms_ni!System.Windows.Forms.Layout.DefaultLayout.ApplyCachedBounds(System.Windows.Forms.Layout.IArrangedElement)+0x24f
00000000`0025c580 00000642`76069d64 System_Windows_Forms_ni!System.Windows.Forms.Layout.DefaultLayout.xLayout(System.Windows.Forms.Layout.IArrangedElement, Boolean, System.Drawing.Size ByRef)+0x204
00000000`0025c6c0 00000642`76069ce6 System_Windows_Forms_ni!System.Windows.Forms.Layout.DefaultLayout.LayoutCore(System.Windows.Forms.Layout.IArrangedElement, System.Windows.Forms.LayoutEventArgs)+0x24
00000000`0025c700 00000642`76069b9d System_Windows_Forms_ni!System.Windows.Forms.Layout.LayoutEngine.Layout(System.Object, System.Windows.Forms.LayoutEventArgs)+0x26
00000000`0025c740 00000642`7604b182 System_Windows_Forms_ni!System.Windows.Forms.Control.OnLayout(System.Windows.Forms.LayoutEventArgs)+0xad
00000000`0025c780 00000642`76069278 System_Windows_Forms_ni!System.Windows.Forms.Form.OnLayout(System.Windows.Forms.LayoutEventArgs)+0x52
00000000`0025c800 00000642`760690d1 System_Windows_Forms_ni!System.Windows.Forms.Control.PerformLayout(System.Windows.Forms.LayoutEventArgs)+0x118
00000000`0025c8c0 00000642`7604bf98 System_Windows_Forms_ni!System.Windows.Forms.Control.OnResize(System.EventArgs)+0x141
00000000`0025c920 00000642`7604c0d2 System_Windows_Forms_ni!System.Windows.Forms.Form.OnResize(System.EventArgs)+0x18
00000000`0025c960 00000642`7605eb0b System_Windows_Forms_ni!System.Windows.Forms.Control.OnSizeChanged(System.EventArgs)+0x32
00000000`0025c9a0 00000642`7605e77c System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds(Int32, Int32, Int32, Int32, Int32, Int32)+0x10b
00000000`0025ca20 00000642`7606fcbc System_Windows_Forms_ni!System.Windows.Forms.Control.UpdateBounds()+0x2bc
00000000`0025cb40 00000642`7605d44b System_Windows_Forms_ni!System.Windows.Forms.Control.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x3c
00000000`0025cbb0 00000642`7610e197 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x27b
00000000`0025cd60 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Form.WmWindowPosChanged(System.Windows.Forms.Message ByRef)+0x27
00000000`0025cda0 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`0025cdf0 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`0025cea0 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`0025cee0 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`0025cf70 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`0025d030 00000000`76f73789 USER32!DispatchClientMessage+0xc3
00000000`0025d090 00000000`770759a6 USER32!__fnINLPWINDOWPOS+0x2d
00000000`0025d0f0 00000000`76f7ac4a ntdll!KiUserCallbackDispatcherContinue
00000000`0025d198 00000000`76f7ac94 USER32!ZwUserMessageCall+0xa
00000000`0025d1a0 00000000`76f7e026 USER32!RealDefWindowProcWorker+0xb1
00000000`0025d270 000007fe`fbfb30d6 USER32!RealDefWindowProcW+0x5a
00000000`0025d2b0 000007fe`fbfcddf2 uxtheme!DoMsgDefault+0x2a
00000000`0025d2e0 000007fe`fbfb6c8a uxtheme!OnDwpSysCommand+0x50
00000000`0025d310 000007fe`fbfb1711 uxtheme!_ThemeDefWindowProc+0x223
00000000`0025d3e0 00000000`76f7bb73 uxtheme!ThemeDefWindowProcW+0x11
00000000`0025d420 00000000`76f7d24a USER32!DefWindowProcW+0xe6
00000000`0025d470 00000000`76f7d80c USER32!UserCallWinProcCheckWow+0x1ad
00000000`0025d530 00000000`76f7d778 USER32!CallWindowProcAorW+0xdb
00000000`0025d580 00000642`7f67b382 USER32!CallWindowProcW+0x18
00000000`0025d5c0 00000642`7605e378 mscorwks!DoNDirectCallWorker+0x62
00000000`0025d660 00000642`760473f4 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.DefWndProc(System.Windows.Forms.Message ByRef)+0xa8
00000000`0025d750 00000642`7605da52 System_Windows_Forms_ni!System.Windows.Forms.Form.DefWndProc(System.Windows.Forms.Message ByRef)+0x74
00000000`0025d840 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x882
00000000`0025d9f0 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`0025da40 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`0025daf0 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`0025db30 00000000`76f73ee2 mscorwks!UMThunkStubAMD64+0x7a
00000000`0025dbc0 00000000`76f7bf9b USER32!UserCallWinProcCheckWow+0x163
00000000`0025dc80 00000000`76f7c20d USER32!DispatchClientMessage+0xc3
00000000`0025dce0 00000000`770759a6 USER32!__fnDWORD+0x2d
00000000`0025dd40 00000000`76f7ac4a ntdll!KiUserCallbackDispatcherContinue
00000000`0025ddc8 00000000`76f7ac94 USER32!ZwUserMessageCall+0xa
00000000`0025ddd0 00000000`76f7e026 USER32!RealDefWindowProcWorker+0xb1
00000000`0025dea0 000007fe`fbfb30d6 USER32!RealDefWindowProcW+0x5a
00000000`0025dee0 000007fe`fbfcedf5 uxtheme!DoMsgDefault+0x2a
00000000`0025df10 000007fe`fbfb6c8a uxtheme!OnDwpNcLButtonDown+0x71
00000000`0025df50 000007fe`fbfb1711 uxtheme!_ThemeDefWindowProc+0x223
00000000`0025e020 00000000`76f7bb73 uxtheme!ThemeDefWindowProcW+0x11
00000000`0025e060 00000000`76f7d24a USER32!DefWindowProcW+0xe6
00000000`0025e0b0 00000000`76f7d80c USER32!UserCallWinProcCheckWow+0x1ad
00000000`0025e170 00000000`76f7d778 USER32!CallWindowProcAorW+0xdb
00000000`0025e1c0 00000642`7f67b382 USER32!CallWindowProcW+0x18
00000000`0025e200 00000642`7605e378 mscorwks!DoNDirectCallWorker+0x62
00000000`0025e2a0 00000642`760473f4 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.DefWndProc(System.Windows.Forms.Message ByRef)+0xa8
00000000`0025e390 00000642`7605d7a0 System_Windows_Forms_ni!System.Windows.Forms.Form.DefWndProc(System.Windows.Forms.Message ByRef)+0x74
00000000`0025e480 00000642`7605d182 System_Windows_Forms_ni!System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)+0x5d0
00000000`0025e630 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
00000000`0025e680 00000642`76278bd9 System_Windows_Forms_ni!System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)+0xb5
00000000`0025e730 00000642`7f67945a System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(Int64, Int32, Int64, Int64)+0x29
00000000`0025e770 00000000`76f7d24a mscorwks!UMThunkStubAMD64+0x7a
00000000`0025e800 00000000`76f7d39e USER32!UserCallWinProcCheckWow+0x1ad
00000000`0025e8c0 00000642`7f67b167 USER32!DispatchMessageWorker+0x389
00000000`0025e940 00000642`7607f1fd mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
00000000`0025e9e0 00000642`7607dfb4 System_Windows_Forms_ni!DomainBoundILStubClass.IL_STUB(MSG ByRef)+0x11d
00000000`0025eb60 00000642`7607d7de System_Windows_Forms_ni!System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(Int32, Int32, Int32)+0x604
00000000`0025edb0 00000642`7607d20d System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)+0x59e
00000000`0025ef00 00000642`80150178 System_Windows_Forms_ni!System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)+0x6d
00000000`0025ef60 00000642`7f67ba32 WindowsApplication1!WindowsApplication1.Program.Main()+0x58
00000000`0025efa0 00000642`7f4bc645 mscorwks!CallDescrWorker+0x82
00000000`0025efe0 00000642`7f4d0496 mscorwks!CallDescrWorkerWithHandler+0xe5
00000000`0025f080 00000642`7f5aee5f mscorwks!MethodDesc::CallDescr+0x306
00000000`0025f2b0 00000642`7f5d3ba4 mscorwks!ClassLoader::RunMain+0x23f
00000000`0025f510 00000642`7f59acfa mscorwks!Assembly::ExecuteMainMethod+0xbc
00000000`0025f800 00000642`7f5e16c3 mscorwks!SystemDomain::ExecuteMainMethod+0x492
00000000`0025fdd0 00000642`7f5c641c mscorwks!ExecuteEXE+0x47
00000000`0025fe20 00000642`7ee69ade mscorwks!_CorExeMain+0xac
00000000`0025fe80 00000000`76e5495d mscoree!_CorExeMain+0x3e
00000000`0025feb0 00000000`77058791 KERNEL32!BaseThreadInitThunk+0xd
00000000`0025fee0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

A call to SetWindowPos API results in a kernel-to-usermode callback to the target WndProc in the case where the thread owns the target HWND. After enough nesting of multiple callbacks, the kernel will just stop making recursive calls to WndProcs and window messages will be dropped.

To understand this better, if we check previous call stack in more detail, we can see what we mean by "SetWindowPos results in a kernel-to-usermode callback":

0:000> kL100
Child-SP RetAddr Call Site
00000000`002589f8 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)
...
00000000`00258dc0 00000642`7605cff5 System_Windows_Forms_ni!System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)+0x52
...
00000000`00259110 00000000`76f73a5a ntdll!KiUserCallbackDispatcherContinue
00000000`002591b8 00000642`7f67b167 USER32!ZwUserSetWindowPos+0xa
...
00000000`002594e0 00000642`80150b74 System_Windows_Forms_ni!System.Windows.Forms.Control.set_Size(System.Drawing.Size)+0x33
00000000`00259520 00000642`7605eb0b WindowsApplication1!WindowsApplication1.Form1+MyPanel.OnSizeChanged(System.EventArgs)+0x1d4
...
...

Basically, we get to OnSizeChanged in parent panel. Parent panel calls set_Size on its child panel. This produces a call to SetWindowsPos API (USER32!ZwUserSetWindowPos) which produces a kernel-to-usermode callback (ntdll!KiUserCallbackDispatcherContinue) to the WndProc of the child panel which finally calls OnSizeChanged on that child panel. And so on. After enough nesting of these callbacks in the same thread, the kernel will just stop making recursive calls to WndProcs.

So why is this happening, if current default nested window limit is 50 on Windows platforms?

The root cause of the issue is available kernel stack space. In user mode, if recursive calls to a function are made enough times you will end up with a stack overflow exception. The same thing can happen in kernel mode. In this case, the application is calling SetWindowPos to resize a window. The window handles the WM_SIZE message by resizing its child windows. This operation is done recursively for however deep the window hierarchy is.

A SetWindowPos call will transition into kernel mode in order to make changes to the specified window’s position. There is then a callback from kernel mode into user mode to call the window procedure of the window to process the WM_WINDOWPOSCHANGING/WM_WINDOWPOSCHANGED messages. Once the messages are handled, the SetWindowPos call returns.

Ultimately what happens is that a stack overflow exception is generated in kernel mode. Rather than not handling the exception (which would "blue-screen" the machine) the window manager handles the exception and returns from the SetWindowsPos call. This problem is not limited to x64 Windows as it can also occur on x86 Windows, although it would take a deeper window hierarchy for the problem to occur in x86 Windows. Why the difference? Well, the size of pointers doubled from 32-bit to 64-bit and the size of the kernel mode stack did not. 

Note that this is not a bug, but a limitation. The fact of the matter is that, even if we were to increase the kernel stack size in the future, applications do not have infinite kernel stack space available. These are the only alternatives we have to deal with this:

1) Redesign the application to have less containers and reduce the number of nesting levels.

2) In order to workaround this limitation, the application has to break nesting by asynchronously setting sizes on the child control from the last control which receives the message.

2.1) In our sample application we can change OnSizeChanged method above to:

protected void ChangeSize()
{
  if (childPanel != null)
  {
    childPanel.Size = new Size(
    this.ClientSize.Width - 2 * childPanel.Left,
    this.ClientSize.Height - 2 * childPanel.Top);
  }
}

protected override void OnSizeChanged(System.EventArgs ea)
{
  if (this.Handle != null)
  {
    BeginInvoke(new MethodInvoker(ChangeSize))
  }

  base.OnResize(ea);
}

All panels get their size changed. Thanks to BeginInvoke only one call to OnSizeChanged happens in the same thread at a given time, and that breaks the nesting of multiple kernel-to-user mode callbacks within the same thread.

2.2) Another way I found to break the nesting of callbacks which may help but I haven't tried personally:

1. Select a window with simple layout of children (for example: single child - a panel with DockStyle.Fill) which will be breaking your recursion . (Not necessarily the last window which receives the positioning message)

2. Override WndProc for the window like this:

internal class MyTabPage : TabPage
{
private const int WM_WINDOWPOSCHANGING = 70;
private const int WM_SETREDRAW = 0xB;
private const int SWP_NOACTIVATE = 0x0010;
private const int SWP_NOZORDER = 0x0004;
private const int SWP_NOSIZE = 0x0001;
private const int SWP_NOMOVE = 0x0002;

[DllImport("User32.dll", CharSet=CharSet.Auto)]
public static extern int SendMessage(HandleRef hWnd, int msg, int wParam, int lParam);

[DllImport("User32.dll", ExactSpelling = true, CharSet = System.Runtime.InteropServices.CharSet.Auto)]
public static extern bool SetWindowPos(HandleRef hWnd, HandleRef hWndInsertAfter,
int x, int y, int cx, int cy, int flags);

[StructLayout(LayoutKind.Sequential)]
private struct WINDOWPOS
{
public IntPtr hwnd;
public IntPtr hwndInsertAfter;
public int x;
public int y;
public int cx;
public int cy;
public int flags;
};

private unsafe delegate void ResizeChildDelegate(WINDOWPOS * wpos);

private unsafe void ResizeChild(WINDOWPOS * wpos)
{
// verify if it's the right instance of MyPanel if needed
if ((this.Controls.Count == 1) && (this.Controls[0] is Panel))
{
Panel child = this.Controls[0] as Panel;

// stop window redraw to avoid flicker
SendMessage(new HandleRef(child, child.Handle), WM_SETREDRAW, 0, 0);

// start a new stack of SetWindowPos calls
SetWindowPos(new HandleRef(child, child.Handle), new HandleRef(null, IntPtr.Zero),
0, 0, wpos->cx, wpos->cy, SWP_NOACTIVATE | SWP_NOZORDER);

// turn window repainting back on 
SendMessage(new HandleRef(child, child.Handle), WM_SETREDRAW, 1, 0);

// send repaint message to this control and its children
this.Invalidate(true);
}
}

protected unsafe override void WndProc(ref Message m)
{
if (m.Msg == WM_WINDOWPOSCHANGING)
{
WINDOWPOS* wpos = (WINDOWPOS*)m.LParam;

Debug.WriteLine("WM_WINDOWPOSCHANGING received by " + this.Name + " flags " + wpos->flags);

if (((wpos->flags & (SWP_NOZORDER | SWP_NOACTIVATE)) == (SWP_NOZORDER | SWP_NOACTIVATE)) &&
((wpos->flags & ~(SWP_NOMOVE | SWP_NOSIZE | SWP_NOZORDER | SWP_NOACTIVATE)) == 0))
{
if ((wpos->cx != this.Width) || (wpos->cy != this.Height))
{
BeginInvoke(new ResizeChildDelegate(ResizeChild), new object[] { m.LParam });
return;
}
}
}

base.WndProc(ref m);
}
}

3. repeat for every 12-th or so window

I hope this helps.

Regards,

Alex (Alejandro Campos Magencio)