Thread local storage (TLS) is useful for application to store thread specific data. It is like most system resources (for example, Security Token leakage issue mentioned by Wei Zhao), we must balance the alloc and free behavior in time, otherwise it will be exhausted. Here I’d like to show a troubleshooting scenario while using TlsAlloc/TlsFree improperly.

 

For Windows 2000/2003 system, the max limit of TLS is 1088, for detailed information, refer to:

http://msdn2.microsoft.com/en-us/library/ms686749.aspx

 

When problem happens, we can take a hang dump of the running application (in my case, it is a web application).  This is one sample happened in X64 bit environment.

Check the TLS status in windbg:

 

0:023> !tls -1
TLS slots on thread: 790.ad8
0x0000 : 0000000000000000
0x0001 : 0000000000000000
0x0002 : 0000000000b36bc0
0x0003 : 0000000000000000
....
0x043d : 0000000000000000
0x043e : 0000000000000000
0x043f : 0000000000000000

The total TLS number is 1088 (from 0~0x43f). That's why the TlsAlloc doesn't work
--- it is out of the TLS index limitation.

 

To resolve the problem, we must ensure the related modules should always free TLS by calling TlsFree properly.  But how to check this while an application has been put into a production environment? The Debugging tool is the good tool to resolve this:

 

We can configure a customized adplus configure file first.

 

------------------------------

<ADPlus>

<Breakpoints>

      <NewBP>

          <Address> kernel32!TlsAlloc+0x5c </Address>

          <Type> BP </Type>

         <CustomActions>j (ebx = 0xFFFFFFFF) '.dump /mfFuht /u c:/temp/abc.dmp;g';'kb;r;!gle;g'; </CustomActions>

          <Actions> Log;Stack;</Actions>

         <ReturnAction> G </ReturnAction>

       </NewBP>

 

      <NewBP>

          <Address> kernel32!TlsFree+0x24 </Address>

          <Type> BP </Type>

         <CustomActions>kb;r;dq rsp+40h l1;!gle;g; </CustomActions>

          <Actions> Log;Stack;</Actions>

         <ReturnAction> G </ReturnAction>

       </NewBP>

      

</Breakpoints>

-------------------------

 

The first <NewBP> is to dump call stack when TlsAlloc is called. Espcially, It will generate dump files when TLS_OUT_OF_INDEXES error happens, TLS_OUT_OF_INDEXES is 0xFFFFFFFF actually.

The second <NewBP> is used to dump call stack, and trace the TLS index values (dq rsp+40h l1), with it, we will know if the TLS index value keeps increasing or stays flat—means balanced.

 

Then the final step is to monitor and trace this problematic application:

 

1. Create folder C:\testsymbols and c:\temp

2. Put tls.cfg in C:\

3. Run the tlssample.exe

4. In the debugging tools folder, run this command to get public symbols from Internet Symbol Server:

 

symchk -ie app.exe -s srv*C:\testsymbols*http://msdl.microsoft.com/download/symbols

 

5.

 

cscript.exe adplus.vbs -crash -pn app.exe -c c:\tls.cfg -quiet -y C:\testsymbols -o c:\temp

 

6. Keep monitoring the app.exe till final fault TLS_OUT_OF_INDEXES happens.

 

The text format debug logs in c:\temp will give us enough information to find out where the TlsAlloc is called without controls, either it happens in our modules or in third parties:

 

-------------------------------------------------------------------------------

--- Breakpoint kernel32!TlsAlloc+0x5c encountered ----

Current stack below ---

 # Child-SP          RetAddr           : Args to Child                                                           : Call Site

00 00000000`0122b940 00000000`065b56cf : 00000000`00000028 00000001`40fa7388 00000000`00000001 00000000`0122bbe8 : kernel32!TlsAlloc+0x10e

01 00000000`0122b980 00000000`05f6658d : 00000000`0a1e73d0 00000000`00000000 00000000`00000000 00000000`05df38ad : App!someFunction_0x12

……

--- Breakpoint kernel32!TlsAlloc+0x5c encountered ----

 

Current stack below ---

 # Child-SP          RetAddr           : Args to Child                                                           : Call Site

00 00000000`05bbaac0 00000000`064856cf : 00000000`00000028 00000001`00edebb0 00000000`00000001 00000000`05bbad68 : kernel32!TlsAlloc+0x10e

01 00000000`05bbab00 00000000`0642658d : 00000000`076e4bc0 00000000`00000000 00000000`00000000 00000000`05cf38ad : App!someFunction_0x12

…..

-----------------------------------------------

 

Regards,

 

Freist Li