Riham Selim's Blog

My SOS cheat sheet

2013-02-11T15:18:00Z

There are some handy sos commands that I need to use frequently while debugging, there are several other handy sos cheat sheets available that I use as well, check this one for example.

Breaking and Breakpoints

Print all CLR exceptions and continue execution

sxe -c "!pe;g" clr

You can run .logopen “C:\MyErrors.log” before this command and .LogClose after it so all you errors will be logged to the file

Break on specific exception type

!soe -Create System.Data.SqlClient.SqlException

Place a breakpoint

Check this link for more details.

!Name2EE Microsoft_TeamFoundation_Framework_Server!Microsoft.TeamFoundation.Framework.Server.TeamFoundationApplication

!DumpMT -MD 000007ff0030fe28

!bpmd -md 000007ff0030fd50

Conditional breakpoint

bp Address ".if (Condition) {OptionalCommands} .else {gc}"

Dumping objects and Arrays

Dump all objects of specific type:

000007fc76a73e48 47 6768 System.Configuration.SectionInformation

!dumpheap -type System.Configuration.SectionInformation

Recursively dump all objects of a certain type

.foreach (obj { !dumpheap -type Microsoft.TeamFoundation.Framework.Server.IdentityDescriptor -short }) { !mdt ${obj} -r}

Dump a specific property of all objects from this type

First get the offset of the property you need to print (easiest way is to just dump one of the objects and copy the offset)

0:033> !do 0000001834b180b8

Name: System.Configuration.SectionInformation

MethodTable: 000007fc76a73e48

EEClass: 000007fc76a14c58

Size: 144(0x90) bytes

File: D:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll

Fields:

MT Field Offset Type VT Attr Value Name

000007fc76a732d0 40001be 8 ...figurationSection 0 instance 0000001834b17c78 _configurationSection

000007fc76a73788 40001bf 68 ...n.SafeBitVector32 1 instance 0000001834b18120 _flags

000007fc76a73740 40001c0 70 ...SimpleBitVector32 1 instance 0000001834b18128 _modifiedFlags

000007fc76a76220 40001c1 60 System.Int32 1 instance 300 _allowDefinition

000007fc76a76280 40001c2 64 System.Int32 1 instance 100 _allowExeDefinition

000007fc76a75ed8 40001c3 10 ...nfigurationRecord 0 instance 0000000000000000 _configRecord

000007fc782faee0 40001c4 18 System.String 0 instance 000000183469a030 _configKey

000007fc782faee0 40001c5 20 System.String 0 instance 0000001834699908 _group

000007fc782faee0 40001c6 28 System.String 0 instance 0000001834699ef0 _name

000007fc782faee0 40001c7 30 System.String 0 instance 0000001834699f28 _typeName

000007fc782faee0 40001c8 38 System.String 0 instance 0000000000000000 _rawXml

000007fc782faee0 40001c9 40 System.String 0 instance 0000000000000000 _configSource

000007fc782faee0 40001ca 48 System.String 0 instance 0000000000000000 _configSourceStreamName

000007fc76a7a048 40001cb 50 ...igurationProvider 0 instance 0000000000000000 _protectionProvider

000007fc782faee0 40001cc 58 System.String 0 instance 0000000000000000 _protectionProviderName

000007fc76a73a50 40001cd 78 ...errideModeSetting 1 instance 0000001834b18130 _overrideModeDefault

000007fc76a73a50 40001ce 80 ...errideModeSetting 1 instance 0000001834b18138 _overrideMode

And then loop through all objects of this type and dump this property

.foreach (foo {!DumpHeap -type System.Configuration.SectionInformation -short}) {!do poi(${foo}+28)}

Sample Output

Name: System.String

MethodTable: 000007fc782faee0

EEClass: 000007fc77c63720

Size: 32(0x20) bytes

File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String: uri

Fields:

MT Field Offset Type VT Attr Value Name

000007fc782fdc90 40000aa 8 System.Int32 1 instance 3 m_stringLength

000007fc782fc1c8 40000ab c System.Char 1 instance 75 m_firstChar

000007fc782faee0 40000ac 18 System.String 0 shared static Empty

>> Domain:Value 0000001833796360:NotInit 00000018337ed9e0:NotInit <<

Name: System.String

MethodTable: 000007fc782faee0

EEClass: 000007fc77c63720

Size: 60(0x3c) bytes

File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String: webRequestModules

Fields:

MT Field Offset Type VT Attr Value Name

000007fc782fdc90 40000aa 8 System.Int32 1 instance 17 m_stringLength

000007fc782fc1c8 40000ab c System.Char 1 instance 77 m_firstChar

000007fc782faee0 40000ac 18 System.String 0 shared static Empty

>> Domain:Value 0000001833796360:NotInit 00000018337ed9e0:NotInit <<

Name: System.String

MethodTable: 000007fc782faee0

EEClass: 000007fc77c63720

Size: 50(0x32) bytes

File: D:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String: defaultProxy

Fields:

MT Field Offset Type VT Attr Value Name

000007fc782fdc90 40000aa 8 System.Int32 1 instance 12 m_stringLength

000007fc782fc1c8 40000ab c System.Char 1 instance 64 m_firstChar

000007fc782faee0 40000ac 18 System.String 0 shared static Empty

>> Domain:Value 0000001833796360:NotInit 00000018337ed9e0:NotInit <<

Performance Testing Cloud Hosted Applications

2012-11-16T19:00:00Z

Building while flying

Performance testing in the cloud can be quite challenging, the best video I have seen so far was one that our test architect Dennis Bass used to show to people asking about what testing in the cloud is like (I am also borrowing a lot of content from him)

http://www.youtube.com/watch?v=L2zqTYgcpfg

The video says it all: the infrastructure, environment, tools, basically everything you rely on is changing, all while you are building your application

Performance matters..

..even more in the cloud, more compute time = more money.

Of course, if your performance sucks, nobody is really going to use your service – but even small performance variations can affect your cost.

For example, while these two code snippets look kind of the same, one of them costs 50% more than the other.

Code 1:

if(ServiceGuid.ToString().Equals(c_AccountService)

{

//Do Something

}

else if(ServiceGuid.ToString().Equals(c_UseService)

{

}

Code 2:

String guid = ServiceGuid.ToString();

 if(guid.Equals(c_AccountService)

 //Do Something

 else if(guid.Equals(c_UserService)

//

(and yes, I know FxCop will scream about my string compare, that’s just an example)

The costs really add up at the end. You need to be really smart about the number and size of your instances

From MSDN:

Windows Azure Compute cost is based on size and instances. The formula is simple:

Size *total instances

The critical factor is size. Choices span from "extra small" to "extra large." (For a list of size details, seeWindows Azure Compute, and How To Configure Virtual Machine Sizes.) To complicate this, running many instances of an extra small machine can be more effective than running one instance of an extra large machine. The final choice depends on the application. If the application does intensive processing, then you require an extra large (or large, or medium) instance. In contrast, you can have an application that does little processing but serves many users. In that case, a small or extra small instance that scales out to many instances in response to demand may be all that is needed.

Planning for your testing

- Estimate the costs of your testing, be realistic about it and set the expectations.

o Testing in the cloud costs thousands, while you can eliminate some of the costs but using simulators or on premise testing, you will need to do final verification on the cloud!

- Add time and resources

o Application and test engineering

o Additional network bandwidth

- Operational testing needs

o Elasticity, Patching

o Backup/Restore, Disaster Recovery

- Deployment

- Test Data

o E.g. Unique Internet Accounts

- Have the right diagnostics in place.

- Keep a heartbeat or ping test running.

- Set realistic expectations

o The cloud is designed for scale, not for performance!

Keep in mind, if anything can go wrong, it will

Even more in the cloud, multitenancy, network issues and all kinds of problems really - Dependent Services will go up/down, speed up, slow down, be updated,..

- Create your own Chaos Monkey or use an existing one, NetFlix team have shared their Chaos Monkey for AWS, also you can try Steve Marx’ Chaos money for Azure.

- RETRY: Everything, all the time, until you are 100% sure it is really broken, try to fail gracefully as much as you can (this will be more common than you think!

It is not always your fault

No matter how reliable your code is, other tenants code might not be as reliable, they might abuse the resources and slow you down, so no matter how confident you are about your code make sure you handle extreme conditions like high CPU and low memory accordignly, this is especially important with SQL Azure.

Test Tools

In the next post, I will be writing about the test tools and strategies for testing a hosted application, stay tuned!

Windbg - Breaking on specific CLR exception

2012-11-15T17:39:00Z

I get this question quite frequently, "I know how to break on all CLR exceptions, but how do I break on a specific exception, e.g. I want to see only Sql Exceptions?"

It is actually quite easy to do in WinDbg using the soe (StopOnException) command found in sos.dll

StopOnException [-derived] [-create | -create2]

<Exception> <Pseudo-register number>

Causes the debugger to stop when the specified exception is thrown,

but to continue running when other exceptions are thrown.

The -derived option catches the specified exception and every

exception that derives from the specified exception.

So after you load SOS (Check Windbg QuickStart guide for instructions on how to do that) you need to run the following:

0:052> !soe -Create System.Data.SqlClient.SqlException

Breakpoint set

0:052> g

(234c.1b28): CLR exception - code e0434352 (first chance)

CLR exception type: System.Net.WebException

"The remote server returned an error: (304) Not Modified."

(234c.1b28): CLR exception - code e0434352 (first chance)

CLR exception type: System.Net.WebException

"The remote server returned an error: (304) Not Modified."

(234c.1b28): CLR exception - code e0434352 (first chance)

(234c.1cd8): CLR exception - code e0434352 (first chance)

'System.Data.SqlClient.SqlException hit'

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

KERNELBASE!RaiseException+0x3d:

000007fe`fdb5cacd 4881c4c8000000 add rsp,0C8h

You will notice that the debugger is skipping all exceptions and breaking only on the exception type you are interested in.

Windbg QuickStart guide (Part 4 - Debugging unexpected program behavior)

2012-03-14T23:31:00Z

Let’s assume I have written some really ugly code like what you see here:

1:  int i = 0;

2:  try

3:  {

4:      i = int.Parse(args[0]);

5:  }

6:  catch

7:  {

8:      Console.WriteLine("Invalid Argument");

9:  }

10: Console.WriteLine(i);

When the user executes the program with invalid arguments, all he will see is an error that the argument is invalid, let’s execute the program with “1”, the user is expecting the output to be 1, but he gets a zero, yet there is no error or exception.

In this case you need to catch the first chance exception, i.e. you want to break when an exception happens even if it is handled in the code.

To break on first change exception you use the sx command, From MSDN documentation:

The sx* commands control the action that the debugger takes when an exception occurs in the application that is being debugged, or when certain events occur.

In this case we want to break on an exception, so we use sxe and since we are interested in CLR exceptions we do sxe clr

0:000> sxe clr

0:000> g

Once I enter my input I see this:

(182c.163c): CLR exception - code e0434352 (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

eax=001aedc4 ebx=00000005 ecx=00000005 edx=00000000 esi=001aee84 edi=00000001

eip=7613b9bc esp=001aedc4 ebp=001aee14 iopl=0 nv up ei pl nz ac po nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212

KERNELBASE!RaiseException+0x58:

7613b9bc c9 leave

If I do !pe – this is what I get:

0:000> !pe

Exception object: 02563754

Exception type: System.FormatException

Message: Input string was not in a correct format.

InnerException: <none>

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131537

To see the stack, I do !clrstack

0:000> !clrstack

OS Thread Id: 0x163c (0)

Child SP IP Call Site

001aeed8 7613b9bc [HelperMethodFrame: 001aeed8]

001aef94 6be0d5b4 System.Number.StringToNumber(System.String, System.Globalization.NumberStyles, NumberBuffer ByRef, System.Globalization.NumberFormatInfo, Boolean) [f:\dd\ndp\clr\src\BCL\System\Number.cs @ 1029]

001aefbc 6b41bad2 System.Number.ParseInt32(System.String, System.Globalization.NumberStyles, System.Globalization.NumberFormatInfo) [f:\dd\ndp\clr\src\BCL\System\Number.cs @ 712]

001af088 6b48bc20 System.Int32.Parse(System.String) [f:\dd\ndp\clr\src\BCL\System\Int32.cs @ 140]

001af094 002e008a BuggyApp.Program.Main(System.String[]) [d:\Projects\BuggyApp\Program.cs @ 15]

001af23c 6f5855e2 [GCFrame: 001af23c]

Another way to do this is to tell the debugger the execute !pe and !clrstack immediately after seeing the exception:

This is the debugger input:

0:000> sxe -c "!pe;!clrstack" clr

0:000> g

(828.d2c): C++ EH exception - code e06d7363 (first chance)

(828.d2c): CLR exception - code e0434352 (first chance)

Exception object: 025e2268

Exception type: System.IndexOutOfRangeException

Message: Index was outside the bounds of the array.

InnerException: <none>

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80131508

OS Thread Id: 0xd2c (0)

Child SP IP Call Site

0028ed84 7613b9bc [HelperMethodFrame: 0028ed84]

0028edf4 00320082 *** WARNING: Unable to verify checksum for BuggyApp.exe

BuggyApp.Program.Main(System.String[]) [d:\Projects\BuggyApp\Program.cs @ 15]

0028ef9c 6f5855e2 [GCFrame: 0028ef9c]

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

eax=0028ec44 ebx=00000005 ecx=00000005 edx=00000000 esi=0028ed04 edi=00000001

eip=7613b9bc esp=0028ec44 ebp=0028ec94 iopl=0 nv up ei pl nz ac pe nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216

KERNELBASE!RaiseException+0x58:

7613b9bc c9 leave

Hope you find that helpful, feel free to post any questions!

Breaking on Module Load

2012-03-14T23:23:09Z

Let’s say we want to break the debugger when a specific dll is loaded, for example I want to enable some SOS commands and clr hasn’t been loaded yet, this is especially useful when you have an exception that happens too early in the program and you can’t rely on manually trying to break at the right time.

For example, right after attaching the debugger to a process I do this, I will get an error since clr hasn’t been loaded yet

0:000> .loadby sos clr

Unable to find module 'clr'

Using the sxe ld command, I can tell the debugger to break when clr is loaded

0:000> sxe ld clr

0:000> g

Right away I get this:

ModLoad: 6f580000 6fc03000 C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll

eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=7efdd000 edi=003cf400

eip=77d7fc42 esp=003cf2d4 ebp=003cf328 iopl=0 nv up ei pl zr na pe nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246

ntdll!NtMapViewOfSection+0x12:

77d7fc42 83c404 add esp,4

Now the command will succeed!

0:000> .loadby sos clr

Windbg QuickStart guide (Part 3 - Crashes)

2012-03-08T07:46:00Z

The approach you will take for debugging crashes is very similar between looking at the live process vs looking at a crash dump, there are some small differences that I will talk about in this post.

The first question: What is wrong?

If the program is throwing an exception and exiting

If the program has just crashed, the easiest thing to do it to attach a debugger to the process and see the last exception in the stack

0:005> .loadby sos clr

The !threads command will show me all threads, the thread with an exception will have an exception on top of the stack

0:005> !threads

PDB symbol for clr.dll not loaded

ThreadCount: 2

UnstartedThread: 0

BackgroundThread: 1

PendingThread: 0

DeadThread: 0

Hosted Runtime: no

Lock

ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception

0 1 14d0 00492eb8 2a020 Preemptive 02639428:00000000 00483e38 0 MTA System.InvalidOperationException 024f2968

3 2 14f8 0049dd78 2b220 Preemptive 00000000:00000000 00483e38 0 MTA (Finalizer)

Now we use the !pe command to dump the exception object

0:005> !pe 024f2968

Exception object: 024f2968

Exception type: System.InvalidOperationException

Message: Path does not exist!

InnerException: <none>

StackTrace (generated):

SP IP Function

001FECE8 00520155 CompareCSV!CompareCSV.Program.Main(System.String[])+0xd5

StackTraceString: <none>

HResult: 80131509

Ok, now we know that it is a path does not exist exception, but which path? We can do this in two ways, if it is a small program like the one I have here I could do !dso

0:000> !dso

OS Thread Id: 0x11fc (0)

ESP/REG Object Name

001FEB88 024f2968 System.InvalidOperationException

001FEBD0 024f2968 System.InvalidOperationException

001FEC20 024f22a4 System.Object[] (System.String[])

001FEC50 024f22a4 System.Object[] (System.String[])

001FEC98 024f22a4 System.Object[] (System.String[])

001FECA8 024f22bc System.String c:\invalid1

001FECC8 024f2968 System.InvalidOperationException

001FECCC 024f2968 System.InvalidOperationException

001FECE8 024f29bc System.String Path does not exist!

001FECEC 024f2968 System.InvalidOperationException

001FECF8 024f22a4 System.Object[] (System.String[])

001FED90 024f22a4 System.Object[] (System.String[])

001FEEF4 024f22a4 System.Object[] (System.String[])

001FEF2C 024f22a4 System.Object[] (System.String[])

Another way to do this would be showing the CLRStack with the parameters

0:000> !clrstack -p

OS Thread Id: 0x11fc (0)

Child SP IP Call Site

001fec2c 7613b9bc [HelperMethodFrame: 001fec2c]

001fece8 00520155 CompareCSV.Program.Main(System.String[]) [d:\Projects\CompareCSV\CompareCSV\Program.cs @ 25]

PARAMETERS:

args (0x001fecf8) = 0x024f22a4

And now I can dump the args passed to main, the !do command will not be very helpful since it is an array

0:000> !do 0x024f22a4

Name: System.String[]

MethodTable: 6b4dfc38

EEClass: 6b1b7a50

Size: 24(0x18) bytes

Array: Rank 1, Number of elements 2, Type CLASS

Fields:

None

The right thing to use is !da, which will list the array items

0:000> !da 0x024f22a4

Name: System.String[]

MethodTable: 6b4dfc38

EEClass: 6b1b7a50

Size: 24(0x18) bytes

Array: Rank 1, Number of elements 2, Type CLASS

Element Methodtable: 6b52cb5c

[0] 024f22bc

[1] 024f22e0

Now I can dump the objects and see the invalid paths!

0:000> !do 024f22bc

Name: System.String

MethodTable: 6b52cb5c

EEClass: 6b15431c

Size: 36(0x24) bytes

File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String: c:\invalid1

Fields:

MT Field Offset Type VT Attr Value Name

6b52e784 40000aa 4 System.Int32 1 instance 11 m_stringLength

6b52d564 40000ab 8 System.Char 1 instance 63 m_firstChar

6b52cb5c 40000ac c System.String 0 shared static Empty

>> Domain:Value 003f7838:NotInit <<

0:000> !do 024f22e0

Name: System.String

MethodTable: 6b52cb5c

EEClass: 6b15431c

Size: 36(0x24) bytes

File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll

String: c:\invalid2

Fields:

MT Field Offset Type VT Attr Value Name

6b52e784 40000aa 4 System.Int32 1 instance 11 m_stringLength

6b52d564 40000ab 8 System.Char 1 instance 63 m_firstChar

6b52cb5c 40000ac c System.String 0 shared static Empty

>> Domain:Value 003f7838:NotInit <<

If the program is Behaving in an unexpected way

There are various types of problems that could cause unexpected behavior, but the one I hit the most is first chance exceptions taking the code in some unexpected path, for more details check This post.

Hope you find that helpful!

Error loading sos.dll

2012-03-04T07:23:00Z

Many times you get this error when you do .load sos

0:080> .load sos

The call to LoadLibrary(sos) failed, Win32 error 0n2
"The system cannot find the file specified."
Please check your debugger configuration and/or network access.
You can solve this by doing .loadby sos clr, this means you are trying to load sos.dll from the same location you loaded clr.dll from.

This is a different error

0:091> .loadby sos clr

Unable to find module 'clr'

The first question I ask myself is whether I am debugging a 32 bit application with 64 bit debugger, if not the I do lm to list all loaded modules

In this case I saw that clr.dll is not loaded, but mscorwks is this is because the application I am debugging is using .net framework 2.0

0:091> lm

start end module name

00000000`1b6d0000 00000000`1b9cf000 System_Data (deferred)

00000000`1e230000 00000000`1e279000 System_Transactions (deferred)

00000000`714b0000 00000000`71556000 System_Core (deferred)

00000000`77610000 00000000`777b9000 ntdll (private pdb symbols) d:\symbols\ntdll.pdb\15EB43E23B12409C84E3CC7635BAF5A32\ntdll.pdb

00000642`ffaf0000 00000642`ffb09000 mscorsec (deferred)

000007fe`dae40000 000007fe`db68b000 System_Data_ni (deferred)

000007fe`e07b0000 000007fe`e0945000 System_DirectoryServices_ni (deferred)

000007fe`e0950000 000007fe`e0a5d000 System_EnterpriseServices_ni (deferred)

000007fe`e0a60000 000007fe`e0b45000 System_Transactions_ni (deferred)

000007fe`eb380000 000007fe`ec25c000 mscorlib_ni (deferred)

000007fe`ec260000 000007fe`ecbfc000 mscorwks (deferred)

...

0:091> .loadby sos mscorwks

That worked!

Unfortunately sometimes is it not that simple, I get this nasty error a lot

Failed to load data access DLL, 0x80004005
Verify that

1) you have a recent build of the debugger (6.2.14 or newer)

            2) the file mscordacwks.dll that matches your version of mscorwks.dll is
                in the version directory
            3) or, if you are debugging a dump file, verify that the file
                mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
            4) you are debugging on the same architecture as the dump file.
                For example, an IA64 dump file must be debugged on an IA64
                machine.

.cordll -ve -u -l works most of the time

if that still doesn't work, try following the steps here

Hope that helps!

Windbg QuickStart guide (Part 2 - hangs)

2012-03-02T03:22:00Z

In part1 we have discussed the steps you need to get ready to start debugging with windbg, in this part we will walk through some steps and commands that might help you troubleshoot a specific problem

The first question is what are you trying to debug?

Troubleshooting a hang

a. The first step is to check if this is a high CPU or low CPU hang?

Open task manager (Start->Run->taskmgr)
Check the cpu for your process, If the CPU is consistently high then you need to investigate a High CPU hang, alternatively you need to investigate low CPU hang.

High CPU hangs

1- Are we spending too much time in garbage collection? Check the following perf counters in perfmon:

.Net CLR Memory
% Time in GC
Allocated bytes/sec

Are these too high? If so you can follow this post

Also detailed information about .net perf counters is available here

2- If that’s not the case, the question is which thread is consuming the CPU? – for this we use !runaway command

0:000> !runaway

User Mode Time

Thread Time

1:358 0 days 0:00:47.408

2:150 0 days 0:00:03.495

0:d8 0 days 0:00:00.000

Ok, now we know that both threads 1:358 and 2:150 are consuming a lot of CPU time, but which one is really stuck? Continue the process for a few more seconds by doing

0:000> g

Or pressing F5 After a few seconds we try again:

0:000> !runaway

User Mode Time

Thread Time

1:358 0 days 0:00:47.408

2:150 0 days 0:00:06.395

0:d8 0 days 0:00:00.000

We can tell easily that thread 2:150 is taking more CPU that it was initially – so we know which thread is stuck, now we can switch the context to this thread and start debugging it:

0:000> ~2s

And then show the managed stack and unmanaged stacks

0:000> !clrstack

What to look for?

Infinite loop
Intensive IO operations
Huge allocations to the Large Object Heap
Any intensive CPU operations.

Useful commands and sequences

~*e !clrstack -> show all managed stacks
~*k -> show all stacks (unmanaged as well)
!bpmd mscorlib.dll System.Threading.Thread..ctor (place a managed breakpoint) can help identifying infinite loops if the stack is too deep.

Low CPU hangs

Critical section

Check the stack traces for all your threads are you seeing a similar pattern?

0:000> ~* kb50

1 Id: 6fc.3d8 Suspend: 1 Teb: 7ffde000 Unfrozen

ChildEBP RetAddr Args to Child

005afc14 7c90e9c0 7c91901b 000007d4 00000000 ntdll!KiFastSystemCallRet

005afc18 7c91901b 000007d4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc

005afca0 7c90104b 004a0638 00430b7f 004a0638 ntdll!RtlpWaitForCriticalSection+0x132

005afca8 00430b7f 004a0638 005afe6c 005afe78 ntdll!RtlEnterCriticalSection+0x46

· !locks command can help us to identify the current owner of this critical section. When used without parameters, this command displays the list of critical sections that are currently held by the application's threads.

· !dlk

Deadlock

Alternatively, you might be hitting a deadlock – try the following command:

0:000> !syncblk

Useful commands and sequences

- ~*e !clrstack -> show all managed stacks

- ~*k -> show all stacks (unmanaged as well)

- !dso ->Dump stack objects

Waiting on SQL locks

Here is another interested deadlock I hit today – after doing ~*e!clrstack I found that all my threads were stuck here:

OS Thread Id: 0xb54 (35)

Child SP IP Call Site

000000002ccdca60 00000000772a78ca [NDirectMethodFrameStandalone: 000000002ccdca60] <Module>.SNIReadSync(SNI_Conn*, SNI_Packet**, Int32)

000000002ccdca20 000007ff00aeaf91 SNINativeMethodWrapper.SNIReadSync(System.Runtime.InteropServices.SafeHandle, IntPtr ByRef, Int32)

000000002ccdcb20 000007ff00aeaafc 000000002ccdcbc0 000007ff00aea870 System.Data.SqlClient.TdsParserStateObject.ReadNetworkPacket()

000000002ccdcc10 000007ff00aedfd4 System.Data.SqlClient.TdsParserStateObject.ReadByte()

000000002ccdcc40 000007ff00aed4fd System.Data.SqlClient.TdsParser.Run(System.Data.SqlClient.RunBehavior, System.Data.SqlClient.SqlCommand, System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.BulkCopySimpleResultSet, System.Data.SqlClient.TdsParserStateObject)

000000002ccdcd10 000007ff00af5538 System.Data.SqlClient.SqlDataReader.ConsumeMetaData()

000000002ccdcd60 000007ff00af5342 System.Data.SqlClient.SqlDataReader.get_MetaData()

000000002ccdcdb0 000007ff00af4bf1 System.Data.SqlClient.SqlCommand.FinishExecuteReader(System.Data.SqlClient.SqlDataReader, System.Data.SqlClient.RunBehavior, System.String)

000000002ccdce20 000007ff00af3c0e System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, Boolean)

000000002ccdcef0 000007ff00af23a0 System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String, System.Data.Common.DbAsyncResult)

000000002ccdcf90 000007ff00af219c System.Data.SqlClient.SqlCommand.RunExecuteReader(System.Data.CommandBehavior, System.Data.SqlClient.RunBehavior, Boolean, System.String)

000000002ccdcfd0 000007ff00af1df5 System.Data.SqlClient.SqlCommand.ExecuteScalar()

This is very interesting, why are all the threads stuck executing a SQL query?

Upon doing !dso, I found that this statement is being executed

sp_getapplock

So interestingly this time we got blocked waiting on a SQL lock, not an app one

Part 3 is coming soon with details on how to debug crashes, high cpu and memory problems!

Windbg QuickStart guide (Part 1)

2012-02-25T13:29:00Z

The purpose of this post is basically to give people a quick start guide to windbg, over the next few posts I will add more details on specific problems, so you can go directly to the steps to debug your specific problem, hope you find it helpful!

Part 1: Before you start

Step1: Get the tools

This tutorial is using Debugging tools for windows, the first question you need to ask yourself is whether the application you are debugging is from a 32 bit or 64 bit application? You can download the tools from here:

32 bit application

64 bit application

Helpful tip

If you are debugging a 32 bit application running on 64 bit platform, you will find this page very helpful to troubleshoot different problems

Step2: Start debugging

Are you debugging a live application or a minidmp?
Live application: File->Attach to process and select your process

Minidmp: File->Open and open your minidmp.

Step3: Load your symbols

Assuming your symbols are at \\symbolsshare\symbols

You will want to cache the symbols locally on your machine, you will also want to cache the windows symbols locally at c:\symbols

.sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols (Tip: .symfix c:\symbols will do the same thing!)

.sympath+ srv*c:\symbols*\\symbolsshare\symbols
.reload

.symfix c:\symbols

.sympath+ srv*c:\symbols*\\symbolsshare\symbols

.reload

Tip: .symfix c:\symbols is the same as .sympath srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Tip: To troubleshoot symbol loading problems, check this post

Step4: Load SOS

SOS.dll is a debugging extension that helps you debug managed programs in Visual Studio and in the Windows debugger (WinDbg.exe) by providing information about the internal common language runtime (CLR) environment. SOS.dll is automatically installed with the .NET Framework.

.loadby sos clr

Tip: if sos.dll is not loading as expected, check the post Error Loading sos.dll

Step 5:
Now you can are ready to move to the next step, depending on your specific problem:

More tutorials to be added soon, please list any specific problem you have and I will work on including it, hope you find that helpful!

Remotely enabling windows azure diagnostics

2011-01-11T21:26:00Z

I frequently need to Enable and configure collecting performance counters remotely in my windows azure deployment, There are a few tasks you are trying to accomplish

1- Get all roles you have for your deployment

2- Get the Diagnostics manager associated with each instance of each role

3- Set your Windows Azure Diagnostics Settings.

I created the following helper class to make it easier for me, hope you find it useful:)

   1:          #region Default Counters

   2:          static string[] defaultCounterNames = new string[]

   3:          {

   4:              @"\Memory\Available Mbytes",

   5:              @"\Processor(_Total)\% Processor Time",

   6:              @"\Processor(*)\% Processor Time",

   7:              @"\Process(*)\Working Set",

   8:              @"\Process(*)\Private Bytes",

   9:              @"\Process(*)\Working Set Peak",

  10:              @"\Process(*)\Virtual Bytes",

  11:              @"\Process(*)\Virtual Bytes Peak"

  12:          };

  13:          #endregion

14:

  15:          public static void SetState(string connectionString, string deploymentId , int sampleRateInSeconds, int transferPeriodInSeconds, string[] counterNames = null)

  16:          {

  17:              if (counterNames == null || counterNames.Length == 0)

  18:              {

  19:                  counterNames = defaultCounterNames;

  20:              }

21:

  22:              CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(connectionString);

23:

  24:              //TODO: Need to remove this if HTTPS is enabled - this allows connecting though http, otherwise the connection will fail.

  25:              DeploymentDiagnosticManager.AllowInsecureRemoteConnections = true;

26:

  27:              //Get the diagnostis manager associated with this blob storage.

  28:              DeploymentDiagnosticManager deploymentDiagnosticsManager = new DeploymentDiagnosticManager(cloudStorageAccount, deploymentId);

29:

  30:              //Get the role instance diagnostics manager for all instance of the a role

  31:              var roles = deploymentDiagnosticsManager.GetRoleNames();

  32:              foreach (string item in roles)

  33:              {

  34:                  //Get the Role instance Diagnostics manager for each instance. and use it to enable data collection

  35:                  var roleInstanceManagers = deploymentDiagnosticsManager.GetRoleInstanceDiagnosticManagersForRole(item);

  36:                  RoleInstanceDiagnosticManager.AllowInsecureRemoteConnections = true;

  37:                  Console.Out.WriteLine("Getting Diagnostics Managers for Azure Role '{0}'", item);

38:

  39:                  //Set the new diagnostic monitor configuration for each instance of the role

  40:                  foreach (var ridmN in roleInstanceManagers)

  41:                  {

  42:                      Console.Out.WriteLine("\tEnabling counters on instance {0} of role {1}", ridmN.RoleInstanceId, ridmN.RoleName);

43:

  44:                      EnableCounters(sampleRateInSeconds, transferPeriodInSeconds, counterNames, ridmN);

  45:                   }

  46:              }

  47:          }

This is the code that enables the counters for each role instance:

   1:  private static void EnableCounters(int sampleRateInSeconds, int transferPeriodInSeconds, string[] counterNames, RoleInstanceDiagnosticManager ridmN)

   2:          {

   3:              DiagnosticMonitorConfiguration dmc = ridmN.GetCurrentConfiguration();// Depending on what you are trying to fo this can be DiagnosticMonitor.GetDefaultInitialConfiguration();

4:

   5:              counterNames.ToList<String>().ForEach(counterName =>

   6:              {

   7:                  string counterNameTrimmed = counterName.Trim();

   8:                  if (!string.IsNullOrEmpty(counterNameTrimmed))

   9:                  {

  10:                      PerformanceCounterConfiguration pcc = new PerformanceCounterConfiguration();

  11:                      pcc.CounterSpecifier = counterNameTrimmed;

  12:                      pcc.SampleRate = TimeSpan.FromSeconds(sampleRateInSeconds);

  13:                      dmc.PerformanceCounters.DataSources.Add(pcc);

  14:                      Console.Out.WriteLine("\t\tCounter {0} Sample Rate {1} seconds", counterName, sampleRateInSeconds);

  15:                  }

  16:              });

17:

  18:              dmc.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromSeconds(transferPeriodInSeconds);

  19:              ridmN.SetCurrentConfiguration(dmc);

  20:          }

Retrieving more than 1000 rows from windows azure storage

2011-01-06T22:15:00Z

I recently hit an issue with my code when trying to retrieve data from Windows Azure Diagnostics table, I noticed that I'm getting only 1000 rows every time even though I have not set any limit on the number of rows I want to retrieve, it turns out that there is a limit to 1000 rows for windows azure storage.

The limit is not only on the number of rows, there is also a limit on the amount of time it takes the query to execute, the Table service may return a maximum of 1,000 items at one time and make take only up to 5 seconds. If the result set contains more than 1,000 items or takes too long to execute the table service will return a Continuation Token that you can use to retrieve the rest of your rows data, when you received all your data your token is null.

Originally this was my code:

1: WADPerformanceCountersTableContext wadContext = new WADPerformanceCountersTableContext(cloudStorageAccount.TableEndpoint.ToString(), cloudStorageAccount.Credentials);

2:

   3:  var items = from x in wadContext.WADPerformanceCountersTable

   4:  where x.EventTickCount >= start.Ticks && x.EventTickCount <= end.Ticks

   5:  select x;

6:

   7:  foreach (WADPerformanceCountersTable item in items)

   8:  {

   9:     //use item

  10:  }

This is the code after adding the use for continuation token:

1: ListRowsContinuationToken continuationToken = null;

2:

   3:  CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(connectionString);

4:

   5:  WADPerformanceCountersTableContext wadContext = new WADPerformanceCountersTableContext(cloudStorageAccount.TableEndpoint.ToString(), cloudStorageAccount.Credentials);

6:

7:

   8:  do

   9:  {

  10:      //Query using the start time and end time, take only n rows

  11:      //The response headers will contain a continuation key that can be used to retrieve the next n rows

  12:      var allItems = (from x in wadContext.WADPerformanceCountersTable

  13:          where x.EventTickCount >= start.Ticks && x.EventTickCount <= end.Ticks

  14:          select x).Take(c_pageSize);

15:

  16:      var query = allItems as DataServiceQuery<WADPerformanceCountersTable>;

17:

  18:      if (continuationToken != null)

  19:      {

  20:          query = query.AddQueryOption("NextPartitionKey", continuationToken.PartitionKey);

  21:          if (continuationToken.RowKey != null)

  22:          {

  23:             query = query.AddQueryOption("NextRowKey", continuationToken.RowKey);

  24:          }

  25:      }

26:

  27:      var response = query.Execute() as QueryOperationResponse;

28:

  29:      foreach (WADPerformanceCountersTable item in allItems)

  30:      {

  31:          //use item

  32:      }

  33:      //now check if there are more rows left to be retrieved, if there are more rows execute another request to get the remaining items

  34:      if (response.Headers.ContainsKey("x-ms-continuation-NextPartitionKey"))

  35:      {

  36:          continuationToken = new ListRowsContinuationToken();

  37:          continuationToken.PartitionKey = response.Headers["x-ms-continuation-NextPartitionKey"];

  38:          if (response.Headers.ContainsKey("x-ms-continuation-NextRowKey"))

  39:          {

  40:             continuationToken.RowKey = response.Headers["x-ms-continuation-NextRowKey"];

  41:          }

  42:      }

  43:      else

  44:      {

  45:          continuationToken = null;

  46:      }

47:

  48:  } while (continuationToken != null);

49:

ListRowsContinuationToken is a simple class defined as follows:

   1:  public class ListRowsContinuationToken

   2:  {

   3:      public string PartitionKey { get; set; }

   4:      public string RowKey { get; set; }

   5:  }

Finally, I found an even easier way to get this done! The trick is to convert the query into a CloudTableQuery - once you do this you will be able to retrieve all rows with a single call (just make sure that this is what you really want to do)

   1:  CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(connectionString);

   2:  WADPerformanceCountersTableContext wadContext = new WADPerformanceCountersTableContext(cloudStorageAccount.TableEndpoint.ToString(), cloudStorageAccount.Credentials);

3:

   4:  Console.Out.WriteLine("Reading data from WAD storage");

5:

   6:  //Query using the start time and end time, take only 1000 rows

   7:  //The response headers will contain a continuration key that can be used to retrieve the next 1000 rows

   8:  var query = (from x in wadContext.WADPerformanceCountersTable

   9:               where x.EventTickCount >= start.Ticks && x.EventTickCount <= end.Ticks

  10:               select x);

11:

12:

  13:  var allItemsAsTableService = query.AsTableServiceQuery();

  14:  IEnumerable<WADPerformanceCountersTable> allItems = allItemsAsTableService.Execute();

15:

  16:  foreach (WADPerformanceCountersTable item in allItems)

  17:  {

  18:  //use item

  19:  }

I chose the last solution because I needed minimal changes to my code, and I didn't really need paging, if you need paging, continuation token will be the way to go!

Error TF30059: Fatal error while initializing web service from server after installing SP1

2008-08-27T20:40:00Z

If you are getting this error from the server right after installing SP1, you can open the event log and check for the crash report that has been created with the error details.

If the error is similar to

Exception: TypeInitializationException
Message: The type initializer for 'Microsoft.TeamFoundation.Server.IntegrationResourceComponent' threw an exception.
Stack Trace:
   at Microsoft.TeamFoundation.Server.IntegrationResourceComponent.RegisterExceptions()
   at Microsoft.TeamFoundation.Server.Global.Initialize()
   at Microsoft.TeamFoundation.Server.TeamFoundationApplication.Init()
Inner Exception Details
Exception: ReflectionTypeLoadException
Message: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.
Stack Trace:
   at System.Reflection.Module._GetTypesInternal(StackCrawlMark& stackMark)
   at System.Reflection.Assembly.GetTypes()
   at Microsoft.TeamFoundation.Server.SqlResourceComponent.RegisterExceptions(Assembly assembly)
   at Microsoft.TeamFoundation.Server.IntegrationResourceComponent.RegisterExceptions()
   at Microsoft.TeamFoundation.Server.IntegrationResourceComponent..cctor()

This sometimes happens when you have BOTH client and server on the machine, but you installed SP for the client only. The best way to fix this is to install TFS SP1 on the server machine as well..

A few VS IDE Tips to make your life easier

2008-02-02T07:57:00Z

This is a number of my favorite VS tips, let me know what you think

Tip #1: Select Code Using Column Mode

This is one of my favorites, if you are a copy paste person like me, you will find this very useful,
Here is how is works, while selecting text in editor press the “ALT” button, when doing this you can select columns instead of entire lines. Assuming you have these 3 variables in your code

You decide that instead of naming you variables a1, b2, c3 you just want to name them a, b & c to do this: press alt, select the numbers column

And click delete

This can be very handy when you copy and paste columns.

Tip #2: Using TODO & the Task like

You can get a list of all your TODO comments using the task list, you need to start your comments with TODO and then go to View -> Other windows-> Task List. You will also see your HACK and UNDONE comments. If you want to change the comments you see you can do this by going to

Tools->Options, then under the environment folder select Task List and add your own.

Tip#3: Store Commonly Used Code in the Toolbox

If there is a part of the code that you use all the time or a comment that you keep re-writing, you can drag it and drop it in the tool box. Whenever you need to re-use this code you can get it from the tool box

Tip#4: Ctrl + drag and drop to copy code

Works the same as copy and paste without changing your clipboard

Tip#5: Code Snippets

The code Snippets can save you a lot of time doing repetitive tasks. One of my favorites is the “surround with” code snippet.

Try to select a potion of your code and press “Ctrl + S, Ctrl + K”

You will get a list of options for what you want to surround you code with

If you select Region and enter the region name, this part of the code will be surrounded in a region.

- Code snippets can also be useful with loops, switches, ..

Try to type switch and press Tab twice

With switch_on still selected, enter the name of the variable you want to switch on and press enter

P.S. Even better: You can create your own code snippets..