Waiting for Godot
Idle hangs are the subject of today’s rant.
They can be pretty tricky things. With desktop applications, there are generally only a few threads and you have a good idea what each thread does. With components that run inside a complex host such as the ASP,NET worker process or DLLHost, things are a bit more complex. Lets look at the basics before we go in to the complex issues.
Threads block when they are waiting on IO or waiting on an event/mutex/semaphore/critical section. Is it a problem if you see a waiting thread while debugging? No, not normally. Threads spend a lot of their lives blocked in real world applications that do things with data, ports and so on. CPU bound applications (such as games) tend to have few threads and they generally don’t block much but applications like this are rare and getting rarer. So, threads that are waiting are not a problem as long as they don’t wait forever.
In previous blog entries, I have wittered on about causes of hangs a bit so I will only recap briefly. An idle hang is when most or all threads are waiting for something that will never happen. They are generally stress problems – they only happen when the system is heavily loaded. They are normally timing sensitive. Idle hangs are not to be confused with an app that is running slowly or a genuinely idle application. I quite often see a process which is reported to have hung but which is actually doing nothing because it hasn’t been asked to do anything – typically because something earlier in the chain has broken. If that is the case, you will see a lot of threads waiting for work.
One common thing when debugging hung systems is processes where nothing is happening and all the threads seem to be stuck in COM calls, normally out of process calls. If you see this then something is hung but it isn’t the process that you are looking at. You need to have a poke around inside the COM component.
Debugging complex processes such as really requires a low level debugger such as WinDbg. You can do it in the VS.NET IDE but it isn’t at all easy and I don’t know anyone who routinely does this. Everyone that I know who does this for a living uses WinDbg or CDB which is a non-GUI version of WinDbg. The other thing that you need is some understanding of what you would expect to see. There are a couple of things that you need if you are to find your way around.
- Symbols. Good symbols make life easy. No symbols make life hard. For more information, try http://www.microsoft.com/whdc/devtools/debugging/debugstart.mspx
- An idea of what is normal in the process. Take a test environment and dump the process when it is working fine. Look at what sort of things you see
- For managed code, you need SOS. Details at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vsdebug/html/vxtsksosdebugging.asp
Next blog, I will be talking about what sort of call stacks you might expect to see in DLLHost or the ASP.NET worker process.
Signing off
Mark
