If broken it is, fix it you should

Using the powers of the debugger to solve the problems of the world - and a bag of chips    by Tess Ferrandez, ASP.NET Escalation Engineer (Microsoft)

High CPU in .NET app using a static Generic.Dictionary

High CPU in .NET app using a static Generic.Dictionary

Rate This
  • Comments 28

A couple of weeks ago I helped out on a high CPU issue in an ASP.NET application.

Problem description

Every so often they started seeing very slow response times and in some cases the app didn’t respond at all and at the same time the w3wp.exe process was sitting at very high CPU usage 80-90%.  This started happening under high load, and to get the application to start responding again they needed to restart IIS.

Debugging the problem

They gathered a few memory dumps during the high CPU situation for us to review and when running the sos.dll command ~* e !clrstack (in windbg) to see what all the threads were doing we found that they were all stuck in callstacks similar to this one:

OS Thread Id: 0x27dc (124) 
  ESP       EIP     
2f77ed24 795b3c5c System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].FindEntry(Int32) 2f77ed3c 795b3835 System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].ContainsKey(Int32) 2f77ed40 209f1932 MyComponent.Settings.get_Current() ... SOME STACK FRAMES REMOVED AS THEY ARE NOT IMPORTANT FOR THIS ISSUE ... 2f77f0a4 209f7545 ASP.MyApp_default_aspx.ProcessRequest(System.Web.HttpContext) 2f77f0a8 65fe6bfb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 2f77f0dc 65fe3f51 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 2f77f11c 65fe7733 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception) 2f77f16c 65fccbfe System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object) 2f77f188 65fd19c5 System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest) 2f77f1bc 65fd16b2 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest) 2f77f1c8 65fcfa6d System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32) 2f77f3d8 79f047fd [ContextTransitionFrame: 2f77f3d8] 2f77f40c 79f047fd [GCFrame: 2f77f40c] 2f77f568 79f047fd [ComMethodFrame: 2f77f568]

In other words, the method MyComponent.Settings.get_Current() was calling ContainsKey on a Generic.Dictionary object and for some reason it was getting stuck when trying to find the entry.

Looking at the MyComponent.Settings.get_Current() method, we found that the Generic.Dictionary it was calling ContainsKey on was a static dictionary and that all threads were working on the same dictionary.

The MSDN documentation about Generic.Dictionary has the following information about the thread safety of Dictionary objects 

A Dictionary can support multiple readers concurrently, as long as the collection is not modified. Even so, enumerating through a collection is intrinsically not a thread-safe procedure. In the rare case where an enumeration contends with write accesses, the collection must be locked during the entire enumeration. To allow the collection to be accessed by multiple threads for reading and writing, you must implement your own synchronization.

What is happening here, and causing the high CPU is that the FindEntry method walks through the dictionary, trying to find the key.  If multiple threads are doing this at the same time, especially if the dictionary is modified in the meantime you may end up in an infinite loop in FindEntry causing the high CPU behavior and the process may hang.

Resolution:

These type of timing issues with static collections are fairly common in ASP.NET apps with high load.

To resolve this timing issue you should take special care to synchronize (lock) around access to the dictionary if there is a possibility that you may have multiple writers working at the same time or if there is a possibility that you write while someone else is reading/enumerating through the same dictionary. 

In general, I would recommend to always read the thread safety information carefully when using static collections as many of them require that you implement synchronization on concurrent read/write operations, to avoid this type of issue or issues with for example HashTables where you may get exceptions like InvalidOperationException “Load Factor Too High”.

Have a good one,

Tess

  • Is it even a good idea to have a static dictionary?  Depending on what it is (and whether it's huge and memory concerns would become an issue) would it not be better to replace the static dictionary with a expirable cached one instead?

  • Ahh! Quite interesting diagnosis and resolution. I enjoyed reading this post.

    Thanks!

  • Neil,  could be, but even so, you could run into the same issue since the problem occurrs when you access the same dict. on multiple threads at the same time.

  • Hi Tess,

    interesting post, but i'm hoping you can also shed some light on this:

    suppose you have a dictionary with some metadata that is initialized at application startup, and then never, ever modified.  can you still run into problems when a high number of threads are iterating over the dictionary concurrently?  I would assume that this would be safe, provided that there are indeed no modifications to the dictionary, but i'm still not sure...

  • Very good post. So far I have not know that objects will have thread safe problems. Thanks mam. I will be careful while implementing collections.

    Thanks,

    Thani

  • Compared with c/c++, .NET give us a lot convenience

  • @Tess yes, my mistake - I actually meant to totally separate the dictionary instances and not share them.  It only really works if they're quite small though, I suppose.

  • hi Tess,

    Nice post.

    Quite Interesting to read such a post. Keep writing on such issues.

  • Great post. Keep up the good work. All the best...

  • Thank you very much Tess. Lesson learned for me. I've used static dictionaries and single instanced dictionaries before in .Net apps for properties instead of creating numerous place holder field variables. I guess this was a mistake and will now know but I usually don't do this anymore with the new .Net 3.5 way of making auto properties.

  • I use a static Dictionary all the time and always lock it with ReaderWriterLockSlim(). It's simply a great cache.

  • Davy,  that depends on how you iterate over it.  Using an enumerator to iterate over it is not a thread safe operation so if you use that you would still have to lock, otherwise the enumerators would just go back and forth.

  • We ran into exactly the same problem with a high load wcf service that had the same symptoms wish I had know more about windbg at the time had to use remote debugging.

  • Hi Tess,

    "Even so, enumerating through a collection is intrinsically not a thread-safe procedure. In the rare case where an enumeration contends with write accesses, the collection must be locked during the entire enumeration."

    Is seems that these sentences confuse others, too:

    http://stackoverflow.com/questions/511205/net-dictionary-is-only-enumerating-thread-safe

    It is still unclear for me: Is it safe to enumerate from multiple threads if it is guaranteed that no writes happen at the same time? If yes, why?

  • Petter,

    Don't take my word for it, but my incling is to say that it's not threadsafe as the current item in the enumerator may then be changed by multiple threads as you iterate through it.

Page 1 of 2 (28 items) 12
Leave a Comment
  • Please add 3 and 6 and type the answer here:
  • Post