I found a set of really nice, short videos by Arturo Toledo on the Silverlight blog that I thought were worth sharing.

He goes through some of the design principles using the Microsoft Expression tools and talks about what you should think about when designing a web page, logo or whatever it is you are designing.

Principles of design:

Balance  - how you can achieve a sense of balance, and avoid the feeling of being pulled too much to any part of the design.
Rhythm – using repeating visual patterns to create appealing visual effects.
Symmetry and asymmetry – how to achieve balance both in symmetrical and asymmetrical designs.
Emphasis – using different factors like form, color, shape etc. to make some elements stand out
Unity – how to make the viewer feel like everything in design belongs there
Points, Lines and Form – using points, lines and forms as markers for other visual elements
Colors – how to choose a color palette where the colors jive well together

Hands-on videos:

Experimenting with User Interface Colors with Expression Design
Creating Mosaic and Tiled Patterns using Expression Design
Learning to create Icons using Expression Design  
Creating a Sample Icon using Expression Design

I’m hoping there is more to come because so far these have been very interesting for someone like me who has always been intrigued by design but has no sense of what works and what doesn’t.

Tess

Recently I had a case where the customer had an issue with session variables.  The claim was that if they use in-proc session state their session variables would work just fine, but if they changed the session mode to stateserver in web.config their sessions were lost.

<sessionState mode="StateServer" cookieless="UseCookies" stateConnectionString="tcpip=127.0.0.1:42424"/>

When looking at bit closer at the repro, it wasn’t really all session variables that were lost, it was only ones set in Page_Error, and when they were read in an error page they redirected to (with the customErrors setting in web.config) would be null.

Why is there a difference in this case between in-proc and out of process session state?

When you use in-proc session state, the session objects are stored in the memory of the process, and thus when you run code like Session[“mySessionVar”] = “some value”;  it is immediately written to that session variable.

With out-of-proc session state on the other hand (stateserver or sqlsessionstate), all session variables for the current session are read from the store at the beginning of each request, and then any changes are committed back to the session store at the end of the request.

In a case where an exception occurs, the request is effectively aborted so none of the changes that are made to the session during that request are written back to the store.  

Old session variables and values will still be intact though…

So how do you handle this situation in a case where you use state server or SQL session state?

If you want to use session state to store values relating to errors in pages that should be available in an error page you can clear the error in page_error and manually redirect to the error page.

The code could look something like this:

protected void Page_Error(object sender, EventArgs e)
{
    Session["ErrorPage"] = "Default2.aspx";
    Exception ex = Server.GetLastError();
    Session["ErrorMessage"] = ex.Message.ToString();
    Server.ClearError();
    Response.Redirect("~/ErrorPages/ErrorPage.aspx");
}

Clearing the error allows the request to finish processing and the session variables to be committed to the store.

Laters,
Tess

If my post on dump debugging in VS 2010 piqued your interest, here is a list of a number of other new debugging features coming up in 2010

http://blogs.msdn.com/ms_joc/archive/2009/05/27/dev10-beta1-free-at-last.aspx

I for one will be following the blog to get the scoop…

On a related note, John Robbins also wrote a nice post about historical debugging in VS2010,  I haven’t tried it yet but I can already see a number of ASP.NET scenarios where it can be applied.

http://www.wintellect.com/CS/blogs/jrobbins/archive/2009/06/16/how-does-vs2010-historical-debugging-work.aspx

Laters,

Tess

I know that for a lot of you this is a bit far into the future, but I thought it would be nice to run through a few of my lab scenarios to show off some of the new features in Visual Studio 2010. 

I am sure that I will miss a bunch of cool features here as this post is just based on my first experience with this, but I will likely post more about these features as we get closer to RTM.  As we are only in Beta, anything and everything may change before RTM and the below is just based on my own impressions so there may be other better ways of doing things.

Why debug dumps with Visual Studio .NET

I love WinDbg and will probably continue using it for many years to come since it is a very simple and clean and powerful debugger.  I do realize though that not everyone debugs production issues (hangs/crashes etc.) every day (at least i hope not:)), and that most .net developers are very comfortable in Visual Studio.NET, so it is nice to be able to stay within that tool where you recognize the IDE and structure.

If you have compiled the app “debug” and set debug=true in web.config if you are working with an asp.net app, debugging a dump is very similar to being stopped at a breakpoint.  Of course, since it is not a live process you can’t hit go, but you can match up source code, see locals etc.  without having to learn a bunch of new commands.

There are a couple of caveats here:

1. .NET debugging only works on 4.0 dumps – with 3.5 and earlier you only get native debugging

2. Visual Studio as opposed to windbg doesn’t seem to use deferred symbol loading (I may be missing a setting here), but this means that when you start debugging a dump you load up all symbols which might take a while (at least the first time).  With windbgs deffered symbol loading, it only loads symbols when it needs them which makes it significantly faster to load up a dump in windbg if you only want to check some quick things.

3. Mileage varies depending on if your app is compiled debug, you have source lined up etc.   If the app is not compiled debug you will not get things like locals or source synchronization.

How do you gather the memory dumps?

You can get dumps with windbg (adplus) as shown in this post or debug diag, or in visual studio.net for that matter (debug/save dump as… when you are debugging)

Typically in a web scenario or in a scenario where you are reproing a problem in a winforms app or windows service on a customer machine you wouldn’t have visual studio installed there, and then adplus or debug diag are good options for gathering dumps.  

Setting up symbols

For Visual Studio to be able to resolve native (non-dotnet) function names/parameters/types etc., it uses symbols, like any other debugger.  For all Microsoft components there are public symbols located at http://msdl.microsoft.com/download/symbols.   If you have your own native components you would have to compile them with symbols in order to resolve function names.

You can set up the symbol paths for your symbols under Tools/options/debugging/symbols and a typical setup would look like this:

The first time it loads up the symbols it will find them in http://msdl.microsoft.com/download/symbols and as it downloads them it will put them in your cache location (c:\webcache).  The next time you look at a dump, it will check the c:\webcache location first, and only download the symbols it can’t find there.   This means that the first time you look at a dump (any dump) it may take a long time depending on your internet speed as these files can be pretty big, but the next time it should be significantly faster.

In here you can also add the locations of any of your own symbols and this setting is persisted, so the next time you open VS, the symbol paths are there for you.

Notice that there is a button for “Load symbols from Microsoft symbol servers”.  This is available once you have started debugging a dump or any process, and this will do the above for you, setting up a local cache if you haven’t done this already.  I like setting it up explicitly as pushing the “Load symbols from Microsoft symbol servers” leaves the UI “hanging” without visual feedback so you don’t know how far you have gotten in the process, while setting up a symbol path allows you to see the modules/symbols as they load.

Debugging a simple hang scenario (Debugging labs - Lab 1)

I recompiled Lab 1 with 4.0 (just switching some settings in web.config), ran a stress and grabbed a dump with adplus.  Look at the Lab 1 instructions on how to repro and get the dump if you are interested in following along.

1. Open the dump in Visual Studio

File/Open/File – select the dump

This will present you with the dump file summary which contains info about the dump, any exceptions that might have occurred (in this case just a wake debugger (0x80000007) from cdb when getting the dump), along with some info about the OS version and the CLR version.

If you move to the right (not shown here) there are also some options to set up symbol paths and debug either mixed or native.  The default is mixed.

2. Click on the super obvious green go button:)  to start debugging

This will start loading symbols and get you in debugging mode where you can see the stacks for all the threads etc. so grab a coffee, chat with your colleagues about the weekend or watch the symbols as they load:)

At the end it may pop up a message box talking about an unknown exception (this is the wake debugger exception so don’t be alarmed, just click ok or break)

Once you are done loading all the symbols you are presented with the current call stack.

If this would have been a dump taken at a crash or exception this stack is what you would have started investigating.  In the case of a hang dump this is rather meaningless since the active thread is thread 0 – just the main thread, which often has little or nothing to do with what you are debugging.

For hangs, we want to see all the stacks, and in particular those running .net code or those that appear blocked for any reason.

3. Open the “Parallel Stacks” window (Debug/Windows/Parallel Stacks)

This is a feature I really like in Visual Studio (and you can use it when you debug normally as well).  What it does is it pulls out all the stacks and looks for commonalities.   If you have 5 threads doing the exact same thing it will group them together so you get a very nice overview of what is going on.

In this case (image below) for example it shows us that we have 16 threads sitting in DataLayer.GetFeaturedProducts.  15 of these are waiting in a Monitor.Enter (note that it shows both .net and native stacks here), and one is sitting in a Thread.Sleep.

From this, and knowing the code we can deduce that all these requests for FeaturedProducts are waiting in a lock (Monitor.Enter) and the lock is probably owned by the thread sitting in the Thread.Sleep.

Note:  So far we haven’t done anything that requires debugging to be turned on, so even if debug=false, you can still get this type of information, and in most cases this might be all you need.

3.  Look at source code and locals.

If you want to go into more detail about what is going on, you can right click on the GetFeaturedProcudts frame for example and choose “Switch to frame”, which brings up the source code (if the source is lined up) and shows us with a green arrow what statement we are on (the lock).

If you click on the “Show threads in source” button on the debugging toolbar…

…it also highlights in gray the location that the other thread is on and you can also switch to that thread if you want to by right clicking the warning sign in the left sidebar and choosing “switch to frame”

In this case we don’t have a whole lot of locals set, but if you did you could get to them in the same way you would if debugging normally.

In summary:

I am really interested to see where this will be going in the future and what additional features if any we’ll see in RTM.  

Based on my first experience with this here are my impressions:

+ Why I would use Visual Studio to debug dumps

+ You don’t have to learn a new environment (windbg) or a lot of new commands (sos) for simple scenarios
+ Really nice visualization of the threads, it makes it very easy to spot the important ones
+ Matching up source code is very useful as with windbg you have to jump through hoops and match IL with source to figure out where you’re at
+ Looking at locals the Visual Studio way is a lot easier than using !do in sos to navigate through it.   As VS can make use of debug info it can also name the locals which is not possible in windbg.
+ The native and managed stacks are intermingled as Visual Studio is a true mixed mode debugger while windbg for example is really native only (using sos as a helper to look at .net stacks)

- Why I would stick with windbg

- If you don’t have debug info, you can’t get locals or items on the stacks.  You can get this with windbg/sos, even if you don’t have variable names it is still very useful information.
- Since Windbg uses deferred symbol loading it is a bit faster to load up a dump in windbg.
- You can debug .net 2.0/3.5/1.1/1.0 dumps etc. in windbg and 4.0 dumps
- In Visual Studio (beta 1) I have not found a way to look at the managed heaps so debugging memory issues is still a task for windbg, or by loading sos in visual studio.
- SOS contains a lot of nice commands that allow you to look at .net data in various ways (looking at exceptions, gc heap info etc.), again you can load sos in visual studio to get this info, but windbg feels more natural for this. 

In reality I will probably use a combination of both + debug diag depending on scenario, but I hope this post has given you some insight into the topic and what’s to come.

Have fun,

Tess

Yesterday I found this really nice Channel 9 interview with Maoni Stephens (Dev Owner of the CLR GC) and Andrew Pardoe (Program manager for the CLR GC) where they talked about the new Background GC in CLR 4.0.

She also talks about it here and there is not much value in me repeating what she already says there but basically the main points of the video and the post are:

Concurrent GC is being replaced by Background GC in CLR 4.0

Concurrent GC is the mode of the GC that you use in desktop applications for example.  The goal of the concurrent GC is to minimize pause time, and it does so by allowing you to still allocate while a GC is in progress (hence the concurrent part).

Concurrent GC is only available in workstation mode. 

In server mode (which is what you use in ASP.NET for example when you have multiple processors/cores), simplified all managed calls are paused while in a GC, which means that you can’t allocate anything.  This means that the process pauses slightly while in a GC but on the other hand what you loose in pause time, you gain in throughput as GCs are made by x number of GC threads concurrently, where x is #procs*#cores.

In concurrent GC you were allowed to allocate while in a GC, but you are not allowed to start another GC while in a GC. This in turn means that the maximum you are allowed to allocate while in a GC is whatever space you have left on one segment (currently 16 MB in workstation mode) minus anything that is already allocated there).

The difference in Background mode is that you are allowed to start a new GC (gen 0+1) while in a full background GC, and this allows you to even create a new segment to allocate in if necessary.  In short, the blocking that could occur before when you allocated all you could in one segment won’t happen anymore.

Background GC’s will be available in the Silverlight CLR as well

The CoreCLR uses the same GC as the regular CLR, so this means that Silverlight apps benefit from this as well…

As Server mode does not use concurrent GC this will not be available in Server GC

Having this in server mode would be incredibly cool as GCs can get pretty hefty, especially in 64 bit apps with very large heaps, but as Maoni mentions in the video and in the post, this work for the concurrent GC lays the foundation for the same work being done in the Server GC. Because of the complexities involved in doing this in Server GC though this is not included in v4.0.

If you do have a lot of latency due to heavy pause times during full garbage collections, there is a feature that was introduced in 3.5 SP1 that allows you to be notified when a full GC is about to occur.  You can then redirect to another server in a cluster for example while the GC occurs.     

I just want to mention that this not being in the Server GC does not mean that you should switch your server apps (asp.net etc.) to use workstation with concurrent,  Server GC is optimized for these scenarios and should still be used there.

Have fun,
Tess

I have put together a quick and dirty debug diag script for troubleshooting .net memory leaks.  (attached to this post)
The reason I put it together was mainly to show how you can create your own debug diag scripts but feel free to use it to troubleshoot memory leaks, knowing that it does string parsing on the output so it is a bit prone to errors if sos changes output formats.

NOTE:  This script will only work on .NET 2.0 x86 memory dumps

Mourad recently published a whitepaper on debug diag that talks about how to use it for various scenarios and he also explains how to create some basic scripts and I used this as my starting point.

My script loads up sos from the framework directory and goes through the following steps.

1. Print .NET Framework Version
2. Print information on the GC (Garbage Collector) heaps and report the amount of memory used for .NET objects
3. Print and categorize the 40 most memory consuming object types (so that you can spot which object types you might be leaking or using a lot of)
4. Print the finalizequeue so that you can see what objects on the heap have finalizers to verify that they are neccessary.  It also reports potential blocking of the finalizer including the finalizer stack so that you can see if/why your finalizer thread is blocked.
5. Print objects on the large object heap
6. Determine and report the size of the cache for each asp.net application in the process
7. Report the number of active sessions in the process
8. Check if the process was in a GC when the dump was collected as this may invalidate heap/object information

How to use the script

1. Install debug diag 1.1 from http://www.microsoft.com/DOWNLOADS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en

2. Unzip the attached file and put the DotNetMemoryAnalysis.asp file in the C:\Program Files (x86)\DebugDiag\Scripts directory (or your debugdiag/scripts directory if you installed it somewhere else)

This will add a new analysis script to the Advanced Analysis tab in debug diag:

2. Generate a memory dump when memory usage is high. With debug diag you can do this by going into the processes tab, right-click on the process and select “Create Full Userdump”

3. In the Tools/Options & Settings menu, set the “Symbol Search Path For Analysis” to SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

4. In the Advanced Analysis tab, select Add Data Files, add your memory dump, select the DotNetMemoryAnalysis.asp script and click Start Analysis  (note that some of the steps may take a while depending on the size of the dump).  If you get an InStr exception that means that you are either trying to debug a 64 bit dump or the sos version in your framework directory is not matching up to the framework version of the dump (i.e. the proper mscordacwks symbol could not be found)

Once the analysis is done you will be presented with a mhtm file containing the report.

Sample report:

Here is a sample report generated for a dump taken for Lab 3 where the finalizer is blocked because of some bad code in the destructor for the Link object.

For this example we can see already from the analysis summary that something is fishy with the finalizer, and in the finalizer section we can also see the finalizer stack.

Scrolling down to the 40 most memory consuming object types we find a lot of Link/Link_aspx and System.Web.UI… objects, so in this case the report is pretty much straight on.  In other cases it might be a bit less obvious but hopefully the suggested articles help narrow it down. 

Analysis Summary   Type Description Recommendation    Warning Number of objects ready for finalization: 35765 This is an indication that your finalizer thread may be blocked. Look at finalizequeue info and finalizer stack to determine why/if the finalizer is blocked    Information Cache Size: 1397928 Bytes This includes memory for objects stored in in-process sessions      Information Number of active in-process sessions: 4000      Information GC Heap usage: 768160596 Bytes Common issues for high .NET Memory usage includes: Blocked finalizers, lots of memory in cache/sessions, lots of large objects and memory rooted in statics You should also revivew the most memory consuming objects.

Analysis Details   PID-5476__W3WP.EXE_-DefaultAppPool-__full_1964_2009-05-07_13-58-54-165_1564.dmp   Type of Analysis Performed   .NET Memory Analysis Machine Name   PRATHER Operating System   Windows Server 2003 Service Pack 2 Number Of Processors   2 Process ID   5476 Process Image   C:\WINDOWS\SysWOW64\inetsrv\w3wp.exe System Up-Time   10 day(s) 05:48:21 Process Up-Time   8 day(s) 23:15:06 Table of contents .NET Framework Version .NET GC Heap Information 40 most memory consuming .NET object types .NET Finalizer Queue Large Object Heap Cache .NET Version 2.0.50727.3053 retail Server mode with 2 gc heaps SOS Version: 2.0.50727.3053 retail build .NET GC Heap Information Number of GC Heaps: 2 ------------------------------ Heap 0 (001aa5c0) generation 0 starts at 0x39d0c6a8 generation 1 starts at 0x39005c0c generation 2 starts at 0x02d10038 ephemeral segment allocation context: none segment begin allocated size 02d10000 02d10038 06cf1098 0x03fe1060(66981984) 16fd0000 16fd0038 1afca200 0x03ffa1c8(67084744) 1efd0000 1efd0038 22fc61f0 0x03ff61b8(67068344) 26fd0000 26fd0038 2afc1338 0x03ff1300(67048192) 2efd0000 2efd0038 32e7c290 0x03eac258(65716824) 36fd0000 36fd0038 3a0e7734 0x031176fc(51476220) Large object heap starts at 0x0ad10038 segment begin allocated size 0ad10000 0ad10038 0ad23b78 0x00013b40(80704) Heap Size 0x16f99b74(385457012) ------------------------------ Heap 1 (001ab888) generation 0 starts at 0x3de2921c generation 1 starts at 0x3d17acc8 generation 2 starts at 0x06d10038 ephemeral segment allocation context: none segment begin allocated size 06d10000 06d10038 0acfe46c 0x03fee434(67036212) 1afd0000 1afd0038 1efaba04 0x03fdb9cc(66959820) 22fd0000 22fd0038 26fa3d7c 0x03fd3d44(66927940) 2afd0000 2afd0038 2efa58d8 0x03fd58a0(66934944) 32fd0000 32fd0038 36c42aa8 0x03c72a70(63384176) 3afd0000 3afd0038 3e0e39b4 0x0311397c(51460476) Large object heap starts at 0x0cd10038 segment begin allocated size 0cd10000 0cd10038 0cd10048 0x00000010(16) Heap Size 0x16cf97e0(382703584) ------------------------------ GC Heap Size 0x2dc93354(768160596)   More information: Compare the total GC Heap size to the number of private bytes in the process when the dump was taken (or the dump size on disk) to determine if most of your memory is on the .NET GC Heap In the !eeheap -gc output above you can also see if most of your .NET GC memory is on the small object heaps or on the large object heap (LOH) (objects over 85000 bytes). If you see that most of your memory is on the LOH, look at the LOH output to see what those objects are. If the GC Heap is relatively small, run debug diag with leaktracking to track native leaks and analyze the dump with the MemoryAnalysis script instead. See debug diag help for more information about this. Related posts: I have a memory leak!!! What do i do? (defining the "where") Are you really leaking .NET Memory? 40 most memory consuming .NET object types MT Num items Tot. size Type 66108d24 33 8316 System.Web.UI.RenderStyle[] 6614e4dc 35 8540 System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6 79323510 363 8712 System.Collections.Stack+StackEnumerator 7932e7d0 185 8880 System.Signature 79331a6c 472 9440 System.RuntimeType 648c710c 636 10176 System.Configuration.PropertySourceInfo 66108884 33 10956 System.Web.UI.HtmlTextWriter+RenderAttribute[] 7a5eca44 576 11520 System.Collections.Specialized.HybridDictionary 79304314 194 11640 System.Reflection.ParameterInfo 7932fba8 215 12040 System.Reflection.RuntimeMethodInfo 0fbf1b90 35 13020 ASP.links_aspx 661087a0 593 14232 System.Web.UI.ControlCollection+ControlCollectionEnumerator 66107950 941 15056 System.Web.UI.Pair 648c8fa0 299 15548 System.Configuration.ConfigurationProperty 79329b58 496 15872 System.EventHandler 7933151c 232 16400 System.Char[] 7a5ecbc0 829 16580 System.Collections.Specialized.ListDictionary+DictionaryNode 648c91d4 353 16944 System.Configuration.ConfigurationValues 66113834 759 18216 System.Web.UI.Html32TextWriter+FontStackItem 79332a88 243 18484 System.Int32[] 79332b38 1627 19524 System.Int32 7a5e7dbc 1329 21264 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry 66106ba4 315 21420 System.Web.UI.DataBoundLiteralControl 66101cd0 630 22680 System.Web.UI.ControlCollection 66101934 379 22740 System.Web.UI.LiteralControl 648c9414 1147 22940 System.Configuration.ConfigurationValue 7933335c 232 26484 System.Byte[] 661068d4 385 35420 System.Web.UI.WebControls.DataListItem 66101680 945 41580 System.Web.UI.Control+OccasionalFields 7933291c 1869 44856 System.Collections.ArrayList 79332f40 910 50960 System.Collections.Hashtable 661493a4 33 51084 System.Web.Caching.ExpiresEntry[] 6611b000 4000 192000 System.Web.SessionState.InProcSessionState 793040bc 4141 284380 System.Object[] 66148eb0 4026 289872 System.Web.Caching.CacheEntry 7933303c 948 349560 System.Collections.Hashtable+bucket[] 0fbf1edc 35999 575984 Link 79330a28 36165 723300 System.Text.StringBuilder 001a9a30 6028 43134680 Free 793308ec 47536 721577968 System.String Total 168583 objects´   Color Object type Red System.Web.UI... objects Blue System.Data... objects Green System.XML... objects Purple Custom objects More Information For more information on how to read !dumpheap -stat output, see !dumpheap -stat explained To get info on how to dig deeper into the !dumpheap -stat output see ASP.NET Memory Investigation Finalizer queue SyncBlocks to be cleaned up: 0 MTA Interfaces to be released: 0 STA Interfaces to be released: 0 ---------------------------------- ------------------------------ Heap 0 generation 0 has 370 finalizable objects (107a9a9c->107aa064) generation 1 has 12 finalizable objects (107a9a6c->107a9a9c) generation 2 has 47 finalizable objects (107a99b0->107a9a6c) Ready for finalization 17909 objects (107aa064->107bb838) ------------------------------ Heap 1 generation 0 has 247 finalizable objects (107bcb40->107bcf1c) generation 1 has 4 finalizable objects (107bcb30->107bcb40) generation 2 has 36 finalizable objects (107bcaa0->107bcb30) Ready for finalization 17856 objects (107bcf1c->107ce61c) Statistics: MT Count TotalSize Class Name 6614bbc0 1 12 System.Web.Configuration.ImpersonateTokenRef 79334808 1 20 Microsoft.Win32.SafeHandles.SafeFileMappingHandle 793347b0 1 20 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle 79317fac 1 20 Microsoft.Win32.SafeHandles.SafeTokenHandle 66147c6c 1 20 System.Web.PerfInstanceDataHandle 6614b038 1 32 System.Web.Compilation.CompilationMutex 7932335c 2 40 System.Security.Cryptography.SafeProvHandle 6612f6d0 2 56 System.Web.Security.FileSecurityDescriptorWrapper 79317928 3 60 Microsoft.Win32.SafeHandles.SafeWaitHandle 79321cd0 2 120 System.Runtime.Remoting.Contexts.Context 7932a0f4 3 132 System.Threading.ReaderWriterLock 79316fb0 7 168 System.Threading.TimerBase 7932b108 14 280 Microsoft.Win32.SafeHandles.SafeRegistryHandle 661483d8 10 280 System.Web.DirMonCompletion 7932a09c 31 496 System.WeakReference 66151384 70 1680 System.Web.HttpResponseUnmanagedBufferElement 66148130 96 1920 System.Web.ApplicationImpersonationContext 66151404 100 2000 System.Web.ClientImpersonationContext 66103cb4 68 2992 System.Web.UI.WebControls.TableStyle 79330ec0 78 4368 System.Threading.Thread 0fbf1edc 35989 575824 Link Total 36481 objects   More Information: !finalizequeue will show all the objects on the heap that have finalizer methods, and have yet not been disposed of. It is a good idea to look through this list and verify that all your "custom" objects on this list really need finalizers/destructors, as having unneccesary finalizers will lead to higher memory consumption and a potential for blocked finalizers. Related posts: Unblock my finalizer To dispose or not to dispose, that's the 1 GB question As the number of finalizable objects is more than 0, please check the finalizer thread to see if it is blocked or active Finalizer Thread OS Thread Id: 0x1534 (14) ESP EIP 0286f8f4 7d61ccc6 [HelperMethodFrame: 0286f8f4] System.Threading.Thread.SleepInternal(Int32) 0286f948 79299225 System.Threading.Thread.Sleep(Int32) 0286f94c 0fbe2262 Link.Finalize() 0286fc1c 79f8df9a [ContextTransitionFrame: 0286fc1c] 0286fcec 79f8df9a [GCFrame: 0286fcec]   Objects on the Large Object Heap ------------------------------ Heap 0 Address MT Size total 0 objects ------------------------------ Heap 1 Address MT Size total 0 objects ------------------------------ total 0 objects Statistics: MT Count TotalSize Class Name Total 0 objects More information: A high amount of large objects (strings and arrays over 85000 bytes) can lead to GC Heap fragmentation and thus higher memory usage in your application. Look through the large objects, to dig deeper you can run !do on the object address in windbg, to see if these objects are expected and if you can minimize their usage in any way, by caching etc. Common reasons for high amounts of large objects are large viewstate and Dataset serialization Size of Web Caches in the process sizeof(06d1cd2c) = 1397928 ( 0x1554a8) bytes (System.Web.Caching.Cache)   More information: There is one System.Web.Caching.Cache object referencing all cached objects, per web application In-Proc session state is stored in the cache, so the size of all session vars is also included in the size of the cache for the specific application Related articles How much are you caching UI objects in session scope

Script Summary   Script Name Status Error Code Error Source Error Description Source Line DotNetMemoryAnalysis.asp Completed

Have fun,

Tess

Bret wrote a post about Managed Stack Explorer, a really nice tool that was developed in 2006 but since then seems to have been forgotten.

It’s excellent if you want to troubleshoot hangs/performance issues in a process but don’t want to go through the hassle of getting and analyzing a dump.   What it will do is basically attach, snap the .net call stack and detach so you can see what all the .net threads are doing without using a debugger.

What is even cooler is that you can copy/run the tool from anywhere which means that you can copy it to a clients machine if you have a runaway thread in a win forms app for example. 

Have a nice weekend,

Tess

I recently got an email with the following question:

“Can you give me some very helpful hints with this one ? I am struggling with the following error in an asp.net application.

Exception Message: Internal Error : Unable to load DLL 'RdbNet.dll': A dynamic link library (DLL) initialization routine failed. (Exception from HRESULT: 0x8007045A) The RdbNet.dll is an .NET dll from Oracle so I can't debug into it.

If I use a windows forms application I never have the error. I have already used procmon, filemon and regmon to search for access denied, not found... and so but couldn't find a reasonable one.”

My note: The actual .net Exception is System.DllNotFoundException

I don’t really have a silver-bullet answer for this one, but I thought I’d discuss how I would go about troubleshooting this if it happened to me or if I got a case on it.

Potential causes

An “initialization routine failed.” problem generally occurs because a) some dll that this dll referenced was not found or b) something in the init method failed.

The problem is that option B could really mean anything, such as the init routine making calls that require permissions that you don’t have, or for example that it interacts with the desktop which is not permitted from a service etc.

The most common issue though is that the component was not properly installed, so that some of its dependencies are not available.

Differences between web forms and win forms (in this context)

The most apparent difference between a web forms and a win forms application is the user context that it runs under.  Unless you’re impersonating or the app pool id is changed, the application pool runs under the network service account which has very limited access. 

Another big difference is that an asp.net service is not allowed to interact with the desktop. 

There are of course a lot of other differences but these are the two that I think may be relevant for this issue.

Where to go next

1. Run filemon + regmon to look for not found or access denied

My first troubleshooting step is exactly what they mention that was done in the email.  Running procmon (filemon+regmon) to look for not found or access denied, specifically focusing on the files that RdpNet.dll depends on.  A quick search gave me this link to the release notes of the Oracle RDB Provider for .NET including the dependencies for it.

I still believe that there might be something in there, but it is a bit tricky to look at these files if you are not exactly sure what you are expecting to find.   A neat trick, since there is a working win forms solution is to run filemon+regmon when it works to see if it accesses some files that it didn’t access in the web forms case.

The other thing that is interesting here is to find out if it was tested in a win forms application on the same machine as the web server.  This would tell us if we can exclude the idea of some dependencies missing.

2. Reinstall the component

If the win forms app was not tested on the actual web server I would simply reinstall the component as this is a quick and easy step to make sure things got registered properly.

3. Test with elevated permissions

If the problem reproduces on a development server or a test server you can temporarily create a new application pool running under a user that has high permissions (admin or similar) to see if the problem still reproduces.  If it does, then you have a permissions issue and it is back to filemon/regmon again.   I wouldn’t recommend running with elevated permissions live of course, but in a test/dev environment it would be an ok step to try.

4. Get a memory dump on the Exception and log stack traces for all first chance exceptions

With debug diag you can set up a crash rule for the w3wp.exe process and configure it to “Log Stack Trace” for unconfigured first chance exceptions, and then use “add exception” to add a specific action for the CLR (.NET) Exception System.DllNotFoundException and ask it to create a full dump when this exception occurs.

For more info on setting up such rules see http://blogs.msdn.com/tess/archive/2009/03/20/debugging-a-net-crash-with-rules-in-debug-diag.aspx

Once you have the log and the dump, look in the log for exceptions (native or .net) that occurred just before the System.DllNotFoundException as this is probably what later bubbled up to generate this exception.

In the dump you can dump out all recent .net exceptions to see if some exception has occurred recently that might lead up to the exception you are seeing now.  

5. Contact the vendor of the component

If you can’t get anything from the above steps I would suggest contacting the vendor of the component.  Sometimes (I don’t believe it is the case with this particular component though) the dll you are trying to use is not mean to be used from ASP.NET because it interacts with the desktop, uses System.Drawing or similar.  

Perhaps you need to configure something to be able to use it from ASP.NET or they can tell you what permissions are needed etc.

If you ran through step 4 and did get a call stack for an exception occurring in one of their components right before the DllNotFoundException occurred, that is usually very useful.

Laters,
Tess

A colleague of mine, John Allen, created an awesome tool way back that displays memory usage in a process very nicely.  I use screenshots from it from time to time in presentations or posts but unfortunately that tool is not publicly available.

Since lots of people have asked me about it I figured I would just do a quick and dirty mock-up (based on the same ideas) and post the VS solution here.

This sample tool will give you a visual overview of the virtual memory space (from a memory dump), show you where your allocations exist and what types of allocations you have.   For example in the screenshot below you can see that at the beginning of the memory space we have a lot of virtual allocations (dark green – committed, light green - reserved), then we have a lot of free space (white) and towards the end of the memory space we can see our dlls scattered out (dark red).

In the bottom screen we can see our GC (.NET) Heaps.  In other words, most of the virtual allocations we see in the top screen are really GC Heaps. One little caveat is that for the GC heaps it doesn’t show what is reserved for the GC heaps, only what is committed, i.e. what we actually use.

I have separated them out so that you can use the tool for non-.net apps as well if you want to.

The purpose of looking at something like this is to get a grasp for how much fragmentation we have, how much reserved vs. committed memory we have etc.  and if we do have a lot of fragmentation, where should we start looking in order to reduce it.

The original tool is a bit more complex as it can read in memory dumps etc. and lets you zoom into different areas to get more details but for the most part what you see above is enough.

<disclaimer> 
   The sample tool posted here is almost completely void of error handling, and does a lot of output parsing so just be aware that it is by
   no means claiming to be robust…   I just wanted to get it out there so you have the functionality
</disclaimer>

To use the sample tool follow these steps:

1. Open the memory dump in windbg and set the symbols properly

.symfix c:\mycache
.reload

2. Run !address and copy the output to a text file.

3. Load sos

.loadby sos mscorwks

4. Run !eeheap –gc and copy the output to another text file

5. Open the tool, click on Load !address and load the !address output,  click on Load !eeheap –gc and load the !eeheap –gc output

Suggestions for features you might want to implement:

There is a lot of info stored in just these two files, including how much memory the process is using, how much is reserved and committed etc.  as well as sizes of GC heaps, number of GC heaps and so on.   

I didn’t have time to add all this but it would probably be nice to show it, as well as automatically gathering the info from a dump, or even doing the right-click on a memory area for more info + zoom in.

Feel free to build on it, and if you do and post it somewhere, please add a comment with a link to your version and share the wealth:)

Have fun,
Tess

Since a .net exception is a .NET object like any other, it gets stored on the GC heap when you (or some code you call) calls new XXException(). 

This means that if you have a memory dump of a process you can dump out all the recent exceptions that have occurred, or rather all exceptions that have not yet been garbage collected, which will give you a good feel for what exceptions occurred recently. 

Doing this is pretty easy if you have sos loaded (.loadby sos mscorwks) in windbg

To dump information about a specific exception you can use !pe 02ea6b0c for example which will show you the stack, exception name etc.

The problem is that you have to do this for all the exceptions which can become pretty tedious if there are many of them, so to speed up the process you can use the .foreach command to automate the process

This will go through all objects on the heap, where the typename contains the word exception and print them out with !pe (short for !PrintException)

Note that this will also include items that are not really exceptions like the System.Text.DecoderExceptionFallback, but since those are not exceptions it will just try to print them out but fail, so you can ignore those.

The output looks like this:

If you also want it to print out the nested exceptions you can just change the command a little bit to say

A couple of related posts:

Are you aware that you have thrown over 40,000 exceptions in the last 3 hours?
.Net exceptions - Tracking down where in the code the exceptions occurred
Questions about .net Exceptions
ASP.NET 2.0 Crash case study: Unhandled exceptions

Laters,

Tess

After installing .NET 3.5 SP1 you may get Validation of viewstate MAC failed exceptions when doing post backs on ASP.NET pages

Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.

Stack Trace:

[HttpException (0x80004005): Unable to validate data.]
   System.Web.Configuration.MachineKeySection.GetDecodedData(Byte[] buf, Byte[] modifier, Int32 start, Int32 length, Int32& dataLength) +289
   System.Web.UI.ObjectStateFormatter.Deserialize(String inputString) +140

[ViewStateException: Invalid viewstate. 
	Client IP: 127.0.0.1
	Port: 34562
	User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; WOW64; .NET CLR 2.0.50727; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.590; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; Zune 3.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8)
	ViewState: /wEPDwUKMTc2NzE0NzA0NmRkmWc0SFS8H55FfURfpUekG2KhS8g=
	Referer: http://localhost/MySite/Default3.aspx
	Path: /MySite/Default.aspx]

[HttpException (0x80004005): Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.]
   System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidationError) +106
   System.Web.UI.ViewStateException.ThrowMacValidationError(Exception inner, String persistedState) +14
   System.Web.UI.ObjectStateFormatter.Deserialize(String inputString) +242
   System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter.Deserialize(String serializedState) +4
   System.Web.UI.Util.DeserializeWithAssert(IStateFormatter formatter, String serializedState) +37
   System.Web.UI.HiddenFieldPageStatePersister.Load() +207
   System.Web.UI.Page.LoadPageStateFromPersistenceMedium() +105
   System.Web.UI.Page.LoadAllState() +43
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +6785
   System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +242
   System.Web.UI.Page.ProcessRequest() +80
   System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context) +21
   System.Web.UI.Page.ProcessRequest(HttpContext context) +49
   ASP.default_aspx.ProcessRequest(HttpContext context) +4
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +181
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +75

Cause:

This will happen if you have specified an action on the forms element, and if the action is different than the page you are browsing to, i.e. in this case the page I am browsing to is Default3.aspx, but the action is set to Default.aspx (as you can see from the Path and Referer in the error message)

<form id="form1" runat="server" action="Default.aspx">

The reason this occurs after installing SP1 for .NET 3.5 is because prior to this service pack, the action attribute was ignored.   Now that it is no longer ignored, the post-back will actually post back to the action page, and of course the view state for Default3.aspx will not be valid for Default.aspx.

You’ve heard it before:)  this is not a bug, it’s a feature…  in this case that’s actually true…

Resolution:

Remove the action attribute, or change it to post to the right page if you want to have viewstate enabled.

Laters,
Tess

A few weeks back me and Micke (one of our Architect Evangelists) had a session at TechDays where we talked about “things that looked good on paper” i.e. things that sound pretty ok in the design/development phase but sometimes turn out to be a disaster in production.

We are both pretty passionate about making the lives of the ops. people easier by thinking about the maintenance of the sites/apps at design time, rather than having it be an afterthought.  I stole the title of this post from one of Mickes talks about bridging the gap between dev and ops. 

The topics we brought up are based on issues that we commonly see in prod. environments and we started off each section with a quote and dissected the pros and cons and what we think people should think about…

Here is a summary:

1. With web services we can use the same interface from both our web apps and win forms apps

While this is perfectly true, there is a right and a wrong time and place for everything.   When you make a web service call, remoting call or WCF call for that matter there is a lot of stuff that goes on behind the scenes, like getting a connection, serializing and de-serializing parameters and return values, spawning up new threads to make the new httpwebrequests etc.

I’ve talked a lot about issues with serialization and de-serialization, specifically when it comes to serializing large sets of data, complex objects or datasets for example.   Serialization of these types of objects generate a lot of memory usage and is often quite expensive when it comes to CPU usage. Also, if you call web services within the same app pool you can run into issues like thread pool exhaustion.

The moral of the story?  Use web services if you need to get data that you couldn’t get by loading up a component in the app.   In other words, if you need to go to a DMZ or a different network to get it. 

If you want to create a web service (hosted on the same server as your asp.net app) so that you can get the same functionality both from your asp.net app and your win forms apps,  a good option is to write a component that does this, and then wrap it in web service calls for your win forms apps to use.

At the very least you should be really frugal with the amount of data you send back and forth.  For example filter the data before bringing it back so you transfer as little data as possible.

More reading:

Case Study: ASP.NET Deadlock calling WebServices
ASP.NET Performance Case Study: Web Service calls taking forever
OutOfMemoryExceptions while remoting very large datasets
Dataset serialization

2. Bob, just turn on tracing on the WCF end point

A lot of app configuration these days is done in XML.  You often hear that XML is so great because it is human readable/writable, but is it really???  Even with XML configuration some things are extremely wordy and require a lot of xml code to configure.  Imagine that you have an issue in production where you need Bob (or Jerry, or Ruth or <replace the name of your favorite ops guy/gal here>) in operations to turn on WCF tracing on all the servers in the web farm.  He probably doesn’t have Visual Studio handy to swap this through the UI so he’ll probably use the incredibly useful configuration tool Notepad to write the 10+ lines of XML needed to enable the tracing. Rinse and repeat for all servers in the web farm.

Is that fair? What if there is a mistake in the XML? 

To make it a bit easier on Bob you could provide him with two web.config files (with and without tracing) and that is at least an improvement, but then there is of course the issue of forking,  if you have to change something in one config you need to change it in both etc.

A nicer way would be to create some powershell commandlets to enable tracing or whatever config items you want, like connection strings or whatever else you might have stored in your configs.  The nice part about this is that it is scriptable so you could create one script and run it on all servers.

While you’re at it, why not implement a powershell provider for your app that allows the ops guys to configure parts of your application, or get values from your application from powershell.   Powershell objects are .net objects so scripts etc. are written in .net.

Taking it one step further, you can even call powershell command lets from an MMC snapin in case you want to configure things from there.

3. Let’s put this data in session scope so we don’t have to go back and forth to the database all the time

Got a tweet from Fredrik earlier this week where he suggested a title for a pod-cast “Session state is the Achilles heel of ASP.NET”.  I definitely agree… Again, everything has it’s pros and cons, and session state is nice for saving SMALL pieces of user specific data, but if you have a high-load web app, I would say that you seriously need to consider going stateless.

Over the years I have seen many many web apps with lots of data in session state.  A favorite seems to be to put datasets in session scope to avoid hitting the database all the time.  Especially if the query to get the data is pretty complex.

Now imagine that this site grows and needs to be replicated on different servers in a web farm so we can’t use in-proc session state anymore.  In that case you would need to put session objects in an out of proc session store like state server or sql server.   Just like with the web service calls you need to serialize and de-serialize data as you are now doing cross process calls, which again uses a lot of mem and a lot of CPU.

Even if you have one server and store it in-proc, there is a real chance that you will rack up a lot of memory if you have a lot of concurrent sessions.

For out of proc session state it gets even worse… without putting it in session scope you would go out and get the data when you need it.  If you have out of proc session scope you will go out and get every single session var for the given user on every single request (with session state enabled), and then put it back in the session store on end request.  That’s a lot of serialization/de-serialization.

Another thing about hoarding data, unrelated to session state that is a bit of a pet-peeve of mine is when apps bring in loads of data from the database and process it in-proc, based on the notion that they don’t want the DB to be a bottle neck.

Just some food for thought there…  I think that very few applications are better/faster at handling/processing data than database engines, hence if the code in the app is not better at data processing than the DB, then aren’t chances pretty good that this will just create a bottleneck in the app instead? 

Some related posts:

Debugging Script: Dumping out ASP.NET Session Contents
ASP.NET Memory: Thou shalt not store UI objects in cache or session scope
ASP.NET Memory Leak Case Study: Sessions Sessions Sessions…

4. HttpUnhandledException, does that mean I should restart the server?

There are a lot of posts and discussions around which logging framework is best, and I think at least some of you agree with me that a lot of time is spent in the design phase to work out which one to use, but how much time do you spend thinking about what to log?

Often when i get cases and ask for event logs the event logs literally look like a nice and very ornate Christmas tree.  Most of the entries contain stack traces, some contain exceptions that are handled mixed with some that are not.  That’s ok, at least the part about stack traces, I love them, they make sense to me and I can use them to troubleshoot once I have waded through the unimportant events with log parser or some other tool.  But… does this really make any sense to Bob in operations?   Unless he has a dev background chances are that it makes no sense at all and unless it is something he has seen a million times before, he probably wont know how to act, or not act on the events.

In my humble opinion Bob should only get about max 5 events a day in his log, and that’s on a busy day.  Every event should have a nice problem description and most importantly action like restart the server, run diagnostics on the DB etc.  Sometimes you don’t know the action or even the problem description and then maybe the action could be “report this unknown failure to dev”.     I bet that your ops guys/gals would be a lot happier…

I am not saying that you should stop logging the exceptions, but preferably not in the same log as the ops logs.

Oh, just one more thing about this… I’ve talked before about apps that throw a lot of exceptions and the perf impact this has even if the exceptions are handled.   In fact, if you forget about the perf impact,  there is another disadvantage to throwing a lot of exceptions and that is that the app is a lot less supportable… why?  because if you need to debug the app and dump or log on a specific exception type, this can become very hard if you have a lot of benign exceptions as you will generate lots of dumps or logs which takes time, disk space, and more importantly it is very hard to find the needle in the haystack.

5. With ASP.NET we can update the sites even when they are live, ASP.NET will handle the rest

When you update the site with a number of new assemblies for example, old requests will finish running with the old assemblies and a new appdomain will be created when the next request comes in with the new assemblies loaded. 

So far so good…

Now, picture that you have a lot of assemblies in your update and that a new request comes in when you’re halfway done copying the assemblies.  In that case new requests will be serviced in a partially updated application, and you may even see locking issues if the load is really high.

If you have a web farm and don’t use sticky sessions a post can be done from one server (updated) to another (not updated) and if you have changed user controls etc. then the view state might become invalid.

So, for low volume sites, updates to a live environment is usually cool, but if you have a lot of load you need to consider taking the server out of rotation before updating, or update when load is not as high.

Related post:
ASP.NET Case Study: Lost session variables and appdomain recycles

6. It works on my machine, let’s go live

Test and load test, with lots of scenarios and with appropriate load levels, Nuff said.

I know that you know this already, and yet we get so many cases where issues that come up in production and become crisis situations could have been avoided if the applications had been properly load tested.

A lot of issues, such as a specific method causing a leak, or a hang, can even be discovered with very simple load testing at the dev stage.   

There are plenty of really good load testing tools and profiling tools out there like Load runner, Ants profiler, Visual Studio Team System Test etc.  I’m sure you have your own favorites.

For poor mans stress testing, that can be done on the dev machine, you can use the free tool tinyget that comes with the IIS 6.0 Resource Kit

7.
- Do we have a plan for crashes? 
- We’ll document it in phase 2

Crashes, hangs and memory leaks is not usually something that people really plan for.  I have seen extreme examples of planning for these types of issues at some of my customers like:

#1 One company that I work with has a full fledged plan for what will happen if a crash/hang/memory leak is discovered in production.  The plan includes documentation for ops with step by step instructions on how to get dumps, how to upload them etc.  They even do fire drills with ops to test that their plans work.

#2 Another company I work with has included code in their app to dump the process under certain conditions and the dumps are then automatically bucketized by issue type and scripts are autorun to debug the dumps and collect vital information about the issue.   In other words, most of their analysis for these cases is automated to the tee.

Not everyone has to go to these extremes, especially if the app isn’t mission critical, but a good recommendation would be to have some documentation for ops on how to act in general cases so that you get the most data possible about the issue.  Like how to gather performance counter data or dumps.

In my last post i described how you can set up rules with debug diag that ops can activate as needed.

8. What do you mean baseline?  I think CPU usage is usually around 40%, maybe 50

When you troubleshoot a problem like a hang, memory leak or crash a key piece of information is often “what is different in the failure state compared to the normal state”.

Setting up a performance counter log that rolls over for example every 24 hours and alerts ops when certain values exceed some predefined number is very easy and has extremely low impact on the system.

Having this history when you troubleshoot something is as i mentioned very useful since you can see things like, right around the time it crashed memory went up to x MB, or we started seeing a large number of exceptions etc.   Although it might not solve the issue it can often give you a good direction to move in.

This article about Performance monitoring and when to alert administrators is from 2003, but except for a few changes in a few of the counters most of it still holds true.   The article describes what counters to look at, suggested trigger values and some typical causes for issues that cause the counter to hit the trigger.  It’s simply a must read.

9. I don’t need to care about memory management, isn’t that what the GC is for?

True, true, the GC manages the .net memory in the sense that it will automatically collect any objects that are collectable and free them so you don’t have to call free or release as you would in native languages.

Memory problems are very seldom caused by the GC not collecting as it should.  Instead they are often caused because the app is unintentionally still holding on to the objects, through references or through not disposing/closing/clearing disposable objects.

I have written tons of articles around memory management and memory issues in .net so rather than listing them all, just look at the Memory #tag to get more info about .net memory issues, how the GC works and how .net memory management works.

The moral of the story here is that even if you have a garbage collector, you still need to make sure that your memory is ready to collect.

I would love to hear your comments on these topics. 

This is by no means supposed to be a complete list, so I would also love to hear about your own tips on how Devs can make life easier for the Ops guys and avoid production issues.

On the soapbox,
Tess

During mine and Micke’s presentation at TechDays this week I showed a demo of setting up rules with Debug Diag to identify the cause of a crash in an ASP.NET application.

Even though debugging might be tricky, setting up rules in Debug Diag is beautifully simple and I personally believe that it would be a good idea for anyone running a web site to have debug diag installed along with a few instructions for the ops personnel on how to set up the rules.   Better yet,  you can set up the rules in advance and just activate or deactivate them as needed.

Here is a recap of that demo…

Problem description

The application crashes with the following event in the eventlog

Log Name:      Application
Source:           Application Error
Date:              2009-03-20 11:12:09
Event ID:        1000
Task Category: (100)
Level:              Error
Keywords:       Classic
User:               N/A
Computer:       MYMACHINE
Description:
Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919413, faulting module kernel32.dll, version 6.0.6001.18000, time stamp 0x4791a76d, exception code 0xe053534f, fault offset 0x000442eb, process id 0x%9, application start time 0x%10.

So the w3wp.exe process crashed due to a 0xe053534f exception (which happens to be stack overflow, but even if you don’t know that it doesn’t really matter)

Setting up Debug Diag rules and gathering data

When you open up Debug Diag you are met by the following screen where you can set up different rules.

If you have a potential memory leak or a memory issue, you should choose the memory and handle leak rule.  This will inject a dll in the process that will track any allocations or de-allocations that occur while the rule is active.  Once you have “leaked” memory you can then get a memory dump with debug diag, and this will now contain info about what stacks etc. allocated the memory that is still in the process.

This works well for native memory issues.  For .net memory issues you should be aware that allocations for .NET GC heaps are made my mscorwks, which means that .net memory issues will show up as mscorwks having a potential leak. I would recommend that you read some of my earlier posts on troubleshooting .net memory issues.

The IIS Hang rule is very useful if you have some pages that sometimes take longer than they should.  With this rule you can set up triggers to get logs or memory dumps if a page takes longer than x seconds.

If you have an exception you want a dump for or a crash you should use the crash rule and that is what we’ll do now…

The next step is to choose a process that the rule should apply to.  Here we can choose either IIS which means that it will apply to all IIS processes like w3wp.exe, dllhost.exe, inetinfo.exe etc.  

In this case we will only set up a rule for w3wp.exe (the asp.net process).  If we just leave it like that, and the process crashes it will reattach the rule next time the process comes up.  It will also apply the rule to all w3wp.exe processes if there are multiple on the system.  If you only want it to apply to one specific w3wp.exe instance you have to check the checkbox for “This process instance only”.

The next window that comes up in the wizard lets you configure what you want to trigger the debug diag action.  This can either be just a crash (in which case you just click next).  You can also set it up to get dumps or log events on specific exceptions (as we will do), or on breakpoints.  The PageHeap Flags option is used if you want to troubleshoot heap corruption issues.

Since we want to trigger a dump when the process gets the 0xe053534f exception, we will click on the Exceptions… button here

0xe053534f is a native exception so we can just enter it in the box for the exception code.  We can optionally give the exception a name (this will just label the dumps that are generated). and for the action type we choose full dump to have it dump on the exception.

Note that you can set an action limit here to avoid that it keeps generating dumps.  This is pretty useful if you want to set up a rule for System.NullReferenceException for example, or something else that might happen a lot.

If we would have wanted to dump on a particular .net Exception we would have just choosen the CLR (.NET) Exception from the listbox on the left and we could have then entered the specific exception name (such as System.NullReferenceException)

Now we are ready with the rules and can just click ok, next, next, finish to activate the rule.

Once this is done we can reproduce the issue and debug diag will generate the dump and put it under the <debug diag dir>\logs\crash rule… directory.

Analyzing the data

At this point we can click the Analyze Data button and have debug diag analyze the dump for us.  This works extremely well if the issue is purely native.   Just a few quick notes about this though…

1. I wouldn’t recommend analyzing it on a production server, so the best thing is to copy the dump to a dev machine, and analyze it (from the advanced analysis tab)

2. You should set up the symbol path under tools/options and settings to

SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols

3. If the issue is not purely managed, you may still need to open the dump in windbg

If we analyze it we get this info at the top of the html report

and thread 25 looks like this:

This tells us that we seem to be crashing because of a stack overflow, but the stack looks very funky so we don’t really know what caused the stack overflow. 

Mscorwks is listed as the module that caused the crash, but this is just because mscorwks is the component that raises the native exception.

For stack overflows as most of you probably know, the most common reason is that we are in some type of recursive loop, so what we really would like to know here is what is on this stack…  The reason why it is showing up with just addresses and not method names, is because debug diag doesn’t understand .net so we’ll have to bring the dump to windbg to analyze it and check out the .net stack.

In windbg we can then load up sos (.loadby sos mscorkws) and run !clrstack on the active stack to get the callstack

Once you get to this point it is pretty simple, clrstack shows a recursion, as expected, which in this case is because of a logic error in the GameInfo constructor + GameInfo explicit cast operator.

This was a pretty long post, but if you have a look at debug diag you’ll find that the user interface is pretty intuitive and again, i would really recommend that you have a look and consider setting up a few rules for operations to activate if the stuff hits the fan.  

Laters,

Tess

I was working on some Silverlight samples and needed an image that could flip over as in the example below.

All the samples I could find on the net were pretty complex and contained a lot of code to do the animation and I wanted something really simple.

To create the illusion of an image that flips over (or any kind of UI element that flips over) you just need 4 things

1. A front image / UI element
2. A back image / UI element
3. A flip animation
4. A reverse animation

The flip and reverse animations can be very simple as you will see below. 

In order to make it reusable in our application we can create a UserControl and call it FlipImage

First create 2 UI elements (front and back), in this case Grids but they could really be any kind of items, like images, datagrids etc.

The animation for flipping the image is very simple. First we make the back image/UI element zero size, then to do the flip animation we just shrink the front image towards the middle, and when it is shrinked to zero size we expand the back image.

To get it to shrink around the middle we need to set the RenderTransformOrigin to 0.5,0.5... and in order to be able to do the shrinking/expanding we need to add <ScaleTransform...> to the UI Elements that we can target in the animations.

With all this, our XAML now looks like this

Now we can add the storyboard to the UserControl.Resources to do the filp animation

There are two animations here... the first one targets front, and goes from 00:00:00 to 00:00:00.2 (200 milliseconds), and in that time it will ScaleX from the current value down to 0.
The second one goes from 00:00:00.2 to 00:00:00.4 (also 2 milliseconds) and will expand back by scaling x wise from the current value to 1 (full size).

sbReverse looks exactly the same, except for that front and back are reversed.

In order to be able to configure the front image, back image and to be able to call Flip and Reverse from the app we can add the following methods and properties for the class  (note:  you need to add a using statement for System.Windows.Media.Imaging for the BitmapImage)

If we want to create a menu similar to the one they use on the Microsoft Italy Student page, you can add the images to our application like this

(In order to be able to use <my:… you need to add a reference to your assembly in the user control definition, eg. xmlns:my="clr-namespace:FlipMenu")

The result looks something like this:

I added a rotate transform to make the menu look pretty, and a Tag that we can use in the mouseleftbuttondown to navigate to the link. The tag is just a property of any control that can be used to store any data you want related to the control.

And the code for this menu page is then extremely simple... we just flip on mouseenter, reverse on mouseleave and navigate on mouseleftbutton down:

And that is all there is to it...

Btw, you need to check out Nikhils Silverlight.FX project with lots of UI goodies.  He has a control in there that does the flip shown above.

Until next time,
Tess

Yesterday I had a nice chat with Richard and Carl at .NET Rocks.  We talked about .net, debugging, memory issues and a bunch of other stuff, and in the end I got a pop quiz that I failed:)  See if you can figure out what caused the outage on their servers…

The show will be uploaded here some time today.

Laters,
Tess