I know the answer (it's 42)

Back to Basics: Memory leaks in managed systems

Someone contacted me over my blog about his managed application where the working set goes on increasing and ultimately leads to out of memory. In the email at one point he states that he is using .NET and hence there should surely be no leaks. I have also talked with other folks in the past where they think likewise.

However, this is not really true. To get to the bottom of this first we need to understand what the the GC does. Do read up http://blogs.msdn.com/abhinaba/archive/2009/01/20/back-to-basics-why-use-garbage-collection.aspx.

In summary GC keeps track of object usage and collects/removes those that are no longer referenced/used by any other objects. It ensures that it doesn’t leave dangling pointers. You can find how it does this at http://blogs.msdn.com/abhinaba/archive/2009/01/25/back-to-basic-series-on-dynamic-memory-management.aspx

However, there is some catch to the statement above. The GC can only remove objects that are not in use. Unfortunately it’s easy to get into a situation where your code can result in objects never being completely de-referenced.

Example 1: Event Handling (discussed in more detail here).

Consider the following code

EventSink sink = new EventSink();
EventSource src = new EventSource();

src.SomeEvent += sink.EventHandler;
src.DoWork();

sink = null;
// Force collection
GC.Collect();
GC.WaitForPendingFinalizers();

In this example at the point where we are forcing a GC there is no reference to sink (explicitly via sink = null ), however, even then sink will not be collected. The reason is that sink is being used as an event handler and hence src is holding an reference to sink (so that it can callback into sink.EventHandler once the src.SomeEvent is fired) and stopping it from getting collected

Example 2: Mutating objects in collection (discussed here)

There can be even more involved cases. Around 2 years back I saw an issue where objects were being placed inside a Dictionary and later retrieved, used and discarded. Now retrieval was done using the object key. The flow was something like

1. Create Object and put it in a Dictionary
2. Later get object using object key
3. Call some functions on the object
4. Again get the object by key and remove it

Now the object was not immutable and in using the object in step 3 some fields of that object got modified and the same field was used for calculating the objects hash code (used in overloaded GetHashCode). This meant the Remove call in step 4 didn’t find the object and it remained inside the dictionary. Can you guess why changing a field of an object that is used in GetHashCode fails it from being retrieved from the dictionary? Check out http://blogs.msdn.com/abhinaba/archive/2007/10/18/mutable-objects-and-hashcode.aspx to know why this happens.

There are many more examples where this can happen.

So we can conclude that memory leaks is common in managed memory as well but it typically happens a bit differently where some references are not cleared as they should’ve been and the GC finds these objects referenced by others and does not collect them.

.NETCF: The mini Y2K

I am sure most people haven’t yet forgotten the Y2K problem. This year our team faced a mini Y2K, but don’t worry we anticipated it and all is well.

If you head over to NETCF Wikipedia page you’d notice the NETCF versions look as follows

<MajorVersion>.<MinorVersion>.<Year><DayofYear>.<Rev>

A sample version number is

3.5.7283.0

This represents .NETCF version 3.5 build on the 283rd day of 2007. I guess by now you can guess the problem. We use a single digit to represent the year. By that nomenclature, the version of the build churned out today would be 3.5.0001.0 which is lower than the one generated the year before and would fail to install.

These numbers are automatically generated by scripts on the server that churns out daily builds. The numbering system was invented a long time ago in early 2000 and no one bothered to fix it. We anticipated that it’s going to fail as we move into the new decade and have updated it to now have 2 digits for the year (and yes we know it will break again in the future, but hopefully that’s too far out to care right now).

Happy new Year.

Some things I have learnt about SW development

Working in the developer division is very exciting because I can relate so well with the customers. However, that is also an issue. You relate so well that you tend to evolve some strong opinions that can cloud your view. While working in the Visual Studio Team System team and now in the .NET Compact Framework team I have learnt some lessons. I thought I’d share some of them

I am not *the* customer (or rather the only customer)
Even though I represent the customer in different avatars (sometimes as a developer, sometimes as an office worker, sometimes just as a geek) I am not THE only customer that the product targets. When your product is going to be used by 100,000 developers/testers the average or even the predominant usage is going to be very different from what you think it will be. Sitting though an usability study is a very humbling experience. Like I believed that everyone should just be using incremental search.

Consistency is important
Geeks and bloggers tend to over-tout coolness. While cool user experience (UX) seems awesome, it is frequently overdone and what seems cool on casual usage tends to tire soon. Consistency on the other hand lets your users get on-board faster and lets them spend time doing stuff they care about and not learning things that should work automatically.

Consistency doesn’t mean that every application needs to look exactly like notepad. Expression Blend is a great example which looks refreshingly cool (appeals to the designers) and at the same time provides an experience that is consistent to other windows apps.

Learn to let go
”If there’s not something you can actively do on a project, if it’s something you can only influence and observe, then you have to learn to provide your feedback, and then let go and allow other people to fail… People don’t learn if they never make any mistakes” ~ Raymond Chen on Microspotting

Corporate software development is very different from Indie development. Large software development projects have a bunch of people/teams involved. It is not necessary that the collective opinion matches yours. At some point you need to learn to let go and do what is required. As an example I can debate on what a Debug Assert dialog should look like or do. However, there are other folks to design and think about the UX, as a developer I need to give inputs and once the call has been made it’s my job to provide the best engineering solution that implements that UX**.

** Do note the debug assert dialog is a fictitious example, I never worked on the IDE side.

Indic Language Input

If you have tried inputting Indian languages in Windows you know it’s a major pain. That is particularly sad because Windows comes with very good support of Indian languages. I had almost given up using my native language Bengali on a computer due to this. Even when I was creating the About Page for this blog and wanted to have a version in Bengali, I had to cut it short a lot because typing it out was so painful.

There are web-based tools like the Google Transliteration tool that works well for entering text into web-pages where they are integrated (e.g. Orkut). However, I wanted a solution that pans the desktop, so that I can use it for say writing a post using Windows Live Writer.

Enter the Microsoft Indic Language Input tool. Head over to the link and install the desktop version. You can install the various languages individually (currently Bengali, Hindi, Kannada, Malayalam, Tamil and Telugu is supported). I personally installed the Bengali and Hindi versions.

Since I am on Windows 7 which comes pre-installed with Complex language support I needn’t do anything special. However, on older OS like XP you need to do some extra steps which are available through the Getting Started link on that page.

Once you are setup you can keep the Windows Language Bar floating on the desktop. The tool extends the language bar to allow you to enter Indic languages using an English keyboard via transliteration.

Go to the application where you want to enter Indic language and then switch to Bengali (or any of the other 6 supported Indic language) using this language bar. Start typing বেঙ্গলি using English keyboard and the tool will transliterates. The moment you’d hit a word terminator like space it inserts the Bengali word.

I tried some difficult words like কিংকর্তববিমূঢ় and it worked amazingly well

I had a very good experience with the tool. The only issue I faced was that the tool was extremely slow with some WPF apps like Seesmic twitter client. However, I got to know from the dev team that they are aware of the issue (it’s for some specific WPF apps and not WPF in general). I hope they fix it before they RTM (the tool is in Beta).

Tip: You can hit alt+shift to cycle the various languages in the toolbar without having to use your mouse (which is handy if you typing using a mix of languages).

NETCF: Count your bytes correctly

I got a stress related issue reported in which the code tried allocating a 5MB array and do some processing on it but that failed with OOM (Out of Memory). It also stated that there was way more than 5 MB available on the device and surely it’s some bug in the Execution Engine.

Here’s the code.

try{
    byte[] result;
    long GlobalFileSize = 5242660;  //5MB
    result = new byte[GlobalFileSize];
    string payload = Encoding.UTF8.GetString(result, 0, result.Length);
    System.Console.WriteLine("len " + payload.Length);
}
catch (Exception e)
{
    System.Console.WriteLine("Exception " + e); 
} 

The problem is that the user didn’t count his bytes well. The array is 5MB and it actually gets allocated correctly. The problem is with the memory complexity of the UTF8.GetString which allocates further memory based on it’s input. In this particular case the allocation pattern goes something like

  5MB  -- Byte Array allocation (as expected and works)

  5MB  -- Used by GetString call
 10MB  -- Used by GetString call
  5MB  -- Used by GetString call
 10MB  -- Used by GetString call

So GetString needed a whooping 30MB and the total allocation is around 35MB which was really not available.

Morale of the story: Count your bytes well, preferably through a tool like Remote Performance Monitor

Silverlight: Where are my colors

The Silverlight System.Windows.Media.Colors class is a trimmer counterpart of the WPF Colors class. E.g. Colors.AliceBlue is available in WPF but not present in Silverlight. However, these standard colors are indeed available in Silverlight XAML.

In effect in Silverlight XAML you can use

<Border Background="AliceBlue" BorderBrush="Coral" BorderThickness="2" CornerRadius="10" >

However, if you try that in code it fails to compile

foo.Background = new SolidColorBrush(Colors.Coral);

This means if you ever need to use standard colors in Silverlight code you have to use RGB values, which for Coral would be something like

foo.Background = new SolidColorBrush(Color.FromArgb(255,255,127,80));

This is kind of painful. Moreover, you need to frequently be able to convert between HTML color, RGB and XAML standard color names when you are developing an Silverlight application. To make the job easier I hacked up a small Silverlight 2 app at http://www.bonggeek.com/Ag/Colors/. You can do the following with this

1. Change colors with sliders, RGB, HTML, XAML standard color names to see what the color really looks like
2. All the format is kept in sync, this means if you change the sliders to get to the color Salmon then the RGB value and HTML will be updated and the drop down will switch to show the selected colors name
3. You can easily create variants of standard color. E.g. select Salmon using the drop down and then push the sliders to get to a slightly lighter/darker shade of the color

Enjoy and extend as you please. Sources are available here.

Global variables are bad

<This post is about C++ (C# folks are saved from this pain)>

One of our devs taking care of NETCF on Nokia S60 reported that after the latest code reached their branch things are working very slow. It was observed that Garbage Collector is getting kicked in way more than it should. Investigation showed something interesting.

I added a global var gcConfig which had a fairly complex constructor and that among other things sets the various member variables like the GC trigger threshold to default value (1mb). All of these works fine on Windows variants.

However, TCPL states “It is generally best to minimize the use of global variables and in particular to limit the use of global variables requiring complicated initialization.”. This is especially true for dlls. We tend to ignore good advice sometimes :)

For Symbian OS (S60) on device complex ctors of global objects are not run (without any warning). The members are all set to 0 (default member initialization). So in the gcConfig GC-trigger was being set to 0 instead of 1mb. The allocation byte count is compared against this value to figure out when it’s time to start a GC. Since it was 0 the collection condition is being met for every allocation and hence for every allocation request GC was being fired!!!

Actually considering that even after that the app was actually running shows that we have pretty good perf ;)

A good alternative of using global vars is as follows

MyClass& useMC()
{
    static MyClass mc; // static ensures objects are not created on each call
    return mc;
}

MyClass& mc = useMC();

Do note that this has some multi-threading issue. See http://blogs.msdn.com/oldnewthing/archive/2004/03/08/85901.aspx.

.NET Compact Framework BCL < .NET BCL

Over twitter some folks are regularly discussing about the fact that there are some chunks of desktop .NET Base class library (BCL) that are missing on the .NET Compact Framework (.NETCF). So I thought I’d post the rationale behind things that are missing and what criteria is used to figure out what makes it in.

First of all, .NET and .NETCF use completely different runtimes. So the BCL code doesn’t work as is. They need to be ported over. Something being available on the desktop reduces the effort of our BCL team but still significant cost is involved in making it work over NETCF. This means that its not as if everything is available on .NETCF BCL and we cut things out (elimination process), but the other way round where we start from 0 and we need to figure out whether we can take something in. In that process we use some of the following rationale.

1. ROI: Does the user scenario that it enables on a mobile device justify the cost
   System.CodeDom.Compiler. Yea right you expected to compile on the phone didn’t you :)
   
2. Easy workaround available: If there is an acceptable workaround for a given API then it drops in priority. A trivial example could be that we ship some method overloads but not the other. E.g. Instead of shipping all overloads of Enum.Parse we drop Enum.Parse(Type, String) and keep only Enum.Parse(Type, String, Bool).
   This applies at the level of namespace or Types as well.
   
3. Lower in priority list: It needs work to get it done and it’s lower in priority than other things that is keeping the engineering team busy.
   If there are a lot of request for these features and not good/performant workarounds available then it will go up the priority list and make it to future version of NETCF
   
4. Too large to fit: Simply too large to fit into our memory limitations
   E.g. Reflection.Emit which leads to other things missing like Dynamic Language Runtime, Linq-to-SQL
   
5. No native support: It uses some underlying OS feature which is either not present on devices or is not relevant to it
   WPF, Parallel computing support

With every release developers ask for new features and we also negotiate to increase NETCF footprint budget so that we can include some (but not all) from those requests. To choose what makes it in we use some of the criteria mentioned above.

Given the system constraints under which NETCF works a vast majority of the desktop APIs will continue to be missing from NETCF. Hopefully this gives you some context behind why those are missing. If you approach NETCF simply from trying to port a desktop application then you would face some frustration on the missing surface area.

BTW: Do post your comments on this blog or on twitter (use the #netcf hashtag).

<Update: I made some updates to this based on feedback />

NETCF: GC and thread blocking

One of our customers asked us a bunch of questions on the NETCF garbage collector that qualifies as FAQ and hence I thought I’d put them up here.

Question: What is the priority of the GC thread
Answer: Unlike some variants of the desktop GC and other virtual machines (e.g. JScript) we do not have any background or timer based GC. The CLR fires GC on the same thread that tried allocation and either the allocation failed or during pre-allocation checks the CLR feels that time to run GC has come. So the priority of the GC thread is same as that of the thread that resulted in GC.

Question: Can the GC freeze managed threads that are of higher priority than the GC thread?
Answer: Yes GC can freeze managed threads which has higher priority than itself. Before actually running GC the CLR tries to go into a “safe point”. Each thread has a suspend event associated with it and this event is checked by each thread regularly. Before starting GC the CLR enumerates all managed threads and in each of them sets this event. In the next point when the thread checks and finds this event set, it blocks waiting for the event to get reset (which happens when GC is complete). This mechanism is agnostic of the relative priority of the various threads.

Question: Does it freeze all threads or only Managed threads?
Answer: The NETCF GC belongs to the category “stop-the-world GC”. It needs to freeze threads so that when it is iterating and compacting managed heap nothing on it gets modified. This is unlike some background/incremental GC supported by other virtual machines.

GC freezes only managed threads and not other native threads (e.g. COM or host-threads). However, there are two exclusions even on the managed threads

1. It doesn’t freeze the managed thread that started GC because the GC will be run on that thread (see answer to first question)
2. Managed threads that are currently executing in native p/invoke code is not frozen either. However, the moment they try to return to managed execution they get frozen. (I actually missed this subtle point in my response and Brian caught it :))

Hope this answers some of your questions. Feel free to send me any .NET Compact Framework CLR questions. I will either answer them myself or get someone else to do it for you :)

I am the Empire

5 years back exactly on this very day I walked into Cyber Towers office of Microsoft India for the first time. If someone asked me at that time how long I plan to work for Microsoft, my answer would’ve been couple of years max. I was still a bit suspicious of the evil empire and it took some time for me to realize that “I am the empire”. The 5 years has been a mind blowing journey.

Even though in our industry working for any company for a 5 year stretch is a huge thing, it doesn’t seem so in Microsoft. Our team has 4 people over 20 years and at least 8 people over 10 years.

Some random thoughts on the last 5 years

1. Made my family relocate 3 cities before settling down in Hyderabad (my wife’s threat that the next relocation would be without her helped).
2. Bought an apartment
3. I have a daughter now who was born around 3 months after I joined MS.
4. Started working in a shared office with Ankur and then worked in various cubicles. Finally I have an office.
5. 3 job designations. SDE to SDEII to Senior SDE. I am fiercely protecting the SDE suffix.
6. It’s fantastic to walk into a book store and see books on your work and see screenshots of UI you’ve designed or description of algorithms you’ve come up with.
7. 327 blog posts on blogs.msdn.com
8. I still do not use Visual Studio to edit code
9. Finally gave up on Linux even at home. Bye bye Ubuntu, welcome Windows 7
10. I have 3 Xboxes in my office and none at home
11. I use Nokia phones
12. Made fantastic friends at work and sadly few of them are left with Microsoft India (either moved to Redmond or have left MS). Hello Amit, Srivatsn, Ankur :)
13. Always wanted to attend a BillG project review but he quit
14. Got into some email wars
15. I am no longer the fire-head I used to be. I actually think before I speak.
16. Went top down from n-tier architectures (TFS), desktop applications (VSTT) and now embedded (NETCF)
17. I prefer Starbucks coffee now instead of Darjeeling tea

How does the .NET Compact Memory allocator work

As I mentioned in one of my previous posts the .NET Compact Framework uses 5 separate heaps of which one is dedicated for managed data and is called the managed heap. The .NETCF allocator, allocates pools of data from this managed heap and then sub-allocates managed objects from this pool.

This is how the process goes

	At the very beginning the allocator allocates a 64Kb pool (called the obj-pool) and ties it with the current app-domain state (AppState).
	All object allocation requests are served from this pool (Pool 0). After each allocation an internal pointer (shown in yellow) is incremented to point to the end of the last allocated object.
	Subsequent allocation is just incrementing this pointer.
	This goes on until the point where there is no more space left in the current obj-pool
	Since there is no more space left in the current pool (pool 0) a new pool (pool 1) is allocated and all subsequent object allocation requests are serviced from the new pool.

The first question I get asked at this point is what happens when one object itself is bigger than 64Kb :). The answer is that the CLR considers such objects as “huge-objects” and they are placed in their own obj-pool. This essentially means that either obj-pools are 64Kb in size and has more than one object in them or are bigger than 64Kb and has only one huge-object in it.

Object allocation speed varies a lot between whether it is being allocated from the current pool (where it is just the cost of a pointer increment) or it needs a new pool to be allocated (where there is additional cost of a 64Kb allocation). On the average managed objects are 35 bytes in size and around 1800 can fit in one pool and hence on the whole allocation speed attained is pretty good.

Verifying what we discussed above

All of the things I said above can be easily verified using the Remote Performance monitor. I wrote a small application that allocates objects in a loop and launched it under Remote Performance monitor and published the data to perfmon so that I can observe how the managed heap grows. I used the process outlined in Steven Pratschner’s blog post at http://blogs.msdn.com/stevenpr/archive/2006/04/17/577636.aspx 

Using the above mentioned mechanism I am observing the managed bytes allocated (green line) and the GC Heap (red line). You can see that the green line increases linearly and after every 64Kb the red line or the GC Heap bumps up by 64Kb as we allocate a new object pool from the heap.

Memory leak via event handlers

One of our customers recently had some interesting out-of-memory issues which I thought I’d share.

The short version

The basic problem was that they had event sinks and sources. The event sinks were disposed correctly. However, in the dispose they missed removing the event sink from the event invoker list of the source. So in effect the disposed sinks were still reachable from the event source which is still in use. So GC did not de-allocate the event sinks and lead to leaked objects.

Now the not-so-long version

Consider the following code where there are EventSource and EventSink objects. EventSource exposes the event SomeEvent which the sink listens to.

EventSink sink = new EventSink();
EventSource src = new EventSource();

src.SomeEvent += sink.EventHandler;
src.DoWork();

sink = null;
// Force collection
GC.Collect();
GC.WaitForPendingFinalizers();

For the above code if you have a finalizer for EventSink and add a breakpoint/Console.WriteLine in it, you’d see that it is never hit even though sink is clearly set to null. The reason is the 3rd line where we added sink to the invoke list of source via the += operator. So even after sink being set to null the original object is still reachable (and hence not garbage) from src.

+= is additive so as the code executes and new sinks are added the older objects still stick around resulting in working set to grow and finally lead to crash at some point.

The fix is to ensure you remove the sink with –= as in

EventSink sink = new EventSink();
EventSource src = new EventSource();
src.SomeEvent += sink.EventHandler;
src.DoWork();
src.SomeEvent -= sink.EventHandler;
sink = null;

The above code is just sample and obviously you shouldn’t do event add removal in a scattered way. Make EventSink implement IDisposable and encapsulate the += –= in ctor/dispose.

Finalizers and Thread local storage

Writing finalizers is generally tricky. In the msdn documentation for finalizers the following limitations are mentioned.

1. The exact time when the finalizer executes during garbage collection is undefined
2. The finalizers of two objects are not guaranteed to run in any specific order
3. The thread on which the finalizer is run is unspecified.
4. Finalizers might not be run at all

#3 has interesting consequences. If a native resources is allocated by a managed ctor (or any other method) and the finalizers is used to de-allocate that then the allocation and de-allocation will not happen on the same thread. The reason being that the CLR uses one (maybe more than one) special thread to run all finalizers on. This is easily verified by the fact that if you query for the thread id inside the finalizers you will get a different id than the main thread the application is being run on.

The thread safety is generally easy to handle but might lead to some tricky and hard to locate problems. Consider the following code

class MyClass
{
    public MyClass()
    {
        tls = 42;
        Console.WriteLine("Ctor threadid = {0} TLS={1}", AppDomain.GetCurrentThreadId(), tls);
    }

    ~MyClass()
    {
        Console.WriteLine("Finalizer threadid = {0} TLS={1}", AppDomain.GetCurrentThreadId(), tls);
    }

    public void DoWork()
    {
        Console.WriteLine("DoWork threadid = {0} TLS={1}", AppDomain.GetCurrentThreadId(), tls);
    }

    [ThreadStatic]
    static int tls;
}

class Program
{
    static void Main(string[] args)
    {
        MyClass mc = new MyClass();
        mc.DoWork();
        mc = null;
        GC.Collect();
        GC.WaitForPendingFinalizers();
    }
}

Here we create a class, use it and finalize it. In all of these 3 methods we print the thread id and also use a special variable named tls.

If you see the tls is marked with the attribute ThreadStatic. The definition of ThreadStatic is “Indicates that the value of a static field is unique for each thread”.

I hope by now you have figured out the gotcha :). Under the hood the CLR uses a native OS concept called Thread Local Usage (TLS) to ensure that the value of tls is unique per thread. TLS uses special per thread data-structure to store that data. Now we have set the value in the ctor and used it in finalizer. Since they run on different threads each will get different values of the same field.

On my system the out put is as follows

Ctor threadid = 5904 TLS=42
DoWork threadid = 5904 TLS=42
Finalizer threadid = 4220 TLS=0
Press any key to continue . . .

As is evident the finalizer ran on a different thread (id is different) and the TLS value is also different from what was set.

So the moral of this story is “Be careful about thread safety of finalizers and do not use thread local storage in it”

What post are you known for?

I was just listening to Scot Hanselman’s podcast where he interviews Joel. They talk about how most bloggers get known for one post. E.g. Joel is known for “The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)”.

Now obviously I’m not a famous blogger and I’m not known for any post but the fact that my most read post is “How many (.NET) types are loaded for Hello World” makes me feel a bit weird. IMO that is one of the least useful posts I’ve ever written.

.NET Code Pitching

The word pitch can have many meanings, but in case of the code pitching it is used in the sense of “to throw away”.

The desktop CLR never throws away or pitches native code that it JITs. However, the .NET Compact Framework CLR supports code pitching due to the following reasons

1. It is targeted towards embedded/mobile systems and therefore needs to be more sensitive towards memory usage. So in some situations it throws away JITed code to free up memory.
2. NETCF runs on RISC processors like ARM where code density is lower than that of x86. This means the JITed native code is larger in size for a given set of IL on say ARM vs that on say x86. Due to this the memory usage overhead is a bit aggravated. However, this isn’t really a primary motivator and there are other work around like using the newer ARM instruction set extensions.

Code pitching can happen due to various reasons like (note this is not an exhaustive list)

1. When the managed application gets WM_HIBERNATE message it fires a round of garbage collection and also pitches code
2. When the JITer fails to allocate memory it attempts code pitching to free up memory and re-tries allocation
3. Other native resource allocation failures also initiates pitching

Obviously code pitching has performance implications and can result in the same method being pitched multiple times. You can monitor code pitching in Remote Performance Monitor.

As the name suggests the GC does drive code pitching but they are not essentially related. E.g. if user code forces GC by System.GC.Collect() code pitching is not done. Code pitching is primarily driven by real low memory scenarios.

Caveat: While pitching the CLR ensures that it doesn’t throw away any JITed method on the current managed execution stack. No points for guessing why that is important.