Akhune's WebLog

Change of pace and country!

Akhune — Thu, 30 Aug 2007 23:29:00 GMT

After a brief stint in India Development Center, Hyderabad I am back in the US. I am currently a senior developer in the newly announced Microsoft Response Point product. Microsoft Response Point is an IP-PBX product aimed at small businesses built with ease of management in mind. Beneath the covers it has some of the most unique and compelling collection of technologies which excited me first to this team. Speech recognition, Automated receptionist dialog systems, SIP/RTP based VoIP and Windows XP Embedded based device to name a few.

We also have a blog. Check it out for more information.

Visual Studio 2005 Device Emulator and Internet Connectivity

Akhune — Wed, 16 Nov 2005 13:49:00 GMT

Device Emulator and Internet Connectivity

With the release of Visual Studio 2005, I have seen some confusion over how to connect the Device Emulator to the Internet. This post is an attempt to clear some of this confusion. Feedback welcome.

Connectivity changes from Beta2 to Final Release

The Desktop to Device Emulator connectivity story has changed quite radically between the Beta2 and the final release version of Visual Studio. Some of the default settings out of the box are different so you need to be aware of them. In the remaining part of this post I will focus only on the final version and point out where it differs from the Beta2 in a major way.

First, if you have Beta2 of Visual Studio 2005 please remove it using the uninstall steps at http://msdn.microsoft.com/vstudio/support/uninstall/default.aspx. If you used the uninstall tool, then it has been update to remove the DMA Transport Update but do make sure to check ‘Add Remove Programs’ to make sure it is not there before you install the final release version.

The rest of the post assumes that you have installed the final version of Visual Studio 2005 and (optionally) Windows Mobile 5.0 SDK.

Starting the Device Emulator

There are three ways to start the Device Emulator from Visual Studio 2005

From Tools->Connect To Device-><pick one emulator from the list> ->Connect
From Tools->Device EmulatorManager, then right click on any emulator from the displayed list and press ‘Connect’.
Create a smart device project and either press F5 or Deploy Solution. Pick a particular emulator when the dialog comes up (in case of managed projects) or from right clicking on your project name in the solution explorer, select Properties and select Deployment (in case of native projects).
There is a fourth way, from command line, to start the Device Emulator that we won’t touch upon in this post.

Visual Studio 2005 running on the desktop (henceforth called ‘Desktop’) interacts with the Device Emulator over a “DMA” channel. In Beta2, the Desktop – Device Emulator interaction occurred over TCP/IP stack. Think of DMA as a direct communication between two Windows processes (Visual Studio and Device Emulator) and hence it’s much more robust and fast than going over the network stack.

Confirm that you have the DMA Transport as the transport between Desktop and the Device Emulator. DMA Transport is the new default in Visual Studio 2005 final version. In Beta2 the default was the TCP/IP transport. When you select the ‘Properties’ for any emulator from Tools0->Options->Device Tools->Devices, you will see “DMA Transport’ under the Transport field. Note that while this can be changed to TCP Connect Transport (if you have the Virtual Network Switch Driver installed) it is NOT RECOMMENDED.

Enabling Internet Connectivity From the Device Emulator

When the Device Emulator starts up it acts like a physical device that is not cradled. As such it has no Internet connectivity (the Device Emulator has no “data” plan from a carrier to get any such connectivity “over the air” J). In order to “Cradle” the Device Emulator we need to start the Device Emulator Manager (Tools->Device Emulator Manager). The Device Emulator Manager would indicate the instance of the emulator currently running with a green arrow.

Right click on the instance of the emulator that is running (presumably only one in your case) that you want to cradle and select ‘Cradle’.

If you have ActiveSync 4.0 installed (the developer preview is available at http://download.microsoft.com/download/c/4/5/c45f8f83-6383-43d7-840b-cb9638484e4d/setup.exe) then ActiveSync will automatically detect that a new “device” has been connected. The Device Emulator Manager and the Device Emulator can work with ActiveSync 3.8 also but Visual Studio 2005 supports ActiveSync 4.0 only. You can either create a Guest partnership or a Standard partnership with the Device Emulator. I usually just hit 'Cancel' which automatically sets up a guest partnership.

Once ActiveSync is in the “Connected” state you should have Internet connectivity from the Device Emulator. If you are on a corporate network and access external web sites via a proxy server you will be prompted to set it up. The settings used here are similar to the ones that you would use to setup your desktop IE (Tools -> Internet Options -> Connections -> LAN settings)

At this point you can access the Internet, e.g. www.microsoft.com, from Pocket IE.

You can also access the local intranet web sites from the Device Emulator.

Enabling Internet Connectivity To the Device Emulator

This scenario is not supported out-of-the-box in Visual Studio 2005. If you need more information please let me know.

(Non-Technical) Biscuits and milk

Akhune — Mon, 05 Sep 2005 11:36:00 GMT

Thanks to a nudge from Sriram I have started blogging again. The idea was also planted by a good friend of mine, Rajnish, several months back. There are a lot of things that have changed in the past year that I am not sure where to begin. The biggest and the most exciting change has been that I have moved back to India (for an extended visit :-)). After almost 7 years in Redmond and 10 years in US I wasn't sure what to expect. But thanks to a lot of support from Microsoft and newly found friends the transition has been amazingly smooth.

The big day arrived on 3rd of August. Originally I was supposed to fly out on the 31st of July but I had to reschedule the flight because of torrential rains in Mumbai. That gave me another 3 days with my wife and daughter (they join me here next month). On the way here my emotions were mixed. For one this just felt right. But leaving behind a lot of good friends and colleagues and most importantly, my wife and daughter, was tough. One of the main reasons to come back was to be with friends and family. And since arriving in Hyderabad I have gotten right down to this very task and have made many new friends, both at work and otherwise. So far so good.

Here in Hyderabad, I have joined the Visual Studio for Devices team in the India Development Center (part of Microsoft Indian (R&D) Private Limited). In my team we would be developing the next generation of Visual Studio features for device developement like the debugger, connectivity to devices and the device emulator. More on that in future posts.

Growing up I used to love "Glucose" biscuits and hot milk. The Indian grocery stores in Redmond have started stocking these and it was fun to eat them again after many many years. So I asked myself: why not enjoy them here in Desh itself? The small canteen in the campus cafeteria doesn't stock the Glucose brand that I used to enjoy (Parle) but does have a Brittanea variety. So one day I brought them (a large pack for Rs. 8, compare that to > $1 in Redmond ;-)) up to the shiproom meetings and now it might just become a tradition.

I hope to keep blogging on the amusing little anecdotes from Hyd... Stay tuned.

Analyzing Common CLR Performance Problems

Akhune — Fri, 11 Jun 2004 21:06:00 GMT

How To Use This document

This document is intended to help you diagnose common CLR performance issues. Over the years we have seen a wide variety of CLR performance issues from our customers. This document tries to classify these issues in broad categories and provide guidance for each class of problems. Many facts mentioned in this document hold for the CLR versions V1.0 and V1.1. If there are differences in behavior between these two versions then they are called out. V2.0 behavior, if documented here, is subject to change.

Minimum requirements for a CLR Performance investigation

Please follow the following guidelines

n If you are analyzing a memory related issue then collect:

1) MUST: .NET CLR Memory perf counters trace log

2) MUST: Vadump snapshot (vadump –o –p <pid> and vadump –v –p <pid>

3) MUST: Dump file so that SOS commands can be run as a post process.

4) NICE: Read this article on GC to gain a better understanding of your problems.

5) NICE: If GCHeap is suspected to be an issue then gather SOS!DumpHeap –stat, SOS!DumpHeap and SOS!EEHeap –gc log files.

n If you are analyzing an execution speed related issue (things are slow, take too long etc.) then collect:

1) MUST: .NET CLR Memory\% Time in GC perf counter, Process\% Processor Time

2) MUST: Sampling profile logs. Please make sure that the symbols are correct and do really point to code in the CLR. These would most likely be functions from mscorwks.dll. Note that GC related functions at the top doesn’t indicate an issue in GC’s performance usually, rather it’s excessive allocations by the application.

3) NICE: Sanity check the machine for excessive paging due to low memory conditions, poor performance due to disk fragmentation, other processes or network interference etc.

PROBLEM: My Application’s Private Bytes Grow Indefinitely

PROBLEM: My Application is leaking memory

1) There are several ways in which you can confirm memory leaks. If you know that the GC Heap is leaking then skip to step 5).

2) Observe “Process\Private Bytes” perf counter or “MemUsage” column in the task manager. If they increase over a period of time (this time could be as little as a few hours or could be a few days) then you have a memory leak. Collect the “.NET CLR Memory\# Bytes in all Heaps” perf counter for your application when the memory leak manifests itself. If this counter is steadily growing (presumably at the same rate as the total memory growth of your application) you have a managed memory leak. The application apparently “leaks” in-spite of the Garbage Collector (GC) because references to objects in the GC heap are still alive. These references or “roots” would hold on to the managed objects and prevent the GC from collecting garbage.

3) If the GC Heap is growing but not at the same rate as the application’s total memory over a period of time then you might have smaller managed objects holding onto larger unmanaged memory. Depending on the memory pressure on the machine the GC might or might not clean up these smaller managed objects. Consider using the Dispose pattern to eagerly garbage collect these expensive resources.

4) If the GC Heap is not growing over a period of time then you have an unmanaged memory leak. Snapshot the application’s working set at regular intervals with vadump. Use vadump –o –p <pid> and redirect the output to a log file. The summary at the end of the log file would appear as in Figure 1. The two interesting numbers are the “Heap” which is the NT process heap and “Other Data” within which the GC Heap is located. If the “Heap” is growing steadily then you have an unmanaged memory leak[1]. If the “Other Data” is growing but the GC Heap perf counter (“.Net CLR Memory\# Bytes in all Heaps”) was not then you have an unmanaged memory leak stemming from calls to either “VirtualAlloc”/”VirtualAllocEx” or large (>256 KB) allocations on the NT process heap. Please use traditional memory leak tools to analyze these problems. E.g. !heap 0 would show calls to HeapAlloc with no corresponding calls to HeapFree.

Catagory                        Total        Private Shareable    Shared
                           Pages    KBytes    KBytes    KBytes    KBytes
      Page Table Pages        29       116       116         0         0
      Other System             8        32        32         0         0
      Code/StaticData       1806      7224      1040      3732      2452
      Heap                   201       804       804         0         0
      Stack                    9        36        36         0         0
      Teb                      4        16        16         0         0
      Mapped Data            141       564         0        68       496
      Other Data             168       672       668         4         0

      Total Modules         1806      7224      1040      3732      2452
      Total Dynamic Data     523      2092      1524        72       496
      Total System            37       148       148         0         0
Grand Total Working Set     2366      9464      2712      3804      2948

Figure 1: VADump output

5) Once you have established that the GC Heap is leaking memory, the next step is to identify the cause of the leak. You can approach the problem in two ways. First, you can try to identify which Type(s) of object is leaking. SOS[2] is an ntsd/windbg extension that can help examine the GC Heap. E.g.

0:001> !DumpHeap -stat -min 100

total 1992 objects

Statistics:

      MT    Count TotalSize Class Name

 d750604        1       116 System.Double[]

  d0373c        1       140 System.Boolean[]

 d9198dc        2       248 System.Web.HttpCachePolicy

  d02edc        2       264 System.UInt64[]

  <snip . . . >

  d026a0      241     49868 System.Int32[]

  d02960      316     67272 System.Collections.Hashtable/bucket[]

  d02364       22    106364 System.Char[]

  d02c2c       78    249752 System.Byte[]

79b4f3f8      995    331976 System.String

Total 1992 objects

large objects

 Address       MT     Size

 90d9350 79b4f3f8    87492 System.String

  <snip . . . >

 90984d8   d0209c     4096 System.Object[]

 90ef980   d0209c     2064 System.Object[]

total 16 large objects

More help on SOS commands can be found by typing in “SOS!Help” in the ntsd/windbg command window. You can also use CLRProfiler[3] (available at this gotdotnet site) and use the “Show Heap Now” view taken for two or more snapshots to identify the leaking objects. Second, you need to identify the frequent allocators of these Types. The GC Heap grows because there are references or “roots” from outside the GC Heap to the managed objects. If you using SOS then use SOS!GCRoot <addr> to handles for the object at <addr>. The object addresses can be obtained from DumpHeap command. If you are using the CLRProfiler then you can simply trace the references and observe how they trace back to the <root>. GC Handles can also cause GC Heap leak. Please refer to 8) for details.

6) Sometimes the Finalizer thread can fall behind in finalizing objects. This can happen either due to Finalizer thread being blocked (e.g. if the main thread is marked STA and doesn’t let the Finalizer thread run) or slow due to excessive load from other similar or higher priority threads. This causes the GC heap to grow and cause a perception of a leak. If the Finalizer thread gets a chance to run then this “leak” would disappear. A combination of calls to GC.WaitForPendingFinalizers and GC.Collect can also empty the finalization queue. WARNING: Use these two calls ONLY to confirm this theory (i.e. the finalizer thread is falling behind). If you do find that this is the case you should consider marking your thread as MTA. If excessive load causes the Finalizer thread to lag behind then the GC would step up the collections and eventually as memory pressure on the system grows[4] the memory would be collected.

7) Pinning can cause perception of a memory leak. Please refer to item 3) in the next section for more details.

8) GC Handle leaks can also cause GC memory to leak. GC Handles are handles to Objects. They leak when these handles are created and forgotten about. Use SOS!objsize to list handles. If this list is large and growing then you have a GC Handle leak.

PROBLEM: My Application’s Virtual Bytes Grow Indefinitely

PROBLEM: My Application experiences excessive memory fragmentation

PROBLEM: My Application CRASHES with OUT-OF-MEMORY exception (while there is still a lot of memory available on the machine)

On 32 bit machines, Virtual Memory of the application is limited to 2 GB[5]. Long running applications can load enough modules and allocate enough memory to run out of Virtual Memory. There are several ways in which you can confirm that your application has hit this ceiling. Observe “Process\Virtual Bytes” perf counter or “Virtual Memory Size” column in the task manager. If it is close to the 2 GB limit then the application is very close to throwing an Out-of-Memory exception. The GC tries to allocate a contiguous memory block of SEGMENT_SIZE but fails to allocate this memory under memory stress and the managed allocation request fails with an exception. On server GC, if there are more than 8 processors, SEGMENT_SIZE is 16 MB for normal heap and 8 MB for Large Object Heap; for more than 4 processors it's 32 MB and 16 MB respectively and for 4 or less processors it’s 64 MB and 32 MB respectively. In workstation GC it's always 16 MB and 16 MB respectively.

1) Snapshot the virtual memory address space with vadump –v –p <pid> at regular intervals. You can also use !inetdbg.vmmap command in ntsd/windbg. Look for patterns of allocations that are odd. You can windiff the vadump logs to look for such patterns. Note that minimum virtual space that can be allocated in Windows NT and XP is 64 KB, hence module loads of VirtualAddress calls with sizes less than 64 KB should “waste” some space. Packing these regions would help.

2) If visual inspection of the memory regions or the allocation patterns don’t provide clues you might have to put a break point at kernel32!VirtualAllocEx and examine stack traces. This process can be automated with post processing tools.

3) Fragmentation can occur in the GC heap when objects are pinned for significant periods of time. When this occurs, the pinned objects may get promoted to generation 2. Over time, other objects around the pinned objects get collected, leaving free space surrounding the pinned objects. Since new allocations always occur in generation 0, the free space in these Gen 2 memory segments is wasted, which can increase the OS memory usage substantially even if the total memory in all generations (as viewed via perf counters) is proportionately low. The best way to verify this is with the SOS debugger extension’s DumpHeap command, which will give you a breakdown of the “Free” space in the heap. If this is significant, then consider helping the application developer reduce pinned objects over long periods of time – pinning should be as short as duration as possible.

[1] If there is a perf counter that tracks process heap usage of a process then please send feedback.

[2] SOS was called Strike in CLR V1.0 and V1.1

[3] Note that SOS debugger extensions and CLRProfiler complement each other.

[4] When 90% of the physical RAM is used

[5] Applications can be run the 3GB mode also in which all arguments apply with the higher limit.

A MUST read for .Net Performance and Scalability issues

Akhune — Wed, 26 May 2004 21:43:00 GMT

http://msdn.microsoft.com/perf

Download the book for free as PDF or read individual chapters online!

Hello World

Akhune — Fri, 21 May 2004 01:32:00 GMT

Quick Introduction: My name is Abhiram Khune. People usually call me Abhi. I am the lead developer in the CLR Performance Team. We investigate and fix performance issues and regressions, help customers with performance design and investigation, write performance tools and plan future performance improvements. My personal focus is on the working set and private pages used by the CLR.

In my posts I would try to help answer your questions and post articles on common performance problems, tips, techniques and any thing interesting that I come across. So stay posted.

Together with Jan Gray, I would be presenting at TechEd 2004 in San Diego next week. DEV 491: “.NET Framework: Writing Faster Managed Code” on 27th May, 1.30-2.45 PM. See you there!

- Abhi