Sandeep Prabhakar's WebLog

Building a community

2005-11-30T13:04:00Z

Community building is an important part of any product release – especially so for a platform like the .Net Compact Framework. One of the things that takes sometimes a significant fraction of my working day is interacting with the community in multiple forms. Since this is such a time consuming process, I thought I'd rationalize my thoughts on the issue.

1. What's its purpose?

To provide a feedback loop to improve the product. This is information flowing from the community to the product team.
To disseminate knowledge from the people with answers to the people with questions. This is information flow in the reverse direction.

So the question is how do we provide a solution to these purposes quicker and more efficiently?

Feedback to improve the product:
1. More aggregated and analyzed views of problems people are facing. This is where things like Watson reports are very useful, since they aggregate the problems into buckets which allow us to focus sparse resources on things that are impacting the most users.
2. Deeper understanding of the underlying problems. Sometimes people complain that x, y and z are not working correctly. It takes some analysis and understanding to know why x, y and z is happening and if we have a solution to the underlying problem A, we automatically solve all other related problems.
To disseminate knowledge:

Dissemination of knowledge is governed by 2 factors:

· Number of people who have the knowledge and are willing to share it.

· What's the reach of the information shared?

Let's discuss what these 2 factors mean.

Product teams developing software is typically very small. Based on the size of the community it provides services to, it is not possible for the product team to answer each and every question in a timely manner. This is the main reason that community building is important around any product. A community is something where people are willing to share information that they gleaned the hard way so that it benefits a large community. This is where MVPs are so beneficial since they provide that spark to the community. The size of this community and the experience of the people in the technology are the principle factors that determine how long it takes for a question to be answered.

Reach of information is also another thing that effects how efficiently the information is disseminated. For example say I solve a customer's problem over email. I might get n other customers over the space of a long time where I might need to revisit the problem and solve it again (or atleast dig out the last email I had). This is where I think blogging/ whitepapers is so useful. It’s a means to leverage your work you did when you solved customer 1's problems. This is beneficial in 2 ways – first, it reduces the number of times the product team needs to look at the same issue again and again and second, it provides a quicker answer to the customer when he needs a solution to the problem he is facing.

2. So what are the channels of communication?

· Product support

· Blogging

· White papers

· KB articles

· Newsgroups

· Interest groups

· Chats

· Email aliases

Each of these channels of communication has advantages and disadvantages. Some are suited for certain types of interaction whereas others are not.

I am always on the lookout for better and more efficient use of time in such a way that it benefits the largest possible community. So if you have any feelings as to what form of communication works the best for you, please send me your comments.

My 2 cents on Performance

2005-10-25T21:09:00Z

One of the most common questions we are asked is how fast is .NET Compact Framework?

There are 2 things that will help you the most for your project:
1. Learning how .NET Compact Framework is architected from a performance viewpoint, you'd get do's and don't while coding up a managed application.

A lot has been already described about this and I don't have anything to add that's not already in these awesome blogs from Mike, Scott and Roman:
http://blogs.msdn.com/mikezintel/archive/2005/03/30/403941.aspx
http://blogs.msdn.com/scottholden/archive/2005/01/05/346770.aspx
http://blogs.msdn.com/netcfteam/archive/2005/05/04/414820.aspx

2. Measuring your key scenarios using kernels early in the project.

Before writing a huge and bulky project in .NET Compact Framework, you should measure the running times using small kernel programs that are representative of your scenarios. It's said that the 3 important keywords to writing a performant application are "Measure, measure, measure...". Make sure that you have performance goals in mind at the very beginning of the project. Sometimes performance can be tweaked as an after thought, but in most cases this might require huge architectural changes and major rewrites of parts of your program. A few pointers when doing this performance measurement:

1. Put the operation you want to measure the time of in a loop. Measure the time before starting the loop and after the end of the loop. Do NOT measure the time for each iteration of the loop. Make sure that the loop has enough iterations to run for a few seconds. Run the test many times and average the times. The reason for there is some measurement error and if you time within each iteration of the loop, you are increasing the error. A simple example of seeing how long it takes to read from an XML File is:

startTime = Enviroment.TickCount();
for(int i=0;i<5000000;i++)
{
// Code to read from XML File
}
endTime = Enviroment.TickCount();

TimePerIteration = (endTime-startTime) / 5000000;

2. A second baseline for comparison, might be to translate the same kernel above to native code and see how long it takes for the same loop to execute. This way you can measure the overhead of managed code compared to native code. In some cases, the managed implementation could be faster than the native implementation due to the performance tweaks that we have done in the Compact Framework.

3. If on the whole, the .NET Compact Framework program meets your performance goals except for a few scenarios (and the native implementation meets your goals), you could look at PInvoking to native code for those specific scenarios. However, we recommend against this as we have a robust framework and there might be some error checking that you might fail to do in your native kernel that gives it better performance. The PInvoke route is an option to not writing the entire application in native code, but just a few key pieces. (Think of it as what we used to do when comparing assembly code kernels in a mostly C program a sometime back...). Also, another option for scenarios that don't meet your performance goals, is to rewrite managed code using a different mechanism. There are multiple ways to perform the same functionality - like for instance, you can program to sockets rather than to web services. Of course, there are ease of use, interop considerations to take into account, but these tradeoffs need to be made early in the cycle rather than later.

4. Sometimes there are variations in the time of the measured benchmark kernel. Turning on performance counters (http://msdn2.microsoft.com/en-us/library/ms172525) you can see if there are things that affect execution time by randomizing it. Things you should look for are GC's, JIT code pitches etc. If you loop is creating a huge amount of garbage, this might not be avoidable (But then if that's your scenario, you would like GC to execute cause then it's typical). If not, and the GC just happened to occur while you were executing the function - then you could add a GC.Collect() before starting the measurement of your kernel to eliminate the effects of GC on your timing.

5. In order to get managed code to execute, there are 2 stages: JIT stage and running the Jitted code stage. In typical long running programs, the 2nd stage is the one that's important and takes the longest time. As a result, one of the common things we do when we measure performance it we prime the JITing process. That is we execute the operation once before we start measuring the time in a loop. This JIT's all the code in the codepath that's being measured before you actually start measuring the time and hence you get a more accurate timing of the 2nd stage which represents execution of the program.

For example:

Foo.Test();

startTime = Enviroment.TickCount();
for(int i=0;i<5000000;i++)
{
Foo.Test();
}
endTime = Enviroment.TickCount();

TimePerIteration = (endTime-startTime) / 5000000;

This posting is provided "AS IS" with no warranties, and confers no rights.

Plugging into Events that happen on a device in managed code

2005-10-21T20:57:00Z

One of the most common requirements of a well written mobile application is to be network aware. For instance, as an app how do I determine if I am in the signal range of a GPRS network?

There's this cool managed class that exposed in the windows mobile 5.0 SDK that offers a ton of properties that you can hook event handlers on. Thus for instance when you come in the range of the GPRS network, poof an event is fired and you code that needs to do some network operation is called.

Hooking in the event handler for getting an event on a GPRS network becoming available is as follows:

SystemState state = new SystemState(SystemProperty.PhoneGprsCoverage);
state.Changed += new ChangeEventHandler(state_Changed);

Check out the information in the SDK for all the events that can happen on a device and how you can plug into those events.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/mobilesdk5/html/T_Microsoft_WindowsMobile_Status_SystemState.asp

This posting is provided "AS IS" with no warranties, and confers no rights.

HowTo: Approach Debugging Networking Problems

2005-10-21T04:15:00Z

One of the things I’ve learn over the last few years and what I find missing is the answer to the question of, “What’s the best approach to debugging a network issue?”. After spending long hours in front of nasty networking issues I think the following are the steps is the best way to go about it.

1. Understand the System
This is probably the most important step to debugging an application. You should find out all the entities you are interacting with and how the communication takes place. Create a picture of what the application is trying to do. For networking problems, I find timeline diagrams to be very useful.

In addition, network sniffing tools are a big help in helping you know whether your model of how the system works is how it actually works. In my mind, there are 3 parts to understanding the system – understand the client, understand the network, understand the server. These are 3 active entities performing operations and without all 3 of them working together, we have a non-functional system.

2. Check the Configuration Options
One of the most common reasons for failure of network applications is due to incorrect configuration. Incorrect configuration could happen at the client, the network and the server. The following lists possible mistakes in configuration that networking applications depend upon:

The client:
• If you are using a Pocket PC or a Smartphone, make sure you have the connection manager settings setup correctly.
• Make sure the client clock is correct and matches the server clock.
• Make sure you have the required certificates installed on the device.

The Network:
• Firewall Issues
• IPSec Issues
• Errors in routing tables (presumably this never happens).

The Server:
• Incorrectly configured redirection
• Server application problems
• Incorrectly configured authentication settings

3. Perform Sanity Checks & Isolate the Subsystem Causing the Problem
Sanity checks are typically simple programs that check the validity of your setup and configuration. This kind of debugging is approaching the problem from the opposite direction as that described in the next section. In this section, we start out with a bunch of small programs each of which tests the sanity of (ideally) exactly one subsystem. This way we could start out small verifying the functionality of small systems which we will then make into larger programs to verify larger and larger subsystems. In the next section, we go in the other direction by taking a large application and removing all components that are not necessary and still be able reproduce the problem. This step is typically better for cases where we get 100% reproducible failures. This typically is caused due to some incorrect configuration and is quickly identified by this step.

A scenario for how this would be used is say the test that tests SSL fails and all other http tests pass, then the options for rectifying the problem would be:
• Check that the dll that provides the SSL services is present on the device. (This is typically not a problem for standard devices, and more of a problem for custom devices).
• Check that the entire chain of CA upto the root CA of the server you are connecting to is installed on the device.
• Check that the system clock shows the correct time.

Examples of checking the underlying components on which your program is built might be trying to verify the following pieces of information:
Does DNS work?
Does a simple socket application work?
Does a simple http application using a GET request work?
Does a simple http application using a POST request work?
Does the authentication subsystem work?
Does SSL work?
Am I going through a proxy and does that work?

4. Create a Repro Case
In order to debug any application, the very first step is to isolate the problem to the smallest portion of code that could be causing the problem. Networking problems are no different. Isolating the smallest piece of code that reproduces the problem might involve stripping your application down to the minimum code that still has the problem. In some cases this might not be possible and you might feel like it is not worth the effort doing this. But if the problem is not apparent, this might be the only recourse to figuring out what the problem is.

5. Check the RFCs/ MSDN Documentation / KB Articles/ Whitepapers/ Blogs/ Search Online
These are useful in general, but more specifically after you hit a problem and you’ve narrowed down the scope of the problem using the information in the previous two sections, this step can really help you to understand how to solve the problem if it’s a known issue. MSDN docs and RFCs might have some caveats hidden in the documentation for some of the APIs or protocols. If the documentation falls short of describing what *should* happen in the situation you have encountered, fallback to checking the frames of reference by seeing how a native application behaves on the same device or how an application built on .NET running on a desktop machine behaves. This will help you to identify correct behaviors or atleast verify that more than one piece of software behaves the same way.

6. Ask for Help

There are multiple avenues available for asking for help.
• If you have friends who are knowledgeable about the technology, it is probably easiest and quickest to ask them. Part of this, is posting on the newsgroups in the technology area of interest. (For instance, the .NET Compact Framework’s newsgroup can be found at http://forums.microsoft.com/msdn/ShowForum.aspx?ForumID=33)
• As a last resort contact the product support people in the organization.

When asking for help make sure you include all or most of the information below. While developers experienced in the technology can spot known problems immediately, the availability of any information helps debugging a problem. Things that stand out that are of immediate benefit for some third party debugging your issue are the output of network packet capture tools. Information that could be of use is as follows.
• Network traces at the client, server and proxy.
• The set of experiments conducted and the information about whether each of them succeeded or failed.

This posting is provided "AS IS" with no warranties, and confers no rights.

How does .NET Compact Framework determine which proxy to use?

2005-10-21T03:40:00Z

This is how .NET Compact Framework determines which proxy to go through while communicating with a server.

The order in which .NET Compact Framework (NETCF) searches for proxy information is:
Step 1: It checks if the programmer has specified this in his program using GlobalProxySelection.Select.
Step 2: If the program does not specify this, .NET Compact Framework uses the System’s webproxy. There are 2 pieces of information retrieved: the proxy’s address and whether Bypass on local address is turned on or not.

The way Step 2 above is performed on Pocket PC and Smartphones differs from how its performed on plain windows CE devices. NETCF tries to obtain the System’s default Web proxy as follows:

For Windows CE devices:

First it tries to get it from Internet Explorer. If IE is not installed it gets it from Pocket Internet Explorer’s registry settings.

There is this property called BypassProxyOnLocal which can be set in Internet Explorer and Pocket Internet Explorer, which bypasses the proxy if the site is considered local.
If (BypassProxyOnLocal == false)
Requests are always sent through the proxy.
If(BypassProxyOnLocal == true)
Requests that are considered non-local and sent through the proxy if the URLs contain a dot “.”. If the URL does not contain a dot, the request is sent directly to the server.

If you are sending data to a server using the IPAddress of the server which contains a dot, the request would be sent through the proxy. To avoid sending the request through proxy, you can do one of the following:
1. Replace the numerical IP address with the machine’s host name and set BypassProxyOnLocal to true.
2. Set GlobalProxySelection.Select to an empty proxy before issuing the request.

For Pocket PC and Smartphones:

NETCF obtains the System’s web proxy by calling into connection manager to request for the proxy information.

This posting is provided "AS IS" with no warranties, and confers no rights.

Static Constructors

2005-10-05T04:30:00Z

There are some things that cannot be done in .NET Compact Framework in a static constructor which are possible in the full .NET Framework. Basically, all static constructors when are executed in a serialized fashion in .NET Compact Framework V2 (Note that this design may change in future versions of the product). As a result, you can result in deadlocks if you are doing certain operations in the static constructors. Take the following program for example. Here, I have 2 threads. In the main thread, I execute a function in StaticClass1 - this calls implicitly the static ctor for StaticClass1. In this static ctor, we wait on an Event that gets set by the Static Ctor of StaticClass2 which gets executed on another thread. Since the execution of static ctors are serialized in the .NET Compact Framework's execution engine, the StaticClass1 will never signal the event and never finish. Also, StaticClass2's ctor cannot execute since we prevent both the StaticClass1's and StaticClass2's ctors from executing at the same time... deadlock is the result. This applies only to .NET Compact Framework and works fine on the .NET Framework.

using System;

using System.Threading;

using System.Net;

public class Test

{

public static SyncContainer sc = new SyncContainer();

public static StaticClass1 static1;

static void Main (string[] args)

{

Thread t = new Thread(new ThreadStart(ThreadProc));

t.Start();

static1 = new StaticClass1();

string str = static1.Func();

}

public static void ThreadProc()

{

Console.WriteLine("Executing ThreadProc");

StaticClass2.Func();

Console.WriteLine("After Executing ThreadProc");

}

public class StaticClass2

{

static StaticClass2()

{

Test.sc.are.Set();

}

public static string Func()

{

return "world";

}

public class StaticClass1

{

static StaticClass1()

{

Console.WriteLine("Executing Static Ctor");

Test.sc.are.WaitOne();

Console.WriteLine("After WaitOne");

}

public string Func()

{

return "hello";

}

public class SyncContainer

{

public AutoResetEvent are = new AutoResetEvent(false);

}

The solution to this problem if you have a similar situation is to take the code from the static ctor and put it in a static property/method and call that before you call anything else on that class.

This situation was found when a Web Service call was made in the static ctor. And since internally in the .NET Compact Framework's Networking code we use this paradigm of setting events to signal completion of networking transfer completion, we result in this deadlock.

This posting is provided "AS IS" with no warranties, and confers no rights.

Of Overloads and Difference in behavior between .NET and .NET Compact Framework

2005-09-26T20:00:00Z

One interesting issue that came up recently and not very obvious at first sight is the following:

We found a difference in behavior in .NET Compact Framework compared to the Full .NET Framework. Basically we found that calling the same function on .NET and .NET Compact Framework had a differing behavior. The reason this came to be was .NET had more overloads for the same function compared to .NET Compact Framework. So for example is .NET had overloads:

Foo(int bar);

Foo(object bar);

whereas .NET Compact Framework had only the overload:

Foo(object bar);

This is pretty common since we try to subset the full .NET Framework to give most of the utility of the .NET with the least amount of code. However as this specific issue came about, we found that there can be subtle differences.

When the code such as the one below was compiled:

Main()

{

bool flag = true;

Foo(flag);

}

on .NET, this would call into the more specific function overload Foo(int) whereas since on .NET Compact Framework since we didn't have that overload would target Foo(object). In this case, any internal implementation differences between the 2 overloads would manifest itself as a difference in behavior between .NET and .NET Compact Framework.

The definitive way to determine if this is the cause of the difference in behavior is using ildasm to view the IL generated code and seeing which overload is being called.

This posting is provided "AS IS" with no warranties, and confers no rights.

Hello World!

2005-08-23T03:10:00Z

True to my geeky background, I'll meet the world using the cliche'd phrase... To introduce myself, I'm a software developer working on the .NET Compact Framework team (usually called the NETCF team in short). I've been on the team for 2 years and its been a fun experience. The best part of being in this team is you walk in every week knowing you're going to be doing something totally different (I can't recall doing the same thing on two different weeks). I did my undergrad in Chennai, India in Anna University and my grad school in Virginia Tech. For the time I've been on the NETCF team, I've poked around with Networking, Globalization, Generics, Localization and other odds and ends as the requirements come up. If you have any feedback about these areas about stuff that doesn't work right or things that are missing or just someone to curse for your problems (in these areas ;) ... no curses like the weather sucks in seattle etc etc) send them on to me.