Krzysztof Cwalina : General Programming

MEF on CodePlex

kcwalina — Sat, 06 Sep 2008 00:00:00 GMT

We have just released an update to MEF. You can get it at http://www.codeplex.com/MEF

The changes are quite significant:

1. The preview ships with sources under a very permissive license (Ms-LPL).

2. We now support constructor injection. Feature that the community asked for.

3. We completely redesigned MEF’s extensibility points. The extensibility points are designed to support writing custom providers of composition data. For example, out of the box MEF requires composable parts to be attributed with attributes that provide metadata describing the composition. We got lots of feedback that this is not acceptable in many scenarios. The new extensibility points make it easier to extend MEF to externalize the metadata (to an XML file for example). Note, that the changes are just the first step toward the goal of making the extensibility easy and powerful. We will most probably keep making improvements in this space in the future, so feedback on the new extensibility points would be more then welcome.

4. We significantly cleaned up the container APIs. But as above, there is more clean up to come in the future.

Framework Design Studio Released

kcwalina — Fri, 04 Apr 2008 21:10:00 GMT

When I was coming back from Mix 2007, I was bored on the plane and so started to write a dev tool. What a geeky thing to do on a plane. :-)

The tool allows comparing two versions of an assembly to identify API differences: API additions and removals. Comparing versions of APIs comes very handy during API design process. Often you want to ensure that things did not get removed accidentally (which can cause incompatibilities), and as APIs grow, you want to review the addition without having to re-review APIs that were already reviewed. The tool, called Framework Design Studio (FDS) supports these scenarios.

Later on, I got lots of help from Hongping Lim (a developer on our team), and David Fowler (our 2007 summer intern). David ported the application to WPF, and Hongping basically took it from an early prototype stage to what it is today and made it possible to ship it externally.

Anyway, you can get the tool at the MSDN Code Gallery, the user guide is attached to this post, and lastly, here is the API diff output that the tool generates. Hope you find it useful.

Job Openings on the .NET Framework Core Team

kcwalina — Fri, 14 Mar 2008 22:49:00 GMT

We have been incubating ideas about building a simple extensibility framework for some time. Now, as plans for the next version of the .NET Framework crystallize a bit more, we decided to productize the project. As a result, we have opened a job position (and most probably will be opening more) on the .NET Framework team. If you are interested, please see details here and send me an email at “kcwalina at microsoft.com.”

So, what is this extensibility framework? Initially, it will be a low level core .NET Framework feature to make it easy for applications to expose extensibility points and consume extensions. Think about what for example FxCop has to do define rule contracts and load rule implemented by the community. These are the basics, and we can talk about the broader and longer term vision when you come to Redmond for an interview :-)

This is a technical Program Manager position in Redmond, WA, and it’s basically exactly the job I did when I joined Microsoft. Besides working on the Framework features, all Program Managers on the core team have opportunities to work on API design and architecture projects.

Video Recording of "Framework Engineering: Architecting, Designing, and Developing Reusable Libraries"

kcwalina — Tue, 08 Jan 2008 21:42:00 GMT

I just received a video recording of a talk I did at the last TechEd. You can find the abstract below, and the WMV file can be downloaded from here. Hope you find it useful.

[UPDATE: I attched the slides in xps format. The ppt file is 10x larger]

Framework Engineering: Architecting, Designing, and Developing Reusable Libraries

This session covers the main aspects of reusable library design: API design, architecture, and general framework engineering processes. Well-designed APIs are critical to the success of reusable libraries, but there are other aspects of framework development that are equally important, yet not widely covered in literature. Organizations creating reusable libraries often struggle with the process of managing dependencies, compatibility, and other design processes so critical to the success of modern frameworks. Come to this session and learn about how Microsoft creates its frameworks. The session is based on experiences from the development of the .NET Framework and Silverlight, and will cover processes Microsoft uses in the development of managed frameworks.

What Do Swimmers Have to Say About Framework Design?

kcwalina — Fri, 05 Oct 2007 03:09:00 GMT

I am starting to feel pressure to finish up slides for my presentation at the upcoming TechEd in Barcelona. I will be talking about framework architecture and design. Here is the abstract I took from the conference’s site:

WIN304 Framework Engineering: Architecting, Designing, and Developing Reusable Libraries

I am super excited about the talk, which I want to make into a continuation to the API design pre-con I did with Brad Abrams at the last PDC (BTW, the same content is available here). The PDC talk was about designing the API surface, but there is so much more to framework design!

I am also excited coming back to the wonderful city of Barcelona. Last time I was there, believe it or not, I was on the Polish swimming team competing in the Olympics (see #13). I hope this trip will bring back some memories from the other and completely different “career” I had in the past.

FxCop Rule for Multi-Targeting

kcwalina — Wed, 03 Oct 2007 02:50:00 GMT

Two months ago, Scott blogged about the multi-targeting support in Visual Studio 2008. I worked on this feature in the planning phase (read “long time ago”), and so I am quite thrilled to see it finally in the hands of developers. Especially, that several years ago I remember our small working group sitting in a room and wondering whether such feature was even possible. The complexities of implementing it in a large project like the Visual Studio seemed quite daunting.

The thing that made implementing multi-targeting in one release possible was the concept of Red and Green bits. You can read about the concept here, here, and here, but quickly: red bits are Framework assemblies that existed in the .NET Framework 2.0 and were serviced in versions 3.0 and 3.5. Green bits are assemblies that were added either in the version 3.0 or 3.5. The servicing changes in Red bit APIs are limited (after all it is servicing) to a very small number of API additions and bug fixes.

We leveraged (and influenced) the decision to limit the number of changes to existing assemblies to drastically simplify the requirements for the multi-targeting system. That is, we made an assumption that the majority of differences between the Framework versions (targets) are on assembly boundaries.

But now I have to confess, there are some limitations in this design that we accepted when we made the original simplifying assumption. There is a very limited number of APIs being added to the Red assemblies and the multi-targeting system is currently not able to detect these. For example, if you set the target to the .NET Framework 2.0 and read the newly-added (albeit obscure) property GCSettings.LatencyMode the program will compile just fine but then fail to run on the .NET Framework 2.0 RTM. The reason is that the property was added to an existing (Red) class GCSettings, not to a new class in a new assembly. Despite that the number of such additional APIs in Red bits is very small (and that we recommend that you still test your programs on all targeted platforms), this can be quite annoying. And so feeling a bit responsible for this (and trying to promote FxCop J), I wrote an FxCop rule that can analyze IL of an assembly targeted at Framework 2.0 and warn you about all calls to members that are not present in 2.0 RTM.

Here is how it works: The test program below uses new LatencyMode enum and calls a new GcSettings.LatencyMode property. As I mentioned above, these APIs don’t exist in .NET Framework 2.0 RTM. Even if I built this project with the multi-targeting target set to Framework 2.0, the system would not complain about calling these APIs. But, as you can see in the error list, the FxCop analysis engine can deal even with this difficult to detect problem. You can think of the rule as a very smart (post) compiler-step.

A sample project with the rule is attached to this post. I provide it without any guarantees, as is, and in fact I am sure it has many bugs and problems. When I find time, I will work on it more to polish it a bit but I don't promise (i.e. treat it as a sample).

BTW, to install the rule, just build it and drop it to the FxCop rules directory. On my machine it is at C:\Program Files\Microsoft Visual Studio 9.0\Team Tools\Static Analysis Tools\FxCop\Rules. In addition you have to go to Project settings of the project you want to analyze and turn on FxCop analysis (the “Code Analysis” tab in the project properties). Lastly, you can either rebuild or right click on a project and choose “Run Code Analysis”. I hope it’s helpful.

Duck Notation

kcwalina — Wed, 18 Jul 2007 21:20:00 GMT

I have been working with the C# and VB teams on design guidelines for LINQ. We started to talk about the so called Query Pattern, which describes what you need to do if you want a custom type to support the new query operators (select, where, group by, etc.).

The query operators use duck typing to determine whether they can operate on a type or not. This means that instead of implementing an interface (static typing) a queryable type will need to have a set of members that follow a specified set of conventions (naming, parameter and return types, etc).

For example, the C#’s foreach operator already uses duck typing. This might be surprising to some, but to support foreach in C# you don’t need to implement IEnumerable! All you have to do is:

Provide a public method GetEnumerator that takes no parameters and returns a type that has two members: a) a method MoveMext that takes no parameters and return a Boolean, and b) a property Current with a getter that returns an Object.

For example, the following type supports foreach:

class Foo {

public Bar GetEnumerator() { return new Bar(); }

public struct Bar {

public bool MoveNext() {

return false;

}

public object Current {

get { return null; }

}

// the following complies just fine:

Foo f = new Foo();

foreach (object o in f) {

Console.WriteLine(“Hi!”);

}

But, as you can see in the yellow highlight above, the describing the foreach pattern in English (or any other spoken language) is quite difficult and not very readable, especially if you contrast it with the simplicity specifying requirements based on static typing:

Implement IEnumerable.

… and having IEnumerable defined as:

public interface IEnumerable {

public IEnumerator GetEnumerator();

}

public interface IEnumerator {

public bool MoveMext();

public object Current { get; }

}

The english description gets much worse for something like the query pattern, which is way more complex than the foreach pattern. Because of that, I was thinking that there must be a better way to specify such patterns based on duck typing. But when I searched the web, to my surprise, I could not find any simple notations to do that. If you know of any, please let me know.

In the absence of an existing notation, I started to think about something like the following:

[Foreachable] {

public [Enumerator] GetEnumerator();

}

[Enumerator] {

public bool MoveMext();

public [ItemType] Current { get; }

}

This seems much easier to parse than the English description of the pattern. What do you think?

How to Fight Complexity in Software (part I)

kcwalina — Wed, 18 Jul 2007 02:02:00 GMT

A couple of weeks ago, Grady Booch gave a lecture at Microsoft. It was a pleasure to hear of my software engineering heroes in person. Grady talked about “the promise, the limits, and the beauty of software.”

The main thing that captured my interest during the lecture was a discussion about the cost of complexity in software.

Grady said that there are many factors influencing the cost of software, but two of these factors have disproportionally high impact on the overall cost: a) the processes used to develop and maintain the software, b) the complexity of the codebase. To fight growing complexity of the codebase, many companies allocate some percentage of their resources for refactoring, cleanup, and other activities that don’t directly result in features for the end user of the software. This really resonated with me. I am a huge fan of this approach. We do some of it here in the developer division (read about the MQ milestones here, here, and here), but I wish we and software industry in general did even more.

FxCop Designers Honored with the Chairman's Award

kcwalina — Wed, 04 Jul 2007 02:21:00 GMT

Last week, during the annual Engineering Excellence week, several Microsoft engineers and managers involved in development of engineering tools and practices were presented with Engineering Excellence Awards. In addition, the principal designers of three static analysis tools were honored with so called “Engineering Excellence Chairman’s Award,” which is given for contributions that our chairman (Bill) considers especially important. The Chairman award was given twice in the history of Microsoft.

Mike Fanning, Brad Abrams, and I received nice glass statuettes and a framed letters from Bill and Jon (our VP responsible for EE). See pictures below. The letter says “The Engineering Excellence Chairman’s Award is the Microsoft’s highest award for engineering group employees worldwide.”

It’s extremely rewarding to receive something like that for what started as a hobby project on which we worked mainly in our free time (at least, before it was officially productized). I would like to thank Brad for being the “let’s do it now” person, pushing the whole effort of design guidelines and fxcop development, and Mike for his passion for quality and being the super-developer on the team.

Of course, many others contributed to the success of fxcop. Once we gather all the names, we will post a new entry or update this one to thank everybody and record all the contributions for posterity. [UPDATE: Brad just posted on this here.]

Generic Methods as Casts

kcwalina — Thu, 07 Jun 2007 20:56:00 GMT

Somebody just asked me which of the following API design alternatives is better. They both do what we could call “casting.”

// Generic Method “Cast”

var foo = someInstance.SomeMethod<IFoo>();

// Simple Cast

var foo = (IFoo)someInstance.SomeMethod();

The Generic method approach can potentially have two benefits. A constraint can tell users what the subset of valid casts is. For example, the following API will allow casts to a collection (a subtype of IEnumerable):

public TCollection SomeMethod<TCollection>() where TCollection:IEnumerable {

…

}

If the user calls this method and tries to “cast” the return value to let’s say an Int32, the compiler with complain. For example,

var int someInstance.SomeMethod<int>();

… will generate the following error:

The type 'int' must be convertible to 'System.Collections.IEnumerable' in order to use it as parameter 'TCollection' in the generic type or method 'SomeClass.SomeMethod<TCollection>()'

Secondly, the Generic method might do different things based on the type parameter. The method relying on a simple cast has now knowledge about the type the user will cast it’s return value to.

public TCollection SomeMethod< TCollection >()

where TCollection:IEnumerable,new() {

TCollection newCollection = new TCollection ();

ArrayList newArray = newCollection as ArrayList;

if (newArray != null) newArray.Capacity = 0;

return newCollection;

}

If neither of these benefits applies in your case, I would use a simple cast. It’s more transparent in terms of what’s going on. The users of a generic method might wonder what kind of “magic” is being done with the generic type parameter.

Silverlight - Program the Web the Way YOU Want It

kcwalina — Sat, 05 May 2007 19:36:00 GMT

I just came back from Mix where we announced details of Silverlight’s programming model. This is pretty much what I have been working on for the last year and a half. It feels great to finally be able to talk about it. I will blog more about the details after I am back from my well deserved vacation.

But more importantly, I am supper excited that there is finally a real programming model for the Web. I am a system software developer, and to me, all existing web programming models felt like smaller or larger hacks. They are all based on technologies invented for something other than writing applications, but adopted to these scenarios over the years. For example, HTML was invented for representing documents with hyperlinks. This is quite natural for immature technologies, so I am not blaming anybody here :-).

When Silverlight’s vision started to crystallize, we realize that we have an opportunity to do the Web programming model right this time:

1. Modern programming languages: In Silverlight, you can program using your .NET language of choice, whether it’s a traditional OO language like C#, VB.NET, or C++, a dynamic language like IronPython or IronRuby, or a functional language like F#, … or choose from the plethora other 3^rd party languages.

2. State of the art markup: XAML really shines when you need the ultimate flexibility in representing graphics and GUI.

3. First class garbage collected runtime: The runtime included in Silverlight (“Core CLR”) is basically the .NET Framework’s runtime with features irrelevant to the web programming removed (e.g. COM Interop).

4. Large ecosystem of tools, information, and services: Silverlight’s type system is the same CIL type system used in .NET and the Silverlight Framework is simply a subset of the .NET Framework. This means the most if not all .NET Framework tools work or can be easily adopted to Silverlight. This includes Visual Studio, FxCop, .NET Reflector, MSDN, and new specialized tools like the excellent Expression Blend.

I already wrote a couple of small applications using Silverlight, and I have to say that it makes web programming natural for the first time. The platform no longer forces you to program its way, while your intuition and years of programming experience tell you to code some other way. It makes me say, for the first time in my life, that I am proud to be a web developer :-)

Nourishing Creativity

kcwalina — Wed, 25 Apr 2007 03:39:00 GMT

For those who haven’t heard about it, TED is a multidisciplinary conference for experts from technology, entertainment, and design industries. They just updated their web site and while browsing around it I found this very interesting presentation about creativity by Ken Robinson.

One of the main points Ken makes is that penalizing people for failing when they try new things is killing creativity. His talk concentrates on education systems, but I think the same mechanisms exist in the industry.

Effective Presentation Skills

kcwalina — Fri, 02 Feb 2007 04:25:00 GMT

I just watched a very interesting presentation. The content of the presentation is interesting, but not very relevant to the work software engineers do. The super interesting and relevant part is that it’s one of the best presentations I have ever seen in terms of the mechanics of presenting, and I think all who do presentations can learn a lot from it.

1. The presenter wanted to communicate just one relatively simple thing that not many people pay attention to, so he had to capture the audience’s attention and he did a very good job doing it.

2. The presentation has lots of content, but the content is just to support the main thesis; it's not a laundry list of things he wanted people to remember (that it almost never successful).

3. The presentation shows the power of visual communication (and software). The same information presented in a static table, bullets, etc. would completely fail in communicating the main message.

How to Design Exception Hierarchies

kcwalina — Tue, 30 Jan 2007 22:47:00 GMT

I still get a lot of questions on how to design exception hierarchies, despite several attempts to describe it in talks, the FDG book, and in posts on this blog. Maybe the guidance gets lots in the in the complexities of the full guidance surrounding exception handling or I am a bad communicator. Let me assume the former :-), and so here is one more attempt at describing the guidance in the most succinct way I am capable of:

· For each error condition you reusable routine can get into, decide whether the condition is a usage error or a system error.

o A usage error is something that can be avoided by changing the code that calls your routine. For example, if a routine gets into an error state when a null is passed as one of its arguments (error condition usually represented by an ArgumentNullException), the calling code can modified by the developer to ensure that null is never passed. In other words the developer can ensure that usage errors never occur.

o A system error is something that cannot be avoided by simply writing code that tries to avoid the error condition. For example, File.Open gets into an error condition when it tries to open a file that does not exist (and it throws FileNotFoundException). This error condition cannot be fully avoided. Even if you check whether the file exists before calling File.Open, the file can get deleted or corrupted between the call to File.Exists and the call to File.Open.

· Usage errors need to be communicated to the human developer calling the routine. The exception type is not the best way to communicate errors to humans. The message string is a much better way. Therefore, for exceptions that are thrown as a result of usage errors, I would concentrate on designing a really good (that is explanatory) exception message and using one of the existing .NET Framework exception types: ArgumentNullException, ArgumentException, InvalidOperationException, NotSupportedException, etc. In other words, don’t create new exception types for usage errors because the type of the exception does not matter for usage errors. The .NET Framework already has enough types (actually too many, IMHO) to represent every usage error I can think of.

· System errors need to be further divided into two groups:

o Logical errors are system errors that can be handled programmatically. For example, if File.Open throws FileNotFoundException, the calling code can catch the exception and handle it by creating a new file. (Side note: this is in contrast to the usage error described above where you would never first pass a null argument, catch the NullArgumentException, and this time pass a non-null argument).

o System failures are system errors that cannot be handled programmatically. For example, you really cannot handle out of memory exception resulting from the JIT running out of memory.

· System failures should result in the shutdown of the process. This might sound scary at the first read, but I would say it's by-definition: a system failure is an error that can neither be handled by the developer or the program. The best way to shut down a process in such cases is to call Environment.FailFast, which logs the state of the system, and that is very useful in diagnosing the problem. The good thing is that system failures are extremely rare in reusable libraries. They are mostly caused by the execution engine.

· Logical errors are errors that can be, and often are, handled in code. The way to handle such errors is to catch the exception and execute some “compensating” logic. Whether the catch statement executes is determined by the type of the exception the catch block claims it can handle. This means that logical errors are are condition (and actually the only conditions) where the exception type matters and when you should consider creating a new exception type.

· If you think you are dealing with a logical error, I would validate this belief by actually writing code or describing very precisely what would the catch handler do when it catches the exception to allow the program to continue its execution. If you cannot describe it or the error can be avoided by changing the calling code, you are dealing with a usage error or a system failure.

· Note that logical errors are still pretty rare. As a rule of thumb, I would say that error conditions in a typical reusable library fall into: <1% of system failures, 5% logical errors, and the rest ~95% are usage errors.

· If you are convinced that your error is a logical error, you need to decide whether to reuse one of the existing exception types from the framework or create a new exception type.

o You should use an existing Framework exception is both of the following are true

§ The exception type makes sense for your error condition. For example, you don’t want to throw FileNotFoundException if your threadpool implementation runs out of the thread quota.

§ The exception will not make an error condition ambiguous. That is, the code that wants to handle the specific error will always be able to tell whether the exception was thrown because of the error or because of some other error that happens to use the same exception. For example, you don’t want to throw (reuse) FileNotFoundException from a routine that calls Framework’s file I/O APIs that can throw the same exception, unless it does not matter to the calling code whether your code or the Framework threw the exception (BTW, it’s quite often that it does not matter).

o If you decided not to reuse an exception type from the .NET Framework, you will have to create a new exception type. When you create a new exception type, as a rule of thumb, I would simply inherit it from System.Exception or from a single subtype of System.Exception representing custom exceptions declared in your component/framework. I would create the root only if you have more than 3 custom exceptions. There are limited cases where it’s good to create a more elaborate hierarchy, but it’s extremely rare. I would say in a typical library 99% of custom exceptions should follow this guidance. The reason is that when you catch exceptions you almost never care about the hierarchy. You almost never care about the hierarchy because you almost never want to catch more than one error at a time. You almost never want to handle more than one error at a time, because if the errors could be handled the same way, they should not be expressed using two different exception types. Now, please note that I said “almost” and “should” in several places. You your case falls into the small number of corner cases, you need to create a deeper hierarchy.

o Lastly, keep in mind that it’s not a breaking change to change an exception your code throws to a subtype of the exception. For example, if the first version of your library throws FooException, in any future version of your library, you can start throwing a subtype for FooException without breaking code that was compiled against the previous version of your library. This means, when in doubt, I would consider not creating new exception types till you are sure you need them. At which point, you can create a new exception type by inheriting from the currently throw type. Sometimes it might result in slightly strange exception hierarchy (for example, a custom exception inheriting from InvalidOperationException), but it’s not a big deal in comparison to the cost having unnecessary exception types in your library, which makes the library more complex, adds development cost, increases working set, etc.

.NET Framework Performance Signatures

kcwalina — Sat, 27 Jan 2007 03:52:00 GMT

I love Rico’s performance quizzes in general, but the last one has something especially interesting: a link to a file listing all members in the Framework and their estimated performance signatures (perf characteristic based on the number of allocations in the member).

Now, I only wish I had an fxcop rule warning me when I call some of the expensive members in loops, low on the callstack, on the UI thread, or in any other performance sensitive places.