Eric Lee - A Humble Blog : Visual Studio Team System

Apologies, forgot to include a file in my Hemi distribution

ericlee — Mon, 20 Nov 2006 22:56:27 GMT

Hey guys,

Sorry, in my Hemi tool, I’m using a 3rd party control from Component One. I have a license for their tools and they allow redistribution. The file you need is located here: C1.Win.C1FlexGrid.2.zip.

Thanks to Grant for spotting this so quickly.

Eric.

Work Item Moving Tool is back!

ericlee — Mon, 20 Nov 2006 12:35:45 GMT

Hey guys, I've finally had a chance to update my work item moving tool to work with the RTM release of Team Foundation Server. Along the way, I made some modifications and changes that I've wanted to do for a while. I haven't thought of a better name than 'work item mover' so for now I've code-named it if you will 'Hemi'.

I’ve posted the tool to: http://www.counterpunchsoftware.com/downloads/hemi.release.zip. You’ll also need: C1.Win.C1FlexGrid.2.zip.

I included the Team Foundation Server binaries that it depends on, but the best way to run the tool is just to run Hemi.exe from the Visual Studio Tools Command Prompt.

Hemi starts up and looks as follows:

I’m thinking about using a MDI type of interface in the future; that would enable you to save and re-run moves that you might do on a regular basis.

To start moving work items, the first thing to do is use the File menu…

This starts the various stages involved in moving work items from one team project to another.

The first thing to do is pick the Team Foundation Server that you want to access.

Depending on the authentication that you are using, you may get prompted for credentials. Once you’ve chosen a Team Foundation Server, the next step is to select the work items that you want to move. This is done by selecting a Team Foundation Server Query.

Pressing this button will bring up the same work item query control that is used in the Excel integration.

Once you select a query, press the Find button on this dialog to bring up the actual work items. Press OK and you’ve established the work items that you are going to move. Now select the Team Project to move these work items to.

Once you’ve done that, press Next to move on to the next stage.

Any query that you might use might encompass a few different work item types, so in this stage, you’ll decide how to map those work item types to the Team Project you are moving the work items to. In most cases, you’ll probably move the work items to the same type in the other Team Project.

But, sometimes work item types are defined differently. For example, in a CMMI project, a bug is defined with two required fields – Symptoms and Steps to Reproduce. These fields aren’t required in an Agile project. So, this stage also gives you the chance to fill these missing values in. Press the Check for Missing Values to see what these missing values are.

If there are any, enter values into the table.

Press Next to move on to the next stage and actually move your work items.

This stage shows you what work items will be moved, as well give you a couple of options for your original work items. Team Foundation Server doesn’t support deleting work items, so your original work items will still exist. By default, Hemi will create a Moved state for these work items and assign them to that state. Also, it will create links between the original work items and their new copies, and vice versa.

Once you move to the next stage, your move will be committed!

If you look at your original work items, what you should see is that it is in a Moved state:

And also, your original work item will be linked to the new one, and vice versa.

Anyways, I hope you find Hemi useful – if you run into any bugs or have suggestions, please let me know. Either through this blog or email me at eric@counterpunchsoftware.com.

Thanks!

Eric.

'Wall Time' as a way of determining correctness

ericlee — Fri, 29 Sep 2006 01:01:00 GMT

Quite a few years back, I joined the Windows Server 2003 team as a developer. Windows Server 2003 is a massive organization – I was part of the UDDI team which resided under Directory Services.

Our QA team had a suite of about 300 automated tests. The collected the ‘wall time’ of a test run – so literally how many minutes/seconds it took for each test to run. The ‘wall time’ amounts were aggregated into a test suite run time.

When I first joined, that suite of tests was taking about 2 hours to complete. That was a concern because the test suite used to only take about 20 minutes. We used a homegrown solution to store our test data and this was 2002, so there was no SQL Server 2005 Business Intelligience to take advantage of

My first task when I joined was to determine if there was a performance problem – the number of tests in the suite had increased, and so had our functionality – and if there was one, to fix it. I spent a long while in our profiling tools (crude compared to what we have today) and tracked the problem down to a Thread.Sleep() that someone had accidently put in the code.

What was ironic was that we had several unit tests that targetted that particular piece of code. Looking back, at the test results for those unit tests, I could see that the ‘wall time’ for those tests had almost tripled. On their own, these slower unit tests only added a little bit of time to the test run. But when the Thread.Sleep() code those unit tests were targetting werer hit by our load/scalability tests, the slowdown because expontential.

What we didn’t have then was a way to express how long a test was supposed to take. We checked for return values, the presence of exceptions, etc, but we couldn’t say that a test would fail if it took longer than some amount of time.

This ‘timeout’ is something that is built into Visual Studio Team System. If you look at the properties of a test, you’ll see a timeout property.

The value of this property is a handy way to define the performance characteristics of a given test. Run times will of course differ based on hardware, but this property is a good sanity test.

Thanks,

Eric.

Building development scorecards

ericlee — Thu, 28 Sep 2006 01:03:00 GMT

Hey guys,

Earlier this year, I did some blog posts about how to create key performance indicators (KPIs) based on the software development data that Team Foundation Server collects.

There are numerous clients for those KPIs, but the one that is probably the most dedicated KPI client from Microsoft is Business Scorecard Manager (BSM); with the upcoming Office 2007 release, this product will become a part of Microsoft Office PerformancePoint Server 2007. The BSM team blog has lots of great information about their product.

Scorecards in general are interesting. In just about any business, there are dozens, if not hundreds of data points to collect when trying to build an overall (or sub-overall ) view of the business so somebody can make a decision. One of my first projects when I joined Microsoft was to build a status collection and display web application for our QA team. Team members would update the status of their test runs, and every week the managers would look at the overall status and decide on strategy. That simple web site was in essence a score card. Maintaining it was a pain, because different teams had different data formats and different preferences for updating their data. Some didn’t want to use a web site, some did, etc.

Gathering and displaying data from across many different teams and normalizing data formats is what Business Scorecard Manager does a really nice job of. BSM takes KPI’s from various sources, including itself, and visualizes them in a consistent manner.

Usually, BSM is shown in the context of business functions like sales and finance, but there is no reason not to use it for software development data as well.

Let’s use the KPI’s from my earlier posts as an example. Below is a screen shot of what one of them looks like inside the SQL Server 2005 KPI browser view.

The first step is to import our SQL Server 2005 KPIs’ into a format that BSM understands. The KPIUtil from Microsoft Downloads does exactly this. The instructions for the tool are excellent; basically you install the sample indicators and run the tool to import your KPIs from SQL Server 2005. The first step is necessary to bring in the indicators that SQL Server KPIs have access to.

After running KPIUtil, we will have access to our LeadTime KPI inside of Business Scorecard Manager. I just used one of the example workspaces (SalesSummary) from Business Scorecard Manager to augment. In a real project, you would probably want to create your own workspace from scratch.

A BSM scorecard can be published into many different forms. One way is to publish the scorecard into Windows Sharepoint Services – BSM will automatically create, format and populate a WSS page for you. Once published, this page will stay in sync with any changes that you might make to your scorecard.

Before adding our LeadTime KPI, the WSS page generated by the example Sales Summary score card looks like this:

Inserting a new KPI into the scorecard is pretty straightforward. Objectives are used to organize data in your scorecard, so you might create an objective for tracking various KPIs for your software development.

There are various aspects of an objective that I can’t do justice to, so I’ll just add the LeadTime KPI to the objective we just created.

Once we re-publish this scorecard, our WSS page is updated as well, and we see our new KPI, displayed pretty much the same way as we saw in the SQL Server 2005 browser view:

I’m not sure that scorecarding is for everyone, but I think it might be more applicable than people think. Whenever I walk through the hallways of our development team, or many others at Microsoft, I see monitors on the walls that show various charts and graphs tracking project progress. Most of these systems are hand created and maintained. I wonder if Business Scorecard Manager would be a more efficient solution?

In any case, I’m always on the look out for a technology that might make software development a little bit easier. There is a lot of capabilities in the upcoming release of Office 2007, so I’m looking forward to doing more posts about that.

Thanks!

Eric.

Building a Virtual Kanban System with Team Foundation Server - Part 1

ericlee — Mon, 11 Sep 2006 23:11:00 GMT

Hey guys,

David Anderson is someone that I’ve had the privilege of working with for the past couple of years. One of the things that I’ve found really interesting about his work is that he often takes practices from other industries and applies them to managing software development.

One example of his is the idea of a Kanban System. Many Japanese manufacturing companies have used this process with great success. The basic idea is to control the amount of raw material in play. Wikipedia has a great explanation of the system in general.

“The Kanban system might be visualised as a "Three bin system" - one bin on the factory floor, one bin in the factory store and one bin at the Suppliers' store. The bins usually have a removeable card that contains the product details and other relevant information - the Kanban card. When the bin on the shop floor is empty, the Kanban card is removed and given to the store. The store then replaces the bin on the factory floor with a full bin which also contains a removeable Kanban card. The store then contacts the Supplier and indicates the need to replenish the Kanban card. The product also containing a Kanban card is delivered into the factory store completing the final step to the system. So it will never run out of product, providing of course, the cards are reliably collected from empty containers”

David Anderson has had a lot of success applying the principles of a Kanban System to software development. It remarkable how effective a relatively lightweight process like a Kanban System can be; his paper describes in detail the success of a Microsoft IT team.

I won’t do justice to the work so I’ll leave it to you to read the paper; what I wanted to explore was how one might implement a Kanban System with Team Foundation Server.

As I mentioned in one of my earlier posts, I think there might be some really interesting integration between BizTalk Server and Team Foundation Server. BizTalk becomes really useful when you have to make a bunch of things and people interact in a reasonable and consistent way; otherwise referred to as orchestration

I’m new to BizTalk Server so I wanted to experiment with how it can be used in conjunction with Team Foundation Server. I’m just scratching the surface of BizTalk so be warned, there might be easier/better ways to do things. The full solution can be downloaded here.

The Orchestration Designer integrates into Visual Studio 2005 and is a key part of BizTalk Server. First, I’ll show you the orchestration as a whole; then we’ll dive down into the details.

Let’s go through each step so we get a basic idea of what the orchestration looks like. The number items below match the numbers in the diagram above.

The orchestration is activated when we get a message indicating that a work item is changed
From a Kanban perspective, we only care about a work item that has changed if it is just been created, made active or closed. We don’t care about any intermediate changes. If this change matches that criteria, we are going to query Team Foundation Server to determine the number of active work items.
If a new work item has been created, and our Kanban is full – i.e. there are more than a certain number of active work items – then we will have to put this work item in our back log.
If a work item is being closed, and our Kanban has at least one open card – i.e. there are less than a certain number of active work items - then we will notify the customer that a free card is available.

That’s all there is to our Kanban System example.

But, you’ll notice that I left out a bunch of details – things like how do we determine the number of active work items, notify the customer, etc. One advantage of using BizTalk Server is that there is a separate between logical and physical ports.

A port in BizTalk Server is something you receive data from and/or send data to. The data being sent or received is in the form of XML. You have to define the structure of the XML immediately, but you can decide what the physical ports are later. You change them as well. So for now, we don’t have to worry about how we’ll get information about the number of work items in Team Foundation Server or how we notify our customer. We know we have to do it, we’ll figure out exactly how to do it later on.

Now let’s look into the details of the orchestration. First, let’s look at how the orchestration is activated.

1 – Port_WorkItemChanged Port

We haven’t done it yet, but we will register for WorkItemChanged events with Team Foundation Server. These event notifications will come into this port.

The designer unfortunately does not allow you to change the width of things, so the name of this port is somewhat obscured. The message we are waiting for is defined by an XML schema that we create. Actually, strictly speaking, we can create the schema from scratch or reference an existing one defined in XSD, as a .NET or as a web service.

The structure of the message we are waiting for is very simpe and looks like:

Ideally, I would have liked to use the WorkItemChangedEvent that is delivered directly from Team Foundation Server’s event notification system. But, the structure of that message makes it somewhat difficult to extract the information that I need from it. For now, I took the easy way out and defined my own message structure that I’ll hyrdate.

2 – Receiving the message

This is just a standard BizTalk Server thing – this receive is what activates our orchestration.

3 – Deciding if we are interested in this event

We’re not interested every time a work item changes; the Expression shape in BizTalk Server allows us to define an expression to find only the ones we’re interested. The expression behind this shape looks like:

The only changes we’re only interested in are the newly created work items, the ones that get closed, and the ones that are reactivated.

When we have a work item in any of those states, we’ll treat them differently based on the number of active work items that we have. The section of our orchestration below is what we use to count up our active work items.

1 – Counting Work Items

We have a web service that queries Team Foundation Server and returns the number of Work Items in the active state.

2 – Web Service Port

BizTalk Server setups up a request/response port for you automatically when you use a port from a web reference.

Once we have the number of active work items, we will combine that information along with the state of the work item.

If the work item is active (either because it is newly created, or because it’s state was set that way), and our Kanban is full, we will ‘punt’ the work item. A Kanban is full when there are no open slots or cards to represent the work. This enables us to control the amount of work the development team has to worry about.

We ‘punt’ a work item by calling another method on our web service port. What that web service does is change the state of our Work Item to ‘BackLog’. This is the end of the orchestration for this branch.

If the work item is closed and our Kanban has an empty slot, then we will notify our team of a free Kanban card.

This is the end of the orchestration for this branch as well.

So, what we have in our orchestration as a whole is the basic implementation of a Kanban system. To summarize, when a work item is created or re-activated, we check to see if we have any open kanban cards. If we do, that’s fine we don’t need to do anything. But if we don’t, then we will assign that work item to the backlog. Every time a work item is closed, we check to see if we have any open kanban cards. If we do, then we send a notification to our team to let them know that a card is open.

Those are the basics of the system – I’ll do a follow up post that delves into some of the details about how the ports are bound and show some examples of the orchestration actually executing.

Thanks!

Eric.

Quick post - 2 really cool features about Work Items

ericlee — Wed, 06 Sep 2006 06:50:00 GMT

I’ve just returned from Tech Ed Australia and New Zealand and I’ve got lots to blog about that.

But before I do that, 2 features about Work Items caught my eye tonight that I really wanted to share.

First, you can copy a single work item to another Team Project by simply right-clicking on it.

When you choose Create Copy of Work Item… you get a choice of what Team Project and Work Item Type you like to make the copy to.

Neat stuff! I am perpetually working on my tool for copying work items en mass, and in all honesty I had overlooked this feature before. Now I wonder if this is exposed in the API somewhere I don’t think it is, but maybe it is something we can do in the future.

Another neat feature is the way process guidance is integrated – let’s say that you’re wondering what the lifecycle of a given work item type is. For example, do bugs go from active to resolved to closed; or active to approved to resolved to close? If you’re ever wondering, just right-click on the work item and view the process template.

This will bring up the process template at exactly the place that is pertinent at the moment. One can easily view every related to this work item type in the guidance.

If you customize your process guidance, then you can be taken to information that pertinent to your organization as well.

Lately I’ve been trying to drill into the depths of work item tracking and these 2 tips caught my eye so I wanted to share. It never ceases to amaze me how dense of a product VSTS 1.0 is.

Anyways, hope you’re doing well, look for a Tech Ed wrap up post from me soon.

Thanks!

Eric.

You meet the nicest people on a Honda...

ericlee — Mon, 14 Aug 2006 05:54:00 GMT

When I moved to Seattle, one of the most enjoyable things I learned how to do was ride a motorcycle. I really enjoyed the open air, speed and thrill of riding.

Both of the bikes that I’ve owned – a Kawasaki Ninja 500 and a Ducati Monster 750 – were very reliable and never caused me any problems; a good thing, since I don’t have any mechanical skills to speak of.

The only thing I had to worry about was learning how to ride. And even that wasn’t very hard – in Washington State, there is a Motorcycle Safety Foundation class that is jointly sponsored by the state and several of the major motorcycle manufacturers – Honda, Kawasaki, Suzuki, etc. The class was just $50 and included 1/2 day of classes and 2 full days of riding.

Just about everything was provided for us. A few weeks before the class, everyone that was registered for the class received vouchers in the mail entitling us to 20% off helmets and gloves at most of the local motorcycle dealerships.

For those who didn’t buy their own gear, there was a reasonable selection of helmets and gloves to use onsite. We had brand-new, friendly, 250cc motorcycles to learn on as well. These bikes were provided by the various manufacturers that sponsor the course. Generally speaking, we had our choice of bikes that represented the major categories of motorcycles – cruisers, standards and dirt-bikes.

The last-half of the course was our endorsement test. Our instructors became our evaluators and one-by-one we did the test. My class had 30 or so people and everyone passed. Many, like myself, had had never ridden any motorized 2–wheelers in their lives before.

After 3 days, the market for motorcycles in Washington State grew by 30

At the time, I took the class a bit for granted. However, had I wanted to learn how to ride a motorcycle in the 1950’s or early 1960’s, things would have been very different.

‘You meet the nicest people in a Honda’ is a popular ad that Honda ran in the 1960’s to promote their new motorcycle, the ‘Super Cub’. This ad campaign and the motorcycle represented an important turning point in the motorcycle industry in the United States.

At the time, the perception around motorcycles in the United States wasn’t very positive. Generally speaking, motorcycle-ists were considered rebels, outlaws and members on the fringe of society. Also, maybe more importantly, the motorcycles at the time were not very attractive for the mainstream. Motorcycles of that era were heavy, difficult to ride and unreliable. You had to have a good understanding of the workings of a motorcycle to perform the near constant maintenance that the bikes of that day required.

Honda’s ‘Super Cub’ changed all of this. It was small, friendly-looking and easy to ride. And it’s engine was nearly maintenance free. The ad campaign that introduced this motorcycle focused on parents, young couples and professionals riding the ‘Super Cub’.

What Honda did was pretty impressive. They identified an appealing thing that a small, niche segment of the market was doing, and, more importantly, they identified what was keeping this niche thing from being popular in the mainstream United States. To name a few examples, motorcycles in the United States at the time:

suffered from a negative image
required its customers to have indepth mechnical knowledge
was thought to have little practical value

To mitigate these issues, Honda did 2 things. First, they built a great product - a fun, easy to ride, bullet-proof motorcycle that could accomodate 2 people easily; this helped increase the broad appeal of the motorcycle. Second, Honda built a friendly ad-campaign to start changing the negative perception of motorcycling in general.

The rest is history, the ‘Super Cub’ sold 200K units in its first year, more than all the other motorcycle manufacturers combined that year and went on to sell 50M units in its lifetime. The ‘Super Cub’ is largely remembered as being the motorcycle that brought motorcyling into the mainstream US market.

So what does this have to do with software? Or Application Lifecycle Management? Well, my hypothesis is that there is an opportunity to bring what customers who are very, very early adopters have been doing into the mainstream.

When we launched Visual Studio 2005 last year, we started talking about something called the Application Platform, which was the combination of BizTalk Server (BTS), Visual Studio and SQL Server.

My hypothesis is that the Application Platform represents what a few of our very early adopter customers have been doing for quite some time.

For example, BTS can do many things, but one of the things it does very well is provide a way to implement complex workflows and flow of data.

From a software development standpoint, BTS is pretty useful. There are lots and lots of complex human workflows that take place in software development – code reviews, buddy builds, check-in approvals are just a few examples. Why not take advantage of a graphical designer to literally orchstrate these instead of relying on word of mouth or documentation? SQL Server is pretty useful as well – I’ve gotten a lot of interest around building Key Performance Indicators from software development data as well as applying data mining to it. Whether you are doing either or both, the great thing about having a data warehouse for your software development is that you can use it in so many other places. Project Server, Business Scorecard Manager and Excel are all clients that can make very sophisticated use of a data warehouse.

If your organization happens to be using BTS and SQL Server for more traditional scenarios, you all of a sudden have a single place for to implement all of your processes – either business-related or software development-related, and you have a common format, a data warehouse, to represent all of your data – either business-related or software development-related.

My hunch is that some customers have been able to do this with BTS 2004, VS 2003 and SQL Server 2000 products, but it hasn’t been easy. There certainly are a lot of hurdles that prevent this type of solution from being accepted by the mainstream.

With the Application Platform bringing together SQL Server 2005, BTS 2006 and VS 2005, I think we are taking a step towards making a solution like this attractive for the mainstream.

For example, BTS 2006 has a bunch of new adapters, but 2 that are really interesting for software development are the email and sharepoint adapters. A BTS adapter allows BTS to accept messages from whatever is being ‘adapted’. With these 2 new adapters, BTS can treat email and sharepoint (SPS and WSS) documents as though they were XML documents and use them in BTS orchestrations. SOAP is another BTS adapter.

Since Team Foundation Server uses email or SOAP for its notifications and automatically creates a WSS sharepoint site for your projects, there is a natural way to integrate with BTS. Team Foundation Server doesn’t have a great workflow story right now, but BTS is all about complex workflows and orchestrations. It’s not hard to imagine a TFS event notification that activates a BTS orchestration which runs its course and reports its results by creating a new document on a Team Project’s portal. I’m sure there are tons and tons of other possiblities.

Another example in terms of making this solution more mainstream is the integration with the Visual Studio client – now without ever leaving the friendly confines of the Visual Studio shell, you can:

write C++ code
design a class in C#
design a multi-tier SOA application
kick off a load test
create a KPI to report your results
data mine to find interesting associations
design your business process

If you know the previous generation tools, then this integration maybe isn’t as important. But if you’re new to an Application Platform-style solution, then centering everything around the familiar – Visual Studio – can really ease the learning curve.

Anyways, just some thoughts on a Sunday evening. The more I dig into SQL Server 2005 and BizTalk Server 2006, the more excited I become about how to apply those technologies to software development. I don’t think we have a ‘Super Cub’ yet, but it’s getting there.

Thanks!

Eric.

When to use Team Projects

ericlee — Thu, 10 Aug 2006 03:18:00 GMT

Team Projects have turned out to a more confusing feature than was intended in Visual Studio Team System. We are in the process of building some guidance for when to create them, but in the meantime, I thought I would shed some light on how we use them internally at Microsoft.

Team Projects were intended to represent the largest unit of work possible in your organization. For us at Microsoft, that means a product or prodct line. ‘Visual Studio 2005’ for example, is a product line that we represent with a single Team Project.

Visual Studio 2005 is made up of a number of products, features and technologies. For example, ASP.NET, Team System and the Common Language Runtime are all part of the Visual Studio 2005 product line.

To keep things organized, we use ‘Areas’ extensively in our single Team Project. Permissions can be set on areas to restrict access to various parts of a given Team Project; I’ll do a video on this shortly.

Here is how are ‘Areas’ are laid out:

‘Areas’ are used all the way down to the feature level; for example, the Team Foundation Server Version Control team uses an Area to define each SCM command:

‘Areas’ are something we’ll be talking a lot more about in upcoming presentations and documentation. They are a really useful way of categorizing within a Team Project. Hopefully showing you how we categorize things within our teams helps a bit. More detailed guidance is on its way.

Thanks,

Eric.

What does a Team Foundation Server CAL look like?

ericlee — Thu, 10 Aug 2006 01:41:00 GMT

Much has been made about when and where you need a Team Foundation Server CAL; but until now, I don’t think anyone has talked about what one looks like.

In my 8 or so years at Microsoft, I’ve never actually seen a CAL of any kind. So, a few weeks ago, I decided to order a Team Foundation Server CAL for myself.

Today, a card board box arrived in my office and it was filled with Team Foundation Server CALs and the mystery was solved

Here is what a Team Foundation Server CAL looks like (incidentally, here is what I look like as well):

The actual CAL, ‘certificate’ if you will is quite detailed – there are lots of holographic images, etching and shiny metal-looking material on it.

As it turns out, a Team Foundation Server CAL is much thinner than I am. I’m in the process of losing some weight, so stay tuned, I’ll be sevlte like our CAL in no time.

Anyways, just wanted to share, take care!

Eric.

Sharing files in Team Foundation Server

ericlee — Fri, 21 Jul 2006 03:52:00 GMT

Hey guys,

As you probably know, sharing is not a feature that is supported by Team Foundation Server. Buck Hodges’ and Brian Harry have both made some excellent posts about this.

Sharing files has some advantages and disadvantages. It is nice to be able to have a common file that is referenced throughout your code base. That shared file might contain a version number, or some useful constants. Sharing is pretty common in web development scenarios I believe as well.

The tricky thing about sharing a single file across multiple projects is that by default Visual Studio will make copies of any existing files that you try to add to your project. That’s good for isolating yourself from breaking changes, but not so good if you’re trying to share a single copy of a file.

I stumbled across a feature in Visual Studio last night that makes this type of sharing pretty easy to do. When you have a project open in Visual Studio and you want to add an existing file, you can use the Add Existing Item context menu option. In the resulting dialog you can navigate to a file and add it to your project. Visual Studio, by default, will create a copy of that file and put it in your project folder.

However, there is an additional option to add just a reference to that file as illustrated below:

This Add As Link option enables you to setup single-file sharing with Team Foundation Server. If you add links to a shared file from several different projects, each of those projects will see any change you make to that shared file immediately.

Again, this type of sharing is not always the right thing to do, but it is an option. I recorded a short video that illlustrates how to combine this feature with Team Foundation Server to enable sharing.

Thanks!

Eric.

Thoughts on David Lempher's brainstorm about Impact Analysis

ericlee — Wed, 19 Jul 2006 21:17:00 GMT

Rob Caron pointed me to David Lempher’s recent blog post about possibly doing impact analysis with Team Foundation Server. It’s a funny coincidence, because Bogdan Crivat and I were brainstorming about how we can take better advantage of the data mining in SQL Server 2005.

Data mining with SQL Server 2005 involves applying one of several algorithms to your data to pull out information that one cannot deduce from casual observation. There are several algorithms that I think would lend themselves to software development.

SQL Server 2005 provides the following algorithms that you can use for data mining:

Microsoft Clustering
Microsoft Association Rules
Microsoft Decision Trees
Microsoft Linear Regression
Microsoft Logistic Regression
Microsoft Naive Bayes
Microsoft Neural Network
Microsoft Sequence Clustering
Microsoft Time Series

The Microsoft Assocation Rules algorithm is used for doing what is known as market basket analysis. The canonical example of market basket analysis is a grocery shopping cart – market basket analysis provides insight into questions like: ‘when people buy diapers, what else are they buying?’. One answer to that question surprisingly is beer; I guess dads must get sent out a lot to buy diapers so they pick up a little treat for themselves Grocery stores use this information to coordinate their promotions – when diapers go on sale, don’t put beer on sale as well, since it is likely to get bought anyway.

So how does this relate to software development? Generally speaking, the Microsoft Association Rules builds rules that can be used to predict the presence of an item based on other items in a transaction. A transaction in this case could be the check-in that fixes a bug. What files are being included? If file foo.cs is being fixed as part of a bug, are there other files that are likely to be included? This could help with impact analysis. Imagine a check-in policy that looks at the files in your change set and tries to predict what else might be there, or should be there. That could help you answer questions about ‘what else get affected if I touch this file?’

Another interesting algorithm to apply is the Microsoft Clustering algorithm. This algorithm creates clusters of artifacts that share similar characteristics. The canonical example of this algorithm in use is in the retail industry. The question to answer might be, ‘in my population of customers, who is buying bicycles?’ Applying a clustering algorithm to this data might show that customers who buy bicycles tend to live in the northwest, are male and between the ages of 25–30; this is just an example of course.

Applied to software development, clustering might be used to pull out source files that happen to share similar characteristics. Maybe it is because they are the ones always being fixed? One could implement impact analysis by looking at whether a file that is about to be changed is part of one of these clusters.

Data mining is also commonly used to produce Profit Charts to help direct-marketers. The canonical scenario is a direct mail campaign. It costs something to send a mailer out to a customer, so businesses want to set a limit on how many they send to reduce their cost. On the other hand, the more mailers they send, the more customers they contact, and the more profit they stand to make.

The crux of the problem is how to find a happy balance. Data mining helps by creating profit charts. Profit charts work hand in hand with lift charts. Lift charts are used to illustrate the accuracy of a prediction. A prediction in a direct mail campaign might be whether the customer responds or not. The value of the prediction will be ‘yes they will respond’, or ‘no they will not respond’.

The accuracy of the prediction depends on the attributes of the customers in the population. Maybe there are some customers who have attributes that strongly favor a prediction, one way or another. Maybe there are other customers who have attributes that make an accurate prediction very hard to come by.

Quoting from the MSDN help, the input into a profit chart is:

Population

The number of cases in the dataset that is being used to create the lift chart. For example, the number of potential customers.

Fixed Cost: The fixed cost that is associated with the business problem. If this were for a targeted mailing solution, the cost would not depend on variables such as the number of telephone calls made or the number of promotional mailings sent.

Individual Cost: Costs that are in addition to the fixed cost, that can be associated with each customer contact. For example, promotional mailings or telephone calls.

Revenue Per Individual: The amount of revenue that is associated with each successful sale.”

A typical profit chart shows an increase in profits up to a point, after which profits decrease as more of the population is contacted.

For example, if the peak of the profit curve is at 55 percent of the population and the associated predict probability is 20 percent, this indicates that to achieve maximum profits you should only contact those customers whose response is predicted with a 20 percent or greater chance.

There may be an application for profit charts in software development. Bogdan and his team have done some experiments on predicting the outcome of a bug. For example, will the outcome be actionable (i.e. fixed) or will it be non-actionable (i.e. duplicate, not-repro). When a schedule gets tight, maybe it makes sense to use a profit chart to only work on bugs whose outcome can be predicted with a 20% or greater accuracy?

Anyways, just some thoughts on data mining and Team Foundation Server. I think we are just touching the surface on what can be done here. I would suggest poking around on the SQL Server Data Mining community site to learn more.

Over the next few weeks, I’ll be doing a few more posts on data mining and Team Foundation Server. Exploring ways of doing impact analysis might be a great question to start with.

David, in case you’re reading, I’ll see you in Australia in a few weeks, looking forward to it!

Thanks!

Eric.

More on the Trend Calculation

ericlee — Wed, 19 Jul 2006 18:26:00 GMT

One of the most useful aspects of the KPI is the ability to report a trend – that is, the value of a KPI compared to its value in the past. This is a really useful way to determine whether you are making progress or not.

I had a chance to meet with Bogdan Crivat – a friend of mine from when we worked together on Visual C++ – and a couple of his colleagues on the SQL Server Analysis Team. They really helped me understand a little bit more about how time dimensions worked.

Below is a video that touches on how time dimensions are applied to our Code Coverage KPI.

Reporting the Trend

The MDX query and results from the video are shown below:

MDX results:

Since we don’t have any previous weeks of data for our code coverage, our KPI reports an upwards trend even with a measly 1.36% of code coverage. As our project matures and more code coverage data is published from our tests, this trend will become more meaningful.

Another thing that came out of my conversation with Bogdan is some more insight into how to data mine VSTS data. Look for another post on this shortly.

Thanks!

Eric.

Just a quick post about being more specific about what data you collect ...

ericlee — Fri, 02 Jul 2004 22:26:00 GMT

The Performance Tools are an integrated part of Visual Studio Team System, but they are also exposed as command line tools.

One of these tools - vsinstr.exe is used to insert timing probes into your binary. These probes report information to us that we use to gather performance-related data about your functions.

By default, vsinstr.exe will instrument all of the methods in all of the classes for a given assembly. This may not always be what you want to do.

vinstr.exe has a /include flag that allows you to specify only the functions that you want to include in your data collection.

/include can be specified any number of times and uses a syntax of:

        /include:[namespace].<class name>::<method name>

You can of course specify nested namespaces - we currently have a bug where we don't handle nested classes correctly.

Once you have instrumented your binary with /include, you can collect data by launching your Performance Session using VS or manually executing your application.