Jeff Beehler's Blog : Dogfooding

How DevDiv uses TFS

jeffbe — Sun, 20 Apr 2008 17:57:27 GMT

Gregg Boer, a program manager on the TFS work item tracking team has started a series of posts outlining in good detail how we used TFS work item tracking within Developer Division to manage the Visual Studio 2008 project. Based on the popularity of my Channel 9 video covering similar topics, I suspect this will be interesting reading for many of you. Check out the first 4 articles in his occasional series:

Hopefully this explanation of how we have implemented TFS internally will help you with your own implementations. Please let us know what you think.

Thanks!

Update on DevDiv dogfooding of TFS

jeffbe — Fri, 28 Mar 2008 22:31:08 GMT

Matt Gertz, Developer Division's manager responsible for our internal development tactics and process, recently posted an article covering the broad post-VS2008 efforts to more broadly adopt TFS within DevDiv for future product development. As regular readers of my blog know, Team System teams have been dogfooding TFS for a long time now so this change is about getting the rest of the division switched over as well. We've certainly had our share of growing pains (ever try unshelving a shelveset with 1 million files?) but we're pretty much through the transition (and fixed a number of scale related issues along the way). I posted a recent report on internal usage stats for TFS and in it you can see the increase of DevDiv users from 1,300 to 2,200 since December '07. Now you know what prompted that considerable increase.

If you're interested in learning more about DevDiv's dogfooding of TFS, be sure to read Matt's article: http://blogs.msdn.com/vbteam/archive/2008/03/28/milestone-quality-dogfooding-matt-gertz.aspx.

Internal usage stats for Team Foundation Server

jeffbe — Mon, 24 Mar 2008 16:54:41 GMT

Joe Schwetz, the group manager of our TFS Solution Provider team, publishes a monthly report which updates us on the internal adoption of Team Foundation Server across the company. You've heard from me before about our internal dogfooding of TFS within the Team System team (including my most recent update for Channel 9) but I haven't spent much time explaining the level of dogfooding within the rest of the company, until now. What you read below is an excerpt from Joe's monthly report. Hopefully this conveys both the level of usage as well as our excitement about continued adoption of TFS within Microsoft. As you can imagine, it's not as simple as throwing a switch to get teams to move over. We have to find the right time in the product cycle to adopt TFS, evaluate their hardware needs, help customize their system, migrate their existing assets over to TFS and ensure their teams receive the proper level of training. As you can see below, since the release of TFS 2008, we've seen a marked increase in the number of internal users not only from DevDiv, but Microsoft's IT teams (MSIT) as well as many other places within the company. It's clear from these numbers that TFS adoption is really picking up steam across the company.

Anyway, take a look at these statistics and let us know if you have any questions or comments.

From: Joe Schwetz
Sent: Friday, March 21, 2008 2:41 PM
Subject: March 2008 - VSTS Internal Service Offering Summary

Internal adoption continues to impress, growing by 1,085 active users taking the total to 11,834 companywide! Overall growth of the service offering continues to be driven by widespread adoption in Developer Division and MSIT with steady growth by several large Divisions. The usage outside of the large product groups and MSIT is also increasing at a strong pace, see the “All Other” users tracked on the “Active Users – Service Offering” chart.

We set a new high in the number of Work Items entered Service Offering wide in a single month, with nearly a quarter of a million being created – 246,051 in March! There were only 2 other months in the history of TFS with > 100,000 Work Items created: December 2007 with 154,926 and February 2008 with 112,441.

There were 104 new Projects created in March, this is the 11^th month in a row with >100 projects being created per month.

Here’s a quick summary of the growth at a high level:

As always, please feel free to share these numbers and statistics with customers.

Active Users Across Service Offering:

Developer Division, MSIT, SQL, MSN/Windows Live, and MSCOM all hit new peaks of active users in March. The growth of all other instances combined is continuing to mirror the growth rates of DevDiv and MSIT.

Active Users & Projects Across Service Offering:

At the beginning of the Fiscal Year, we set the goal of having 12,000 Active Users and 1,800 Projects in TFS. As you can see, we’ll be blowing those numbers out of the water by the end of the fiscal year.

Work Items – Stored in TFS 2008 & Rosario IR:

Here are details on the number of Work Items stored based on releases this month (TFS 2008 RTM and on the early Rosario bits). March was a new record for Work Items being entered, with 246,051 being created across the Service Offering!

The following shows the highest number of Work Items per individual TFS instance for TFS 2008 and Rosario IR:

Team Build – Service Offering:

There was a major increase this month in the use of Team Build. As you can see the numbers literally have shot through the roof with MSIT and one of the Consolidated instances greatly increasing the use of Team Build. For details on number of builds on specific instances see the table below.

Team System Web Access Power Tool released!

jeffbe — Tue, 31 Jul 2007 02:27:28 GMT

Brian's recently announced the availability of the the Team System Web Access Power Tool. We've been using it internally for the past 3-4 months and as with most dogfooding efforts, we've found and fixed a bunch of issues. I think you'll find this release to be easier to use and have better performance than previous releases of what was formerly known of as Teamplain 1.0 from DevBiz. While it's still not an official part of the Team Foundation product, it's certainly close enough for most people to adopt and use it on a regular basis. Be sure to check it out!

TFS adoption at Microsoft

jeffbe — Tue, 24 Jul 2007 18:01:38 GMT

Frequent readers of my blog know full well that we take TFS dogfooding very seriously here. In fact, just this month we're celebrating 3 years of using TFS within the Team System team...pretty amazing. As you might expect, usage is taking off in other groups as well. To learn more, I'd recommend reading Brian's recent article on internal adoption of TFS. In it he writes:

Our internal adoption team produced an update on our status this week and I thought I'd share the results with you. We're now up to 21 TFS instances in production, hosting a total of 734 projects about 5,600 users. 2 of those instances are running an Orcas Beta 2 build. 3 are running a very early build of Rosario. and the other 16 are running TFS 2005. We plan on upgrading the remaining TFS 2005 instances to Orcas (TFS 2008) in August.

Read on to learn more and see some cool graphs showing # of users over time.

Customizing TFS

jeffbe — Mon, 04 Jun 2007 06:09:30 GMT

As I previously mentioned, I've been collecting data from current TFS users to understand what sorts of customizations to TFS they are currently making in preparation for my upcoming TechEd talk. I've received quite a few responses (thanks to everyone who responded!) and while no two are identical, patterns have emerged which are pretty interesting (at least to me). Hopefully you agree.

Here are some of the reasons offered up for why people customize TFS...they appear to fall into two broad categories:

Integrate with existing tools, systems and process
- compatibility with existing process
- comply with external regulations
- enact company standards
Innovate incremental process improvement
- track new kinds of work
- establish deeper integration and traceability between artifacts
- automate process

Let's look at each in turn:

Integrate with existing tools, systems and process

Very few of us live in a world where we can start completely from scratch and store everything we need in a single place such as TFS. Instead, we're part of larger organizations which influence and, in some cases, dictate how we need to operate. I'm sure it'll come to no surprise to anyone that we have many existing tools and systems here at Microsoft. Before adopting TFS to track our work, we used something known internally as Product Studio. Some of the teams within DevDiv still use that system so for now we mirror bugs between both systems using a predecessor to the Migration and Synchronization toolkit. To keep those systems in sync with one another, we had to customize our definition of "bug" to track the corresponding ID in Product Studio. This is a simple example of a very common sort of customization we're seeing.

Many teams are actually a small part of larger organizations which have existing process already in place. It could be as simple as having a tradition of tracking a bug's severity instead of it's priority. So, in order to use the MSF Agile Process Template, the customer would first want to make that change to the "bug" work item within the process template. Internally, because our process has evolved over many years and is influenced by the many teams involved in our divisional releases, our process around bugs is quite complicated. It seems that each team has their own specific needs which we have worked hard to support within our overall definition of bug. Here's a glimpse at our current bug form:

Some customers have to comply various external regulations such as SOX or something known as CFR 21 Part 11 (related to medical research). In these cases, the the customer's ability to influence the standard is often much lower and the criticality of compliance is much more important. Often, this implies additional levels of work flow often implemented as additional work item states. In some cases, only a handful of approved team members are allowed to change the work item state or assign work items to other team members. All of this can be achieved through the customization of work item types.

Another common customization pattern we've seen relates to the initial state of newly created Team Projects. Some companies have established a standard version control tree that they want to be used for each project. Others have established a common set of area and iteration paths for projects depending on the overall type of project. These changes are encoded into the Team Project templates which helps ensure consistency across all projects without requiring significant work after creating each project.

Innovate incremental process improvement

Many customers come to us looking for advice on how to improve their development process. As you might expect, we encourage them to start with one of the variants of MSF which are available as part of TFS. For some, that's sufficient. Others are interested in building their own process from scratch while others want to take advantage of existing process implementations such as Conchango's SCRUM, or Ivan Jacobson's Essential Unified Process. You can see a more complete list of TFS Process Template providers here.

One of the innovations that we've observed is to use TFS to track more work than the standard set of work items in MSF allow. For instance, within my team, we use TFS to help us track exit criteria for each of our major releases such as betas during a product cycle. This allows us to effectively track many aspects of our release readiness by aggregating data from the many involved teams regarding many different aspects of our quality gates. I wrote about this in detail a few months ago if you'd like to learn more.

We also track our feature development through a series of new work item types and reports. This has been done in conjunction with a sweeping change in the way we plan, develop, test and track features. As we've implemented these process improvements, we've iterated on the associated work item types, reports and guidance necessary to successfully make the cultural shift.

I collected the following list of custom work item types currently running on our dogfood server to give an idea of the breadth of additional aspects of the software development lifecycle which we track:

Exit criteria OGF Test case Test issue

Experience Feature Product idea Value proposition

Backlog Help topic KB article Release note

Patent Build request Change request TAP site

Another reason customers customize TFS is to establish even deeper integration and traceability between artifacts. I've seen examples where customers annotate their code and tests with Requirement work item IDs and then are able to identify which requirements have sufficient amount of testing based on code coverage results. Needless to say, as we look forward, we want to enable this type of traceability out of the box since the progress and readiness reporting that this enables would be super valuable. We've also seen customers implement various types of work item hierarchies which helps them understand and manage the relationship between work items. This is another area that we want to enable without customization in future releases.

I'd like to hear from you on this topic. Are you seeing these patterns as well? Are there other forces at play that should be called out?

Also, now that I've discussed the 'why customize' question, it's clear that I need to discuss the question of 'how to customize' as well. Look for information about that in the near future.

Over 500 Million served

jeffbe — Sat, 21 Apr 2007 17:57:56 GMT

I'm old enough to remember when McDonald's would display the number (in millions and billions) of hamburgers sold on their signs. Since they now serve 54 million customers a day, the signs now just say "billions and billions". As a kid it was fun to drive by and notice when the number had changed.

I'm having the same experience with TFS dogfooding stats (and so it seems is Buck). It seems like every time I read one of Brian's dogfood stat updates, we've passed another milestone. This time around, the notable ones for me were:

200,000 work items
75,000,000 files under source control
500,000,000 local file copies

yes, that's 1/2 of a billion rows...that's a lot of data.

Next thing I know, Brian will be saying "billions and billions" just like McDonalds. I just hope he doesn't start singing "i'm lovin' it". :-)

The many faces of quality

jeffbe — Sat, 21 Apr 2007 16:59:21 GMT

Brian's been compiling a great series of posts regarding managing quality which I would highly recommend reading. His latest post about community feedback provides a view in on how we think about this important aspect of our development process. People are regularly asking me how we do things on the Team System team and these are great real world examples of the quality assurance systems we have in place. Being someone that pours over reports and is regularly thinking about how to communicate information (and not just data) through them, I'm fascinated about what we're doing in this area. It's so important that people remember that 'quality' isn't just about tracking bugs.

Do yourself a favor and read this informative series of posts:

Perhaps someday we'll convince Brian to write a book on the topic...but don't hold your breath...I'm not sure he could sit still long enough. :-)

Managing Orcas Beta exit criteria by dogfooding TFS

jeffbe — Mon, 05 Mar 2007 21:57:49 GMT

I've written a few times about our usage of TFS while we develop newer versions of Visual Studio and TFS. It's one of my favorite topics so I'm excited about a recent update that we've made to our system to help us manage the end game of our beta release of Orcas. Part of getting ready to do a major release of our flagship product is ensuring that many different criteria are met by the various teams that contribute to the overall release. Since much of our execution and accountability is with the product units within DevDiv (as I believe it should be) we manage many of these "exit criteria" at the product unit level and then roll up status into a divisional view. This is something that TFS is well suited for.

To start with, we created a new work item type to track each exit criteria for each product unit. We currently have 28 exit criteria categories for beta 1 (things like performance results, test pass rates, and stress pass rates). To learn more about some of these read Brian's recent series on Managing Quality (Build, Automation, Performance, Stress, Dr. Watson). Each of these criteria are measured by each of the 24 product teams delivering some components to the release. These work items are assigned to someone on the product team who is responsible for managing and reporting back on these criteria.

The interesing fields of this work item type are as follows:

Title - common name of the exit criteria (one of 28)
Product unit - one of 24 teams that participate in shipping Orcas
Assigned to - who is the program manager or tester responsible for managing the specific exit criteria?
Due date - when will this be signed off on?
Modified criteria - has the product team modified the divisional criteria for their team?
Criteria modifications - if so, what changes were made to the critera?
Sign off status - a drop down with the following choices
- Meets Expectations: a team has determined that they have met the requirements (or the modified criteria) for this exit criteria.
- On track (Doesn’t meet expectations): although a team has not yet met the requirements, they believe that they are on track for meeting the requirements by the due date specified.
- At risk (Doesn’t meet expectations): a team believes that they are not on track for meeting the requirements by the due date specified
- Blocked: a team has not met the requirements for the exit criteria and are unable to make progress due to factors outside of their control
- No data: this is the default state of the work item and indicates that no update has been made
- N/A: this exit criteria does not apply to this portion of the product

From there, we wrote a report which translates the states for each team and criteria to show us an all up view. Since I represent only the Team System teams, I cut out the results for other teams:

Each of these squares in the grid can be clicked on to drilldownto the work item details. Since this is build on TFS, we have information in the work item about history and the links necessary to track and communicate our plans across our various teams. I utilize this regularly so that I can understand the hotspots as we close down our beta release. We will use this information during our product unit reviews as we plan for future milestones and the eventual RTM release of Orcas. Knowing early on where our hotspots are will help us ensure that we have a plan to get green across the board by the time we ship.

Reporting from Visio?!

jeffbe — Sat, 03 Mar 2007 17:58:20 GMT

When I first read Ameya's new post regarding using Visio as a reporting tool for TFS, I thought he must have lost his mind. However, after watching his two videos and how easy it is to provide an alternative view on data flowing from the Team Foundation data warehouse, I can see the power of this integration. I think I'll still be using the Excel connection to the TFS warehouse on a daily basis but I'm going to start looking for chances to use Visio as well.

Cool stuff!

TFS Server Manager

jeffbe — Thu, 25 Jan 2007 17:30:00 GMT

Brian has recently posted about a tool that he introduced to me way before we released TFS. I found it immensely valuable to really get a handle on what was going on with our dogfood server and how much usage it had seen recently. He's been plugging away at this for quite some time and it's really turned into quite a tool. I'm excited to hear that he's considering publishing it as a Power Tool. Judging by the response to his post on the topic, I'd say he's pretty likely to push hard in the next few months to get this out to folks. Take a look and let him know what you think.

Tell us what you think about reporting

jeffbe — Thu, 18 Jan 2007 17:52:41 GMT

The teams responsible for Team System reporting are interested in your feedback as they map out a course for the future. Reporting is one of the most powerful components within Team System. In fact, I use the reporting capabilities of the system on a daily basis to help me get a handle on our various projects. We have created numerous custom reports (which I hope to show you over time) and are always looking for more ways to leverage the information available in the system. However, I'm afraid most customers aren't realizing the full capabilities of the system. So, I'm interested in learning how important reporting is to you and your teams, what data you look at and what information you'd like to see but can't seem to get at. Part of this will inform our investment in reports available out of the box as well as understanding what areas need to be easiest to customize. I know what I, as a customer want, but I'm even more interested in hearing what you want.

Please take 10-15 minutes to tell us: http://www.zoomerang.com/survey.zgi?p=WEB22632D8HGS3

Thanks!

More details on managing work for Team System

jeffbe — Mon, 15 Jan 2007 16:58:24 GMT

I've been meaning to write a more detailed post describing the mechanics for managing work for Team System. Turns out Siddharth beat me to it with his article on Tracking Customer Value. Take a look, it's very informative.

Managing work for Team System

jeffbe — Tue, 17 Oct 2006 17:13:26 GMT

Martin Woodward, one of our newest Team System MVPs, was recently asking me about how we manage work within our team and if I'd be willing to share our TFS work item types to give a real world example. Our dogfood usage of TFS is one of my favorite topics and I'm interested in sharing with you our experiences both good and bad in this space. I don't for a moment argue that we have the ideal implementation of TFS. We're frequently trying new things, some work well and others don't. Plus we're part of a larger team (Developer Division) and as a result, there are constraints and requirements placed on us that don't map super well to how we'd like to run our Team System team. Probably sounds familiar to many of you.

Looking at our dogfood server, we currently have the following work item types in use for the Orcas project:

Value Prop: I've written previously about our use of Value Props in our planning process. They represent the high level, customer focused, value that we hope to deliver in an upcoming release. In the context of managing work in TFS, Value Props are used as top level containers for Experiences (see below) which are in turn containers for Features. We have created reports that show this work breakdown and the progress we've made against each of these Value Props. Because the linking functionality in TFS v1 is somewhat limited, we've had to institute various policies and create special reports to ensure that the parentage from Value Prop to Feature is correct. This is clearly something we're pushing to address with future versions of TFS (I love dogfooding!)
Experience: We break Value Props into a series of user Experiences as a way to help us organize and plan our features. We found that this logical step was necessary because the jump from high level Value Props to low level features was just too difficult conceptually. Now that we're executing, Experiences are not used much except as a connection between Value Props and Features.
Feature: This is where the rubber hits the road and so the Feature work item type is very rich and I should probably spend an entire post or two describing how we use it. It's the thing, more than any other, that we use to track our progress towards shipping. Orcas has something like 350 features planned to be delivered. Features go through a number of states as they get planned, approved, worked on and completed. As this happens, they get slotted into various iterations for implementation and testing. We track start and end dates as well as estimated work complete and remaining through this work item type. It's also the place that we report on the state of our various quality gates (like level of code coverage from automated tests) that are required to be satisified before we check the feature into the main source branch. Like I said, I need to spend a lot more time on this area because there's a lot going on here.
Task: We don't require all teams to track their tasks in TFS but a lot of Team System teams do. Many enter and maintain Tasks through the Excel and Project integration and update them on a regular basis. Tasks can be associated to a Feature through a simple ID field so that it's easy to query on all tasks associated with a particular feature (you can't use links in this way, unfortunately...something we're working on for upcoming versions). We use the MSF CMMI version of Task with a few more date and categorization fields. They are either active or closed. Teams use these to estimate time completed and remaining and roll that up through Excel or Project to update the Feature work estimates on a weekly basis.
Issue: There are always a number of open issues that we need to drive to closure as a team so we use an Issue work item type within TFS to manage these. Here we track owner, description, attention level (is it a divisional issue or something that only one team cares about), group affected and owner(s). We used to manage these through a simple spreadsheet...I love having them in TFS so I can easily assign issues to folks on the team, maintain history of the issue and easily query for the latest information.
Bug: Here's another work item type that I could write a book about, and might if people want. With over 70 fields on the bug form, we collect a lot of information about bugs to track where it came from, how we found it, what the customer impact is, who owns it, what the resolution was and a root cause analysis. Fortunately, of those 70 fields, only about 10 of them are required for a new bug to be entered (more are required through the lifecycle of a bug) so it's not too overwhelming to report a newly found problem.
Icon: This work item type was curious to me since I haven't been involved in its usage. Turns out we have 147 instances of this work item type and it appears we use it to track the lifecycle of graphical elements associated with Visual Studio. Pretty cool actually...I would have assumed that a standard Task would work okay here but upon closer inspection of this work item type, it's clear that there are special needs for this work. For instance, they track the format info (type, format, color depth, transparency, size, etc) as well as well as filenames and various approvals required for inclusion.
UE Release Work Request: Here's another work item type that I wasn't familar with. The User Education (documentation) release team uses TFS to manage the incoming requests for assistance from their various internal customers. This customized work item type focused on the specifics of integrating, building, releasing, and troubleshooting the documentation.

I'm excited that we're always finding new ways to utilize Team System to manage Team System. It drives goodness into the product as we solve the problems we find as we push the system in new directions. There are a number of additional ways that I'd like to push the system including tracking our minor releases (such as TFS PowerToys) and feature dependencies. As we do this, I'll be sure to share with you.

I'd be happy to further explore and describe any of these work item types as people find valuable. Let me know what you want to learn more about and I'll see what I can do to share what we've got.

Thanks!

tasting fresh dogfood

jeffbe — Sun, 26 Jun 2005 10:03:00 GMT

As I mentioned before, we’re in the process of updating our internal version of VSTS with updated bits. We made good progress last week. For instance, Monday night we resolved all known dogfood blocking issues. I emphasize the word “known” because this really means that we’ve cleared out the backlog of issues but doesn’t mean we’re ready to go. Instead it signals that we’re ready for important next steps including branching, a mini-test pass and dry runs all of which are currently underway.

Branching: we've created a separate source branch for the dogfood refresh sources. This allows us to minimize the source churn which in turn helps limit inadvertent regressions. In this branch, we only permit fixes that address dogfood blockers. After we go live, we have a place to fix showstoppers without worrying about other destabilizing changes.

Mini-test pass: we’ve started a two week mini-test pass. While it’s not specifically focused on finding dogfood blocking issues, they’re certainly found from time to time. Anyone that finds a dogfood blocker is instructed to enter the bug, mark it against the “dogfood refresh” iteration and to send mail to our triage team explaining why they think it should be fixed in the dogfood branch.

Dry runs: this is an activity solely focused on determining readiness for going live with this updated version of VSTS. In addition to using the latest good build, we migrate all of our existing data into the system. This helps expose issues in the migration scripts as well as issues only apparent when we have large amounts of data. And we’re talking a lot of data…36,000+ work items, 298,000+ files, representing 700+ workspaces and 1,400 shelvesets. We usually iterate through the bug bashing / dry run process. Typically it takes 3 dry runs to get things ready to go live and we’re wrapping up round #1.

I spent an hour or so on Friday afternoon using the results of the first dry run. Compared to other first dry runs from the past (this is our 4th major update), this dogfood tasted pretty good. It’s certainly not ready for prime time yet but I was able to do many of my most common operations with the system without too many problems. Of course before we roll this out to hundreds of folks, we need to clear up these and other blocking issues so that we don’t impede productivity more than absolutely necessary.

So, we've got so more work to do but we're certainly getting closer. After we go live with this refresh, we'll share the final dogfood build with you as a CTP so that you can let us know how you think it tastes.

jeff

Exit criteria	OGF	Test case	Test issue
Experience	Feature	Product idea	Value proposition
Backlog	Help topic	KB article	Release note
Patent	Build request	Change request	TAP site