drwill's daemon

Where Have I Been?

drwill — Wed, 11 Nov 2009 08:23:53 GMT

After trying to start up blogging, I failed to make it part of my regular routine. I think I found it hard to come up with topics. Working on a v1 product is fun, but it also means no existing customer base or products to blog about. I think I find it much easier to blog about product features and customer scenarios than abstract concepts.

Well now that we’ve shipped Beta 2 of VS2010 I’m overwhelmed with topics and have been able to keep up a daily post on our team blog, and I’ve been doing it for about three weeks! That is a record for me, and it is becoming more of a habit. I don’t plan on keeping a one-a-day pace forever, but I’m hoping this experience makes it easier for me to keep up a regular pace.

Also, I’ve been using a netbook at home now which I find incredibly convenient. I’m online much more often in the evenings now, while watching tv and such. With the help of Windows Live Writer, adding a new blog post is trivial, so all I really need is a topic.

Cheers

MSDN Chat - April 9th

drwill — Fri, 21 Mar 2008 23:39:50 GMT

Join members of the Visual Studio Team System product group to discuss features available in Team Foundation Server, Team Suite, Architecture Edition, Development Edition, Database Edition, and Test Edition. In addition, discuss what's new for these editions for Visual Studio 2008.

Add to Calendar

April 9, 2008
10:00 A.M. Pacific Time
Additional Time Zones

Talking To Customers

drwill — Fri, 21 Mar 2008 08:15:02 GMT

At Microsoft, we have developers and testers. Those two disciplines make up the traditional engineering roles that most companies have. Where we seem to differ the most is in the project management and feature design role. Some companies call them business analysts. We call them program managers.

From group to group at Microsoft, the role varies however. It's not surprising when we have hundreds of different groups and needs in each group. Some PMs are UI focused. Some are extremely technical and design APIs. The PMs on my team focus primarily on user scenarios, application interfaces, and feature prioritization. They seek user feedback in a variety of ways to validate the designs. Since we are a v1 product, we suffer the typical problem of having very little user feedback. You often hear joke that MS apps reach their full potential on version 3. A big part of that is learning how to get it right as far as users are concerned. Sometimes we need 2 iterations of trial and error accompanied by user feedback to find out. Our goal is to improve that v3 groundhog day.

Our PMs are doing some amazing work. They have so many different groups of customers (or at least target customers since we don't have a TCM application available to buy yet) from whom they gather feedback. They PMs spend a great deal of time talking to customers, doing presentations, sending surveys, going over data, coordinating usability studies, and meeting with user groups.

Hopefully, if we've done our job right our v1 product will be closer to perfect. If so, we'll learn considerable amount from this process. Some efforts will prove to be more helpful than others. Regardless, talking to customers is the obvious first start to making a v1 product useful to the people who might use it.

Web Test Variables

drwill — Thu, 06 Mar 2008 02:07:09 GMT

A customer emailed me today with a follow up question. The original question was about web tests recording AJAX-style client side scripting which initiates server queries. In VS2005 we didn't record those requests and our work around was to use Fiddler2 to record the scripts.

In VS2008 we fixed that which is great for customers because it's much easier to record the web test file directly rather than having to switch to another tool. No offense to the author of Fiddler2. It is an amazing web debugger that one should have installed anyway. I'm using it for debugging a for-fun app I'm writing on the weekends to download some special podcasts.

Anyway, the new question was when I parameterize the web server name, how do I change it to a temporary value for a one time execution? This feature has been in since VS2005 and I tried it to make sure before I responded... but it didn't work.

I looked up the topic on MSDN and it turns out you have to prefix the variable. I asked one of our web test gurus and he said it's always been that way, but I guess I forgot.

If your variable name is WebServer1, then at the command-line set the environment variable name to be Test.WebServer1.

e.g. C:> set Test.WebServer1=www.MyCustomServer.com

Here's the direct link to the help topic: http://msdn2.microsoft.com/en-us/library/ms184806.aspx.

Ellen

drwill — Thu, 06 Mar 2008 01:43:32 GMT

Have you heard about Mort, Elvis, and Einstein? No, no... I mean in context of Visual Studio. :) We use these 3 personas to help guide feature development.

I did a search for these names to see what has already been published. MS employees have already blogged about the personas, and some critics have come out against them.

A brief review of some of the criticism seems to be based on a misunderstanding of them and how they are used. I'm not writing this to defend them, but it does bother me that the basis for the dislike is founded on falsehoods.

My understanding of the personas is that they represent work styles. For instance, there are people who like to open the code editor and just start typing. They use Intellisense to discover new features and learn by trying. Others like to read up first and understand all sides of a technology before trying it out. Chances are all of you have tried both methods at one point or another, though over time you probably trend towards one or the other. Neither method is wrong or better. They just "are" and it's important for us who are implementing new features to recognize that both work styles are common. Customers expect both styles to be supported by Visual Studio.

If you want to know more about personas or the controversy around them, here are a few links to start from.

What I want to talk about is how we, in TeamTest, don't find Mort, Elvis, and Einstein to be proper work styles to describe how a manual tester approaches work. Our usability team interviewed a lot of target customers. They watched people working, asked them questions, and even invited us in the product group to take turns with them getting to watch and talk to these target customers. I got to do customer visits at Safeco in the U District, and another at Getty Images in Fremont.

The new persona is called Ellen. Whereas the other personas focus on programming approach, Ellen is not about programming at all. She does have an entirely different history and motivation as one would expect. She is so different from the existing 3 personas, you can see why it was important for us to create a new persona to guide our feature development.

One example of how she has impacted our direction is in our decision to build a test case management solution outside of the Visual Studio shell. We're simplifying the user interface (getting rid of many actions and tools that she won't need) and reducing the time to install. We want it to be easy to use, providing appropriate amounts of guidance along the way to explain artifacts and activities, and fully support her mainline scenarios with the least amount of manual work to accomplish them.

If you've seen an early preview of our work, you can plainly see we haven't yet achieved the spirit I write about. We are dreaming of it though, and we're laying the foundation to provide all of that with each step. You've seen CTP 8 and 10. Soon you'll see 12. I hope you've seen great steps forward in each CTP, and I can't wait for you to see the post CTP12 stuff we're working on now.

Customers

drwill — Mon, 03 Mar 2008 01:10:28 GMT

In the three years I've been working in the Developer Division at Microsoft, I've been impressed with the focus our divisional leaders have placed on customer focus. The benefit to customers is obvious. We're responsive to your concerns and it helps us build a product that better meets your needs.

It also benefits us a lot too! We have specific customer feedback to drive decisions in feature development. A lot of it is anecdotal, but still very, very valuable.

I wanted to share the ways my team and I regularly interact with customers. You might have participated in one of the many ways, but not known other ways to get to know us and have an impact on feature direction.

The MSDN Forums are a place for people to post questions and comments. There are many different forums for specific features in Visual Studio and other products. My product team has two forums. One is for Web and Load Testing, and the other is for all general Team Test issues. Every member of our team spends some time during the week reviewing questions and posting answers. We also encourage others in the community to contribute as well. We try to touch each post within 2 days, and strive for an answer in 7 days.

About once a month, we schedule live chats on MSDN. We'll recruit 2-3 members of each product unit (TFS, TeamDev, TeamTest, TeamData, TeamArch) to attend and respond live to questions. Watch this blog and others for dates, times, and links to the planned chats. The transcripts are posted later for those who were unable to make it, and I suspect for search engines. (Interestingly, the last transcript posted seems to be April of 2007. I'll have to follow up on that and find out if the latest ones are posted elsewhere or if they just haven't made it up yet.)

Blogging is primarily a one way communication mechanism, but it does increase the transparency of what is happening inside of Microsoft. I prefer transparency, even though I realize we can't always have it in full due to IP, etc. We use blogging to show off new features made available in Customer Technical Previews (CTPs), Betas, and final releases. See David Gorena's post about our test case management features available in a recent CTP, for instance. We like to get comments on the blogs about what you see, though we don't get nearly as much feedback as we'd like. If we blog about something you like, ask us to do more. If we blog about something you don't care about, let us know that too. If you have more questions, we'd love to hear them and we'll try to respond in another comment or in a new blog post.

We interact with the community in other ways as well, but these 3 are the things we do on the most regular basis and with the most number of people. If you haven't gotten to know us in any of the ways I've listed, give it a shot and let us know how we can do better.

Blogs of interest:

http://blogs.msdn.com/vstsqualitytools

http://blogs.msdn.com/slumley

http://blogs.msdn.com/dhopton/

http://blogs.msdn.com/edglas/

Our Scrum Cycle

drwill — Mon, 25 Feb 2008 06:26:40 GMT

I've mentioned in a previous post about how our team has adopted Scrum and agile methodology... at least in spirit. :)

I thought I'd share some of the specifics about how we're running it. This is primarily based on direction from our Engineering Manager, Mark Mydland. He's brought Scrum and agile to other teams, both in and out of Microsoft.

At first, our sprint duration was 4 weeks. This is a very drastic change to typical Microsoft engineering. We usually break up work into 3 month milestones. Our iteration duration has but cut into a third!

Recently we increased it to 5 weeks based on feedback from others in our product unit. I'm not as pleased with the 5 week duration because it increased our development time by 50%. I'll explain the details later.

We start with a week of design. We meet in the afternoon for 4-5 hours. Each developer gets to pick up a design area. In the morning or in days before their target design day, they are responsible for preparing their design discussion and collaborating with a QA owner. Long gone are the 15 page development and testing specs. We use a unified design doc for both dev and test. The questions asked in the design doc include:

What is my general solution to the problem
What other alternatives are there?
What is the general pattern?
List of dev tasks
What is it I'm trying to test?
What is my general approach for testing?
What will customers do with this feature?
List of test tasks

By the end of design week, engineers enter their tasks into TFS and link them to a Deliverable (a custom work item type used to group up tasks related to a specific feature). I review the TFS tasks and sign off on each deliverable. Each deliverable has a rank value that indicates the order in which we'll address each feature.

Weeks 2, 3, and 4 are primary implementation weeks. I'll go into how I manage the work during those weeks in more detail in a future blog.

The final week is about stabilization. Some feature work may continue but the focus is on finishing up remaining work, evaluating overall application testing, and getting the application polished. In the developer division we also have things called Quality Gates which include things like making sure you are FxCop clean and that your binaries have 70% code coverage from automation. We'll make sure all those Quality Gates are met in the last week as well.

Clearly our Scrum cycle is unique. I'm curious how others do Scrum. I've read a few books, but I'd like to know how others do it in practice. What do you like? What would you like to improve? Any suggestions for me?

Engineering Lead

drwill — Wed, 20 Feb 2008 00:04:04 GMT

In the last six months, I've transitioned from a QA Lead to an Engineering Lead position. I'm still a persistent and tenacious driver of quality. Now I get to influence the development process as well. It is a great opportunity for me to optimize development for the benefit of quality, as well as a great career challenge.

I began my career as a product support engineer. I was a temporary Microsoft employee for two years. Despite all the negativity you might associate with that, I found it to be an excellent opportunity for me to learn and prove myself. I started in phone support for Windows 95. It was 2 weeks before Win95 was released and it was a very exciting time. Product Support Services (PSS) was a fantastic environment for me to learn. There was a lot of knowledge about Windows, networking, memory management, etc. I just soaked it in. So many people were happy to help others learn. It was easy to find experts. It was the ideal environment for me to learn since I could ask dynamic questions of an expert.

After 10 months of that, I knew I needed a change. Although I enjoyed helping customers, the number of unique issues I encountered on a regular basis wasn't growing. I was tired of hearing about how these issues negatively impacted customers and I wanted to go to the product team to find these issues before they were released.

So I became a manual tester. My job then was as close as I've ever been to the persona we use to to design our Test Case Management product. I tested third party applications to make sure they continued to work on new versions of Windows and Internet Explorer. Next I worked on NetMeeting, testing the multi-conference scale and configurations.

During those two years, I was fascinated with what one could do with programming to automate things. Whether it was a simple Excel macro to make a compelling report, a log parser to normalize disparate log formats, or automation to more quickly test an application... I was hooked. I think at our core, all programmers are a bit lazy and we're attracted to the idea we can invest a bit in an application or script to do work for us. :)

I finally made my entrance into the world of programming as a Software Design Engineer in Test (SDET) and became a full time employee. Back when I started it wasn't the most common type of tester we have at MS. SDETs are developers who use their programming knowledge to test the product. It could be API testing, performance, or scalability. We've even tried to automate UI tests with many essential attempts and failures over the last decade. We learned a lot of lessons and each iteration gets a bit better than before.

I've worked on many different MS apps as an SDET including Microsoft Works, MSN Small Business, ActiveSync (desktop sync for Windows Mobile), and Exchange ActiveSync (wireless sync for Windows Mobile).

Eventually I became an SDET Lead. I didn't think I'd like being a people manager, but I've learned differently since transitioning. More on that in a future blog. As an SDET Lead I championed quality, trained new testers, triaged bugs, and took part in product design. My goal was to drive for quality at every stage.

It was difficult to convince my development leads to change their approach. Now that I'm an Engineering Lead, I have the authority to make direct changes up stream. I'm not out to ruin development at the expense of quality. Now I feel I can make both better. Scrum and agile have been a great guide for bringing that to my team.

Trying Scrum

drwill — Tue, 19 Feb 2008 19:58:16 GMT

Over the last several months, our team has decided to try all this "agile" stuff we keep hearing about. In large part, our new manager is responsible for putting us on this course but many of us were curious with what it would offer and fed up with some of the problems we'd been perpetually dealing with.

My biggest pet peeve was the huge bug backlog we'd always acquire. I always dreaded getting to code complete and then spending months just bug fixing. The developers hated it as much as they loved spending all their time coding before code complete. Overall I think devs were less happy with the pendulum swing facet of their work. I thought they'd be much happier if they had balance in their life where it was always a mix of bug fixing and feature work. Plus, they wouldn't have the feeling of dread associated with a massive bug backlog.

Going forward our goal was to keep bug counts to a minimum.

For QA, the feeling was almost opposite of dev. While the feature work was interesting, they had a nagging feeling about the Pri2 and up bugs not being addressed. They felt guilty and depressed that we might ship with those bugs. "Why can't we just fix them now?" After code complete, they were finally content to start addressing the bug backlog. Of course since the end of the release is imminent there's always bugs that won't get fixed due to what seemed to be some arbitrary date. "Why can't we just be quality driven for releases?" In the end, QA was almost always discontent with the release. It takes an SP1 to relieve the shame. And why do dev and QA have to be at odds? Don't we both want the same thing ultimately?

Going forward our goal was to keep dev and QA in sync, working towards the same prize, and both driven for the same outcome.

We've been doing this for six sprints now and I've learned a lot about agile (small A as my manager reminds me since we're following the spirit of Agile but not adopting every nuance of the methodology). We've made tweaks each sprint, some minor, some major, to make it work for us. Mostly, the engineers have embraced it but change isn't always easy and it hasn't been unusual to encounter resistance at the start.

TechEd 2007 sessions posted

drwill — Thu, 26 Jul 2007 23:50:00 GMT

You can find the 2007 TechEd sessions posted at http://msdn2.microsoft.com/en-us/teamsystem/bb676080.aspx, including my talk on Web and Load testing in VS2008 (codenamed Orcas).

Visual Studio Team System Chat – August 1st

drwill — Tue, 24 Jul 2007 20:43:00 GMT

Join members of the Visual Studio Team System product group to discuss features available in Visual Studio Team Foundation Server, Team Editions for Architects, Developers, Database Pros, and Testers. In addition, discuss what's new in the latest Visual Studio 2008 CTP.

We will be holding two sessions:

Join the chat on Wednesday, August 1^st, 2007 from 10:00am - 11:00am Pacific Time. Add to Calendar | Additional Time Zones

-or-

Join the chat on Wednesday, August 1^st, 2007 from 4:00pm - 5:00pm Pacific Time. Add to Calendar | Additional Time Zones

Visual Studio Team System Chat – July 3rd

drwill — Thu, 21 Jun 2007 00:29:00 GMT

We will be holding two sessions:

Join the chat on Tuesday, July 3^rd , 2007 from 10:00am - 11:00am Pacific Time. Add to Calendar | Additional Time Zones

-or-

Join the chat on Tuesday, July 3^rd, 2007 from 4:00pm - 5:00pm Pacific Time. Add to Calendar | Additional Time Zones

Team System chat

drwill — Wed, 06 Sep 2006 01:08:00 GMT

Want to chat with members of the development team for Team System? We're holding a chat Wednesday the 6th of September at 10am Pacific. We love to hear from you and answer your questions. Below is the official chat invitation.

Cheers,

drwill

Visual Studio Team System Public MSDN Chat

Come and join members from the Visual Studio Team System product group to discuss features available in Visual Studio Architect, Developer and Tester editions and Team Foundation Server. There will be experts on hand to answer your questions, so we hope to see you there!

Join the chat on Wednesday September 6th, 2006 10:00am - 11:00am Pacific time.

To add this to your calendar, click here.

To see your local time of when this chat is, click here.

The Perfect Bug

drwill — Sat, 16 Jul 2005 03:43:00 GMT

http://blogs.msdn.com/vstsqualitytools/archive/2005/07/15/439454.aspx

Running Tests with Code Coverage

drwill — Thu, 09 Jun 2005 02:27:00 GMT

I posted in our team blog, with help from Dominic Hopton, about running tests with code coverage. You can view it here.