scooblog by josh ledgard : Testing

Ship It

jledgard — Thu, 22 Feb 2007 23:15:42 GMT

Scoble is right. All the research, value proposition positioning, and market playbooks can't save you from looking like a copycat. Waiting that extra year or even months to perfect something for the web certainly doesn't win you any customer mindshare.

The read write web means that people become more invested in a web site. If I'm just reading sports news it's easy for me to switch from Espn.com to foxsports. But If I'm posting pictures of my baseball games on flicker and I've built up a post history in a sites community I've just become invested in your site in a way that makes it harder for me to leave. So you're better off being the one with less features and more personal investments than the also-ran trying to win a feature race to make people switch.

Not shipping early also means you subject yourself to the classic "moving target" software problem whereby your clients needs are changing on a regular basis. Spending too much time on the "boil the ocean approach" means your exposing yourself to this risk of putting momentum behind ideas that are constantly shipping.

It's hard to make people realize that Apple's secrecy is NOT the key to their success. I'd wager that their leadership position in the digital audio market has a LOT more to do with shipping a limited, but cool, design... and constantly innovating on top of that design. Even though there are now going to be iPhone competitors it will be Apple's execution that determines if they are the winner. So success is more a factor of timing and consistent execution than world changing platforms in today's web world.

What do you think about CTPs?

jledgard — Thu, 09 Nov 2006 22:28:00 GMT

Brian Harry wants to know. We've always had a concern about the quality of the CTPs we release, but are torn between increasing the quality, but reducing the frequency.

What we found with the monthly frequency of CTPs as opposed to the classic "wait for beta quality" approach is that we received more valuable real time feedback on features in development that allowed us to make changes.

How would you like to see us walk this line?

Will Live QnA suffer the Same Fate as Yahoo Answers?

jledgard — Wed, 09 Aug 2006 20:51:41 GMT

You tell me. I have 10 invites to hand out to the first folks that send me thier e-mail address. I'll post your comments about the service to my blog if you send them back to me as well.

Feedback Request on Visual Studio 2005 Look & Feel

jledgard — Wed, 01 Mar 2006 02:44:00 GMT

Aaron is responsible for a lot about how Visual Studio looks, feels, and behaves as an application. He's looking for some feedback about Visual Studio 2005 and has some specific questions. Feel free to leave your comments/suggestions here:

http://blogs.msdn.com/aaronbrethorst/archive/2006/02/15/532814.aspx

I know my opinions count less since I work here, but my biggest complaints are mirrored by the community.

New tabs should not open on the left. It just puhes the files I care about in large projects off the screen.
I still want to be able to tear off both windows and document objects onto my second or third monitor. Influencers use multi-monitors and I'm sticking to it. :-)
VS 2005 looks WAY better than VS 2003. Sometimes lipstick on the pig can go a long way.
I love the new tool window docking UI... a very slick solution to a common/hard problem.
I wonder when we'll have "ribbons" too. You think word has too many hidden commands... try digging through our menus in team suite.
Tools->Options is a mess. "Show all Options" is a start, but really just hides the problem.

Of course I should share some of the responcibility here since I was the test lead for a while on the IDE team.

Feel free to take Aarons survey and let him know what you think. Your opinions really do make a difference.

Communities Test Team Hiring

jledgard — Fri, 09 Dec 2005 20:20:00 GMT

Microsoft people forward me lots of Job descriptions. Some of them are more interesting than others. I'm sure there are still a few people reading my blog interested in software testing. If I was looking for an SDET job the following description is from one team I would seriously consider.

Why would I get out of bed in the morning for this job?
Yeah, yeah, you're probably already all "eww, test, that's what the *dumb* developers do!" You're wrong. The way Microsoft does test has virtually nothing to do with the way it's done at other companies. "Testers" here are not UI monkeys. They aren't powerless. They don't do what everyone tells them. They have equivalent or better technical skills than the developers. The projects have equal funding for the development and test disciplines.

It's a serious career path, not a "starter role", and in my opinion it's harder to do properly than development. You beat up every other discipline from day one to make sure no one slips up, and that what pops out at the end is what our customers and the business people actually want.

What do you people do all day?
We deliver three-tier dotnet/sql server applications, reporting/metrics infrastructure, and some other things I can't talk about just yet because they're too cool.

You get as much customer face time as you want - how else can you find out the requirements are wrong? We have little to no bureaucracy, and you have virtual carte blanche to define how your do your job. We care about results, not process. Many of our projects have hybrid insource/outsource models, so there's "opportunities" to work with vendors in both China and India, including oversight/management roles if that's your deal.

What have you shipped?
Here’s a few projects we’ve recently shipped updates to:

Gotdotnet: http://www.gotdotnet.com/ (Deployment updates ongoing, so may be spotty)
Forums: http://forums.microsoft.com
DHTML Customer Chats: http://www.microsoft.com/communities/chats/chat2/chatunmoderated.aspx
Blog Portal: http://www.microsoft.com/communities/blogs/PortalHome.mspx

What do you want from me?

Know how to think about technology. If you're demonstrated you can learn what you need to and you can think, you can learn it here.
Show me you systematically think about risk and can make profitable tradeoffs. Be able to defend your decisions and get them implemented when the entire team disagrees.
Be able to compile code. Have a theoretical CS background and way of thinking, even if you don't have the degree.
Know how to properly insult other people's code and make it better.

How do I get this job?

Reply to Vikas Ahuja (remove .online. from e-mail address) with a resume and he’ll get things started

Here are the official job descriptions.

http://members.microsoft.com/careers/search/details.aspx?JobID=49585a06-a92e-408e-a246-ece6cdb984c6

http://members.microsoft.com/careers/search/details.aspx?JobID=BDFADDD8-583A-44B4-9094-61E676F17D18

Hire People that are Good for your Customer Community

jledgard — Wed, 02 Mar 2005 22:32:00 GMT

We're trying to change the culture here. Its no secret. We want people who work on Developer Tools to have regular interactions with our customers. Part of a culture change involves a shift in the people. I guess this is sort of my plea to Microsoft employees who might be reading this and interviewing people in the future. Make sure that you consider community interaction part of the "Microsoft Culture Fit" you are looking for.

Ideally, every month, 100% of the organization has touched a customer in some way. Its not asking much; Answer one newsgroup question... post to a blog... respond to a bug from the MSDN Product Feedback Center. Go to a user group meeting. Sure,some people are better at these activities than others, but getting over 80% should be easy right? In February roughly 32% of people who work in the developer division touched a customer in some way. Not bad (albeit failing according to every High School teacher I ever had), but how do we get above this mark on a regular basis?

Part of the answer is to make sure we hire and retain people who are both good at developing software AND at forming connections to customers. I'm not suggesting we hire someone just because they are good for the community, but that we hire people who share the values of the community and understand how to work within the community and as a champion for their customers.

Everyone should be able to represent their customers, but the members of the QA team are generally the last line of defense and to properly test a product they had better understand how customers are using and intend to use the product. When I was SDET Lead these are some of the questions I would pose to potential test organization hires.

How do you know if a proposed design change or bug fix is right for your customers?
Describe a time that you championed to fix a bug in your product that was reported by a customer.
Talk about how you would identify and work with members of your target customers communities to make your product better.
Describe a time when you were able to make a customer more successful with your product than they had previously been.

I now ask similar questions to any PM candidate I interview and I hope that potential developers get similar questions. I'm not sure there are "best answers" to these questions, but I believe you can learn a great deal about how much a candidate values their customers through carefully listening to their responses.

As I said, everyone should be accountable for interacting with their customers on a regular basis. We'll be better off as a company if we continue to hire people who are capable of working with customers to both make them more successful (and therefore happy with MS solutions) and improve the quality of our products through customer understanding. Hire great people, but make sure they are a fit for your community.

My Thoughts on the MSDN Feedback Center

jledgard — Thu, 29 Jul 2004 01:10:00 GMT

Overall I know it’s a great thing and there are some issues that need to be improved over time for it to be truly successful. I may be a little close to the project since my co-workers are responsible for the Feedback Center, but I think I’m still allowed to have opinions and air them in this space. :-) Recently I was asked some specific questions that do a good job framing my opinions on this project.

Do you like this kind of transparency?
I love this transparency and think it’s only the first step towards a more transparent future of developer products at Microsoft. Why do I like it so much?

In case it’s not clear… when you enter a bug through the feedback center it is also opened directly in our internal bug tracking solution alongside of bugs being found by our own testers. Internally we can’t escape these bugs since they are in the database we scrutinize every day on our way to shipping the product. You get to see when, how, and who fixes your reported issues and you get notified of its progress along the way. We also require that team members provide quality responses to every reported issue to further encourage accountability and close the feedback loop.

Historically we’ve had crash reports and statistics that helped us know which crashes we should focus on based on the number of reports. Triaging non-crashing issues has always led to debates over how we think customers might feel about an issue versus what rating and how many customers might actually be affected by an issue. This will eventually help us triage bugs much more efficiently and gain an even better customer understanding of the impact of bugs we find internally.

We will only get more transparent. Soon you may be able to see our actual internal bug counts and even see internal bugs alongside customer reported issues. When we are in a world where you might have access to any build of visual studio this will be even more important.

How does this compare with the closed, semi-private, bug reporting on beta.microsoft.com ?
I’ve used both and the MSDN feedback center is much better simply because you can search, see, and vote on bugs reported by other users. You also get to add your comments and workarounds to issues as well. With betaplace (beta.microsoft.com) you only get to see issues that you report.

Microsoft products will still have private beta’s for a long time so there will always be a need for a non-public site, but there are teams talking about creating closed group instances of the Ladybug application moving forward because it is seen as a step up from existing solutions. So you can imagine that beta.microsoft.com could eventually be leveraging the Ladybug solution.

What do you think about the quality of submitted bugs?
It could be improved. I’ve spent a lot of time in my career here triaging bugs as a Test Lead and I’ve also been involved in triaging the bugs from the Feedback Center. In general feedback center bugs require more “translation” time to understand what customers are trying to report and what they may have been trying to do when they encountered the bug. Some teams have suggested that the triage time per bug is up to 10 times as much when compared to bugs reported internally. I’m not sure I buy that number (given the statistics you’ll see below), but it does take more time. The extra time required is probably because of the following reasons:

Lack of Additional Background Info: There is a set of information that each team feels should be collected when a bug is logged on their feature. When this information is missing the issue requires an extra round trip with the customer to attain it. We would rather not burden every customer with these specific requests, since we want the bug reporting to remain easy, but we may eventually run an active X control that automatically fills out portions of a longer form for users when they attempt to file a specific bug. This information would include what profile settings you chose at first launch, what other versions of VS you might have installed, what type of project you were working on, etc.
Lack of Clear Repro Steps: We should provide you guys with better bug report examples. I’ve seen a bugs that look like “Sometimes I seem to get an error message that says Foo” without explaining what they were doing that led to the error message. The feedback center is a community in its infancy and the unspoken rules/expectations of a community have not been fully explored yet. Over time, with better examples and additional practice submitting bugs it will improve. The quality here is not unlike the quality of bugs you might get from a new-hire testing the product and, like the new-hire, I’m sure it will improve over time with a little guidance.
Lack of Expected Behavior Reports: Good bugs should not only contain the description of the behavior experienced, but also the expected behavior from your perspective. Instead of just saying “When I click X, FOO occurs” it can be helpful to say “When I click X, FOO occurs, but I was expecting BAR because of Y”. I also believe this will get better over time.
Lack of Good Microsoft Responses: Yes, this something we can improve on that will help quality. A running joke on our test team was that we should have all been filing our bugs through beta-place or Ladybug as customers because we’d get better responses about why issues are “Won’t Fix” or “By Design”. A tester internally will often just see these two word responses and it would be up to the tester to push for a better explanation if they felt the bug was important. Strangely enough we felt we should encourage people to treat the community members with a little more dignity than our own employees.
- Providing clear explanations about bug resolutions is something that is new for a lot of people. Speaking as a former tester I’ll say that it is about time. Currently we’ve had some rough spots and bits of information that are Lost in Translation. Eventually we will adapt by improving our accountability, learning better “customer-speak”, and finding more common ground within this community. This is new for us too. We are investigating providing a direct “Contact the Developer” form for MSDN Feedback center bugs so you don’t get the rather blind “Resolved by Microsoft” text. You would then see who resolved it any be able to contact them directly.

What do the results look like so far?
So we’re listening and guaranteeing a response. IMO this is only part of the solution. In the end we’ll also be judged by what we do with what we hear. It’s unfortunate that this feedback mechanism was opened so late in the cycle of Visual Studio 2005 because a lot of this great feedback will only start to get leveraged for the next version. This is especially true for suggestions that would require more than a trivial amount of new development work. In the mean time I’ll share some raw statistics since late June when the feedback center became public that I took a snapshot of today.

For comparison I thought it might be interesting to look how these percentages compare to internal bugs and suggestions opened during the same time period. For these, unfortunately, I don’t have a good measure of time to response so the raw % will have to do. Also, just assume the total numbers for internally reported issues are larger. :-)

MSDN Feedback Center Bug Stats

Total # Opened So Far 1679
Responded to 75%
Avg # of Days to First Response 6 Days
Still Active 50%
Resolved Fixed 24%
Resolved No Repro 9%
Resolved By Design 7%
Resolved Postponed 6%
Resolved Won’t Fix 4%

Internal Bug %’s for the same time period

Still Active 55%
Resolved Fixed 20%
Resolved By Design 8%
Resolved Won’t Fix 5%
Resolved No Repro 5%
Resolved Duplicate 6%*
Resolved Postponed 1%

MSDN Feedback Center Suggestion Stats

Total # Opened So Far 1174
Responded to 79%
Avg # of Days to First Response 6 Days
Still Active 34%
Resolved Postponed 27%
Resolved Fixed 16%
Resolved Won’t Fix 12%
Resolved By Design 9%
Resolved No Repro 2%

Internal Suggestion %’s for the same time period

Still Active 52%
Resolved Won’t Fix 17%
Resolved Fixed 11%
Resolved Postponed 10%
Resolved By Design 5%
Resolved Duplicate 5%*
Resolved No Repro <1%

*If an MSDN Feedback bug is considered a duplicate it is associated with the primary internal bug so customer bugs can only be duplicates of other customer bugs. That number has been very small since customers can “+1” an existing bug rather than opening new entries. So the % duplicate of MSDN Feedback bugs was not worth reporting.

What’s important to note is that the jury is still out on around half of these issues that are still marked as active. Because of this the resolution percentages are very likely to change over time.

What I learned through this is that, so far, we’ve actually fixed a higher percentage of customer reported issues than internally reported ones! It will be interesting to see if this keeps up over time.

Now that you’ve read this far I’d like to know some things:

Are these numbers interesting to you guys?
Would you like to see more?
If so, what other information would you like to see?
How would you like to see top contributors to the Feedback Center rewarded by Microsoft?

How can Model Testing help test low level functions?

jledgard — Thu, 10 Jun 2004 20:58:00 GMT

It has taken me way to long to respond to this comment from Scott.

re: For Testers: Nihit Kaul Talks Up Model Based Testing

It appears that model based testing forces you to define the various actions your application can take (add, edit, delete, rename, etc.) and when it can take those actions.

But I don't understand how lower level functions are tested in this model. I mean, in your 'add' function has a screen with 80 text boxes that need to be filled in, how is model based testing going to help? I still have to write code to fill each of those boxes, and the code I write is still "static script".

It appears model based testing is a series of static scripts instead of one giant static script. Is that correct?

It's true that Model Based testing may not be best suited for everything. If your low level function doesn't depend on any states you can define, doesn't have any transitions, and just has a lot of parameter arguments then you are probably better off with the application of Pair Wise testing which, with the pairing of argument combinations has the ability to reduce the testing matrix. It's based on the assumption that most bugs in software occur with a combination of one or two inputs. The pair wise combination will then help you reduce the number of scripted tests you might have otherwise tried to generate. If you know something about how the function works (if some pairings aren't valid) you can limit the combinations even further. However, if your low level function depends on some states from classes around it or the system it is running on then you should leverage both techniques.

Depending on the tool used to generate your model tests it could be a set of static scripts that are generated off of a query from the model. The difference is that the scripts are generated by the query of the abstract model (shortest paths, all transitions, etc) rather than trying to enumerate all the correct scripts yourself. For further efficacy you would have a tool that automated your feature straight from the states/actions of your model. Then you can simply “run” the query and not regenerate test code if you make changes to the model (other than the main model code you wrote).

For Testers: Nihit Kaul Talks Up Model Based Testing

jledgard — Tue, 01 Jun 2004 17:00:00 GMT

It was my belief as a test lead and is still my belief. To test a product with any amount of interesting complexity you need to find good ways to reduce a reliance on manual processes. If you test manually you need automation. If you are writing a test plan you need to have a machine figure out the details. If you start writing automation you need to reduce the cost of writing automation. One way to reduce costs and start abstracting the testing problems is through model based testing. Nihit has a good introduction article to the practice you should read.

Internally we have gone so far as to write automation that abstracts Visual Studio, write a model that represents a feature, and have that model make calls to the automation. This reduces the cost of writing scripted automation. Only a few teams have started doing this and I think it has a long way to go, but I would be surprised if the use of such practices don't take off quickly.

New MS Testing Tool for Verifying Accessibility Now Available

jledgard — Wed, 12 May 2004 04:15:00 GMT

Sara publicly released MSAAVerify today. She has been working on it part time for two years and I know is excited to have “shipped“ it. This tool has recently been picked up by several teams at Microsoft for verifying the accessibility of their software and is even taught as part of the “Testing Excellence” class taken by new testers at Microsoft. To quote her Powertoys entry:

“MsaaVerify will verify 9 Microsoft Active Accessibility properties for 10 Microsoft Active Accessibility Role types. It doesn't matter whether these controls are standard Windows controls, managed, owner drawn, or custom drawn, just use MsaaVerify!”

She also gives a little more history on her blog.

“At first, I wanted to automate screen reader testing, but I quickly realized that I needed to take smaller steps....I’ve been working on this tool for so long that it feels like I need to host a ship party this afternoon... Now, our customers can use the same tool that we use in-house.”

Good Times!

NUNIT and Code Coverage Usage

jledgard — Mon, 22 Mar 2004 17:02:00 GMT

Two topics I’d been asked to cover were tackled today.

Gunnar explains “How to Shoot Yourself in the Foot With Code Coverage”. Some choice quotes that I have also found to be truisms:

To efficiently use any of this a time needs to be invested in training people, choosing proper metrics, explaining metrics to all team members, and picking key things to measure.

Whenever I read articles when somebody recommends that one should strive for 100% code coverage I want to start screaming.

Based on my experience getting more than 80% of code coverage gets extremely tricky and there’s very little in return for your investments.

Rido found, what he describes as, the best add-in and wrote a macro to go along with it for integrating NUNIT into VS.NET 2003.

Great Posts! Enjoy.

Mandatory Reading for People Reporting Bugs (Even in Newsgroups!)

jledgard — Mon, 15 Mar 2004 16:52:00 GMT

Having spent the last two years in my previous role of Test Lead in a LOT of triage meetings I found this post to be both hilarious and educational for people opening bugs. If everyone would do the opposite of the recommendations listed there when opening bugs I could have spent a lot less time in triage and more time actually testing. The Core IDE team has actually linked to the post from our triage guidelines page for quick internal referencing.

Also, I’d still love to hear if anyone has opinions regarding Newsgroups VS Web Forums???

Software Testing: The GUI Part 1

jledgard — Wed, 03 Mar 2004 06:21:00 GMT

I've been asked to talk a bit about UI testing. Keep in mind that none of my testing posts are meant to be a complete comprehensive guide on the subject at hand. My goal is simply that everyone reading walks away with a useful idea.

You want to test your GUI early because it could take weeks of dev time to actually build something that you’ll just scrap and start over with. There are a few pieces of UI in Whidbey that have been re-built several times at the expense of more bug fixing and potentially useful features. Please create mock ups and put them in front of as many people as you can. Photoshop them, create prototypes, slides, etc so that you can walk through your ideal user experience. When doing so, you should be asking at least the following questions:

Is it obvious to the user who has never seen this before what has to be done to complete the intended task?

This question was either never asked or answered properly when we were designing the tool window docking in VS.NET 2002 or 2003. If you drag a tool window from its docked position there is no real indication that there are 5 ways to dock it to each of the other windows you might have open. We’ve made this much more obvious in Whidbey. :-)

Does the UI make sense for the task?

Are you using a textbox when you are asking the user for a date? Do you have a dropdown where there will ever be only two intended true/false type options? Are you cluttering the UI with helper text that would be more appropriate in a help topic?

Does the UI appropriately restrict the user input?

Lets pretend there is an autosave feature in your product and you want the user to set the interval time between autosaves. You should think about restricting the interval input to numbers that would make sense. If it could take up to 2 minutes per save you might not want to allow the user to get into a state where they are stuck in an autosave loop. The max should also be a reasonable number that doesn’t defeat the intent of the feature. No point in letting people pick an interval as large as 9999 minutes since that would be only once per week. Negative numbers, symbols, or characters might also not be advisable to allow.

Properly restricting the inputs in the UI will also help to ensure data consistency on your back end and help prevent crashes due to buffer overruns or other issues that could be hidden below the surface due to how the input is manipulated. Plus, the more restrictions you put on user input the more you can reduce the testing required to test the feature.

What will happen if I do X?

For this question it’s best to sit the UI in front of people who have not seen it yet. You should ask them to guess what will happen if they click the button that says “Start”. If the UI is intuitive they should be able to get pretty close. You should ask them what limits they would expect, what paths should be defaulted to, and if the default values make sense, etc in order to help determine how effective the design gets the intent across.

As discussed this list is not comprehensive, but should get you started as you attempt to determine if you are actually close to what your final UI should look like. Again, the more dogfooders of these prototypes you have the better.

Speaking of dogfood… my wife and I picked Spot, the smart one. His new name is Duncan and you can check out his dogster profile. :-)

Enjoy...

Exciting Times

jledgard — Wed, 03 Mar 2004 05:15:00 GMT

The KDE Quality Team was covered on Slashdot today. Maybe working in a test org makes me biased, but this is a great concept for the open source world, and one of the missing links for OSS IMO. Some of the tasks, like writing documentation, may fall out of the typical QA realm, but should work really well as a way to get more non-technical people to feel they too can participate in the various projects. It’s something I’ve experienced with the VS PowerToys. There are a lot more members of each project than there are members willing or able to code a tremendous amount that could probably be really helpful if they did things as simple as verifying that bugs are indeed fixed for the other members before a release is posted. Eventually you could see an OSS quality team writing unit tests or other basic verifications that could also be checked into each project.

I’ve often thought about something simple like opening up our test plans for the community so people could say “Yeah, that should cover how I work” or “No, you’re totally missing these extreme cases I run into every day”. It’s exciting to see “the competition” push us at Microsoft further down the road towards being more open about our work and involving the community to help shape our own projects for the better.

VS Settings and Early Testing (Mindless Linkage Ahead)

jledgard — Mon, 16 Feb 2004 17:41:00 GMT

If you like C# and you have opinions on default Visual Studio settings you should check out this post.

Another testing article I’d like to write is about effective early testing. At Microsoft we use a BVT process that has been nicely explained here.

josh