Five Testers From VC

VC IDE Testing Tactics (and a beta1 plug!)

FiveTestersFromVc — Fri, 30 Jul 2004 03:30:00 GMT

[Rob]

You've probably had enough time to read the last post...Ok, so maybe we were reallllly busy getting our beta product release squared away so you can all get your free Visual Studio 2005 Express Beta1 downloads! Grab it, have fun, win an Xbox, tell me about what was good and what needs more! (There are also versions focusing on other programming languages and even web development!). http://lab.msdn.microsoft.com/express/default.aspx

***

Tactical thoughts

Thinking about all the things that go into testing feature...is the API working? Are there any attack paths through this feature (is it secure)? Does it integrate with existing features? Does it depend on features (our group or another?!) that might change before lock down? Is the UI compliant with suite requirements? Is the UI clear? Are UI redundancies appropriate? What will our users think about our answers to these questions? What happens to the UI when the product is localized (translated to another language)? What happens to the behavior of the feature when the product is localized? (Add a number of other questions from networking to user permission levels depending on the feature category...).

The list appears endless, but luckily we have limited time (some sarcasm on that use of 'luckily', if you didn't notice) to do our testing so we work with a finite set that makes sense for the feature in question.

The highest urgency task is to get the feature tested for main stream functionality. What will our customers need this to do? What will our customers want this to do? What assumptions may be made that could get this feature into trouble (and put the user into a confused or frustrated state)?

We also want to see that the feature discoverable and useable. The UI needs to indicate what will happen and let users know about progress if this takes time. Keeping UI consistent so different tasks at least 'feel' familiar in the welcoming sense is valuable...but difficult when taking legacy features into a new arena (a challenge for many still useful feature sets now housed in a building of .NET splendor). Is there a point to upgrading old UI to match the new style if the old features are stable as is (and use the resources instead for new features and to fix bugs)? Is old UI style a bug or an acceptable artifact of evolution?

These questions are what test cases are all about. Each test case seeks an answer. Customer feedback, experience, and research help us to ask the right questions for our current product. Once we have a base set of questions (Test Case Planning is a cyclical process, just as implementation is) we can move forward to measure the status of product drops based on the answers we get on test passes. We build our automation library which allows us to gather answers faster and feel more secure against regressions that could be introduced every time developers touch the product. At the end of the cycle, we have an automation set that allows us to have a level of confidence that sustained engineering will be able to do service pack work while maintaining a high level of regression test coverage (while the front line team moves to the next generation product).

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

Channel 9 Visits VC

FiveTestersFromVc — Thu, 10 Jun 2004 21:51:00 GMT

[Ron]

As I type this, some of the Channel9 dudes are interviewing one of the compiler PM's (Kang Su Gatlin) right outside my office. I wonder how long the lead time is before the video makes it onto the Channel9 site...

well, since you asked twice...

FiveTestersFromVc — Fri, 28 May 2004 03:05:00 GMT

[Rob]

Woon asked in his comment about the % automation we have with our IDE tests.

Currently, we are at roughly 50% automated in the IDE compared to the total number of active test cases. These tests range from DTE manipulations to actual UI interations the way the user would (drag and drop, button clicking, resizing, ...). One way to get at the UI elements is through MSAA. This works fairly well for standard controls. Custom controls make the story more difficult (which is why we resourt to DTE calls if our test harness can't handle certain controls).

It takes about 2-3 weeks for 15 people to cover all the manual test cases we currently have for the VC IDE. And this doesn't include common shell features that the VS team handles.

Our automation's primary purpose is to give us rapid general status on build drops each day (although our automation is split up between daily and weekly test sets). The automation set does find bugs, but by far the most bugs are found through manual testing of new features. Automation is our 'security blanket' as we can get a breadth of coverage in 24 hrs with a second depth after ~72 hours (looking for regressions from the previous known state) rather than waiting for three weeks (which also uses up the whole team's attention) with manual. Both are vital.

Another new VC Tester Weblog

FiveTestersFromVc — Tue, 11 May 2004 22:30:00 GMT

[Ron]

Eric Fleegal, a tester on the compiler back-end test team, has joined the weblogging movement. His area of interest and expertise is floating point code generation. (He authored the recent MSDN whitepaper on the new Whidbey floating point code generation model). I guess it's time to update our VC bloggers list on the left...

Cool thing I learned about the IDE today.

FiveTestersFromVc — Fri, 07 May 2004 03:42:00 GMT

[Michael] The VS IDE has a ton of cool UI features - many that I'd never ever head of before.

If you're a command line kind guy, you can get some use out of the “Find combobox”. I'm sure this feature is in Visual Studio 7.1 + but I also think it's in Visual Studio 7.0.

If you hit CTRL+D from the editor (and I think most other places in the shell) it will take you to the find combo, this combo box right next to the binoculars sitting in an open folder. From there you can type text you want to find, and then hit F3 to search for it.

What's equally as cool is that you can use the “>” operator to put yourself in command mode, and from there you get an autocompleting list of all the commands you can issue to the IDE's shell directly.

For example, I placed the “>” character in that edit box, then typed “F” and it dropped down with a list starting with “File.AddExistingProject”. With autocomplete, I got it to type “File.OpenFile d:\admin.txt”

I was personally impressed that we support autocompletion from the File.OpenFile command.

There are a ton of other commands available, some of which you might find easier to uncover this way.

It's another way to get around the IDE without having to mouse around, especially if you don't remember all the keyboard shortcuts.

Boing... Boing... Boing...

FiveTestersFromVc — Sat, 01 May 2004 03:32:00 GMT

[Ron]

I noticed Shaykat (over in the C#/debugger team) posted about bug bounces. For our test team., these come in threes.

The first one we encounter is ZBB - zero bug bounce. This is driving active product bugs down to zero. This is the bug bounce that Shaykat is talking about in his post. Test is involved in this bounce becuase sometimes we have active bugs assigned to us. These could be bugs where the dev is asking for more information or a repro case or, in some minor cases, it could be because test owns development of some product areas.

The second bounce is ZRBB - zero resolved bug bounce. This is driving the resolved product bugs down to zero (i.e., verifying and closing them). Bugs in this category are important because, until the fix is verified, we can't be totally sure it's fixed or hasn't caused some other regression. Testing tends to be the most heavily involved in this bounce since (ideally) they have opened most of the bugs and would be the ones to verify the fixes. They are also involved in bugs that weren't opened by testing since those bugs should cause us to ask why we didn't find those bugs and add to our testing if that indicated a test hole.

The third and final bounce is ZTBB - zero test bug bounce. This is driving the active bugs against our test automation down to zero. This is important since a test that isn't working can't find bugs or verify the funtionality of the product.

This last bounce doesn't get as much attention as the other two; it's generally only an effort for the test team but we do treat our test automation as a real development project. That means our test automation code is checked into a source code control system, bugs against our automation go into an issue tracking database, and we do periodically drive our bug count down to make sure we're not missing issues or test coverage.

Prolific, We're Not

FiveTestersFromVc — Fri, 30 Apr 2004 03:19:00 GMT

[Ron]

Geez the five testers haven't really been posting much lately. I just asked Michael about this and he said that he feels there is a lot of pressure to be absolutely correct in what we say. My take: we've been really busy lately. It's interesting, there's probably different classes of webloggers out there. The folks like Dave Winer and Scoble seem to post with the same frequency as they breathe. We haven't been doing that.

Maybe Michael is right and we feel as though we need every post to be a masterpiece. I certainly have been thinking about a post I've promised on compiler correctness tests a long time ago (but I still haven't posted anything). Maybe we should not worry so much about this and just post what's happening or on our minds more frequently. I guess this post is a beginning attempt to do that.

We Got Slashdotted

FiveTestersFromVc — Mon, 19 Apr 2004 20:06:00 GMT

[Ron]

http://developers.slashdot.org/article.pl?sid=04/04/18/0146234

This is about the VC++ 2003 command line toolkit:

http://msdn.microsoft.com/visualc/vctoolkit2003/

VC IDE Testing: cycle check points

FiveTestersFromVc — Fri, 16 Apr 2004 04:14:00 GMT

[Rob]

General IDE testing cycle status points (iterations on/between any two points…)

Specification review
Test Plan Design
Test Plan Review
Developer Implementation design
Developer Implementation design review
Test case design [exploratory, user feedback, regression]
Automation Design
Automation Implementation and review
Manual Test Pass
Automation run
Project status evaluation and reporting

Since the organization is broken up between disciplines (Program management, Development, and Test), cross discipline communication helps deal with the implied dependencies in the eleven points above.

A typical cycle contains zero or more of the following obstacles based on the above points:

PM (program manager group) delivers spec(s) late, preventing review by schedule
test plans delayed by changing specs/parallel product cycle effort lowering priority
test plan reviews are cursory, resulting in missed holes fund later which cost more to correct
the design is the code
well, we can review the code…or just use the feature…
test cases ended up being designed without complete specifications…requires revisions regularly at added cost
automation gets pushed back due to UI/feature churn making tests obsolete an hour before they are checked in
Automation harness issues; changing requirements; dogfooding delays (a good thing, but causes progress trade-off) [dogfooding: using VC drops to develop automation for testing VC drops…]
Manual Test Pass: can always count on them, but they take too long if automation not strategic or complete (temptation is to delay automation to get complete manual coverage creating an ever deepening hole…bite the bullet and do the automation right so manual is only what can’t/shouldn’t be automated).
matrix explosion (platforms * SKU’s * architectures * permission levels; pair wise testing save us!)
What do you mean, “make it green”?! [balancing quality driven vs. date driven product requirements]

Some random elaborations:

There’s a lot of overlap at the beginning/end of product cycles. This really prevents any one discipline from getting a ‘fresh start’, as wrap up work often takes priority over new work (reverse that for PM). Thus, the new work gets delayed quite often.

A note on ‘priority’: Everything is ‘pri 1’, because if it wasn’t, it was cut. This is because there isn’t enough time to do everything that ‘has to happen before we ship’. So, ‘priority’ has no meaning. Just ignore it. Treat it as someone saying, ‘um…’ or ‘er…’. I’m making it a priority to remove that word from tester vocabulary….

At one point, our team had fallen significantly behind our goals for automation development. We found ourselves in a position where we needed to keep testing up on implementation churn by providing immediate testing of any dev patch/check in. This constant impromptu testing continuously delayed automation development between formal test passes. We are in a much better state at this time because we followed through on the difficult decision (made with the dev org) to forego the interrupt driven manual testing completely for several milestones and focus on automation (formal test passes still occurred, of course, to cover all test cases for each milestone).

One angle which has been used with automating IDE features can be summed up with a call to “do everything as the user would”. Reality set in as conflicts with shared UI continuously broke tests over multiple groups. Since feature innovation can’t be stalled just to maintain an automation test bed, it became necessary to follow an academic rule: “a test should test only what it tests”. In other words, if you aren’t specifically testing the UI, if at all possible, the test should not fail because of a UI change. Praise be the DTE! (http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp). For visual studio, feature area automation authors can often bypass UI that they don’t need to directly test and get at what does need to be addressed with the current test. This has made automation faster and more reliable for our current test harness.

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

What kind of people are testing this stuff anyhow?

FiveTestersFromVc — Fri, 26 Mar 2004 03:58:00 GMT

[Rob] I know I said the next installment would be on tactics…but I think in response to feedback Woon Kiat I’ll go with ‘resources' first.

Before that, very quickly, there was a comment from AT on alpha/beta users submitting automation test cases along with bugs. Generally speaking, repro steps for bugs are added to the regression test case set and automated where appropriate. Directly accepting scripts, however, would cause security concerns. It would be a nice world though to just be able to build a customer ‘drop acceptance’ automation test case database… J Now you have me thinking about a tool to allow steps to be entered through a web page that would build a safe script to run against the product...hmmmm….I could build that using the VC++ IDE once I figure out the design…

Resources for Test in VC

Test candidates right out of school rarely have formal test training. The experience level is also limited to personal code and maybe a few small group projects. This gives us the challenge of identifying those candidates that have a rough talent for ‘finding bugs’ or ‘breaking things’ J along with a way to navigate through a product and identify where priority bugs may hide. We also look for people who can adapt quickly and apply techniques ‘just learned’ to new problem areas.

We write test automation and our product is a development tool. Therefore we also look for people who have development proficiency with C++ (knowledge, style, problem solving, … pretty much our customer base).

Our company has training courses (all disciplines) that supplement on the job experience. Everyone on the team is a mentor to some degree or another for every new hire we get. Formal and informal mentors are encouraged in addition to an individual’s team lead. Acquiring a mentor from outside the team is one way we continue to mix ideas, technologies, and techniques throughout the company. We also like to occasionally reorganize teams to allow people to move to new feature areas and ‘stay fresh’ by providing new challenges.

Product specific training is hands on. For Visual C++ we focus heavily on dogfooding (use the current drops for real world coding of team tools and personal hobby projects) the product. Plenty of texts exist on previous versions of the product to help people understand where we’ve been. Specification documents (or, when not available, talks with program managers and feature developers) bring us up to date for what to expect in the current version. When one is new to a feature area, reviewing the old test case base and the bug database helps bring you up to date on the current status and where testing has gone before. Particularly interesting are those bugs found by customers, as they show where customers went with the product that the previous test plans did not anticipate (allowing us to learn better what to give priority to in the limited time we have to test).

Some books found on my office shelf today (more at home where I have a bigger shelf…oh, and I’m not saying these are all ‘fantastic’ books, though some are, I’m just giving a snapshot of what’s behind me):

Various language books (C++, VB, C#)

Various manuals from internal microsoft training courses

Various manuals from Test conferences (most recent: PNSQC and SQE)

Debugging Applications for .NET and Windows, Robbins

Lessons Learned in Software Testing, Kaner et al

How to Break Software, Whittaker

Design Patterns, Gamma et al

Quality Software Management, Weinberg

Writing Secure Code, Howard et al

Automation Programmer’s Reference, Microsoft Press

Applied Microsoft .NET Framework Programming, Richter

Programming with Visual C++ .NET, Shepherd et al

Programming Windows, Petzold

Inside COM, Rogerson

;-)

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

VC IDE testing (issues and processes)

FiveTestersFromVc — Fri, 19 Mar 2004 04:26:00 GMT

[rob] Since Ron has been entering thoughts and comments on compiler testing, I’d like to add a perspective on IDE testing from VC. I’ll leave it in notation format to encourage discussion of points for which readers would like more clarification.

Issues to address through IDE testing:

Changing specifications over time (some features are completely ambiguous until devs implement, others have varying levels of specs prior to implementation)
Library of regression test cases, authors having moved to other teams; QA work is more cumulative than dev work in that regression testing currently takes a larger toll than code base maintenance.
Dependecies (teams depend on us, we depend on them)
Platform coverage
SKU coverage
Multiple paths through UI
CodeModel path around UI
Localization; Data handling and UI
SxS, Compatability
Performance
XP Logo

Process:

Theory
- Schedule and cost for the milestone before you begin (sounds like a line from Willy Wonka: [school teacher] “…switch our Friday schedule to Monday, which means that the test we take each Friday on what we learned during the week will now take place on Monday before we’ve learned it.”) Update schedule as milestone progresses to measure against target dates.
- Review feature specifications from PM team, providing feedback (expect updates during cycle).
- Review implementation specification from Dev team, providing feedback (updates expected).
- QA drafts high level test plan and submit to review by Dev and PM teams; update.
  - Reference previous product and similar feature test plans
  - Incorporate customer feedback and market expectations for setting priorities
  - Test team contracts attempt to prevent assumptions from leading to test holes across team boundries
- Author test cases according to test plan; update plan as needed.
- Author test case automation
- Run automation test passes on coverage matrix
- Run manual test passes on matrix subset
- Log bugs, analyze trends, verify fixes.
- Analyze automation code coverage (block level) results from instrumented builds.
  - Combine results with test plan holes to target next automation set.
- ‘Dogfood’ the product
  - (Semi-)stable builds are selected by individual testers to use for developing automation (some data driven, some code).
  - App week: break into teams and develop short term projects (some become team tools).
- Themed bug hunts: “Bug Bash”
  - team targeting of a specific Logo, accessibility, localization,.... categories.
  - Periodic re-education of feature owners on testing aspects
- Iterate and ship!
Reality, as enforced by those pesky J market needs and restrictions…
- Level of clear specifications varies greatly by feature and product cycle. This is not something QA can (historically) count on. Working closely with the PM team allows for the necessary information to be focused on creating effective test plans. Various unexpected needs (from marketing to division to group to feature) continuously challenge the proposed schedule (you know, that thing we all agreed to before the first feature specification was actually written?). Adapt and overcome; priority trade off.
- Implementation plans tend to be the actual code from Dev. Monitoring check-ins and dev check-in test reviews helps to prepare for proper coverage
- Some high level test plans get bogged down from lack of feature specifications and test cases develop ad hoc during exploratory test passes.
- Authoring automation competes with running manual test cases. The need to report the current product status, in a sense, delays the ability to report a more complete status more often... (Catch 22, but without the airplanes).
- Automation requires a test harness.
  - A test harness requires development and maintenance.
  - VC has moved from an internal tool to a divisional tool
    - Still requires development/maintenance participation
    - Supporting/following the gander isn’t what the goose used to do
- Coverage matrix bulges and balloons; OS varieties, SKU variations, Side by side installations, run time compatibility issues, pre-release forks, internal drops, priority customer drops, priority customer quick fixes, alphas, betas, somewhere in there we have the primary branch we plan to ship…
- Again and again activities we want to do (Pri 2 and 3 test cases; driving pri 2 and 3 bug fixes; driving pri 2 and 3 features) are restricted by schedule needs (competing in the market).
- dependencies get out of sync and one team may be blocked on another team

Next installment: Tactics...

Extreme Programming and Ninja Gaiden...

FiveTestersFromVc — Fri, 12 Mar 2004 05:09:00 GMT

[Michael] Hello everyone. I haven't been very active on the blog lately - we're in the middle of a big test pass. Let's see if I can start a discussion.

How many of you out there are using Extreme programming? How large are your teams? Some of us have been having a discussion about it and we keep coming back to the point that someone has made - that no one knows of any large teams using extreme programming. I hope to hear some good stories about it.

For a small glimpse into my personal life - I've been playing the Xbox game Ninja Gaiden. It's been a long time since I've played a game that was this challenging - yet still made me want to keep playing. It's also probably the best looking game I've ever played on a console. I'm starting to wish I was younger though. Perhaps I should fly one of my ten year old cousins up here. I'm having a hard time reacting fast enough to the bad guys.

Answers to Comment Questions

FiveTestersFromVc — Fri, 20 Feb 2004 08:58:00 GMT

[Ron]

A couple of questions from Woon Kiat to my previous post on the different types of testing we do on our C++ compiler:

>>Is understanding the inner working of a compiler vital to compiler testing?

I would say so, at least at some level. In a general sense, I would say it's hard to be really successful at testing anything without understanding the domain area of what you are testing.

That said, I don't think many folks came to our compiler test teams with a lot of understanding of the inner workings of a compiler, especially the back end. This is something that we pick up from doing the work.

>>How would you verify the binary generated from the compiler is correct? Is it fall under Correctness Tests?

In general, we write programs to expose the behavior that demonstrate the particular compiler behavior we expect to see from the feature we are testing. This does fall into the category of correctness tests. I'll try to include some examples of how we do this in my next post on C++ compiler testing.

Thanks for the questions, Woon!

New Whitepaper: Floating Point Code Generation in Whidbey

FiveTestersFromVc — Fri, 20 Feb 2004 02:22:00 GMT

[Ron]

One of the major new features in the area of code generation for the upcoming Whidbey release of VC++ is our new floating point code generation model. One of the members of my team, Eric Fleegal, has written a nice white paper about this feature for MSDN:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dv_vstechart/html/floapoint.asp

Why I'm blogging...

FiveTestersFromVc — Fri, 13 Feb 2004 03:40:00 GMT

[Michael] I've got a circuitious style - so if you're impatient, you may want to skip this thread. I'm going to talk about one of the big reasons I'm participating in this blog, and what I'm hoping to gain from it. I'm also big on analogies and stories - and as a fan or Robert Fulghum, I hope this one all makes sense in the end.

I went to a physical therapist a while ago - had some pain in my shoulders and arm. I was in a sour mood. Wasn't as productive at work as I wanted to be. I got there and we talked. I complained a lot about my shoulder and arms. Then a funny thing happened. The PT felt around for the tension, listened to what I had to say about that, and we started working on my lower back. After a while, the focus shifted onto my shoulders and arm and neck. However, most of the time was spent on my back. It wasn't until I was there that I realized that my pain was really in my back.

My back had hurt so long that it had stopped registering with my mind that it was a problem. My neck and arms were compensating for the back pain. I'm glad that the PT didn't just work on my neck and arms. If that had been the case, I'd be right back where I started from. But if the PT has just done what I said without establishing a real dialogue with me, we wouldn't have solved my real problem.

There is something different about the information you get from a simple statement and the information you get by "being there" or "being in touch". There is a lot of context missing from one off communication.

This is one of the reasons I hope this blogging experiment will work.

We have charts and graphs and concerns and suggestions filling up databases. We often have more information than we could practically sort through. And we try. But it doesn't tell the whole story. Without the right context, we might not make as good a decision as we could have with the context customers can provide - if there is real dialogue.

Hopefully, by being out here we can establish a useful and meaningful dialog. Blogs seem to be a compromise in a way - they can't scale if everyone who could read one asked a personal question - but hopefully it is a step toward establishing a strong dialog that will provide us and you, our readers, the context to understand each other. And hopefully solve some problems and relieve some customer "pain".
That's what my opinion of the job of Visual Studio is. To make it easier for you guys to write your applications.

Dialog works both ways too. If you don't know what we're doing, or why, then you probably won't be very eager to buy our product. You might feel like we aren't paying attention to your needs. I hope, I really do, that we're spending more of our time working on your problems than your symptoms. Please let us know if that's not the case.