[Rob]

You've probably had enough time to read the last post...Ok, so maybe we were reallllly busy getting our beta product release squared away so you can all get your free Visual Studio 2005 Express Beta1 downloads!  Grab it, have fun, win an Xbox, tell me about what was good and what needs more! (There are also versions focusing on other programming languages and even web development!). http://lab.msdn.microsoft.com/express/default.aspx

***

Tactical thoughts

Thinking about all the things that go into testing feature...is the API working? Are there any attack paths through this feature (is it secure)? Does it integrate with existing features? Does it depend on features (our group or another?!) that might change before lock down? Is the UI compliant with suite requirements?  Is the UI clear?  Are UI redundancies appropriate? What will our users think about our answers to these questions?  What happens to the UI when the product is localized (translated to another language)? What happens to the behavior of the feature when the product is localized? (Add a number of other questions from networking to user permission levels depending on the feature category...).

The list appears endless, but luckily we have limited time (some sarcasm on that use of 'luckily', if you didn't notice) to do our testing so we work with a finite set that makes sense for the feature in question.

The highest urgency task is to get the feature tested for main stream functionality.  What will our customers need this to do?  What will our customers want this to do?  What assumptions may be made that could get this feature into trouble (and put the user into a confused or frustrated state)?

We also want to see that the feature discoverable and useable.  The UI needs to indicate what will happen and let users know about progress if this takes time.  Keeping UI consistent so different tasks at least 'feel' familiar in the welcoming sense is valuable...but difficult when taking legacy features into a new arena (a challenge for many still useful feature sets now housed in a building of .NET splendor).  Is there a point to upgrading old UI to match the new style if the old features are stable as is (and use the resources instead for new features and to fix bugs)?  Is old UI style a bug or an acceptable artifact of evolution?

These questions are what test cases are all about.  Each test case seeks an answer. Customer feedback, experience, and research help us to ask the right questions for our current product.  Once we have a base set of questions (Test Case Planning is a cyclical process, just as implementation is) we can move forward to measure the status of product drops based on the answers we get on test passes.   We build our automation library which allows us to gather answers faster and feel more secure against regressions that could be introduced every time developers touch the product.  At the end of the cycle, we have an automation set that allows us to have a level of confidence that sustained engineering will be able to do service pack work while maintaining a high level of regression test coverage (while the front line team moves to the next generation product).

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

[Rob]

Woon asked in his comment about the % automation we have with our IDE tests.

Currently, we are at roughly 50% automated in the IDE compared to the total number of active test cases.  These tests range from DTE manipulations to actual UI interations the way the user would (drag and drop, button clicking, resizing, ...).  One way to get at the UI elements is through MSAA.  This works fairly well for standard controls.  Custom controls make the story more difficult (which is why we resourt to DTE calls if our test harness can't handle certain controls).

It takes about 2-3 weeks for 15 people to cover all the manual test cases we currently have for the VC IDE.  And this doesn't include common shell features that the VS team handles. 

Our automation's primary purpose is to give us rapid general status on build drops each day (although our automation is split up between daily and weekly test sets).  The automation set does find bugs, but by far the most bugs are found through manual testing of new features.  Automation is our 'security blanket'  as we can get a breadth of coverage in 24 hrs with a second depth after ~72 hours (looking for regressions from the previous known state) rather than waiting for three weeks (which also uses up the whole team's attention) with manual.  Both are vital.

[Michael] The VS IDE has a ton of cool UI features - many that I'd never ever head of before.

If you're a command line kind guy, you can get some use out of the “Find combobox”. I'm sure this feature is in Visual Studio 7.1 + but I also think it's in Visual Studio 7.0.

If you hit CTRL+D from the editor (and I think most other places in the shell) it will take you to the find combo, this combo box right next to the binoculars sitting in an open folder. From there you can type text you want to find, and then hit F3 to search for it.

What's equally as cool is that you can use the “>” operator to put yourself in command mode, and from there you get an autocompleting list of all the commands you can issue to the IDE's shell directly.

For example, I placed the “>” character in that edit box, then typed “F” and it dropped down with a list starting with “File.AddExistingProject”. With autocomplete, I got it to type “File.OpenFile d:\admin.txt”

I was personally impressed that we support autocompletion from the File.OpenFile command.

There are a ton of other commands available, some of which you might find easier to uncover this way.

It's another way to get around the IDE without having to mouse around, especially if you don't remember all the keyboard shortcuts.

[Ron]

Geez the five testers haven't really been posting much lately.  I just asked Michael about this and he said that he feels there is a lot of pressure to be absolutely correct in what we say.  My take: we've been really busy lately.  It's interesting, there's probably different classes of webloggers out there.  The folks like Dave Winer and Scoble seem to post with the same frequency as they breathe.  We haven't been doing that.

Maybe Michael is right and we feel as though we need every post to be a masterpiece.  I certainly have been thinking about a post I've promised on compiler correctness tests a long time ago (but I still haven't posted anything).  Maybe we should not worry so much about this and just post what's happening or on our minds more frequently.  I guess this post is a beginning attempt to do that.

[Rob] 

General IDE testing cycle status points (iterations on/between any two points…)

1. Specification review
2. Test Plan Design
3. Test Plan Review
4. Developer Implementation design
5. Developer Implementation design review
6. Test case design [exploratory, user feedback, regression]
7. Automation Design
8. Automation Implementation and review
9. Manual Test Pass
10. Automation run
11. Project status evaluation and reporting

Since the organization is broken up between disciplines (Program management, Development, and Test), cross discipline communication helps deal with the implied dependencies in the eleven points above.

A typical cycle contains zero or more of the following obstacles based on the above points:

1. PM (program manager group) delivers spec(s) late, preventing review by schedule
2. test plans delayed by changing specs/parallel product cycle effort lowering priority
3. test plan reviews are cursory, resulting in missed holes fund later which cost more to correct
4. the design is the code
5. well, we can review the code…or just use the feature…
6. test cases ended up being designed without complete specifications…requires revisions regularly at added cost
7. automation gets pushed back due to UI/feature churn making tests obsolete an hour before they are checked in
8. Automation harness issues; changing requirements; dogfooding delays (a good thing, but causes progress trade-off) [dogfooding:  using VC drops to develop automation for testing VC drops…]
9. Manual Test Pass: can always count on them, but they take too long if automation not strategic or complete (temptation is to delay automation to get complete manual coverage creating an ever deepening hole…bite the bullet and do the automation right so manual is only what can’t/shouldn’t be automated).
10. matrix explosion (platforms * SKU’s * architectures * permission levels;  pair wise testing save us!)
11. What do you mean, “make it green”?!  [balancing quality driven vs. date driven product requirements]

Some random elaborations:

There’s a lot of overlap at the beginning/end of product cycles.  This really prevents any one discipline from getting a ‘fresh start’, as wrap up work often takes priority over new work (reverse that for PM).  Thus, the new work gets delayed quite often.

A note on ‘priority’:  Everything is ‘pri 1’, because if it wasn’t, it was cut.  This is because there isn’t enough time to do everything that ‘has to happen before we ship’.  So, ‘priority’ has no meaning.  Just ignore it.  Treat it as someone saying, ‘um…’ or ‘er…’.  I’m making it a priority to remove that word from tester vocabulary….

At one point, our team had fallen significantly behind our goals for automation development.  We found ourselves in a position where we needed to keep testing up on implementation churn by providing immediate testing of any dev patch/check in.  This constant impromptu testing continuously delayed automation development between formal test passes.  We are in a much better state at this time because we followed through on the difficult decision (made with the dev org) to forego the interrupt driven manual testing completely for several milestones and focus on automation (formal test passes still occurred, of course, to cover all test cases for each milestone).

One angle which has been used with automating IDE features can be summed up with a call to “do everything as the user would”.  Reality set in as conflicts with shared UI continuously broke tests over multiple groups.  Since feature innovation can’t be stalled just to maintain an automation test bed, it became necessary to follow an academic rule:  “a test should test only what it tests”.  In other words, if you aren’t specifically testing the UI, if at all possible, the test should not fail because of a UI change.  Praise be the DTE!  (http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp).  For visual studio, feature area automation authors can often bypass UI that they don’t need to directly test and get at what does need to be addressed with the current test.  This has made automation faster and more reliable for our current test harness.

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

[rob] Since Ron has been entering thoughts and comments on compiler testing, I’d like to add a perspective on IDE testing from VC.  I’ll leave it in notation format to encourage discussion of points for which readers would like more clarification.

Issues to address through IDE testing:

• Changing specifications over time (some features are completely ambiguous until devs implement, others have varying levels of specs prior to implementation)
• Library of regression test cases, authors having moved to other teams; QA work is more cumulative than dev work in that regression testing currently takes a larger toll than code base maintenance.
• Dependecies (teams depend on us, we depend on them)
• Platform coverage
• SKU coverage
• Multiple paths through UI
• CodeModel path around UI
• Localization; Data handling and UI
• SxS, Compatability
• Performance
• XP Logo 

Process:

• Theory
  • Schedule and cost for the milestone before you begin (sounds like a line from Willy Wonka: [school teacher] “…switch our Friday schedule to Monday, which means that the test we take each Friday on what we learned during the week will now take place on Monday before we’ve learned it.”)  Update schedule as milestone progresses to measure against target dates.
  • Review feature specifications from PM team, providing feedback (expect updates during cycle).
  • Review implementation specification from Dev team, providing feedback (updates expected).
  • QA drafts high level test plan and submit to review by Dev and PM teams; update.
    • Reference previous product and similar feature test plans
    • Incorporate customer feedback and market expectations for setting priorities
    • Test team contracts attempt to prevent assumptions from leading to test holes across team boundries
  • Author test cases according to test plan; update plan as needed.
  • Author test case automation
  • Run automation test passes on coverage matrix
  • Run manual test passes on matrix subset
  • Log bugs, analyze trends, verify fixes.
  • Analyze automation code coverage (block level) results from instrumented builds.
    • Combine results with test plan holes to target next automation set.
  • ‘Dogfood’ the product
    • (Semi-)stable builds are selected by individual testers to use for developing automation (some data driven, some code).
    • App week:  break into teams and develop short term projects (some become team tools).
  • Themed bug hunts:  “Bug Bash”
    • team targeting of a specific Logo, accessibility, localization,.... categories.
    • Periodic re-education of feature owners on testing aspects
  • Iterate and ship!
• Reality, as enforced by those pesky J market needs and restrictions…
  • Level of clear specifications varies greatly by feature and product cycle.  This is not something QA can (historically) count on.  Working closely with the PM team allows for the necessary information to be focused on creating effective test plans.  Various unexpected needs (from marketing to division to group to feature) continuously challenge the proposed schedule (you know, that thing we all agreed to before the first feature specification was actually written?).  Adapt and overcome; priority trade off.
  • Implementation plans tend to be the actual code from Dev.  Monitoring check-ins and dev check-in test reviews helps to prepare for proper coverage
  • Some high level test plans get bogged down from lack of feature specifications and test cases develop ad hoc during exploratory test passes.
  • Authoring automation competes with running manual test cases.  The need to report the current product status, in a sense, delays the ability to report a more complete status more often... (Catch 22, but without the airplanes).
  • Automation requires a test harness. 
    • A test harness requires development and maintenance.
    • VC has moved from an internal tool to a divisional tool
      • Still requires development/maintenance participation
      • Supporting/following the gander isn’t what the goose used to do
  • Coverage matrix bulges and balloons;  OS varieties, SKU variations, Side by side installations, run time compatibility issues, pre-release forks, internal drops, priority customer drops, priority customer quick fixes, alphas, betas, somewhere in there we have the primary branch we plan to ship…
  • Again and again activities we want to do (Pri 2 and 3 test cases; driving pri 2 and 3 bug fixes; driving pri 2 and 3 features) are restricted by schedule needs (competing in the market).
  • dependencies get out of sync and one team may be blocked on another team  

Next installment:  Tactics...

[Ron]

A couple of questions from Woon Kiat to my previous post on the different types of testing we do on our C++ compiler:

>>Is understanding the inner working of a compiler vital to compiler testing?

I would say so, at least at some level.  In a general sense, I would say it's hard to be really successful at testing anything without understanding the domain area of what you are testing.

That said, I don't think many folks came to our compiler test teams with a lot of understanding of the inner workings of a compiler, especially the back end.  This is something that we pick up from doing the work.

>>How would you verify the binary generated from the compiler is correct? Is it fall under Correctness Tests?

In general, we write programs to expose the behavior that demonstrate the particular compiler behavior we expect to see from the feature we are testing.  This does fall into the category of correctness tests.  I'll try to include some examples of how we do this in my next post on C++ compiler testing.

Thanks for the questions, Woon!

[Michael] I've got a circuitious style - so if you're impatient, you may want to skip this thread. I'm going to talk about one of the big reasons I'm participating in this blog, and what I'm hoping to gain from it. I'm also big on analogies and stories - and as a fan or Robert Fulghum, I hope this one all makes sense in the end.

I went to a physical therapist a while ago - had some pain in my shoulders and arm. I was in a sour mood. Wasn't as productive at work as I wanted to be. I got there and we talked. I complained a lot about my shoulder and arms. Then a funny thing happened. The PT felt around for the tension, listened to what I had to say about that, and we started working on my lower back. After a while, the focus shifted onto my shoulders and arm and neck. However, most of the time was spent on my back. It wasn't until I was there that I realized that my pain was really in my back.

My back had hurt so long that it had stopped registering with my mind that it was a problem. My neck and arms were compensating for the back pain. I'm glad that the PT didn't just work on my neck and arms. If that had been the case, I'd be right back where I started from. But if the PT has just done what I said without establishing a real dialogue with me, we wouldn't have solved my real problem.

There is something different about the information you get from a simple statement and the information you get by "being there" or "being in touch". There is a lot of context missing from one off communication.

This is one of the reasons I hope this blogging experiment will work.

We have charts and graphs and concerns and suggestions filling up databases. We often have more information than we could practically sort through. And we try. But it doesn't tell the whole story. Without the right context, we might not make as good a decision as we could have with the context customers can provide - if there is real dialogue.

Hopefully, by being out here we can establish a useful and meaningful dialog. Blogs seem to be a compromise in a way - they can't scale if everyone who could read one asked a personal question - but hopefully it is a step toward establishing a strong dialog that will provide us and you, our readers, the context to understand each other. And hopefully solve some problems and relieve some customer "pain".
That's what my opinion of the job of Visual Studio is. To make it easier for you guys to write your applications.

Dialog works both ways too. If you don't know what we're doing, or why, then you probably won't be very eager to buy our product. You might feel like we aren't paying attention to your needs. I hope, I really do, that we're spending more of our time working on your problems than your symptoms. Please let us know if that's not the case.