Five Testers From VC : VC IDE Testing

VC IDE Testing Tactics (and a beta1 plug!)

FiveTestersFromVc — Fri, 30 Jul 2004 03:30:00 GMT

[Rob]

You've probably had enough time to read the last post...Ok, so maybe we were reallllly busy getting our beta product release squared away so you can all get your free Visual Studio 2005 Express Beta1 downloads! Grab it, have fun, win an Xbox, tell me about what was good and what needs more! (There are also versions focusing on other programming languages and even web development!). http://lab.msdn.microsoft.com/express/default.aspx

***

Tactical thoughts

Thinking about all the things that go into testing feature...is the API working? Are there any attack paths through this feature (is it secure)? Does it integrate with existing features? Does it depend on features (our group or another?!) that might change before lock down? Is the UI compliant with suite requirements? Is the UI clear? Are UI redundancies appropriate? What will our users think about our answers to these questions? What happens to the UI when the product is localized (translated to another language)? What happens to the behavior of the feature when the product is localized? (Add a number of other questions from networking to user permission levels depending on the feature category...).

The list appears endless, but luckily we have limited time (some sarcasm on that use of 'luckily', if you didn't notice) to do our testing so we work with a finite set that makes sense for the feature in question.

The highest urgency task is to get the feature tested for main stream functionality. What will our customers need this to do? What will our customers want this to do? What assumptions may be made that could get this feature into trouble (and put the user into a confused or frustrated state)?

We also want to see that the feature discoverable and useable. The UI needs to indicate what will happen and let users know about progress if this takes time. Keeping UI consistent so different tasks at least 'feel' familiar in the welcoming sense is valuable...but difficult when taking legacy features into a new arena (a challenge for many still useful feature sets now housed in a building of .NET splendor). Is there a point to upgrading old UI to match the new style if the old features are stable as is (and use the resources instead for new features and to fix bugs)? Is old UI style a bug or an acceptable artifact of evolution?

These questions are what test cases are all about. Each test case seeks an answer. Customer feedback, experience, and research help us to ask the right questions for our current product. Once we have a base set of questions (Test Case Planning is a cyclical process, just as implementation is) we can move forward to measure the status of product drops based on the answers we get on test passes. We build our automation library which allows us to gather answers faster and feel more secure against regressions that could be introduced every time developers touch the product. At the end of the cycle, we have an automation set that allows us to have a level of confidence that sustained engineering will be able to do service pack work while maintaining a high level of regression test coverage (while the front line team moves to the next generation product).

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

well, since you asked twice...

FiveTestersFromVc — Fri, 28 May 2004 03:05:00 GMT

[Rob]

Woon asked in his comment about the % automation we have with our IDE tests.

Currently, we are at roughly 50% automated in the IDE compared to the total number of active test cases. These tests range from DTE manipulations to actual UI interations the way the user would (drag and drop, button clicking, resizing, ...). One way to get at the UI elements is through MSAA. This works fairly well for standard controls. Custom controls make the story more difficult (which is why we resourt to DTE calls if our test harness can't handle certain controls).

It takes about 2-3 weeks for 15 people to cover all the manual test cases we currently have for the VC IDE. And this doesn't include common shell features that the VS team handles.

Our automation's primary purpose is to give us rapid general status on build drops each day (although our automation is split up between daily and weekly test sets). The automation set does find bugs, but by far the most bugs are found through manual testing of new features. Automation is our 'security blanket' as we can get a breadth of coverage in 24 hrs with a second depth after ~72 hours (looking for regressions from the previous known state) rather than waiting for three weeks (which also uses up the whole team's attention) with manual. Both are vital.

VC IDE Testing: cycle check points

FiveTestersFromVc — Fri, 16 Apr 2004 04:14:00 GMT

[Rob]

General IDE testing cycle status points (iterations on/between any two points…)

Specification review
Test Plan Design
Test Plan Review
Developer Implementation design
Developer Implementation design review
Test case design [exploratory, user feedback, regression]
Automation Design
Automation Implementation and review
Manual Test Pass
Automation run
Project status evaluation and reporting

Since the organization is broken up between disciplines (Program management, Development, and Test), cross discipline communication helps deal with the implied dependencies in the eleven points above.

A typical cycle contains zero or more of the following obstacles based on the above points:

PM (program manager group) delivers spec(s) late, preventing review by schedule
test plans delayed by changing specs/parallel product cycle effort lowering priority
test plan reviews are cursory, resulting in missed holes fund later which cost more to correct
the design is the code
well, we can review the code…or just use the feature…
test cases ended up being designed without complete specifications…requires revisions regularly at added cost
automation gets pushed back due to UI/feature churn making tests obsolete an hour before they are checked in
Automation harness issues; changing requirements; dogfooding delays (a good thing, but causes progress trade-off) [dogfooding: using VC drops to develop automation for testing VC drops…]
Manual Test Pass: can always count on them, but they take too long if automation not strategic or complete (temptation is to delay automation to get complete manual coverage creating an ever deepening hole…bite the bullet and do the automation right so manual is only what can’t/shouldn’t be automated).
matrix explosion (platforms * SKU’s * architectures * permission levels; pair wise testing save us!)
What do you mean, “make it green”?! [balancing quality driven vs. date driven product requirements]

Some random elaborations:

There’s a lot of overlap at the beginning/end of product cycles. This really prevents any one discipline from getting a ‘fresh start’, as wrap up work often takes priority over new work (reverse that for PM). Thus, the new work gets delayed quite often.

A note on ‘priority’: Everything is ‘pri 1’, because if it wasn’t, it was cut. This is because there isn’t enough time to do everything that ‘has to happen before we ship’. So, ‘priority’ has no meaning. Just ignore it. Treat it as someone saying, ‘um…’ or ‘er…’. I’m making it a priority to remove that word from tester vocabulary….

At one point, our team had fallen significantly behind our goals for automation development. We found ourselves in a position where we needed to keep testing up on implementation churn by providing immediate testing of any dev patch/check in. This constant impromptu testing continuously delayed automation development between formal test passes. We are in a much better state at this time because we followed through on the difficult decision (made with the dev org) to forego the interrupt driven manual testing completely for several milestones and focus on automation (formal test passes still occurred, of course, to cover all test cases for each milestone).

One angle which has been used with automating IDE features can be summed up with a call to “do everything as the user would”. Reality set in as conflicts with shared UI continuously broke tests over multiple groups. Since feature innovation can’t be stalled just to maintain an automation test bed, it became necessary to follow an academic rule: “a test should test only what it tests”. In other words, if you aren’t specifically testing the UI, if at all possible, the test should not fail because of a UI change. Praise be the DTE! (http://msdn.microsoft.com/library/en-us/vsintro7/html/vxgrfAutomationObjectModelChart.asp). For visual studio, feature area automation authors can often bypass UI that they don’t need to directly test and get at what does need to be addressed with the current test. This has made automation faster and more reliable for our current test harness.

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

What kind of people are testing this stuff anyhow?

FiveTestersFromVc — Fri, 26 Mar 2004 03:58:00 GMT

[Rob] I know I said the next installment would be on tactics…but I think in response to feedback Woon Kiat I’ll go with ‘resources' first.

Before that, very quickly, there was a comment from AT on alpha/beta users submitting automation test cases along with bugs. Generally speaking, repro steps for bugs are added to the regression test case set and automated where appropriate. Directly accepting scripts, however, would cause security concerns. It would be a nice world though to just be able to build a customer ‘drop acceptance’ automation test case database… J Now you have me thinking about a tool to allow steps to be entered through a web page that would build a safe script to run against the product...hmmmm….I could build that using the VC++ IDE once I figure out the design…

Resources for Test in VC

Test candidates right out of school rarely have formal test training. The experience level is also limited to personal code and maybe a few small group projects. This gives us the challenge of identifying those candidates that have a rough talent for ‘finding bugs’ or ‘breaking things’ J along with a way to navigate through a product and identify where priority bugs may hide. We also look for people who can adapt quickly and apply techniques ‘just learned’ to new problem areas.

We write test automation and our product is a development tool. Therefore we also look for people who have development proficiency with C++ (knowledge, style, problem solving, … pretty much our customer base).

Our company has training courses (all disciplines) that supplement on the job experience. Everyone on the team is a mentor to some degree or another for every new hire we get. Formal and informal mentors are encouraged in addition to an individual’s team lead. Acquiring a mentor from outside the team is one way we continue to mix ideas, technologies, and techniques throughout the company. We also like to occasionally reorganize teams to allow people to move to new feature areas and ‘stay fresh’ by providing new challenges.

Product specific training is hands on. For Visual C++ we focus heavily on dogfooding (use the current drops for real world coding of team tools and personal hobby projects) the product. Plenty of texts exist on previous versions of the product to help people understand where we’ve been. Specification documents (or, when not available, talks with program managers and feature developers) bring us up to date for what to expect in the current version. When one is new to a feature area, reviewing the old test case base and the bug database helps bring you up to date on the current status and where testing has gone before. Particularly interesting are those bugs found by customers, as they show where customers went with the product that the previous test plans did not anticipate (allowing us to learn better what to give priority to in the limited time we have to test).

Some books found on my office shelf today (more at home where I have a bigger shelf…oh, and I’m not saying these are all ‘fantastic’ books, though some are, I’m just giving a snapshot of what’s behind me):

Various language books (C++, VB, C#)

Various manuals from internal microsoft training courses

Various manuals from Test conferences (most recent: PNSQC and SQE)

Debugging Applications for .NET and Windows, Robbins

Lessons Learned in Software Testing, Kaner et al

How to Break Software, Whittaker

Design Patterns, Gamma et al

Quality Software Management, Weinberg

Writing Secure Code, Howard et al

Automation Programmer’s Reference, Microsoft Press

Applied Microsoft .NET Framework Programming, Richter

Programming with Visual C++ .NET, Shepherd et al

Programming Windows, Petzold

Inside COM, Rogerson

;-)

[This posting is provided "AS IS" with no warranties, and confers no rights. Use of any included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.]

VC IDE testing (issues and processes)

FiveTestersFromVc — Fri, 19 Mar 2004 04:26:00 GMT

[rob] Since Ron has been entering thoughts and comments on compiler testing, I’d like to add a perspective on IDE testing from VC. I’ll leave it in notation format to encourage discussion of points for which readers would like more clarification.

Issues to address through IDE testing:

Changing specifications over time (some features are completely ambiguous until devs implement, others have varying levels of specs prior to implementation)
Library of regression test cases, authors having moved to other teams; QA work is more cumulative than dev work in that regression testing currently takes a larger toll than code base maintenance.
Dependecies (teams depend on us, we depend on them)
Platform coverage
SKU coverage
Multiple paths through UI
CodeModel path around UI
Localization; Data handling and UI
SxS, Compatability
Performance
XP Logo

Process:

Theory
- Schedule and cost for the milestone before you begin (sounds like a line from Willy Wonka: [school teacher] “…switch our Friday schedule to Monday, which means that the test we take each Friday on what we learned during the week will now take place on Monday before we’ve learned it.”) Update schedule as milestone progresses to measure against target dates.
- Review feature specifications from PM team, providing feedback (expect updates during cycle).
- Review implementation specification from Dev team, providing feedback (updates expected).
- QA drafts high level test plan and submit to review by Dev and PM teams; update.
  - Reference previous product and similar feature test plans
  - Incorporate customer feedback and market expectations for setting priorities
  - Test team contracts attempt to prevent assumptions from leading to test holes across team boundries
- Author test cases according to test plan; update plan as needed.
- Author test case automation
- Run automation test passes on coverage matrix
- Run manual test passes on matrix subset
- Log bugs, analyze trends, verify fixes.
- Analyze automation code coverage (block level) results from instrumented builds.
  - Combine results with test plan holes to target next automation set.
- ‘Dogfood’ the product
  - (Semi-)stable builds are selected by individual testers to use for developing automation (some data driven, some code).
  - App week: break into teams and develop short term projects (some become team tools).
- Themed bug hunts: “Bug Bash”
  - team targeting of a specific Logo, accessibility, localization,.... categories.
  - Periodic re-education of feature owners on testing aspects
- Iterate and ship!
Reality, as enforced by those pesky J market needs and restrictions…
- Level of clear specifications varies greatly by feature and product cycle. This is not something QA can (historically) count on. Working closely with the PM team allows for the necessary information to be focused on creating effective test plans. Various unexpected needs (from marketing to division to group to feature) continuously challenge the proposed schedule (you know, that thing we all agreed to before the first feature specification was actually written?). Adapt and overcome; priority trade off.
- Implementation plans tend to be the actual code from Dev. Monitoring check-ins and dev check-in test reviews helps to prepare for proper coverage
- Some high level test plans get bogged down from lack of feature specifications and test cases develop ad hoc during exploratory test passes.
- Authoring automation competes with running manual test cases. The need to report the current product status, in a sense, delays the ability to report a more complete status more often... (Catch 22, but without the airplanes).
- Automation requires a test harness.
  - A test harness requires development and maintenance.
  - VC has moved from an internal tool to a divisional tool
    - Still requires development/maintenance participation
    - Supporting/following the gander isn’t what the goose used to do
- Coverage matrix bulges and balloons; OS varieties, SKU variations, Side by side installations, run time compatibility issues, pre-release forks, internal drops, priority customer drops, priority customer quick fixes, alphas, betas, somewhere in there we have the primary branch we plan to ship…
- Again and again activities we want to do (Pri 2 and 3 test cases; driving pri 2 and 3 bug fixes; driving pri 2 and 3 features) are restricted by schedule needs (competing in the market).
- dependencies get out of sync and one team may be blocked on another team

Next installment: Tactics...