You have already used the bug tracking system (WIT) in TFS before. So, we integrated this bug tracking system with MTR so that you can file bugs with a single click from the runner itself. What's the big deal about this you might ask?
Well, the big deal is that we not just create a bug for you from the runner itself, we also populate it with the test case that you have just been executing when you opened the bug. And that too in this neat little repro steps control with a tab of its own in the bug form. (Yes, we modified the MSF template to include some extra tabs in the default bug) So, all comments that the tester has written, notes she has taken, attachments that she has made are all uploaded automatically to the bug from the first step to the last failed step.
Now, how often have you hit a bug during testing that would just repro on your test machine but simply would not repro on the dev machine. And how often has it turned out that it was a configuration related issue - different service packs, different CPU usage, different memories, the list is endless. To ensure that the developer gets a complete picture of the environment where the bug was actually produced, we fill in system information about the test machine automatically into the newly created bug. Check out the "system info" tab within the bug form:
Now, even with all of these, what if the developer still cannot repro the bug? Perhaps it was a domain expert that filed the bug and not a technical tester. He/she may not have noticed subtleties that might have affected the system and caused the bug. Or the bug repro steps or description maybe woefully inadequate for the developer to produce a reasonable repro. In such cases, the video recording that the runner makes in the background during all test execution, will be considerably useful. Playing back the video will give the developer a very detailed and accurate picture of the exact steps leading up to the bug. This video is filed as an attachment to the bug
What if the test itself took 10 long minutes. Now, the failure is in the last step perhaps. The developer may want to look at only the failed step instead of the whole long video. Video markers are the way to go for that. Each time the tester marks a test step in the test case, a video marker is inserted with that timestamp in the video file. These markers are then populated against the test steps in the repro steps of a bug filed via the runner. When I click on these markers, I can play back from the point where the previous step was marked. This gives the developer finer control over what portions of the video file may be useful.
So, how do you like our actionable bug? Are there any other ways we could help developers get the maximum effective amount of information from the bugs?
Right after we RTM-ed Orcas last Monday, we are out with the November CTP of Rosario VSTS. Go on over here and give it a spin. We have a ton of stuff that's new with VSTS in there - more on BHarry's blog.
But, of course, I am excited about the manual test and AFN portions that we built out of my team here. Check out the awesome background recording that we do while you are out executing your manual tests in MTR. Next time you want to run the same test, just "play back" the recording of your manual tests. We support web apps for now, winforms is on the way...
We've also re-done the test case manager from the Aug CTP - there is now this sleek and stylish stand-alone test case manager where you manage all test activity from. We figure Visual Studio may not be the best environment for testers to go do all of this work from - so we have it completely stand alone now. Our team blog has a nice walkthrough of the TCM features - do skim through the post.
So, what are you waiting for? Go try it already and tell us what you think...
PS: Oh, given the huge size of the VPC, you might just want to install it in convenient parts as provided on the download page.
So, we've been heads down building the latest Rosario CTP of VSTT for you guys. Some good new stuff heading your way... more details when we release it ;-)
Meanwhile, like you must be knowing already, Orcas is on it's way to RTM. VSTT has some great stuff lined up on the load testing side - do check it out. And as with all releases, we are waiting nervously for this one to get out of the door safe and sound. Onto Rosario then!
Among other news, we had an EE day here at IDC, where we get to hear about cool new tech/processes emerging in other parts of the company. MSFT, being as big as it is currently, we certainly need a forum where we can share and learn new stuff across teams. The keynote was very Sinofsky - enjoyable and interesting as always. Peter Spiro did a fiery session as well on day 2 - where he spoke about his journey inside and outside of Microsoft. If I were given a 100 rupees for each time he swore, I'd be rich! LoL! Hard hitting and candid as usual - this guy inspires awe from even the most paranoid of people.
Alan Page gave a talk on testability, that was pretty neat. He also did this course for senior testers; that I'd highly recommend... lots of useful nuggets stringed well and sprinkled with some amusing and interesting anecdotes from him! And it was good seeing him in person - now I have a face to match the blog/mail/IM :)
Oh - and there will be a couple of guest posts from folks at my team on Rosario bits. They wanna test drive blogging and check if it's really their cup of tea...so my blog will be their guinea pig for now :)
Yes - it was bound to happen - sooner rather than later. Brian Harry is now Technical Fellow at Microsoft!! Totally kickass!
For the uninitiated, here's a description of what Technical Fellow is: Technical Fellows represent the highest stage of technical achievement at Microsoft. To be a Technical Fellow at Microsoft, one must truly be a thought leader and drive innovation that has industry-wide influence and impact. We expect these most senior technical leaders to embody the best qualities of technical leadership, innovation, communication, results and customer focus.
I remember the first time I interacted with Brian - I had just joined MSFT right out of college then. I was involved in a flame war with him because he resolved a security bug I filed as "By Design". It hurt my ego greatly and I had to get even with the moronic dev who didn't understand security was a prime concern for product development at MSFT <snigger> So, in the flames that ensued, Brian promptly demonstrated how fixing that "bug" would actually mean locking an admin out of his own system :) Well, well, did I say moronic?
From writing blogs on the simplest facets of the product to driving transparency with customers to levels that DevDiv has never seen before, his community engagement is truly exemplary. His ultra close involvement with the product code is well known the org. I mean, this guy still checks in code and does code reviews on a regular basis along with writing some nifty tools in between. He personally publishes our TFS dogfood stats and also updates it on his blog regularly. I guess his day consists of 48 hours.
Well, in a nutshell, I'm delighted that we have him here at Team System! Congrats Brian! Rock on!
Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester and make her life much easier than it currently is. The primary intent of the MTR is to help the manual tester create an actionable bug with which the developer is much more empowered to debug issues without the bug ping-pong happening between the dev and tester. Like I have mentioned before in my blog, it is pretty tedious for a tester today to even capture screenshots while filing bugs. MTR gives the tester one-click screenshot capture which operates in 3 different convenient modes and attaches itself to the test result as well. There is also a one-click bug filing option in which the bug is pre-populated with test step details from the test case that has already been loaded up in the MTR.
In this blog entry, I will describe the basic end to end scenario using the MTR to execute test cases and publish test results. The last entry, we saw how to create test cases and test runs in the TCM client. To launch MTR, open the VS command prompt, and type msrun. This launches the runner window on your desktop. Here, point to File->Open. You will be prompted with a dialog to connect to a TCM server similar to the "Connect to server" dialog that pops up when you open the team explorer window to connect to a TFS server. Choose the TCM server (in case of the VPC, the server is on the same machine) and choose the team project on the server.
Now, you will see the "Open Test Run" dialog, where you can pick any test run that you want. Let's choose a test run and say OK.
The runner window is populated with the test cases that are the result of the query associated with the run and selected in the dialog. Now, you can open your application under test (aut) and go about executing the tests that you loaded in the run. You can mark each test step with results of pass/fail or inconclusive through multiple ways including a handy single click.
You can enter comments in the comments text box or add attachments using the "Attach" option in the menu. Once the test steps are marked, click on "End test case" to compute the result of the test case from the test step results. You could also mark test cases globally without getting into the test steps at all. And if dissatisfied, you can just right click on the test case and say "Reset test case" to erase all results marked till now and start afresh. Once the test case is complete, you can publish the test case result to the server through the "Publish" button on the toolbar. These results are then published to the server and the test case grayed out to prevent further marking of steps.
In future entries, we will look at screenshots and bug filing in more detail. And of course, our glitzy video recording functionality will feature in its own entry as well. :)
Rosario has test case management in a full fledged fashion as part of the VSTST feature set. So, now test cases are first class citizens in the VSTS world with them having their own exclusive test case manager tool window inside the IDE. Test cases are a separate work item type with pre-defined fields like Owner, Priority, Test steps etc.
Click on the Test->Windows->Test Case Manager option in the IDE to launch the test case manager window shown at the right most side of the IDE below. The test case manager shows the selected team project TestTeamProject with its set of test cases, test run definitions, test runs. You can create a new test case from the TCM window. A new test case pops up with the test case form.
The test case form has a "Steps" tab as shown below:
You can enter descriptive steps for the test case detailing all the actions that need to be performed as part of the test case. You can mark a test step as "Validate step" explicitly using the icon shown in the pic (highlighted using a red dot next to the icon). The test steps shown are also parameterized using data driving. More on that in later posts. Now, you can define a test case query akin to the work item query. All those defined queries are shown in the test cases node in the TCM tool window as shown above.
With the test cases created, you can also create a run definition with the test case query specified as shown in screen shot below. This run definition is used to create a logical cluster of test cases - say tests for a certain component/feature etc.
Now, you can instantiate a new test run using the test run definition created above. The test run can be associated with a build so that the test run results can be published against the build.
Once the test run is created, you can open the test run from the TCM by querying all test runs. The current state of another sample test run in a graph is as shown below:
Now that we have a test case, run definition and test run created, we will delve into using MTR to execute our test cases in further blog posts.
If you still(!!!) haven't heard, Rosario CTP is out in the form of a VPC here :) Doug's whitepaper should give you enough info to take it out for a spin.
Manual Test Runner(MTR) is a key part of the CTP. It is a tool designed to help manual testers to load up their test plans and execute those with the support of additional tools like one-click screenshot, one-click bug with pre-filled repro and more such exciting stuff. Test case management(TCM) is another key piece where we have tried to fill the missing piece in the existing VSTT SKU. There is a tool window right inside VS IDE where you can create/modify/manage test cases, test run definitions and test runs and save them in the TCM server which has your team project's source code and work items. Then, load up the test runs in MTR and go. Publish the test results back into the TCM server and look at the cool reports that show how your test team is doing.
Do check it out and tell me how it goes!
Rosario is the next in line version for Team System after Orcas. And we are doing this CTP when Orcas has not even RTM-ed yet! Isn't that pretty cool? It's been a bit crazy managing multiple releases, but then, we want Rosario to hit the shelves in a short time after Orcas releases - faster new release. And no - I am not allowed to define "short time" :P
I've been working on the VSTT portions of this CTP for some time now and this is certainly one of the most exciting set of features I have seen for testers. Though I cannot wait to talk about all the awesome new stuff in the release, I'll not give out any more info until the CTP is out. Stay tuned for more...
We'll put it out as a VPC download on our website so that there are no installs that you need to do separately. Just set up the VPC and go...
Since I am a tester on the Test edition of VSTS - I get this q a lot of times: "Do you use VSTST for testing VSTST?" And the answer is "You bet, I do!" :)
As you might know already, this is called "Dogfooding". Many Microsoft products are dogfooded extensively before release - Vista, MSN Search, Visual Studio, Team System...
Our devs use unit test types to write their developer regression tests. Our QA team uses unit tests to write our functional tests. Yes, I know this is not a perfect fit for UI heavy testing, but it does suffice for the time being. We use generic tests to wrap our legacy tests that may be scripts or batch files. Load tests are added to the test bed to do stress testing on the app. All of this goes into the TFS server that we have deployed for the entire Developer Division and testing performed out of it.
Like any other user, I have my own set of pet features and pet peeves in TST. I absolutely love the code coverage coloring, the "Run test" from the editor (Orcas feature), the ease with which load tests can be configured and the controller-agent functionality. It makes my life so much easier as a tester. Not to mention that it's incredibly easy to point out to devs where the problems are and associate results with bugs since we use TFS for work item tracking. Now for the bad parts - I think the test list feature is absolutely useless for large test projects. The UI confuses me - more than once, I have selected a test in the window, right clicked and said "Run checked tests", but some other tests start running!! That's because selecting a run is not the same as checking it. :( I have a wishlist as well - stuff like test case management being easier to handle, the test run results being easier to analyze etc.
The best part of dogfooding is that this is a good chance to know which features are likely to be real relief and which ones are mostly going to tick the users off. Testers being the first customers of the tool is literally true in this case. Plus the QA team gets additional testing in the process so that bugs/suggestions are filed before outside users get a chance to go at it.
So, do you eat your own dogfood? ;-)
Well known test consultant James Bach has now coined a new term for manual testing - it's called sapient testing. Well, the part of manual testing that is not drudgery is called sapient testing at least. James makes a great point about how manual testing is actually a composite set of actions that can be divided into sections that CAN be automated and sections that can't. Yeah - I agree manual testing is a pretty overloaded term that way. Go read the article to find a lucid articulation of the whole manual testing process.
Most of us testers usually crib about not having a common language/jargon to talk to other testers. I have seen zillions of discussions about how QA and QM and testing are all completely different things. Innocent sounding words like test case and scenario are rendered ambiguous too. However, I wonder if we are collectively heading to the other part of the spectrum with too much terminology to remember? Trust us to solve one problem to lead to another :)
Hey - but the next time you talk to me about sapient testing, at least I won't give you a blank look :D
Technorati Tags:
manual testing
There are some really sweet videos over on the asp.net site to help kick start users of VSTS. The videos are typically 8-10 minutes long and to-the-point. Very useful for those that are looking for quick starters or help with a specific scenario. Personally, I love these kind of videos as compared to the lengthy hour long tutorials and talks. Short and fast - gets me started right away.
Eric Lee gives a short intro to unit testing with VSTS as well as using code coverage inside of VS. There are some good videos on load testing and web application perf profiling by Chris Menegay as well. Check out the videos and let us know if you want any specific videos up!
TechMela is the "biggest technical event of the year" in India and will replace the Tech Ed events that Microsoft traditionally had in many cities in India. I'll be part of a VSTT talk on 16th June, specifically talking more about how we use our own Team System features internally. Brian Harry does a fantastic job of bringing transparency into this with his mind boggling set of stats that he uploads monthly on his blog. I'll extend into how we dogfood TST as well - so testers outside of MSFT can get a peek at how we use our own products for testing. Abhishek Mathur, our PM will present the new Orcas features of VSTST as well.
So, do drop by to say hi or rave/rant about VSTS if you are planning on attending. (Yeah - I know, even though my blog maybe read by a grand 2 and a half readers, I still wrote this to humour myself :)
OMG! This has got to be the best blog writer tool I have ever seen! Forgive my drooling...but Live writer has just enchanted me with it's great features.
The setup is a ZERO hassle process akin to the marvelous setup experiences I've had with Outlook 2007. The rich editing and inline spell check features are a blast and the html has got to be the cleanest ever. Removing the extra bekaar tags that get inserted with other authoring tools was a sufficient deterrent to stop me from continuing to use those. The UI looks ultra-sleek and gives a squeaky-clean feeling. Recent posts show up and allow editing/preview with single clicks. Sweet!
Of course, I went berserk trying different plugins - and as with most products, some of the plugins seem to be more like features that fit in perfectly but did not figure into the prioritized list for the tool. Most importantly, I lovvvvved the "insert picture" feature - that has got to be the NUMBER ONE problem that most bloggers I know seemed to face. It's such a breeze now! And just for kicks, here is one of my favorite pics - Partha with a friend.
Thanks, Abhinaba and Srivatsn for pointing me to it. 
(oh - I just wanted to use the smiley from ScottIsAFool's plugin 
)
When I see some of the tools available to "aid" testers, I alternate between shock and anger and disappointment! Don't get me wrong - there are some wonderful tools out there; but soooo many tools seem to make an assumption that the tester is more or less a moron who cannot point out an error when he sees one! I am sure many folks resonate with this crib - here is one.
The last discussion that we had about video recording is in my mind generated a conflict in my mind as well. As an oversimplification, let's divide testers into 2 categories: those that debug and those that won't. If I offer a feature like "video-recording-inserted-in-bug" to the former kind, on its own, it's almost always a waste. When I see an issue that does not repro, I try to bunch all invariants and eliminate the changing factors one by one to arrive at the culprit factor that is triggering the bug. In that case, a .wmv file by itself is useless. I'll need data like proc usage, mem usage, env vars set at the time etc. to arrive at the answer. Instead of doing that, if I just attach a video, it hardly helps the dev either - no one benefits.
But consider the latter category: these maybe domain experts, tech support, beta testers, bug bashers - basically a set that may have debugging skills, but are not willing to use them in this situation or a set that do not have debugging skills at all. In that case, attaching a video file may be helpful to recount parts of the bug tale that the testers may not have observed. The video clubbed with system information captured automatically as well as event logs form a powerful set of debugging aids for the developer.
But, abuse of these features can often lead to tester skills being underdeveloped. For instance, if a tester attaches the video and says "Okay - here is the video. I dunno if it were the mouse middle click or the multi monitor or the fact that my CPU was 90% busy that caused the crash; but hey, use the video and go figure." That is a pretty stupid thing to do. Essentially, the feature ends up making the tester hesitant to narrow repro steps, debug the issue or even think up similar cases. You are making my testers dumb!! Or worse, lazy! Thankfully, this is a minor problem where just regulating the QA processes comes in handy. From what Bj said in the last post, a product tester would probably have debugged more into the issue, tried real world testing and found the cause of the bug. But as Chris and Sumod point out, perhaps the repro is coming in from people that are not interested in/don't have time for debugging into this issue at all.
With testers coming from such a wide range of background/experience and requirements, I find the current testing tools available in the market to be woefully inadequate as well as ill targeted (both in the automation and manual testing space). Do write to me about reqs that you have had a tester that have not been addressed by any testing tool at all - perhaps we can collectively devise a solution for our common problems.
And as a postscript, please don't take the title too literally and flame me and accuse me for calling testers that do not want to debug as "dumb" - that's certainly not my intent.
Have you ever video recorded your desktop while executing a test case and attached the video as a repro in your bug to the dev? I haven't. I agree it sounds cool to be able to do it, but in 4 years of testing, I have never seen anyone do it either. Most people I talk to think it's cool and proffer possible scenarios of usage, but no one seems to have ever used it at all.
Can someone who has used desktop recording and attached video files in their bug reports please reply or leave a comment as to why you guys love this concept? All feedback greatly appreciated!
Other senior testers I spoke to at MSFT seem to have a divided opinion on its usefulness, but I could not locate anyone who actually used it regularly. So, I spent some time thinking... why would I ever attach a video file in my bug repro? Because I couldn't repro the issue again? Ummm....unlikely. If I can't repro the issue I saw 5 mins ago, I couldn't possibly expect my dev to do that looking at a lousy video. Perhaps the video'll help me investigate the issue myself? Yeah - to a certain extent. Armed with the system info, event logs and stack trace info, this will be a great additional supplement. Also, if I were doing exploratory testing, it'll certainly help me to remember how I arrived at the buggy state in my system and perhaps, help me trace that path again if I am lost. But like I mentioned above, so far, I have been satisfied with system info, event logs and system logs to point me to the buggy state. I see tools like Camtasia and Test Explorer even have audio recording thrown in along with desktop recording. So, someone has gotta be using these features!! Will that someone help me please?