Getting a video recording of your test to attach to the bug you file on your developer can be a really cool thing. And the Microsoft Test and Lab Manager tool makes it a breeze to do that.
However, if your MTLM is crashing or erroring out when you try to pick up a recording,try the following:
1. Check if WME encoder 9 is installed on the box. If not, install it from here
2. If you are running Vista, Windows 7 or Windows 2008, install the patch for the encoder from here
As Thomas Schissler, one of our most active Team System MVPs points out, the docs indicate this patch is required for Vista, but you need it for Win7 and W2K8 also
When testing UI applications, specially web apps, we frequently encounter the problem of window title changing with the state of the app or operations done on the app. For instance, when checking web mail, the title of the page could read “Inbox(35) - Microsoft Internet Explorer” where 35 really represents the number of unread email in your inbox. Now, when I open it at any other time, it might read “Inbox(36) - Microsoft Internet Explorer” based on unread mail. For GUI testers, this poses a problem since object identification works usually on the basis of window title and the title changing each time I run the test causes object identification to fail. :(
To solve this problem, the API of VSTS 2010 has a “Smart Match” feature built into it. Thej Kumar is our smart young developer that works on the record and play engine of Team Test and today, he will explain how smart match works. So, here you go:
Baseline assumptions:
- The assumption is that the title of the window is of the form <Title> - <Application Name> or <Application Name> - <Title>.
- The assumption is that the last hyphen separates the title from the application name.
How does the heuristic work:
- If the application name matches completely, we ignore it in the calculation of score for a window. In the above example, “Microsoft Internet Explorer” is a complete match. Hence the scores are calculated only for Inbox(35) and Inbox(36).
- Whichever window has the longest common string will have a higher score. Example: “Inbox, 45 unread” and “Inbox, 46 unread” will be a 50% match because the longest common string is 8 (Inbox, 4) out of the total length of 16. On the other hand, “Inbox, unread(45)” and “Inbox, unread(46)” will be excellent matches because the longest common string is 15 out of 17.
Special case for MDI windows:
- Title changes based on whether or not the MDI window is maximized or restored. Eg: if the title of the main window is A and the title of MDI window is B, if B is maximized, the title of A changes to A – [B]. To handle these cases, we give a high score to windows whose one half matches and the other half of only one of the windows is empty.
- If two or more windows get a good score, the window on the top is picked up.
Good things to do regarding titles to ensure smart match picks works correctly:
- Ensure that the last hyphen separates the title from the application name and in general, avoid too many hyphens in the title.
- Ensure that part of the title that changes is present either in the beginning or in the end.
- Ensure that the window which you want to be picked up is present on top of the other windows.
Sometimes on a rare occasion, smart match does end up being oversmart ;-) and gives false positives, but those scenarios can be worked around. Tell us what you think of smart match!
As I sit here @ Tech Ed 2009 in Hyderabad, that’s what I am wondering looking at the sparse audience in the first session of the day. (Yeah right, I was expecting half of the million developers to turn up here ;-) ) Sigh - given the current economic condition and the fact that Tech Ed is still a fairly new event in Hyderabad, the participation might just be expected to be not-so-high after all.
Anyway, AmitC gave a great demo of all the wonderful features we’ve built as part of VSTS 2010. It covered all our value propositions in breadth and there were several “wow” moments in the crowd when he demo-ed test impact, no more no repro and the virtualization capability added newly in VSTS 2010. Bunch of interesting qs from the audience around specific features in test.
SteveB’s keynote opened the talk – he talked a little bit about how IT is impacted by the current economy, gave a peek of the product lineup we have coming out this year from MSFT and answered a host of qs around Azure, Win7 and VSTS 2010. (Oh – our branding is now official for Test and Lab)
Neelesh gave a terrific talk on the load and web test features in Orcas. It is actually funny to see so many folks say they have VSTS 2008 Team Suite, but a large chunk of them not know any of the Team Test features. :-) Of course, with 2010, we have a complete first class test offering – so Team Test should really make a splash with our users. Looking forward to feedback on our Beta1…
Check out this totally kickass feature in VSTS 2010 aka Dev10 – the Coded UI Test (CUIT). This is a brand new test type in VSTT that lets users do functional UI automation. As GUI testers will already know, UI automation is one of the most tedious to write and get all the timing right. What CUIT will do for you is to automatically generate C# or VB.NET code from a series of recorded actions that you did while testing your app. Adding validations is a breeze – just point and click. Point to the object you want, pick the property you want to validate and type in the expected value – this validation gets automatically generated.
For more on CUIT, check out this cool video on Channel 9 – this shows an end to end demo of how to create and run a CUIT. Let me know what you think!
Check out this article in ITMagz.com that covers the latest trends in software testing. This is written by our very own industry expert, Tanuj Vohra that runs the Program Management team for VSTT.
Virtualization testing is something that has taken off pretty well lately - and we have been using it a fair bit internally at the empire too. With the Lab Management SKU being the latest addition to the VSTS family, we have been attempting to dogfood it in our test lab. Before the dogfood though, we have an internal team that's built a tool called Virtual Lab which can create disposable virtual machines super-fast based on similar technology. This tool has clearly upped the productivity of the team - no more fighting for repro machines, elaborate installs etc. Everyone has a host machine that can put up upto 4 VMs.
Tell me if your team uses virtualization to speed up your testing!
Here at MSFT, specifically at DevDiv, we have specific testing categories called "Exit Criteria" - basically gates that you need to pass through in order to ship a release at any milestone. These criteria represent different flavors of testing that your product has to go through in order to be pronounced "fit for release". For eg: localization, accessibility, performance. At DevDiv, we have 26 different exit criteria that we need to pass through before we ship!
Now, as is the case with any "process", this one has also been the subject of passionate debate among the MSFT denizens as to the value this really brings to our product quality. I have heard arguments around how this is a lot of overhead at a very busy time in the ship cycle. Well - I have to concede that these set of tests really do not bring in a lot of bugs. Especially, criteria like "branding and copyright" - where we just test to see if the product is carrying the right trademarks everywhere, the tool names are mentioned correctly etc. Also, when criteria like "configuration matrix" is in question - this is basically the gate where we ensure our products run ok on different operating systems, different versions of CLR, different versions of IE, different bitness...some of us testers tend to go berserk and many times, products get overtested. And of course, there is this eternal danger of manual testing (which is still the best source of the most important bugs) getting undermined and sacrificed at the altar of checking off items in a checklist.
In spite of all of this, I still opine that the exit criteria proc is one of the best engineering practices I have seen here at home. If you are a tester testing a product with a customer base numbering tens of thousands or more, I am sure you appreciate the sheer breadth of things you need to cover in your test planning. The number of dimensions is really mind boggling - security, performance, localization, globalization, backward compatibility, accessibility, stress, endurance, limit...it sure gets crazy. Exit criteria really bring a semblance of sanity in this whole universe of testing. I would term this the distilled essence of the different testing dimensions that go into testing a world class product. It serves as a map to help testers navigate the maze of testing requirements and builds confidence into the quality of the product step by step.
The Globalization EC for instance, is so very well thought out - we have different configurations like Italian, Turkish, Arabic; with each configuration defined by what is different in that config and what are the kind of bugs you can expect to hit. For instance, ITA is one of the "most localized" OSes, which help catch issues where we hardcode stuff like security group names. I remember Team Build had this bug in 2005, where we had hardcoded to check if user belongs to "Administrators"! The test passed on JPN, because the admin group was still "Administrators" on the JPN OS, but it failed on German OS, since the group was "Administrateren" on DEU! Turkish has the infamous sort problems with the quirky dotted/undotted I's. Testing on Arabic configs throw up specific problems rendering bidirectional text or right-to-left UI mirroring etc. To me, this is a treasure trove of info put together by several years of testing experience. Who wouldn't want to use this cheat sheet? :-)
More on exit criteria in later blogs.
As we get closer to shipping, it is getting more exciting here...hope you guys like Dev10...
You have already used the bug tracking system (WIT) in TFS before. So, we integrated this bug tracking system with MTR so that you can file bugs with a single click from the runner itself. What's the big deal about this you might ask?
Well, the big deal is that we not just create a bug for you from the runner itself, we also populate it with the test case that you have just been executing when you opened the bug. And that too in this neat little repro steps control with a tab of its own in the bug form. (Yes, we modified the MSF template to include some extra tabs in the default bug) So, all comments that the tester has written, notes she has taken, attachments that she has made are all uploaded automatically to the bug from the first step to the last failed step.
Now, how often have you hit a bug during testing that would just repro on your test machine but simply would not repro on the dev machine. And how often has it turned out that it was a configuration related issue - different service packs, different CPU usage, different memories, the list is endless. To ensure that the developer gets a complete picture of the environment where the bug was actually produced, we fill in system information about the test machine automatically into the newly created bug. Check out the "system info" tab within the bug form:
Now, even with all of these, what if the developer still cannot repro the bug? Perhaps it was a domain expert that filed the bug and not a technical tester. He/she may not have noticed subtleties that might have affected the system and caused the bug. Or the bug repro steps or description maybe woefully inadequate for the developer to produce a reasonable repro. In such cases, the video recording that the runner makes in the background during all test execution, will be considerably useful. Playing back the video will give the developer a very detailed and accurate picture of the exact steps leading up to the bug. This video is filed as an attachment to the bug
What if the test itself took 10 long minutes. Now, the failure is in the last step perhaps. The developer may want to look at only the failed step instead of the whole long video. Video markers are the way to go for that. Each time the tester marks a test step in the test case, a video marker is inserted with that timestamp in the video file. These markers are then populated against the test steps in the repro steps of a bug filed via the runner. When I click on these markers, I can play back from the point where the previous step was marked. This gives the developer finer control over what portions of the video file may be useful.
So, how do you like our actionable bug? Are there any other ways we could help developers get the maximum effective amount of information from the bugs?
Right after we RTM-ed Orcas last Monday, we are out with the November CTP of Rosario VSTS. Go on over here and give it a spin. We have a ton of stuff that's new with VSTS in there - more on BHarry's blog.
But, of course, I am excited about the manual test and AFN portions that we built out of my team here. Check out the awesome background recording that we do while you are out executing your manual tests in MTR. Next time you want to run the same test, just "play back" the recording of your manual tests. We support web apps for now, winforms is on the way...
We've also re-done the test case manager from the Aug CTP - there is now this sleek and stylish stand-alone test case manager where you manage all test activity from. We figure Visual Studio may not be the best environment for testers to go do all of this work from - so we have it completely stand alone now. Our team blog has a nice walkthrough of the TCM features - do skim through the post.
So, what are you waiting for? Go try it already and tell us what you think...
PS: Oh, given the huge size of the VPC, you might just want to install it in convenient parts as provided on the download page.
So, we've been heads down building the latest Rosario CTP of VSTT for you guys. Some good new stuff heading your way... more details when we release it ;-)
Meanwhile, like you must be knowing already, Orcas is on it's way to RTM. VSTT has some great stuff lined up on the load testing side - do check it out. And as with all releases, we are waiting nervously for this one to get out of the door safe and sound. Onto Rosario then!
Among other news, we had an EE day here at IDC, where we get to hear about cool new tech/processes emerging in other parts of the company. MSFT, being as big as it is currently, we certainly need a forum where we can share and learn new stuff across teams. The keynote was very Sinofsky - enjoyable and interesting as always. Peter Spiro did a fiery session as well on day 2 - where he spoke about his journey inside and outside of Microsoft. If I were given a 100 rupees for each time he swore, I'd be rich! LoL! Hard hitting and candid as usual - this guy inspires awe from even the most paranoid of people.
Alan Page gave a talk on testability, that was pretty neat. He also did this course for senior testers; that I'd highly recommend... lots of useful nuggets stringed well and sprinkled with some amusing and interesting anecdotes from him! And it was good seeing him in person - now I have a face to match the blog/mail/IM :)
Oh - and there will be a couple of guest posts from folks at my team on Rosario bits. They wanna test drive blogging and check if it's really their cup of tea...so my blog will be their guinea pig for now :)
Yes - it was bound to happen - sooner rather than later. Brian Harry is now Technical Fellow at Microsoft!! Totally kickass!
For the uninitiated, here's a description of what Technical Fellow is: Technical Fellows represent the highest stage of technical achievement at Microsoft. To be a Technical Fellow at Microsoft, one must truly be a thought leader and drive innovation that has industry-wide influence and impact. We expect these most senior technical leaders to embody the best qualities of technical leadership, innovation, communication, results and customer focus.
I remember the first time I interacted with Brian - I had just joined MSFT right out of college then. I was involved in a flame war with him because he resolved a security bug I filed as "By Design". It hurt my ego greatly and I had to get even with the moronic dev who didn't understand security was a prime concern for product development at MSFT <snigger> So, in the flames that ensued, Brian promptly demonstrated how fixing that "bug" would actually mean locking an admin out of his own system :) Well, well, did I say moronic?
From writing blogs on the simplest facets of the product to driving transparency with customers to levels that DevDiv has never seen before, his community engagement is truly exemplary. His ultra close involvement with the product code is well known the org. I mean, this guy still checks in code and does code reviews on a regular basis along with writing some nifty tools in between. He personally publishes our TFS dogfood stats and also updates it on his blog regularly. I guess his day consists of 48 hours.
Well, in a nutshell, I'm delighted that we have him here at Team System! Congrats Brian! Rock on!
Manual Test Runner (MTR) is one of the key portions of Rosario with which we are hoping to capture the imagination of the manual tester and make her life much easier than it currently is. The primary intent of the MTR is to help the manual tester create an actionable bug with which the developer is much more empowered to debug issues without the bug ping-pong happening between the dev and tester. Like I have mentioned before in my blog, it is pretty tedious for a tester today to even capture screenshots while filing bugs. MTR gives the tester one-click screenshot capture which operates in 3 different convenient modes and attaches itself to the test result as well. There is also a one-click bug filing option in which the bug is pre-populated with test step details from the test case that has already been loaded up in the MTR.
In this blog entry, I will describe the basic end to end scenario using the MTR to execute test cases and publish test results. The last entry, we saw how to create test cases and test runs in the TCM client. To launch MTR, open the VS command prompt, and type msrun. This launches the runner window on your desktop. Here, point to File->Open. You will be prompted with a dialog to connect to a TCM server similar to the "Connect to server" dialog that pops up when you open the team explorer window to connect to a TFS server. Choose the TCM server (in case of the VPC, the server is on the same machine) and choose the team project on the server.
Now, you will see the "Open Test Run" dialog, where you can pick any test run that you want. Let's choose a test run and say OK.
The runner window is populated with the test cases that are the result of the query associated with the run and selected in the dialog. Now, you can open your application under test (aut) and go about executing the tests that you loaded in the run. You can mark each test step with results of pass/fail or inconclusive through multiple ways including a handy single click.
You can enter comments in the comments text box or add attachments using the "Attach" option in the menu. Once the test steps are marked, click on "End test case" to compute the result of the test case from the test step results. You could also mark test cases globally without getting into the test steps at all. And if dissatisfied, you can just right click on the test case and say "Reset test case" to erase all results marked till now and start afresh. Once the test case is complete, you can publish the test case result to the server through the "Publish" button on the toolbar. These results are then published to the server and the test case grayed out to prevent further marking of steps.
In future entries, we will look at screenshots and bug filing in more detail. And of course, our glitzy video recording functionality will feature in its own entry as well. :)
Rosario has test case management in a full fledged fashion as part of the VSTST feature set. So, now test cases are first class citizens in the VSTS world with them having their own exclusive test case manager tool window inside the IDE. Test cases are a separate work item type with pre-defined fields like Owner, Priority, Test steps etc.
Click on the Test->Windows->Test Case Manager option in the IDE to launch the test case manager window shown at the right most side of the IDE below. The test case manager shows the selected team project TestTeamProject with its set of test cases, test run definitions, test runs. You can create a new test case from the TCM window. A new test case pops up with the test case form.
The test case form has a "Steps" tab as shown below:
You can enter descriptive steps for the test case detailing all the actions that need to be performed as part of the test case. You can mark a test step as "Validate step" explicitly using the icon shown in the pic (highlighted using a red dot next to the icon). The test steps shown are also parameterized using data driving. More on that in later posts. Now, you can define a test case query akin to the work item query. All those defined queries are shown in the test cases node in the TCM tool window as shown above.
With the test cases created, you can also create a run definition with the test case query specified as shown in screen shot below. This run definition is used to create a logical cluster of test cases - say tests for a certain component/feature etc.
Now, you can instantiate a new test run using the test run definition created above. The test run can be associated with a build so that the test run results can be published against the build.
Once the test run is created, you can open the test run from the TCM by querying all test runs. The current state of another sample test run in a graph is as shown below:
Now that we have a test case, run definition and test run created, we will delve into using MTR to execute our test cases in further blog posts.
If you still(!!!) haven't heard, Rosario CTP is out in the form of a VPC here :) Doug's whitepaper should give you enough info to take it out for a spin.
Manual Test Runner(MTR) is a key part of the CTP. It is a tool designed to help manual testers to load up their test plans and execute those with the support of additional tools like one-click screenshot, one-click bug with pre-filled repro and more such exciting stuff. Test case management(TCM) is another key piece where we have tried to fill the missing piece in the existing VSTT SKU. There is a tool window right inside VS IDE where you can create/modify/manage test cases, test run definitions and test runs and save them in the TCM server which has your team project's source code and work items. Then, load up the test runs in MTR and go. Publish the test results back into the TCM server and look at the cool reports that show how your test team is doing.
Do check it out and tell me how it goes!
Rosario is the next in line version for Team System after Orcas. And we are doing this CTP when Orcas has not even RTM-ed yet! Isn't that pretty cool? It's been a bit crazy managing multiple releases, but then, we want Rosario to hit the shelves in a short time after Orcas releases - faster new release. And no - I am not allowed to define "short time" :P
I've been working on the VSTT portions of this CTP for some time now and this is certainly one of the most exciting set of features I have seen for testers. Though I cannot wait to talk about all the awesome new stuff in the release, I'll not give out any more info until the CTP is out. Stay tuned for more...
We'll put it out as a VPC download on our website so that there are no installs that you need to do separately. Just set up the VPC and go...
Since I am a tester on the Test edition of VSTS - I get this q a lot of times: "Do you use VSTST for testing VSTST?" And the answer is "You bet, I do!" :)
As you might know already, this is called "Dogfooding". Many Microsoft products are dogfooded extensively before release - Vista, MSN Search, Visual Studio, Team System...
Our devs use unit test types to write their developer regression tests. Our QA team uses unit tests to write our functional tests. Yes, I know this is not a perfect fit for UI heavy testing, but it does suffice for the time being. We use generic tests to wrap our legacy tests that may be scripts or batch files. Load tests are added to the test bed to do stress testing on the app. All of this goes into the TFS server that we have deployed for the entire Developer Division and testing performed out of it.
Like any other user, I have my own set of pet features and pet peeves in TST. I absolutely love the code coverage coloring, the "Run test" from the editor (Orcas feature), the ease with which load tests can be configured and the controller-agent functionality. It makes my life so much easier as a tester. Not to mention that it's incredibly easy to point out to devs where the problems are and associate results with bugs since we use TFS for work item tracking. Now for the bad parts - I think the test list feature is absolutely useless for large test projects. The UI confuses me - more than once, I have selected a test in the window, right clicked and said "Run checked tests", but some other tests start running!! That's because selecting a run is not the same as checking it. :( I have a wishlist as well - stuff like test case management being easier to handle, the test run results being easier to analyze etc.
The best part of dogfooding is that this is a good chance to know which features are likely to be real relief and which ones are mostly going to tick the users off. Testers being the first customers of the tool is literally true in this case. Plus the QA team gets additional testing in the process so that bugs/suggestions are filed before outside users get a chance to go at it.
So, do you eat your own dogfood? ;-)
