Workflow of the IDE QA team

Workflow of the IDE QA team

  • Comments 12

Hello everyone, I am Anca Miclea. I work as a SDET (QA) in the VC IDE team. This is my first post on VC blog and I would like to share with you how we test the IDE for VC++.

 

Yes, we do test the IDE and this is not an easy job ;) For each new feature that we add to the product, we need to write a Test Plan (TP). Each TP is reviewed by the whole IDE QA Team and if the feature is a large one, like the recent "MFC" update, a review meeting will be held with the developers and the Program Managers (PMs) that work on that feature. 

 

As soon the TP is reviewed, one or more QA folks will design each test case and implement them. In order to be able to implement our automated tests we use as much DTE available support as we can. EnvDTE is an assembly-wrapped COM library containing the objects and members for Visual Studio core automation.  For the parts of the IDE that cannot be covered by DTE we use an internal API that helps us record the mouse events and generate the code that will be used by our test.  This API is mostly used for black-box testing for UI features.

 

As our software is used by developers all over the world, it is very important that our tests are written in a way that will pass on Vista OS using a Normal User and also pass on localized OSes and VS SKUs. In order to have confirmation that a test is good and robust we need to run it in our lab environment, with different configurations, like "Win2k3 ENG x64 + VSTS ENG", or "Vista JPN x86 + VSTS JPN", etc. Running tests in these configurations is a way to validate not only the product itself, but also to validate the automated tests as well.

 

If an IDE product bug is found, usually we try to find an easier set of steps that reproduce the issue (these steps form so called "repro case"), investigate if that repro case is a regression from a previous product version and then open a bug that will be assigned to the appropriate developer.  Once the bug is fixed, QA verifies the “repro case” with the version of IDE that has the fix and close the bug.  It is our QA team’s responsibility to make sure a specific issue does not regress in later versions, by providing an automated test case for each product bug that is fixed.

 

Thanks,

Anca

  • PingBack from http://www.absolutely-people-search.info/?p=2307

  • Anca,

    Great to know about the QA process in Visual Studio.

    Currently, I'm testing VS 2008. No bugs found yet :D only minor "cosmetic bugs", such as a flicker in the integrated web browser: For example in MSDN pages, when I move the mouse pointer over the popup menus.

    Same problem in Microsoft Document Explorer 2008.

    This pages is an example:

    http://msdn2.microsoft.com/en-us/subscriptions/bb608344.aspx

    Which is the appropriate way to report these problems?

    - Davor

  • > "Vista JPN x86 + VSTS JPN",

    Was that really tested?  Here, try this:  In a cpp source file with enough member functions, when the combo box with the list of member functions drops down, the list has a crash bar.  The crash bar is supposed to be a scroll bar, but it's really a crash bar.  This is with SP1 and the extra hotfix for Vista.

    You know that testing on an ENU (or ENG) OS + JPN MUI pack is not the same as testing on a JPN OS?  In Vista we can do the opposite now, we can install the JPN OS + ENU MUI pack, but the result still isn't 100% the same as the ENU OS, even though 99% is respectable.  When testing, we have to use the same real OS that end users will use.

  • The smallest amount of code to crash Visual C++ compiler (single line)

    crash.cpp:

    class{}(

    Other variations are working (er.. crashing) as well:

    struct{}() or struct { int i; }(0)

    try compiling it about you come up with an error:

    fatal error C1001: An internal error has occurred in the compiler. (compiler file 'msc1.cpp', line 1411)

  • C1001 isn't a very severe.  It doesn't make you lose all the files you were working on.

    The first time Microsoft gave me an internal compiler error though, it was cute to see how Microsoftical Microsoft was.  The error message told me to view some Microsoft web page in order to report the error.  The web page said that I could pay Microsoft US$190 in order to report Microsoft's error to Microsoft.  I didn't report it.  (Things have improved since then.  Now on the Connect site we can get "won't fix" and "not reproducible except outside of Microsoft" for free.)

  • Today's trials:  XP JPN with SP2 + Visual Studio 2005 JPN with SP1.  Did QA try to do a repair install, I doubt it.  Did QA try to uninstall VS2005's SP1, I doubt it.  Did QA try to uninstall VS2005 completely, well in a few hours I might have a guess, but there's a feeling that I already have a guess.  Now wondering if there's any way to repair without doing a FORMAT C:.

  • Hello Koroskin,

    Thank you for reporting this issue.  You are correct that our C++ compiler is crashing on illegal C++ class definition.  Instead, we should have been giving a descriptive compiler syntax error something like "expected an identifier or ;".  

    I will open a compiler bug to be fixed in the next version.

    Ulzii Luvsanbat

    Visual C++ Team

  • Hello Norman,

    First of all thank you for sharing your expertise around software localization & globalization.  As with any software testing, there are specific areas and configurations we may have overlooked or insufficiently tested.  However, I can assure you that we do test Visual Studio localized in multiple languages (including JPN) on their respective localized OSes.  And yes, we do separate testing on ENU OS + MUI versus Localized OS.  That said, if you do see discrepancies or issues (looks like you do already with crash bar vs. scroll bar) please report them as bugs through our Connect site.

    Ulzii Luvsanbat

    Visual C++ Team

  • Hi Davor,

    You can report bugs on www.connect.microsoft.com

    We try to repro each bug reported by customers on our machines. It would be nice if you can provide a detailed description and a simple (as possible) repro case.

    Thank you!

    Anca Miclea

    Visual C++ Team

  • "if you do see discrepancies or issues (looks like you do already with crash bar vs. scroll bar) please report them as bugs through our Connect site."

    I gave up with the Connect site when Microsoft said that Microsoft couldn't deal with Microsoft's non-English error texts on the Connect site.

    The quantities of "won't fix" and "not reproducible except outside of Microsoft" had already been rather discouraging.

  • Hi Norman,

    I'm dissappointed that you are having these issues with Connect site on non-English text.  I've looked through all the bugs you have opened against VC++, indeed I found few "Won't Fix" and "Not Repro" bugs (appended below).  We will go through these bugs again with proper translation, if there wasn't any before, and make sure we made correct decisions.  I want to also let you know that please do not be discouraged to report these kinds of bugs.  There's nothing wrong with the Connect site itself in dealing with non-English text, we just need to be extra careful when understanding non-English bugs that come from customers.

    12562 Norman Diamond Fixed Resource language is set half-wrong

    18831 Norman Diamond Not Repro TEXT("")

    12426 Norman Diamond Not Repro BK1506

    15495 Norman Diamond Won't Fix [Cust-MSDN]VC++ Intellisense picks up headers from Wrong folder (Wrong SDK's version of winnt.h gets included)

    12675 Norman Diamond Won't Fix Visual C++ 2005 generates bugs in DLLs

    Ulzii Luvsanbat

    Visual C++ Team

  • Thank you Ulzii Luvsanbat for your interest, which I observed a bit late.

    Regarding any of my submissions which might need translation, maybe an additional explanation is needed.  When I used to report bugs in Longhorn betas (before they became Vista and 2008), some of them were closed with no explanation, and later Microsoft explained that they were closed because I said I was writing in English when I was writing in English.  Subsequently Microsoft changed its mind several times about whether it was better to report in English or Japanese, but it really didn't matter much, because the Longhorn team was even more opposed to fixing bugs than your team is.  Anyway, this is one reason why some of my submissions were in Japanese, despite my inadequate skill in that language.

    However, I notice you didn't mention the particular response which really made me give up with the Connect site.  I quoted Microsoft's error text exactly, so that was Microsoft's Japanese not my poorly written Japanese.  I further provided a rough translation to English.  Microsoft's response was that Microsoft could not contend with submissions of Microsoft's error texts in Japanese.  By itself this would just be laughable, but in the context of all the won't-fixes it just made me give up.

Page 1 of 1 (12 items)