What Does a VC++ QA Do Daily?

What Does a VC++ QA Do Daily?

  • Comments 19

Hello everyone, this is Smile Wei, an SDET (QA) in the VC Compiler team. This is not only my first blog on vcblog, but also my first ever blog.

As you might have guessed, I’m a new face around here, joining Microsoft only 3 ½ months ago. Here I’d like to give a brief introduction to the daily life as a VC QA and, to answer the following two questions that my friends kept asking me:

                How do I like life at Microsoft, especially on the VC team?

                What do I usually do every day?

For the first question, I love working at Microsoft, especially as a QA in the VC Compiler team. Usually people might think a QA may not be as technical as a dev, but this is not true in our team. As a QA in the VC compiler team, the test input are not simply numerical data or operations, in most cases they are C/C++ source code. When you write a test, you need to worry about not only how to write test processes with script languages like Perl, but also how to write test input code in C/C++ to capture both the correct behavior (positive test) and the wrong behavior (negative behavior) of the compiler. Same situation when you write regression tests. Creating a new test usually requires a lot of discussions with devs, because we need to make sure our tests will exactly reflect the new features or changes in the VC compiler parser, even some subtle ones.

Having source code as test inputs also increases the difficulty of bug investigations and reproduction. Because of the complexity of the compiler, some compiler bugs only fail under certain types of machine architectures (x86, amd64, or ia64), some bugs only fail under certain runtime environment (native, /clr, or /clr pure) on a particular architecture, while some bugs only fail with a particular cl.exe parameter (among hundreds of parameter combinations) under a particular runtime environment on a particular machine architecture. Usually you’ll feel that half the work is done once a bug is successfully identified and reproduced. That’s why a large portion of my time is spent on bug investigations.

Although this position is so challenging, I love it so much because I can learn so much about the VC compiler. All the details I learn from this position perfectly complement my previous high-level computer knowledge learned in college. Can you imagine how proud I am to easily find the bug in my friend’s code while it has harassed him for several days? That’s why I love this job. J

For the second question, my major responsibility includes the following tasks:

1.       Bug investigations on test runs. Microsoft has daily test runs on every product with all existing test suites not only before the product is shipped, but also after. Any failures in the test runs will be investigated as the 1st priority task and get resolved in the earliest possible version.

2.       Test fixes and writing regressions. This is a follow-up task of the first one. Once a VC compiler dev fixes a bug, he’ll publish his private built to me for a more comprehensive test even after he has already performed a sniff testing on his own machine. This double assurance mechanism further reduces the risk to introduce new regression bugs by the bug fix. Some extremely crucial bug fix may involve many tests with substantial complexity. For example, to verify a recent fix, I ran a 4 X 3 X 3 X 2 X 2 X 2 test matrix, and each test run in the matrix involved creating and testing a large 23MB PCH. In Microsoft, a regression MUST be written to address the fix if no previous tests were targeting that source code. With more and more regression tests being added, a complex product like Visual Studio might have huge number of tests associated with it. For example, I own a test suite called the “compiler suite”, which contains about 20,000 test cases, this is only one of the 10 test suites of the VC++ compiler team. Please notice that these are only the tests for the VC++ compiler team, not including the tests for the VC++ library team, IDE team, backend team and release team, not to mention other teams in Visual Studio such as the VB and C# teams. The daily test runs on huge number of tests go some way to guaranteeing  the quality of Microsoft products.

 

3.       Verify QFE (Quick Fix Engineering) fixes. Yes, your understanding is correct, the QFEs are those hotfixes occasionally popped out and automatically installed on your machine. The usual QFE process is that the support team collets bug reports from our customers, finds a bug and passes it to the dev team. After the dev team fixes the bug, they alert the QA team so we can test it. The QA team will need to perform a two-round verification. The 1st round is before sending the fix back to the customer who identified the bug, and the 2nd round is after the customer verifies the bug fix, to perform a final verification before it’s officially published into hotfixes. Testing QFE fixes is always our 1st priority tasks to solve customers’ problems in the 1st place.

 

4.       Provide cross-team test support. Occasionally I need to provide test support for other teams - to give a different test prospective for the other team. For example, I recently finished two projects falling into that category and did find some bugs. One project was to test the VS 2008 installation process for the release team, and the other one is to test VS 2008 MSDN samples for the library team. I believe this type of test may help find those bugs that might not have be found by the original test team because of their testers’ familiarity with the correct behaviors and processes of their product. This exactly matches the spirit of a well-known Chinese saying “You won’t have the overview of the mountain if you are already inside it”.

 

5.       Develop internal test tools. As I mentioned in the first two responsibilities, Microsoft performs daily test runs on all of its products, and each test run may execute hundreds of thousands or even millions of tests. How does Microsoft achieve this? The trick is that the whole test execution, test result collection and comparison are fully automated with the support of a large set of test tools. Microsoft employees are encouraged to develop all kinds of internal test tools that can improve the test efficiency and some tools may be adopted and widely used in the whole division or even the whole company.

 

6.       Write tests for new features. Currently the VS team has started working on the next delivery of Visual Studio (the version after VS 2008). A QA will be assigned to work out a whole set of tests targeting each new feature. The test process starts in a very early phase of the product cycle rather than after all the coding is done, which I believe sticks to the agile model instead of the traditional waterfall model. The reason we use an agile model is because the later a bug is found in the product cycle, the more cost will be incurred. Currently most of my responsibilities focus on the first five tasks and I believe in my next vcblog you may see more of my work on new features.

 

7.       Other job responsibilities. This includes everything else on my job commitments or not on my commitments. For example, support campus recruiting, make training slides and videos for new employees, interviewing candidates, and yes, write vcblogs. J

 

Hope you are not bored by my long introduction. J If you happen to have some interest, comments, or suggestions, why not reply to this blog entry – I look forward to interacting with you? If you happen to have more interest on this challenging position, why not come join us? J

 

Really appreciate your patience and hope all of you and your families are happy and healthy. Thanks.

 

Smile Wei

VC Compiler Team

  • PingBack from http://msdnrss.thecoderblogs.com/2007/09/12/what-does-a-vc-qa-do-daily/

  • > a well-known Chinese saying "You won't have

    > the overview of the mountain if you are

    > already inside it"

    In English:  "Can't see the forest for the trees".

    Does QA participate in deciding whether a bug will be fixed?  Or only in verifying whether the code matches the design?

  • Thanks for leting me know another form of this saying. :)

    For your question, yes, QAs definitely participate in deciding whether a bug will be fixed. Whether to fix usually is decided after the negoations between QAs and Devs. A good QA usually does a lot of research before talking to a dev. He needs to find out whether the failed behavior matches the design, figures out stable reproduce steps, investigates potential causes of the failure and the risks of not fixing it. All those information will help in making the decision.

  • My question had a bug because it wasn't clear enough.  Here is my real question:

    Does QA participate in deciding whether to fix all kinds of bugs, such as design bugs, or differences between the design and the ISO standard, or differences between the design and MSDN?

    > needs to find out whether the failed behavior

    > matches the design

    Yes, if the design has a bug but the behaviour matches the design, then the coding doesn't have a bug.  But if the design has a bug then the product still has a bug, so that is why I asked.

    (I even argued with my boss about that.  I tried to say that product quality includes usability by users, but my boss said that quality only depends on whether the code matches the design.  The design depends on how much the OEM pays.  Business rules.)

  • What to say and how to say.I envy you so much that I hope I can join you or your work as a professional developer one day.Thank you for your introduction.As a student ,as you before,I am intrested in developing.

  • Hi, Norman. Good questions. Comments in line:

    1. Does QA participate in deciding whether to fix all kinds of bugs, such as design bugs, or differences between the design and the ISO standard, or differences between the design and MSDN?

    [Ans] It depends which team a QA is in. Visual Studio is a huge product and each QA will focus on the quality control of one aspect. For example, in compiler team, we focus on verifying the behaviors of the compiler, while the QAs in library team will take care about all the MSDN libraries, samples, side-by-side issues. So I would say a QA should definitely participate in making decisions on all kinds of bugs under his responsibility, and will also responsible on all bugs he finds during cross-team support activities.

    2. Yes, if the design has a bug but the behaviour matches the design, then the coding doesn't have a bug.  But if the design has a bug then the product still has a bug, so that is why I asked.

    [Ans] This is an interesting point: what if a design is not good? Are all the bugs match that wrong design are also not good?

    I would say, the design phase in VS team is an long and thorough-thinking phase involved with many technicians. After this phase, usually the design is relatively robust. Even in some rare cases that a few customers do complain about a design, most of them, as you mentioned, are usability issues. Your manager is right on this point, whether to fix such a design issue need to consider a lot of factors, such as the cost and risk analysis on fixing it. Such decisions are usualy out of QAs' control and need be decided by managers or architects.

    If a bug is found to match the design, even a design under discussions whether to be changed, it'll be marked as "by design". If this design is finally decided to be changed in next deliverables, all associated tests will be updated then.

  • Hi, primer_plus:

    So glad that you are interested in my job.:)

    Since you are a student now, why not apply for some summer internship positions in Microsoft? I beliebe you'll learn a lot of skills and knowledge that are not taught in college.

    Keep on track and you'll realize you dream. :)

    Thanks,

    Smile

  • Smile.wei,thanks for your detail introduction.i'm a chinese,working on window kernel driver development.

  • Hi morning, you are very welcome and nice to meet you.:)

  • Just to add to what Smile is saying, and respond further to Norman's questions regarding QA participation as to whether to fix a bug, it definitely depends on the person, their team and their experience as Smile says.  When we report a bug, there are three stages it goes through - active, resolved and closed.  It starts at active when we open it.  It then gets assigned to someone and the resolve it - either by fixing it, won't fixing it, marking it as by design or a few other options.  At that point the bug comes back to the person who opened it to close.  If they agree with the resolution, they close it, but you can also reactivate it if you don't.  In this sense you definitely participate.  If you decide to reactivate it, it's best to give your reasoning in the bug and be convincing.  If there is still disagreement, you work with the dev and maybe also the program manager to resolve the issue.  

    People who are known for thinking things out well and are reasonable about what should and should not be fixed given the context of cost, risk, timeframe are well regarded and their opinions definitely count.  You can also push things through by being obstinant, but that isn't helpful and people will be less likely to consider your opinion in the future.  

    I definitely agree that one of the responsibilities of the test team is to assure the quality of the product.  To this end, on the VC team, QA is involved from the very beginning of spec'ing out features, reviewing specs, dev and test plans and making sure new features are both useable for the end users and testable for the test team.  In the end, QA's job is to make sure things are right for the customers, and if we think things may not be going that way, or something is blocking us from doing so, it's our job to call that out.  However, it's equally important to see the big picture and realize that there are good reasons not to fix some bugs, like the risk to other areas of the product, cost vs effect and that sort of thing.

    Thanks,

    Ben Anderson

    Visual C++ Libraries QA team

  • Hi Smile,

    I found your job very interesting. Thanks for sharing some experience with us.

    I give you one challenge for the "by design" vs "bug":

    If you change an idl file then do an incremental build the binary will not be correct. This is because it will not embed the new tlb. Seems like a bug no?

    What happens it's probably not a bug:

    The tlb is changed but the rc compiler will not be triggered because the tlb file is not part of the project and not an input for the compiler.

    From my point of view this is a bug. The incremental build does not work correctly in some cases. I do not trust it and I have to use

    clean builds everytime I release something.

    Best Regards

    Cosmin

  • The idl file also compiles to some .c and .h files, not only to the .tlb file.  The ATL interface depends on those .c and .h files.  So I think it is a bug that the incremental build doesn't work.

    Since I have the habit of doing full rebuilds, I even forgot about that bug.  Now I remember it, but if I post it to the Connect site then it will be resolved "won't fix".

  • The resource compiler does not depend on the .h or .c files so it will no be triggered.

  • My requested VS feature is to fully support the agile method within the VS environment including test case management, test case execution, a wiki or other application for creating and manually running test cases (a Fitnesse replacement).  

  • Hello Cosmin:

    Thanks for taking the time to read my post and for sending me your reply/question.

    If I understand your question correctly, I would say that in order for the IDE to know a file/step is part of the project (and therefore that it needs to be compiled when doing a build - or checked and potentially compiled if doing an incremental build) then that files has to be known to the project system (for example, just including one of the auto-generated file inside one of the project’s source files is not sufficient for the IDE to know how to generate it correctly.) If some file is external to the project (such as an IDL file and/or some of its auto-generated files including its type library) then the project system must be explicitly told by the developer it is part of the project.

    I am not sure which version of the compiler/IDE. But I did an experiment on VS 2005 (built 14.00.50727.762), which seemed to have shown correct changes after an incremental rebuilt over an .idl file change. It will be very helpful if you can provide us the detailed steps, process/sample codes that trigged the problem you’ve met. If you submitted this problem to MS support team before, a link to that submission will be helpful too.

    Another thing you might want to pay attention to is that I would always do a complete/clean build before releasing a final version of my product even if I am doing incremental builds during development.

    Thanks

    Smile

Page 1 of 2 (19 items) 12