Min Kwan Park's bLog : Let's talk about Testing..

Catch your elusive bugs..

mkpark — Tue, 13 Mar 2007 05:03:00 GMT

Few weeks ago, I did presentationt to my QA team about how they can found elusive bugs faster and easier. I think that this information can be useful information to any one who tests a Windows application.

Please feel free to share your opinion on this PPT.

We are building city than just a building...

mkpark — Thu, 02 Jun 2005 06:46:00 GMT

Few weeks ago, I got some time to think about how we should approach to our S/W design/test engineering.

IMHO, in many cases, we end up like building construction. I think that it is making sense to see an application as a building which has many internal components and aspects. it has fairly complex interdependency between each features and components. However one thing, which I could see in many chaotic or messy projects in engineering stand point, is that team was just focusing on how to build a building and forget about how each buildings in the over all blue print can work together.

We need to remember that we are building harmonic echo system as one big city than just one small application.

my opion on TDD...

mkpark — Sun, 18 Jul 2004 10:40:00 GMT

Around me, I'm seeing some people who are very deep into TDD.

Well, as QA, whenever I heard their mythical belief on TDD, It makes me think why TDD is believed that much.

TDD, it sounds like very testing oriented. yes it is. I agree. and it is good approach. but is it all? and can it find all issues? I don't think so. but many of TDD believers want to believe that it is the final solution for everything.

IMHO, I think that it is just a trend which is good trend because it is focusing on the quality of code at the beginning. however, I don't think that it is everything or final thing which can solve every problem or prevent all issues from your code.

Adding to TDD, we need to understand the bigger picture. you should see the forest with trees, too. otherwise,you will lose harmony in your overall design and process.

While you do test your program...

mkpark — Wed, 14 Jul 2004 11:29:00 GMT

In most times, while I do dogfood/test VisualStudio 2005, I put them under debugger to monitor several things...

- First Chance AV

- Unhandled Exception

- Userbreaks from System to process under debugger

- Appverifier(a tool which is part of Application Compatibility Toolset) notification via Int3(User break)

*ACT is downloadable from http://www.microsoft.com/downloads/details.aspx?FamilyID=7fc46855-b8a4-46cd-a236-3159970fde94&DisplayLang=en

If you are using debug build for testing, you don't have to run your app under debugger in most of cases because your debug build may have enough asserts to show errors in early stage.

However for many QAs, using retail build which will be shipped to customer is more general. so monitoring app with Debugger and appverifier is very interesting ways to do.

For example, I saw fair amount of issues that the thread by RPC call crashes and the AV caught by RPC handler. so caller may handle it without actual crash but it ends up as random and catastropic errors. however under debugger, you can see this kind of issues. so you can easily figure out the real cause of random issues.

Adding to this, with various Appverifier options, you can expose memory and handle issues much easier by reducing the tolerance of memory/handle mis-usages. it is really useful to improve overal stability of your product.

Last minute issues...

mkpark — Sun, 04 Apr 2004 02:21:00 GMT

You may read many articles and books about Testing and many of them said that the earlier you find issues, the better overall.

Is that really true? well, based on my experience, Yes, it is true. However it is always possible for all issues? No it is not.

There are some issues which we may find only when the product and QA are mature and ready.

I'm not trying to say that QA doesn't have to be part of early planning and reviewing to find out issues. Yes it is very important to find out design level defects. it will help to improve the productivity of overall process and it will reduce the issues later. However, even with all efforts, you will encounter the last minute issues when product is mature and fully integrated, also QA(team or person) have enough knowledge about the product in user's perspective. (it is different topic to talk about why QA can't understand everything at the beginning. so I'll talk about this later.)

Why? in my opinion, it is very obvious that the problem was not there at early stage of product or it was hidden some way and waits until the time comes. it may sound strange. but in my opinion, it is true.

if so, how can we reduce these last minute issues? Well, I don't think that there is gold bullet for these. but I think that there are some things that we can consider the below for improvement.

- Keep in mind the integration at the beginning at design level and QA should look at the level of each individual tree and also the big forest. not just your component integration, but also at the final product level. it doesn't have to be done by everyone. but some one in the org should look into this as early as possible.

- Have better communication channel between teams. modern S/W is very complicated and has complex dependency between components. so if you don't understand the changes and progress of depending components, you can do properly QA work at all. so the information should be delivered appropriately.

- Education on the base technology of the product for QA. why? the more you know, the wider and deeper you can see.

- Have various types of user scenarios in various level. you need to have feature level user scenario, component level, and integration level and real world user level scenarios. just doing feature level unit test will not let you avoid the issues.

- Nothing is wrong. you just need to validate what you are assuming... Generally we assume how the product will be used. and QA team is directed to focus on main line scenario all times. Yes, I agree that it is important and proper way to do approach to product with limited resource. However, we need to validate our assumption in someways like beta feedback and news group monitoring or userability study. Well when you see the result, you will be supprised by the result. there are many different ways for our customers to do their work with product.

There are many things to consider besides these. Anyway, important thing is that we need to understand the process and stage of product and we need to prepared for coming stage and do proper testing.

Testing System vs Testing component....

mkpark — Fri, 12 Mar 2004 05:31:00 GMT

Recently I got question on myself about how I do balance between System level testing and component level testing.

First of all, I may need to notify that I'm QAing on heavily UI involved components which are working with many other components from other teams.

System level testing means that you are testing your product like customer at the top level and looking for the harmony with all different components. and component level testing(or Unit testing) means that as you are focusing on the functionality of the component without considering much on how it works with others.

Well, in my opinion, both side has their own importance. and it is not easy to say which one is better. but in practical way, System level testing is more important for most of S/W development cycle which doesn't have enough resource to test component thoroughly, because we can ship something which is working in a good enough degree for general cases with some glitches in specific cases but if each component is working great alone but not working together, Well we did something wrong and we can't ship the product.

However, I don't think that this approach is always correct and proper. there are areas which we should take care of every detail at the component level first like airplane control software, Library codes, and etc. also even though I emphasized the system level testing, I think that certain degree(definitely enough amount; how much is enough is different question) of component level testing is necessary and should be done.

QA should find out balance about how to distribute the limited resource for each side in given situation. and it should be decided as early as possible at least in guide line level.

Remove OLE Exception filer

mkpark — Fri, 27 Feb 2004 05:37:00 GMT

Sometimes, we are seeing random hangs or crashes with our testing product. well when we actually look into the post issue state, in many cases it is not easy to understand what causes the problem.

One of the reasons not to see the cause is because the errornous situations are already dealt automatically. and OLE Exception filter is the very one in many cases.

To disable this, you need to do

Set the registry key to turn of Exception filter like [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Ole] IgnoreServerExceptions"="Y“
reboot your machine.

it will give you better chance to find out the very cause of nasty issues.

One thing to remember, after you set this registry key, you may see abnormal crashes with other applications which you couldn't see before. well this registry key is for entire system. so maybe some of application expose their internal issues to you. :)

Why QA need to read source code...

mkpark — Tue, 17 Feb 2004 04:51:00 GMT

Many cases, QAs are working based on test plan which is generated based on

- Spec of feature

- User scenarios

- Feedback from Dev/PM

- Previous experiences

- Etc...

However we oftenly miss the apect of understanding the feature at the code level. well some people may not agree with me and say that it is not necessary to do white box testing all the time and we'd better be at the user level perspective because we have too complex system and don't have enough time to understand whole detail.

Yes, I agree that it is not easy to spare resource to understand whole detail with limited resource. and Yes, it is important to see the product in customer's perspective which is not based on how the code is implemented at code level.

But, in many cases, if we as QA engineer don't understand the internal structure and data flow with a certain degree of understanding code, we may be ended up with missing test coverage here and there. and the overall quality of product will be just mediocre.

Understanding product at code level will give us benefits like

- better test case generation

- efficient approach in test to the feature

- Can make faster and accurate decision on the impact and scope of changes

- Prove self-esteem and confidence

Well, However we need to have balance to be realistic. we only have limited resource(time and men). so it may not be necessary to understand every single line. drawing the line appropriately will be key to success....

Test code, Can they be re-used for next version?

mkpark — Fri, 06 Feb 2004 06:27:00 GMT

I have worked on same product about 5 years. and I have shipped three different product cycles and now I'm working on next version.

Over this period, whenever we moved over to the new version, one of problem was to convert or rewrite test code. you may think that it is not a big deal. but if you have more than 2000 P1 test cases(which have multiple scenarios) and most of scenarios should test multiple languages with different types of configurations. so it was head-ache.

However my team could learn that though the product is changed and new features are added for each product cycle. we could see that basic functionalities and conceptual unit actions are not changed. so we took an approach to bring up one middle layer which keeps the definition of all concept unit actions and make our automation scripts are using the middle layers for most of cases. then we could move around our automation between different automation harness and products.

Yeah, still we need to implement new internal logic for each different testing targets(like new product), but the cost to move over old scripts could dramatically reduced comparing to our previous experience.

Why QA?

mkpark — Fri, 30 Jan 2004 00:08:00 GMT

QA is very important part in other industries. especially for those which produce human safety related products like cars, medical equipment and others...

But why in S/W industry QA(Quality Assuring) is less valued in general?

Well, in my work place, I can see that it is valued higher than other places(at least which I used to work for). However we as QA sometimes are frustrated with what we are doing and how QA org is working..

I think that it is time to change our mind set on QA. otherwise we will see huge disaster which may be caused catastrophical manner. Why?

Now in this world, S/W is in many many places. your cellphone, car, PDA, PCs, appliances, elevators and many many other places. well some people will say. “Who cares them? they are working fine now and though they fail, it will be ok. so what is the matter?“. No, it is not true. they may work for now. but How do you know it will work for all time? For example, if your ECU(Emition Control Unit) in car is malfunctioning by internal S/W mis-calculation, when you are on highway. it can be very dangerous. and if All your computers are just starting to spitting bad things into internet at once by a small fraction of bad code in your daily use S/W, it can down all your stock trade and bank account and many others.

Well, if you get that situation. as customer, you will be so pissed and you may want to sue the company which made the S/W or product. However as engineer who needs to prevent this sort of issue. we should re-evaluate how important QA is and what should be in place for QA on product we make.

First of all, QA should be valued high. we need to hire better people for QA position. it is not just a position to run a script. it is position to see the big picture and also co-ordinate the details to live together in harmony.

Second, QA process should be in place from the beginning of product design. you will be supprised when you see how many issues and bugs can be found at the design level by the sharp eyes of well prepared QA person.

Third, QA should have broad and in-depth knowledge on product and also customer expectation. QA engineers should be better to understand code in the manner of integration. Devs know detail and they will be great for digging into the specific things. but QA should be able to see how each component work together. otherwise when product is finalized, testing can be done in specific manners, but the integration story will be disaster.

Forth, We as QA engineers should have high self-esteem. and we should develop ourselves to be ready for improving quality of product. we may need to understand system which your product will run on. and we need to have better understanding of customers. and we need to be objective and critical of our process to be improved. same old just good is not the sentence we should take. we always need to figure out point of improvement in our process to make our product better overall.

there are many more things to be done. however important thing is to start now. not wait until the problem comes up. it will be too late to do QA work at that time.