Data Tools and Software Testing : Model Based Testing

Spec Explorer 2010

nihitk — Wed, 02 Dec 2009 16:51:03 GMT

I am very excited to see the latest version of Spec Explorer (which is available even for Visual Studio 2010 Beta 2) now available publicly for everyone in the world to leverage. It is available via the MSDN DevLabs page and the Spec Explorer team (the fact that there is a team dedicated to pushing this tool makes me immensely optimistic about it’s future) even has a wonderful new MSDN blog dedicated to Spec Explorer.

Kudos to the Spec Explorer team and I look forward to wonderful improvements coming from them in the years ahead to help leverage Model Based Testing in a more pervasive way across the software industry.

Elements of MBT - Part 3 – Actions and Action Parameters

nihitk — Fri, 06 Jul 2007 02:45:00 GMT

In the last 2 posts we have looked at what we mean by a model of a system and defined what States and State Variables are. In this post we'll look into Actions and Action Parameters.

We can define an Action as something that modifies or exercises the system. Thus "TurnIgnitionOn" (in the car model being used in this series) can be an action which causes the system to change in some way. When considering Actions you need to think about what exactly is it that changes in the system and which all state variables are impacted by those changes. An action is therefore simply something that causes the system to change states.

Actions become much more powerful and useful when combined with the concept of Action Parameters which can be considered to be similar to method parameters. Like a method (in a programming sense) does some work which can be tweaked or controlled by the method parameters, the effect of an Action can be affected by Action Parameters. In the car driving model an example of an Action Parameter would be whether the ignition could be turned on remotely (via the keychain) or locally. In that case we might want to add an Action Parameter to the model such as TurnIgnitionOn(Mode) where the "Mode" action parameter could take values "Remote" or "Local" similar to the way state variables can have different values.

A really interesting point that emerges here is that there is no one way to model the system. In the example above if for some reason you actually wanted to track how you had opened the car, maybe for a later stage in the model when you were actually locking the car then you would need to store this information in a state variable. Anytime you find yourself using the word "track" you should think about having that value as a state variable. We could therefore break our TurnIgnitionOn action into 2 different types of actions, TurnIgnitionOnRemotely and TurnIgnitionOnLocally and track how we opened the car as a new state variable, say S3, with values Remote or Local. Our model would therefore look something like this:

I will end this post with a word of caution. Even though it is quite easy to add a state variable to the model to track some aspect of the system, you should be very careful when adding state variables to your model and strive to minimize this number. The reason for this guidance is to limit the amount of state explosion that takes place as a result of defining these additional state variables. Remember that every time you define another state variable the number of possible states in the model is increasing exponentially. To keep your model manageable you should not add a state variable till it is absolutely critical to the behavior being captured.

UPDATE: Just noticed that this is actually my 100th post on this blog - WOW! Never thought I'd see the 100th post when I started out...it's been fun! :)

Elements of MBT - Part 2 – States and State Variables

nihitk — Thu, 28 Jun 2007 03:52:00 GMT

This is a continuation of the Elements of MBT series of posts which I started last time with Part 1 and covered what is meant by a model of a software system. In this post I shall be covering additional concepts which you will need to know to gain a better understanding of Model Based Testing.

One of the most fundamental concepts required to define a model of a system is that of State Variables which in turn are needed to define what a State is. A State Variable is simply some aspect of the system which changes over time. For e.g. in the car driving model, one aspect that changes (and which you care about tracking for the purpose you are building the model for) is whether the car is moving or not. So this can be a state variable, say "CarMotionState", which can have values such as "Moving" and "NotMoving". Couple of important things to keep in mind is that a state variable can have multiple values (not just 2 as in the example above) and that these "values" are really just semantic terms to help us define the model. What the actual values of these do and result in (especially from an automation point of view) is tied in later when you are implementing the code for the model.

A State then becomes much easier to define as simply a combination of all the State Variables in the model with particular values assigned to each state variable. Thus if your model contains 3 state variables S1, S2 and S3 with multiple values for each state variable, a state will be defined by a unique set of values for each of the state variables S1, S2 and S3. You can therefore think of a state as simply a snapshot of all the state variable values at a given point in time. From the Part 1 car driving model we can now see that the 3 states in the model are comprised of the following unique combinations of state variable values.

S1 = { IgnitionState = IgnitionOff; CarMotionState = NotMoving; }

S2 = { IgnitionState = IgnitionOn; CarMotionState = NotMoving; }

S3 = { IgnitionState = IgnitionOn; CarMotionState = Moving; }

One thing to notice here is that not all possible states need to be defined for a particular model. Depending on the system and the reason you are creating the model or the abstraction level you want to capture, you might not include certain states in your model. Or certain states simply won't make any sense to model such as the state "IgnitionOff" + "Moving" if you do not want to capture the behavior of the car sliding down hill when the ignition has not been turned on. The great thing though is that it forces you to think of that scenario and to consider if that is something you want to capture and if so then what the expected behavior would be.

In the next post I shall cover Actions and Action Parameters and update the model to add an Action Parameter to one of the actions.

Elements of MBT - Part 1 - What is a model?

nihitk — Sat, 24 Mar 2007 03:54:00 GMT

In this series of "Elements of MBT" posts, I will be covering some basic concepts used when trying to build models of software systems for use in Model Based Testing. The aim is to give an explanation of the various elements involved in defining a model of a software system and hopefully this should get us on the same page and allow us to explore the complexities in more detail.

Let us start by going over what we mean by a "model" of a system. Put simply, a model is a description of the system and is much simpler than the actual system itself. What this means is that the model hides a lot of the details about the system and hiding these details allows us to work at a higher level of abstraction. For e.g. a simple model of driving a car could be:

One thing to keep in mind is that there is no perfect model of a system, since the same system can have completely different models depending on what you plan to use the model for. Are you creating the model of a system simply to capture the behavior and requirements of the system? Are you creating the model to auto-generate the code for the model implementation itself? To describe the data flowing through the system? Or are you generating the model to actually test a particular portion of the system?

A quote which captures this unusual thing about models really well is as follows:

All models are wrong; some models are useful.
- George Box

What is worth hanging onto from this definition is that a model that you create should strive to be "useful" more than anything else. It might be rough round the edges and might not tell you everything about the system (for e.g. you don't know from the above model how the gear is to be changed or what happens when a gear is changed) but if it is useful for the purpose it was created it is a correct model.

Another aspect of defining a model is the notion that the model should help predict what the behavior of the system would be under certain conditions. So if you are in the "Not Moving" state and you "ChangeGearAndAccelerate" then you reach the "Moving" state. Are there certain assumptions in the model? You bet there are! Such as assuming that the ignition is on, that the car has wheels, that the car is not hanging in the air, that there is oxygen in the air, we can go on and on and on. The hiding of these details and assumptions is what helps us look at just this behavior we are capturing. If anything is important for the purpose (say testing) for which we are creating the model, we should consider adding that detail into the model. So if we thought that it was important to know whether the ignition was on or off we would change the model to capture this as well.

Now we know that if you are in the "Ignition Off" and "Not Moving" state then if you "TurnIgnitionOn" you reach the "Ignition On" and "Not Moving" state since simply turning on the ignition does not cause the car to start moving in the model that we have created. To start moving you will need to "ChangeGearAndAccelerate" which should take you to the state of having the "Ignition On" and the car "Moving".

Hope this gives you an idea of what a model contains and what it can be used to capture. In the following posts we will look into each element of a model in more detail.

Model for Getting Drunk

nihitk — Tue, 09 Jan 2007 04:35:00 GMT

You know your team members are really catching onto this whole model-based way of thinking when they pull your leg over a few drinks in a bar and say (tongue in cheek): "So Nihit, the 'Action' of having this drink will take me from a 'Sober' state to a 'Drunk' state, right?".

Yes - exactly - it will (or maybe we need to accommodate degrees of drunkenness as well - not everyone is drunk after a single drink!?) and for completeness sake you can even have an Action Parameter for the type of drink being passed into the "DrinkAlcohol" action! Isn't that neat!? :)

Anyway, here's wishing everyone a wonderful 2007 ahead. Happy New Year!

Model Based Testing – The Yellow Brick Road

nihitk — Sat, 29 Jul 2006 04:04:00 GMT

The future was here – now it’s the past. I am, however, wiser about the ways-of-the-model-based world. Still taking baby steps really, but starting to stand up on my feet. The real-world turned out to be a whole different ball-game from what I had envisioned (and read and expected) but it has been a fun, FUN ride and the road ahead looks fascinating – though long.

In the eons since I posted my original introduction to model based testing, a lot has happened to the world. For e.g. Wikipedia now has an article on MBT. There is an Association for the Advancement of Model Driven Quality Assurance (though they don’t actually state how one can join them!). There are a host of companies dedicated to doing only model based testing (for e.g. Reactive Systems and Confirmiq). There is an IEEE workshop on Advances in Model-based Software Testing (scheduled for Nov 2006 and looks really interesting – would love to hear from someone attending it!). There is even a new acronym for the technology – MDQA for Model Driven Quality Assurance (that’s when you know something has really taken off – when people call the same thing by different names!). I find all this amazing and reflective of the direction in which the testing industry will be moving in the next 5, 10, 20 years.

So what have I been doing all this time? Well, in a nutshell, simply using MBT to actually test something that gets shipped – in this case that happens to be portions of Commerce Server 2007 that I worked on (that should be another selling point used by the sales team and on our info sheet – “Tested Using Bleeding Edge Model Driven Testing Technology”). Along the way I cleared the hurdles and burnt my fingers while trying to keep the eye on the ball – which is always simply to test as best as you can. I now plan to expound in future posts on how to adopt MBT better, what to expect and more importantly, what not to expect from this fascinating methodology.

For this post, I will just mention the most startling thing I discovered – that this dream is going to take a long time to realize. By “this dream” I refer to that futuristic world where model based approaches are entrenched in our processes and systems. Where a tester thinks first in a model based approach whenever he/she is given something to test (even a regression!). Where the metrics are tailored to encourage testers to adopt a model based approach. Where the feature teams contribute to and strive to keep their models up-to-date. And where the tools have the features and reliability that is crucial to be able to develop models and have high levels of confidence in them.

All of these are non-trivial problems and there are no easy answers that anyone can give. But maybe that’s part of the challenge? So, all in all, buckle up and put on some good music – it’s gonna be a long, FUN ride and we’ll change the world on the way!

Pesticide Paradox

nihitk — Sat, 17 Jul 2004 05:03:00 GMT

Just posted an article on the Pesticide Paradox and how Model Based Testing can help alleviate that problem.

Among others things Terrarium 1.2 seems to have been released. I have been wanting to create an organism and play with Terrarium for some time now (try and use Evolutionary Psychology tricks perhaps?). Hope I get to do so with 1.2 at least.

Anyone know of any similar games (.Net based) out there?

Model Based Testing - The future is here!

nihitk — Sun, 30 May 2004 23:47:00 GMT

Not sure if my article on Model Based Testing warranted an article or not (am not sure when exactly does a post qualify as an article - anyone know of any guidelines?), but I did put it up, partly because I wanted to try out the Article section of the .Text blogging application (on which the MSDN blogs are hosted). Does an article differ from a blog post in any concrete way (I know it doesn't show up on the home page where the aggregation of all posts is shown)? Is it any more 'permanent' than a normal post? One advantage I see is that an article might be something more 'updatable' since many people do not consider editing a post blog entry as 'correct'. So I could possibly go back and update an article with more relevant content, links etc. Is that correct?

I am super excited about model based testing and would love to hear from you about what you think is good/bad/ugly about this testing methodology. Will also hopefully post more about the pros and cons of MBT and share some tips and tricks that I discover on my way.