Found this exceptionally inspiring, interesting and informative talk (I saw an internal version of the same talk) by James Whittaker (please subscribe to his blog if you remotely care about software quality), who is a pioneer in the software testing world. He also has a series of blog posts on this topic in which he covers more details.
In fact it was so inspiring, that when I tried to save this post in Windows Live Writer, I got the following dialog + a crash:
instead of a simple message followed by a SaveAs dialog which would have been so much better if there was simply no space to save this on disk.
In defense of the Live Writer test team though there were quite a few "different" things that I was doing and it just makes the same point of how diverse the usage scenarios end up being from our labs.
- I have my "My Documents" setup to automatically redirect to a share for backup purposes.
- I also had the blog I was targeting set to a different one instead of my MSDN blog. This is my personal site blog and since I blog on it from home, I cannot use the externally visible form of www.sitename.com (since I am behind my router on which my server is also located), I was forced to use the http://server format for that blog setup.
- I was creating the draft from work, where the http://server format does not work for my personal blog anyway.
- I tried to save and switch the blogs (from personal to MSDN) at the same time.
Ah, but I shouldn't be surprised. First I used to think that I was an innate tester who just naturally found bugs. But over the years, I have sadly realized that almost all our users are just as gifted as me. Thanks to the quality of our software.
Just today morning as well I got all excited reading about Microsoft Research AutoCollage software and took it out for a spin. But at the end when I tried saving and exiting the app it crashed on me. The weird part was that all along, I was expecting it to crash so it didn't bother me. Why? I think subconsciously for a couple of reasons - it was MSR so I was not expecting released product quality from it, even though we are selling the software on our brand spanking new www.MicrosoftStore.com and secondly because it was doing a lot of fancy, CPU intensive image processing which made me just hold my breath and wait for the inevitable crash (though to their credit it didn't quite occur during the processing). Oh! We have such a long way to go.
Anyway, this talk is all about the future and not the present. The present sucks, we all know that. But fear not, for we brave, intrepid testers will fix all of that (or at least find all of that - we'll need some devs to fix it for us - we can't touch the product code you know!) :)
Right on the heels of VS 2008 SP1 the VS 2010 CTP is available for download as a Virtual PC 2007 image (so it is HUGE - best to use a tool to download the image).
Check out the new WPF based XML Editor (basically building on top of the new WPF Text Editor) as well as the brand new XML Schema Designer (more of a viewer currently though) which adds to the XML Schema Explorer in Orcas SP1 and add multiple designer views such as a Graph View and a Content Model View to make working with XML Schemas really easy and intuitive.
Here are some screenshots of the improvements that we shipped in this CTP. We would love to hear your feedback about any of these tools.
(New WPF based XML Editor)
(Content Model View of the new XML Schema Designer)
(Graph View of the new XML Schema Designer)
has released. w00t!
For the XML Tools team this is an especially big release since SP1 is the vehicle in which two of our big investments went out for the first time. Congratulations to the feature teams involved in getting these out to customers.
XML Schema Explorer which adds to the already rich set of XML Tools such as XML Editor and XSLT Debugger that we ship in VS.
ADO.Net Entity Designer which is a first class designer to help customers adopt and work with the new ADO.Net Entity Framework shipped in .Net FX 3.5 SP1 for the first time.
Both of these tools are incredibly cool (since they are WPF based) and provide a good sense of the direction all our tool offerings will be taking in future versions of VS. Please provide all your suggestions and comments through this MSDN Forum.
It's great to see a lot of the work that has been going on in our product unit finally get into the hands of customers with the release of Visual Studio 2008 SP1 Beta.
Nice to read Scott's post on the Beta in which he talks about among other things some new XSD Browser integration support upcoming in VB (in the final RTM release of SP1) and the cool new Entity Framework Designer which let's you graphically create and edit EDMX files. The XML Editor also includes bug fixes for customer reported issues such as around the way Outlining works in XML documents (much better with SP1).
Sure - it's a little late in coming, but still feels good to say that Visual Studio 2008 has shipped! You can try the 90-day trial versions or the FREE Express Editions of Visual Studio 2008 right now and get started.
There are a ton of cool new features and improvements on the XML Editor and XSLT Debugging (not available in the Express Editions) fronts that can help you be more productive working with XML data.
A couple of my favorite features are the automatic XSD generation from XML files which allows you to have validation and intellisense on any XML file you are working with and the ability to set breakpoints in the input XML files that an XSLT is actually transforming.
A great way to get introduced to the XML / XSLT tools in VS 2008 is through this article by Stan Kitsis in this special issue of the CoDe Focus Magazine which is dedicated to Data Programmability technologies:
XML Tools in Visual Studio 2008
There is another interesting article in an earlier issue written by Chris Lovett (creator of the immensely popular free tool XML Notepad 2007) which provides a much more detailed view of how the XML Editor is evolving into a platform which other tools and designers can use to work with XML documents:
How XML Tools Use the Visual Studio SDK
You should also check out the set of data programmability technologies that will allow SQL Server 2008 to be a very rich experience for developers.
After a memorable ~5 years on the Commerce Server team, I recently changed gears and have joined the world of SQL Data Programmability.
There are a ton of cool things happening on this front and the area my immediate team will be working on currently is the XML Editor and XSLT Debugger (yes - there is an XSLT Debugger and it completely rocks!) tools in Visual Studio.
Looking forward to blogging more on this new (to me!) world of XML and Data Programmability Tools.
Great to see the MSDN Tester Center go live! You can expect to see this evolve as a central portal for all-things-test and a window into a lot of the great test minds working at Microsoft!
Check out the great set of Whiteboard Videos already available on this site. Kudos to the team that got this off the ground!
Oh well...I am biased since I worked on this...but I really think this is a COOL demo of silverlight and AJAX and what eCommerce scenarios can look like.
I especially like the use of videos which I was initially skeptical about but I think can actually influence the impulsive buyers a lot while creating a really immersive shopping experience for the other visitors as well. You will of course need to install Silverlight if it's not already installed on your machine.
Here is a page with video on it.
And on this page you can see a sunglass and zoom in easily and see various colors etc. of the sunglass via hovering. Try adding to shopping cart for a very snazzy UI experience as well.
All in all I think it is a great demo - Kudos to everyone who worked on it!
In the last 2 posts we have looked at what we mean by a model of a system and defined what States and State Variables are. In this post we'll look into Actions and Action Parameters.
We can define an Action as something that modifies or exercises the system. Thus "TurnIgnitionOn" (in the car model being used in this series) can be an action which causes the system to change in some way. When considering Actions you need to think about what exactly is it that changes in the system and which all state variables are impacted by those changes. An action is therefore simply something that causes the system to change states.
Actions become much more powerful and useful when combined with the concept of Action Parameters which can be considered to be similar to method parameters. Like a method (in a programming sense) does some work which can be tweaked or controlled by the method parameters, the effect of an Action can be affected by Action Parameters. In the car driving model an example of an Action Parameter would be whether the ignition could be turned on remotely (via the keychain) or locally. In that case we might want to add an Action Parameter to the model such as TurnIgnitionOn(Mode) where the "Mode" action parameter could take values "Remote" or "Local" similar to the way state variables can have different values.
A really interesting point that emerges here is that there is no one way to model the system. In the example above if for some reason you actually wanted to track how you had opened the car, maybe for a later stage in the model when you were actually locking the car then you would need to store this information in a state variable. Anytime you find yourself using the word "track" you should think about having that value as a state variable. We could therefore break our TurnIgnitionOn action into 2 different types of actions, TurnIgnitionOnRemotely and TurnIgnitionOnLocally and track how we opened the car as a new state variable, say S3, with values Remote or Local. Our model would therefore look something like this:
I will end this post with a word of caution. Even though it is quite easy to add a state variable to the model to track some aspect of the system, you should be very careful when adding state variables to your model and strive to minimize this number. The reason for this guidance is to limit the amount of state explosion that takes place as a result of defining these additional state variables. Remember that every time you define another state variable the number of possible states in the model is increasing exponentially. To keep your model manageable you should not add a state variable till it is absolutely critical to the behavior being captured.
UPDATE: Just noticed that this is actually my 100th post on this blog - WOW! Never thought I'd see the 100th post when I started out...it's been fun! :)
This is a continuation of the Elements of MBT series of posts which I started last time with Part 1 and covered what is meant by a model of a software system. In this post I shall be covering additional concepts which you will need to know to gain a better understanding of Model Based Testing.
One of the most fundamental concepts required to define a model of a system is that of State Variables which in turn are needed to define what a State is. A State Variable is simply some aspect of the system which changes over time. For e.g. in the car driving model, one aspect that changes (and which you care about tracking for the purpose you are building the model for) is whether the car is moving or not. So this can be a state variable, say "CarMotionState", which can have values such as "Moving" and "NotMoving". Couple of important things to keep in mind is that a state variable can have multiple values (not just 2 as in the example above) and that these "values" are really just semantic terms to help us define the model. What the actual values of these do and result in (especially from an automation point of view) is tied in later when you are implementing the code for the model.
A State then becomes much easier to define as simply a combination of all the State Variables in the model with particular values assigned to each state variable. Thus if your model contains 3 state variables S1, S2 and S3 with multiple values for each state variable, a state will be defined by a unique set of values for each of the state variables S1, S2 and S3. You can therefore think of a state as simply a snapshot of all the state variable values at a given point in time. From the Part 1 car driving model we can now see that the 3 states in the model are comprised of the following unique combinations of state variable values.
S1 = { IgnitionState = IgnitionOff; CarMotionState = NotMoving; }
S2 = { IgnitionState = IgnitionOn; CarMotionState = NotMoving; }
S3 = { IgnitionState = IgnitionOn; CarMotionState = Moving; }
One thing to notice here is that not all possible states need to be defined for a particular model. Depending on the system and the reason you are creating the model or the abstraction level you want to capture, you might not include certain states in your model. Or certain states simply won't make any sense to model such as the state "IgnitionOff" + "Moving" if you do not want to capture the behavior of the car sliding down hill when the ignition has not been turned on. The great thing though is that it forces you to think of that scenario and to consider if that is something you want to capture and if so then what the expected behavior would be.
In the next post I shall cover Actions and Action Parameters and update the model to add an Action Parameter to one of the actions.
In this series of "Elements of MBT" posts, I will be covering some basic concepts used when trying to build models of software systems for use in Model Based Testing. The aim is to give an explanation of the various elements involved in defining a model of a software system and hopefully this should get us on the same page and allow us to explore the complexities in more detail.
Let us start by going over what we mean by a "model" of a system. Put simply, a model is a description of the system and is much simpler than the actual system itself. What this means is that the model hides a lot of the details about the system and hiding these details allows us to work at a higher level of abstraction. For e.g. a simple model of driving a car could be:
One thing to keep in mind is that there is no perfect model of a system, since the same system can have completely different models depending on what you plan to use the model for. Are you creating the model of a system simply to capture the behavior and requirements of the system? Are you creating the model to auto-generate the code for the model implementation itself? To describe the data flowing through the system? Or are you generating the model to actually test a particular portion of the system?
A quote which captures this unusual thing about models really well is as follows:
All models are wrong; some models are useful.
- George Box
What is worth hanging onto from this definition is that a model that you create should strive to be "useful" more than anything else. It might be rough round the edges and might not tell you everything about the system (for e.g. you don't know from the above model how the gear is to be changed or what happens when a gear is changed) but if it is useful for the purpose it was created it is a correct model.
Another aspect of defining a model is the notion that the model should help predict what the behavior of the system would be under certain conditions. So if you are in the "Not Moving" state and you "ChangeGearAndAccelerate" then you reach the "Moving" state. Are there certain assumptions in the model? You bet there are! Such as assuming that the ignition is on, that the car has wheels, that the car is not hanging in the air, that there is oxygen in the air, we can go on and on and on. The hiding of these details and assumptions is what helps us look at just this behavior we are capturing. If anything is important for the purpose (say testing) for which we are creating the model, we should consider adding that detail into the model. So if we thought that it was important to know whether the ignition was on or off we would change the model to capture this as well.
Now we know that if you are in the "Ignition Off" and "Not Moving" state then if you "TurnIgnitionOn" you reach the "Ignition On" and "Not Moving" state since simply turning on the ignition does not cause the car to start moving in the model that we have created. To start moving you will need to "ChangeGearAndAccelerate" which should take you to the state of having the "Ignition On" and the car "Moving".
Hope this gives you an idea of what a model contains and what it can be used to capture. In the following posts we will look into each element of a model in more detail.
And so can you. Simply while chatting with your buddies. What I find quite amazing about this initiative is the no-limit on the amount of donation made to each organization.
i’m is a new initiative from Windows Live™ Messenger. Every time you start a conversation using i’m, Microsoft shares a portion of the program's advertising revenue with some of the world's most effective organizations dedicated to social causes. We've set no cap on the amount we'll donate to each organization. The sky's the limit.
You know your team members are really catching onto this whole model-based way of thinking when they pull your leg over a few drinks in a bar and say (tongue in cheek): "So Nihit, the 'Action' of having this drink will take me from a 'Sober' state to a 'Drunk' state, right?".
Yes - exactly - it will (or maybe we need to accommodate degrees of drunkenness as well - not everyone is drunk after a single drink!?) and for completeness sake you can even have an Action Parameter for the type of drink being passed into the "DrinkAlcohol" action! Isn't that neat!? :)
Anyway, here's wishing everyone a wonderful 2007 ahead. Happy New Year!
Question to the community: How can you best test a delete operation?
For e.g. When you test the following API - DeleteObjectFoo("ABCD") - what should we test to verify that this function actual deleted "ABCD"? This API is just a placeholder for an API that deletes something. The question is that, in general, how do we verify that "something" was indeed deleted?
Should we simply test that "ABCD" is deleted from the database (or whatever the underlying storage is)? That is typically what is done. The question then becomes how do we catch a bug which not only deletes "ABCD" but also "XYZ"???
A colleague some time back had raised this issue since we ran into exactly this sort of a bug and found to our dismay that the existing tests were not catching this bug.
What do we change to test for a delete operation in general so that the testing becomes foolproof - or can it even be made foolproof?
The
Process Monitor looks like a great tool to help in debugging issues and includes a bunch of features from a diverse set of sysinternal utilities such as RegMon, FileMon etc. I think this is the first tool offering to come out since the acquisition of
Sysinternals.com and more importantly perhaps,
Mark Russinovich (who has bunch of interesting posts about debugging issues). Looks like there is even a dedicated
Sysinternals Blog at TechNet and if you are not familiar with the sysinternal family of tools there is a free
Tour of the Sysinternals video available.
