Darryl Russi's Blog

Debugging Tip: Viewing Address of Managed Code

2012-02-08T07:39:00Z

Ever need to write managed code that then talked to unmanaged code elsewhere? Sometimes when debugging about why things are not working the way you expect it is useful to just inspect the values on both systems to understand the differences. One of the difference with managed code is that you rarely need to access the actual memory address, but there are times it is useful. In Visual Studio this can be done simply by using the immediate window.

For instance if I wanted to know the address of a byte[] from the following code.

You can dereference the item to get the memory address by opening the Immediate Window & entering the following:

Now you can drop the address of the memory into the memory window to see how it is laid out.

Challenges in Test–Ensuring Testability/Influencing Design

2011-08-30T19:14:00Z

Products are primarily designed for customers, yet a test team must test these products. Who is thinking about how you might test the product? If you are a tester, you better be thinking about it and providing feedback to your developers about their design. Your feedback should both be for the customer and how to better enable you to test the product effectively.

For some of technologies there may be tools you can use such as the IAccessible interface for many windows forms or the tools that we ship as part of Visual Studio Test. Yet you may be on a product for which there are no other people testing that technology & therefore less existing items you can leverage. A tester needs to know what technologies are out there & be willing to dig in and implement similar technologies if they do not exist for their environment.

For example how can you do UI automation on an Xbox console? The console does not have public interfaces for manipulating UI elements & is completely different than a traditional WinForm. Additionally, it uses a different processor architecture (PowerPC) than what many Microsoft products use (x86, x64, & more recently ARM), so some tools & libraries won’t work on it. What does it’s UI need to support for you to be able to have reliable automation? What would you ask the developer to do for you to get better automation? Below are a couple items that you could use when designing your product to improve testability. As always with test, there is no one magic bullet but instead they are tools that we can leverage where it makes sense for our development.

Model View Controller (MVC)

Design plays a large role in how you can test a product. It influences how devs write their code & how you can then write your test code. There are many concepts out there to help improve the testability of a product design. One example is Model View Controller (MVC). The idea with this design pattern is that you minimize the logic that is in your UI (aka View) and push your business logic into the model. The controller then manages the interaction between the two. With most of your business logic pushed into the model, this means you can now manipulate the model directly possibly via APIs and do a large amount of testing that way instead of relying on the UI. Traditionally API testing is much more reliable & robust than UI testing as UI changes more often and has timing issues with regards to knowing when you click on something when the next screen of the UI is ready for input. This design also has the added benefit of allowing the developers to quickly add additional views with out much effort.

A newer pattern which is similar and targeted at several of the latest Microsoft UI development platforms (Windows Presentation Foundation & Silverlight) is Model View ViewModel (MVVM). This adds a ViewModel which is an abstraction of the view used in data binding of the model to the view.

Mock Objects

How are you going to test error conditions in your code? What if components you require are still under development? Are you testing a UI component which requires a service? Mock objects are a useful way to tackle these types of problems. If you design your code to allow for the ability to use mock objects, you can simulate them. In order to do this your product needs to be written clean enough that you can swap mock objects & the real objects.

Take for example a product that relies on a SQL database. You likely have a data layer that handles your calls out to SQL. When implementing this data layer, you can implement a mock object to simulate the SQL server. This would allow you to do things such as simulate the results of the database or if the network is down. Mock objects are also useful at the unit testing level as the developer can isolate their component to only rely on these mock objects. This is especially useful if they depend on another component also under

Technorati Tags: SDET,Test

development which may not be ready for consumption yet. If you are following test-driven development (TDD) then it is also useful for being able to write tests prior to implementing the product code.

Conclusion

Similar to developers, testers need to stay abreast with the various design patterns that can be used in development. As products continue to grow in complexity gone are the days of a tester being able to strictly white box test and black box test an application after development is code complete. Doing so will likely result in a product that is highly difficult to test and result in automation that is less robust and reliable. Instead we expect and involve our testers from the earliest stages of design both as a sounding board for designing a solution to a problem and as a source of input to ensure the product we develop is highly testable.

Challenges in Test – Impeccable Infrastructure

2011-08-22T09:00:00Z

In order to test a product there is always a fair amount of infrastructure needed to keep the wheels running. Just a few examples of the types of infrastructure that test requires are:

Test harness
Logger
Common code for building test automation
Reporting websites
Databases for storing various results
Machines for running tests in the lab
Topology for your product to simulate customer deployments
Tools for managing your infrastructure
Tools for testing & possibly for customers
Etc.

As you might imagine this can become quite an undertaking and in larger teams this is sometimes delegated to a dedicated lab team to support.

On top of all this work is the set of tests that your team is running & many times it also supports your development team for their development needs. A high quality infrastructure is something that can sometimes be taken for granted, until there is a problem which can at best slow down your team’s efficiency or at worse invalidate your testing due to some bad assumptions.

Bugs are an equal opportunity employer so similar to the previous post how you need to prove your product is correct, that extends to your test infrastructure. When you find a bug are you ready & able to prove that it resides in the product code & is not a problem with your test infrastructure? Here are a couple real world examples of both to further demonstrate the need to focus on how you are building your test infrastructure.

Will this thing ever finish?

Working on a sync engine for Microsoft Identity Integration Server (MIIS) we were investigating the ability of the product to scale. As such a tester had instrumented some counters to track the rate that sync occurred as it ran through a couple million objects. This process was taking a couple days, which was expected due to the scale being tested. The tester was then leveraging the data about their rate of sync to determine the performance of the synchronization engine and then extrapolate out an ETA for the test to complete. As he plotted this out each day the rate would drop and he would give an updated ETA. Below is a hypothetical table for the data that was being collected.

Day	Rate (Objs/day)	Objects Remaining	Est. Days remaining
1	50,000	2,000,000	40.0
2	50,000	1,950,000	39.0
3	49,000	1,900,000	38.8
4	48,000	1,851,000	38.6
5	45,000	1,803,000	40.1
6	42,000	1,758,000	41.9
7	40,000	1,716,000	42.9

Note: This data is completely made up just for illustration as it was years ago & I don’t have the actual data.

After several days of work, we realized that every day the rate moved enough that it would never complete. Sound the alarms, we have a bug!

Now you spin up your developer & proudly point to a newly found bug. After some investigation through it is found out low & behold yes there is a bug, but it is the test infrastructure causing it. What is that you say? How can my test cause a product bug? In this case it turned out to be that the way the tester implemented the counters for monitor the product were doing an expensive query via WMI on a VERY aggressive basis. This query was tying up system resources & therefore causing the slow down in the product.

Lesson learned is to be aware of the observer effect and try to ensure your test code is as unobtrusive as possible when measuring items like performance.

Test A says pass, but Test B says fail?

In another example of working on performance on Forefront Identity Manager (FIM) we were in the midst of a major improvement to performance. We had some very talented developers on the project & as a great developer they had written some excellent unit tests to help test their progress closer to the API level. Similarly we had some tests which covered end to end UI scenarios for performance. The developers reported great progress in improving our performance and so we kicked off our tests to see how we were looking end to end. It so happened that our management was meeting to discuss our project progress with their managers, so they were eager for an update for performance. We kick off the UI performance tests & find that everything is timing out!

We have two very different results, so which should we tell our managers to report? We have to figure out which test is correct and why is there a difference. Well a good start is to minimize the variables, so we took the tests out of the picture by doing some manual testing. Sure enough our manual results were inline with what the UI tests were showing,

Technorati Tags: SDET,Test

but why are the tests showing different results? We then started profiling the SQL server using both the unit tests & the UI test. Sure enough the queries reported the differences we saw in the amount of time taken. This goes back to knowing your product, as based on our knowledge of the product we had a good idea of where to start looking. We then ultimately debugged into the unit test & realized that it had a config switch that added some additional optimizations which the dev had forgotten to enable in the product.

Build it & they will come

All this to say whenever you build any piece of code, there are sure to be bugs to come from it. Teams must not view something as just test code & therefore hold it to a lower quality bar, otherwise it will quickly cost them time & invalidate their results.

Similarly as a tester you get to build all kinds of infrastructure to help make the team more efficient. This also means you get to learn all kinds of new technology. Need to build a new report? What tech do you want to use to build it? Why not write it in Silverlight or WPF to learn something new? You get to be the PM, Dev, & Tester for these types of items. Just remember you want to test it thoroughly, otherwise there might be a bug that will bite you later.

Challenges in Test–Proving your feature

2011-08-15T09:00:00Z

Technorati Tags: SDET,Test

When you take on a feature to test, how does one prove it is right? Generally you might work off some checklist of buckets or areas like functional, globalization, security, boundary, localization, performance, stress, etc. Then say once you have run all the tests you came up with & made sure they passed, then it must be done right? As alluded to earlier there is also the question of did you build the right feature for the customer?

The below picture has circulated before on how the various components of product development looked at a simple problem.

One might add to this “How the tester tested it”

Part of test’s responsibility is to prove that their feature is correct. This includes not only that it is functionally correct but that it is actually what the customer wants\needs.

Customer Correctness - Involve your customers

I covered this in the previous post about knowing your customer, but it is worth reiterating as it is highly important. Involving your customer as early & as often as possible will help you increase the likely hood of delivering what they really need. There can be obstacles such as maybe you are creating something brand new so your customer has not seen anything like it before or maybe you have competitive requirements to limit disclosures. Yet these are usually not insurmountable, at the bare minimum you can solicit feedback from co-workers on the work that you are doing. Is what you are working on targeted at a less technical customer & all your co-workers live & write code, then what about your business admin?

Functional Correctness - Quod erat demonstrandum (QED)

A typical question in a product release is how do we know that the product is done & ready for release. This can fill up many meetings & even books about the subject. One aspect of this I wanted to focus on was thinking beyond the procedure of manually coming up with a set of test cases, build them, & then run them. Instead, here at Microsoft many testers have degrees in Computer Science\Engineering and many have advanced degrees including Masters or PhDs. As such we should borrow from our math background & think how you might prove the feature is correct (i.e. QED)

This is more aspirational, but I have seen concepts & ideas used in test to attempt to do this exact thing. As such this is one of the hard problems in test. More broadly these are collectively known as Formal Methods and this is one area which MSR has made considerable investment.

Model Based Testing

One example of how you can prove a feature is correct is model based testing. It is an interesting aspect of how you might prove that a system is correct. I am personally still learning more about model based testing & how we can use it in our testing tool belt when developing products. So this is just an intro to the concept for those unaware.

The basic concept is that if you can map out a conceptual model of your product, then you can use an algorithm to walk your model which will then generate many of the test cases for you. Just taking the time to create a model will greatly help in understanding your product and typically cause you to find questions which will then feedback into your dev & PM specs. For example, what happens if we are in X state & then Y happens?

Microsoft Research has done some work around Model Based Testing (MBT) and since has been released as a power tool for Visual Studio 2010 called Spec Explorer. You can even use Spec Explorer to convert your understanding of the spec into a programmatic model and then when the model is explored find if there are invalid transitions that occur based on some constraints helping you find bugs in your spec & design early on.

One key consideration when building out a model of a system is the scope of a given model. If you attempt to model in detail a complex system, then you will run into state explosion where you will have way too many permutation & combinations to be useful. Instead you should focus on modeling some specific area you are testing to minimize this problem.

Oracles

In order to verify any test result, a tester must develop an oracle to determine if the result of a test should pass or fail. Sometimes these oracles are extremely simple. The simplest being hard coding the answer to each of their tests. Yet this limits the number of tests to the ability of the tester to generate tests & answers to those tests. Those answers must be maintained as the product changes over the course of development. On the other extreme is to perform complex calculations which simulate the product. The problem here is the classic of if test is testing product code who is testing this complex test code?

MBT solves this by using the model you create about your product. Tools like Spec Explorer explicitly capture state in the model and allow for the model to return values, which can then be mapped to actual values which the system will return. So tests generated by Spec Explorer essentially have the oracle embedded in each generated test case. But to do this, the model may have to closely reflect the algorithmic complexity of the system being tested. This can lead to high cost of model building. MBT approaches suggest to solve this by using a collection of models, including the possibility of building composite models (i.e. A more complex model built on top of simpler ones)

There are also other possibilities. Take another example of how one might leverage mathematic ideas is how you can test something as Query & Set calculation in Forefront Identity Manager (FIM). Given a complex problem to test, the test writer needs to create an oracle or way to verify the results that the product is producing. The complexity of tests can be limited by the imagination & ability of the tester to validate their results. Yet if you can leverage something as an oracle to verify the correctness of another item. In the case of query if we can express a dataset in another form & translate our query against that dataset into a different form, then we can create a simple oracle. So taking Query in FIM, we know that the query is given in XPath. Conveniently enough XPath is typically used for querying an XML document. So now if we can represent our dataset as XML, then we could use XPath with .Net as that oracle.

This leverages the concept of transitivity in mathematics. Basically if A=B & B=C then A=C. Or in our case if it is true that .Net XPath returns a valid search result for a XML document and we represent our dataset as a XML document, then this will give us our answer\oracle. This has the advantage of us being able to issue complex tests & leverage existing work to validate our new work. Otherwise we would have to hard code our own test inputs & expected outputs or create a verification algorithm which could reach the complexity of the product’s algorithm.

There are of course some assumptions & limitations here. First we are assuming that the .Net XPath of a XML Document returns the correct answer, but there may be bugs that cause it not to. Secondly we may have data or queries that we are unable to represent appropriately. These are items the tester has to evaluate as they are determining their approach & may choose to do additional testing around the limitations of their approach.

Now that we have Query’s oracle we can use transitivity again and use Query as an oracle for Sets. Sets makes a bunch of complex calculates on the fly. As such we can build a test case generator based on some descriptive language & then just turn around and verify that all the Sets that are calculated on the fly are correct by walking the Set memberships & checking them with a simple query.

Note: I don’t claim credit for the above solutions using in testing Query & Sets in FIM. These were great ideas that other testers on the team developed for dealing with testing these hard problems.

Conclusion

As a tester you should consider not just does your feature conform to the spec that you have been given, but is the spec & design correct to start with. This must include rationalizing it against your customer’s needs. Additionally a tester should move beyond just simplistic functional testing of a complex system but how they can prove the correctness of their feature. Proving correctness should move beyond static validation of 1+1=2 and more into leveraging the testers backgrounds in mathematics and other algorithmic approaches where applicable.

Ultimately when developing your test strategy here are a couple questions you should strive to answer:

Is this the right feature for my customer?

To be right it must include solving a customer problem & delivered in a way that they can consume it.

How am I proving that the feature is functionally correct?
What are the limitations & assumptions of my approach?
Am I testing my feature in a way to cover how my customers expect to use it?

Pasting Code

2011-05-09T16:19:00Z

I have been looking around for a bit for a way to paste code for blog posts. My requirements were pretty simple:

Able to paste code from blog - such that readers could copy & paste code into their own editor
Syntax highlighting – Syntax highlight just makes things easier to read
Minimize scrollbars – Avoid requiring scrollbars to scroll through the code

As such, I have checked out several & the one I found to work the best so far is the Paste As Visual Studio Code add-in for Live Writer. Ideally, I could use something not specific to Live Writer such as Copy As HTML add-in for Visual Studio, but that is currently only supported for VS 2008.

If you have a favorite tool for copying & pasting code, let me know. One addition I would like is the ability to hide\show code in my blog posts to avoid cluttering it up too much.

DataContract Serialization with Generics & Read-only Properties

2011-05-04T22:25:25Z

Recently I have been messing around with figuring out how to serialize some data back & forth between two applications. The code in particular leveraged several interfaces, classes with readonly properties, & generics. This caused a couple problems with serialization which I thought might be useful to share for others that run into a similar problem.

Take for instance the following code:

static class Program
 {
     public static void Main()
     {
     
         Class1 m = new Class1(new MyClass<string>("Test"));
         string xml = m.Serialize();
         Class1 m2 = Class1.Deserialize(xml);
 
         Debug.Assert(m.Value.ToString() == m2.Value.ToString());
     }
 }
 
 [Serializable]
 public class MyClass<T> where T : IComparable
 {
     readonly T item;
 
     public T Item { get { return item; } }
 
     public MyClass(T obj)
     {
         item = obj;
     }
 }
 
 [Serializable]
 public class Class1
 {
     public Class1(object val)
     {
         value = val;
     }
     
     public string Serialize()
     {
         XmlSerializer serializer = new XmlSerializer(typeof(Class1));
         StringWriter writer = new StringWriter();
         serializer.Serialize(writer, this);
         writer.Close();
         return writer.ToString();
     }
 
     public static Class1 Deserialize(string xml)
     {
         XmlSerializer serializer = new XmlSerializer(typeof(Class1));
         StringReader sr = new StringReader(xml);
         Class1 item = (Class1)serializer.Deserialize(sr);
         return item;
     }
     
     readonly object value;
 
     public object Value { get { return value; } }
 }

Parameterless Constructor

If you run this through & try to serialize & then deserialize you will run into a problem just serializing this. The first error you will hit is:

System.InvalidOperationException was unhandled
Message=There was an error generating the XML document.
Source=System.Xml
StackTrace:
       at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id)
       at System.Xml.Serialization.XmlSerializer.Serialize(TextWriter textWriter, Object o, XmlSerializerNamespaces namespaces)
       at System.Xml.Serialization.XmlSerializer.Serialize(TextWriter textWriter, Object o)
       at MyNamespace.Class1.Serialize()
       at MyNamespace.Program.Main()
InnerException: System.InvalidOperationException
       Message=MyNamespace.MyClass`1[System.String] cannot be serialized because it does not have a parameterless constructor.
       Source=System.Xml

Xml Serialization requires you to have a parameterless constructor, so you can either add one, or switch to using WCF DataContract Serialization. This is as simple as changing the attribute on your Classes from [Serializable] to [DataContract] and then updating your Serialize & Deserialize methods. This will allow you to successfully serialize & deserialize the objects without an exception.

[DataContract]
 public class MyClass<T> where T : IComparable
 {
     readonly T item;
 
     public T Item { get { return item; } }
 
     public MyClass(T obj)
     {
         item = obj;
     }
 }
 
 [DataContract]
 public class Class1
 {
     public Class1(object val)
     {
         value = val;
     }
 
     public string Serialize()
     {
         StringBuilder xml = new StringBuilder();
         DataContractSerializer serializer = new DataContractSerializer(typeof(Class1));
         using (XmlWriter xw = XmlWriter.Create(xml))
         {
             serializer.WriteObject(xw, this);
             xw.Flush();
             return xml.ToString();
         }
     }
 
     public static Class1 Deserialize(string xml)
     {
         Class1 newItem;
         DataContractSerializer serializer = new DataContractSerializer(typeof(Class1));
         StringReader textReader = new StringReader(xml);
         using (XmlReader xr = XmlReader.Create(textReader))
         {
             newItem = (Class1)serializer.ReadObject(xr);
         }
         return newItem;
     }
     
     readonly object value;
 
     public object Value { get { return value; } }
 }

Serializing Read-only Properties

Next you will his the Debug.Assert which was a check to ensure we are serializing the read-only property was correctly being populated & in this case it is not. The reason being is that XmlSerialization will automatically attempt to serialize all public properties, but WCD DataSerialization does not automatically attempt to serialize any properties. So you need to decorate any item you want serialized with a [DataMember] tag. This is great because now for readonly properties you can just mark the underlying private data as the DataMember.

[DataMember]
 readonly object value;
 
 public object Value { get { return value; } }

Serializing Generics

The next problem you will hit is a SerializationException about not being able to Serialize the generic MyClassOfString, aka MyClass<string>

System.Runtime.Serialization.SerializationException was unhandled
Message=Type 'MyNamespace.MyClass`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]' with data contract name 'MyClassOfstring:http://schemas.datacontract.org/2004/07/MyNamespace' is not expected. Consider using a DataContractResolver or add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
Source=System.Runtime.Serialization

To resolve this you will need to use the KnownType attribute & there are a couple options:

Generically implement serialization
Explicitly list the known types

Generic KnownType

Generic serialization is nice as the serialize since you don’t have to decorate your code with the various possible classes used for your generic. To implement it you add the KnownTypes attribute to your class with a string of the method name to call.

[DataContract]
 [KnownType("GetTypes")]
 public class MyClass<T> where T : IComparable
 {
     readonly T item;
 
     public T Item { get { return item; } }
 
     public MyClass(T obj)
     {
         item = obj;
     }
 
     static Type[] GetTypes()
     {
         return new Type[] { typeof(MyClass<T>) };
     }
 }

Explicit KnownType

If you do the above then you will wind up with an exception during deserialization now since Class1 does not how to deserialize MyClassOfString. As the deserializer you will need to explictly state what types you expect. In this case you can do it with a single attribute on the top.

[DataContract]
     [KnownType(typeof(MyClass<string>))]
     public class Class1

Final Results

At this point the class will successfully serialize & deserialize fully. Due to the nature of my example, the step of using the GetTypes method is not required since I am doing both serialization & deserialization in Class1, but it is useful when that is not the case.

static class Program
 {
     public static void Main()
     {
     
         Class1 m = new Class1(new MyClass<string>("Test"));
         string xml = m.Serialize();
         Class1 m2 = Class1.Deserialize(xml);
 
         Debug.Assert(m.Value.ToString() == m2.Value.ToString());
     }
 }
 
 [DataContract]
 [KnownType("GetTypes")]
 public class MyClass<T> where T : IComparable
 {
     readonly T item;
 
     public T Item { get { return item; } }
 
     public MyClass(T obj)
     {
         item = obj;
     }
 
     static Type[] GetTypes()
     {
         return new Type[] { typeof(MyClass<T>) };
     }
 }
 
 [DataContract]
 [KnownType(typeof(MyClass<string>))]
 public class Class1
 {
     public Class1(object val)
     {
         value = val;
     }
 
     public string Serialize()
     {
         StringBuilder xml = new StringBuilder();
         DataContractSerializer serializer = new DataContractSerializer(typeof(Class1));
         using (XmlWriter xw = XmlWriter.Create(xml))
         {
             serializer.WriteObject(xw, this);
             xw.Flush();
             return xml.ToString();
         }
     }
 
     public static Class1 Deserialize(string xml)
     {
         Class1 newItem;
         DataContractSerializer serializer = new DataContractSerializer(typeof(Class1));
         StringReader textReader = new StringReader(xml);
         using (XmlReader xr = XmlReader.Create(textReader))
         {
             newItem = (Class1)serializer.ReadObject(xr);
         }
         return newItem;
     }
     
     [DataMember]
     readonly object value;
 
     public object Value { get { return value; } }
 }

Challenges in Test – Knowing your customer

2011-03-22T17:03:34Z

So you know your product & the code, what else is there? Your customer! They are the ones that will be using your product to solve their real problems.

Knowing your customer

So your PM, Dev, & you have implemented the greatest feature. Do you know what problem this is going to solve for your customer? Do you know how they expect to use it? If they are going to be deploying your product do you know what type of topology they will be using?

A good example of this is the Mars Rover that was lost in 1999. One team designed their system using English units, but the other team was actually using Metric units. I bet their tester ensured the feature worked wonderfully but there was one costly assumption that was overlooked & after a 286 day journey the probe was burned up in the atmosphere.

As a test team you will be charged with figuring out how you will be testing the product. The assumptions you make will wind up impacting your customer & their experience with your product.

Developer Bias – “It works on my box”

During the course of development it is easy for something I will call developer bias to set in. You have been working on the product & seen it evolve over time. You can easily grow accustomed to issues or why something might work the way it does even though it is not intuitive to your customer.

One refrain that exemplifies this is a phrase you might hear from a developer of “It works on my box.”, when you report an issue with the product.

Take for example a server product which uses SQL as a datastore. During development most of the team will likely install the product all on a single box. Who wants to maintain multiple machines if they don’t have to after all. But customers won’t be running in that configuration & will likely have a SQL machine & run your server on a separate machine. Does your product work in that topology? Oh & how did you install your SQL server? SQL server can have named instances. Can you install to a named instance? Does your product work when installed to a named instance?

Customer Personas – “It works great for me”

Although you may be one of the target customers for your product, are you a true representative of all your customers? What does a teenager expect form your product vs. a stay-at home mom? Each customer has their own unique needs & scenarios that they are interested in, you need to be aware of them and how you can cover them in your testing.

Let’s look at something such as Kinect for Xbox 360 & a feature as simple as someone waving to launch the Kinect Hub. Seems pretty straight forward right? Interestingly enough eHow even has instructions that are as simple as 1, 2, 3.

Lift your hand toward the person to whom the wave is being directed.
Spread the thumb away from the other fingers of the hand and hold the rest of them together.
Pivot at the elbow & move your hand from side to side several times

It all seems so simple, but do a sampling of people & have them wave at you. You will find a large variance in how they actually wave to you. Some might just slightly move their wrist, others might move their hand back & forth very rapidly. Are kids part of your customer set? They can wave even more differently with their hand almost sideways moving more up & down.

Conclusion

But I am just a tester, isn’t PM the one that talks to customers? They do, but do they understand the assumptions you are making in testing? Have you used the many resources you have to find out information from your customers? At Microsoft there are several avenues to get this information: TechNet Forums, Pre release programs through connect.microsoft.com, various conferences, external user groups, internal distribution lists, MVPs, etc. Work with your team to reach out to your customers, I am sure they will be more than happy to help you build a great product that will help solve their problems.

Challenges in Test – Knowing the code & sometimes ignoring it

2011-03-14T18:54:33Z

A tester just needs to master the art of understanding their product & then file bugs based on their findings right? Wishful thinking! You better have the technical understanding of the code your developer is writing if you intend to properly test it.

Knowing the code & sometimes ignoring it

So understanding how your product works is a good start, but as mentioned earlier a tester needs to have breath & be able to go deep. Part of that depth is understanding the code behind their product.

There are a couple examples of why you will need to have this depth:

Designing your automation frameworks
Debugging your product

Designing your automation frameworks

While development is designing & implementing the product you will need to figure out how you are going to test the product, while also being involved in the product design itself. Understanding how the code is implemented is going to help with designing your automation. For example if you have a typical 3 tier application, how is the UI talking to the middle tier? Are those APIs public? Do they give you what you need to write automation? How can you extend them for what you need? Is there a better way to design the product to make it more testable?

Debugging your product

Part of being a tester means you are one of the first customers of your product. You are developing tests against a moving target as developers are implementing new features & making changes. One day a test may be passing & the next it fails. Is it a test issue or a product issue that is causing it to fail? Your product may not have much manageability yet so there are no good events in the log that you can use. The only thing left is being ready to strap a debugger to the product & step through the code to understand where things are going wrong. If it is a product change that broke your tests you may need to understand how to update your tests to work again. Oh & when you are done, you’ll want to file a bug to improve the manageability of the product. You are doing something a customer would be doing and they won’t be able to strap a debugger to it. You’ll want enough information in the logs that ideally a customer could resolve the problem.

Ignoring the code

So you now know your product’s code inside & out. You are able to debug & even provide your developer with proposed fixes. You might even have helped write some of the features in your product. Now forget everything you know about the code.

What?! You might say, I worked so hard to figure it all out. Yep & if you are not careful you may make some of the same assumptions that the developer did when they wrote the feature. That means you run the risk of designing your test cases around the code & missing some juicy bugs. You need to step back & challenge any and all assumptions that you might be making.

Along with challenging the assumptions made in the code, don’t forget to challenge & look at everything from the customer’s eyes. Does the instructions we provide in the UI make sense to the end user or are we using 3 different terms to reference something because it’s name has changed over the course of development. One powerful tool here is to swap features with another tester & try each other’s area out. If they can’t figure out how to use your feature, don’t dismiss it as they are new & don’t understand the product enough. They are likely stumbling over some areas that a real customer would & since you are neck deep in your feature & code, of course how to use it is obvious to you. It likely won’t be obvious to your customer, so look for how you can improve that also.

Challenges in Test – Knowing Your Product

2011-03-07T20:01:45Z

So you accepted the challenge of being a tester, got your machine setup with the latest version of the code, installed your product, and your manager has given you a hot new feature to test. It seems so easy. You think you can just focus on that feature, but you better think again.

Problem introduction

Take for example a feature called Query in Forefront Identity Manager (FIM). You start out on the feature & learn that it takes a query in the form of XPath & then returns the results of that query. You have some standard set of operators that you expect to find such as =, >, <, and or, not along with some functions like starts-with, ends-with, dateTime, etc. The specific operators are not the important component here, but if you are so inclined you can view the details here.

Knowing your product

As a tester you might be prone to take this problem & run with it to create a bunch of test cases around the various combinations of operators & data values. Initially you find several bugs & you are keeping your developer pretty busy. You even think about some of the various buckets of tests like boundary, localization, globalization, performance, stress, security, etc. Are you now ready to declare your testing complete?

One of the things a good tester needs to do is fully understand the problem space they are working in & also how it may be changing over the course of the product lifecycle. For example in this case the design of FIM leverages the XPath filters of Query in a feature called Sets. These Sets are then used in another feature called Management Policy Rules (MPRs). MPRs are core to the product in determining both if you are allowed to do something. These MPRs also can trigger another feature Workflows which can then perform various actions in the product.

Yikes! You have just gone from a tester of a single feature to something that is deeply involved in 4 different features which are ultimately core to your product. And note that these features are likely covered by different developers, testers, & PMs. So you may not be responsible for these other feature areas, but you will need to know & understand how your feature interacts with them. Even better if you are in active development the design may be rapidly changing as your team reacts to requests in future milestones, so what you knew yesterday may not be true today.

What’s the Impact?

With knowing your product you can now start to evaluate the level of coverage you have & where you might want to add more coverage. One example of how you will need to use this knowledge is justifying the impact of a bug. Let’s say you found a bug in Query which when searching for “Smith” exclusively returns all people with the last name of Smith, but also Smithers are returned at the end. It is the day before you ship the product & the question will invariably be asked: “Can this bug wait until the next patch, service pack, or release?” Or put more simply, “What bad things will happen if we don’t fix this right now?”

Now your knowledge of the product is key. When the feature was originally designed it was just used for searching a list of users or groups, so returning a few extra users would be no big deal right? I mean who would page through the list of all Smiths in a phone book to see that there are a few extra results at the end? But wait remember from above what you learned about the product & understand where else your feature gets used which is granting rights in the system & kicking off workflows. So now this bug has increased in importance since you would potentially be granting rights to users & starting processes which are not intended.

So as you can see as a tester you better know your product almost as good as anyone else to help ensure you have the appropriate level of coverage & you don’t get the luxury of just knowing a single feature area but need both depth & breath across features. If you want to really know your product then no better way than to test it.

Challenges in Test – Introduction

2011-02-28T19:53:00Z

This time of year at Microsoft is when we have what we call our Mid Year Career Discussion (MYCD). Note that this is not a Mid Year Review, but instead a Career Discussion as the intent is for you to specifically reflect on & discuss your career development. As such I was reflecting on a topic that I have seen discussed externally. What is an SDET & why would I want to be one?

This series of posts is not by any means authoritative, but instead meant to give you some food for thought about the types of challenges you can expect to face. So if you are considering a position as a SDET with Microsoft & wondering what you might be able to expect, or just a customer curious about the role of a SDET at Microsoft I hope you find the coming posts useful & informative.

In the following series of posts, I will cover the following things that test must do to help develop a successful product.

Know your product – You may be assigned a couple feature areas, but many times just knowing your own area is not enough to ensure good test coverage.
Knowing the code & sometimes ignoring it – Improving test coverage is about knowing how the product code works & at other times ignoring how it works.
Knowing your customer – Your team has developed a great new whiz bang product, but do you understand the problems that it will solve for your customer & how they will use it?
Proving your feature – It is one thing to test your feature, but how can you prove it is correct?
Impeccable Infrastructure – Your tests are only as good & useful as the infrastructure that you use to run them.
Ensuring Testability & Influencing Design – Your team is planning the next big thing, using some awesome new technology, but do you have any idea on how you can actually test it?

So feel free to follow along & let me know if this information is helpful to you or there are other questions you have.

FIM Service Database size data point

2010-02-04T10:00:00Z

It is an exciting time as customers plan & start to roll out their FIM 2010 deployments. One of the initial questions asked is how much hardware do I need? The team is working on providing some more documentation to help answer these types of questions, but for now I will share with you a data point that may help.

In an earlier post I included the hardware we are using to test performance in our lab. In this post I would like to give you a data point around scale & size of a given db used on this hardware.

Scale

First is a breakdown of the top objects in one of our test databases. This should help give you a reference point for thinking about if your deployment is bigger or smaller than what this db represents. Size will additionally vary based on the configuration, number of attributes, etc that are specific to your deployment.

Object Type

Count

Requests

9 million

Expected Rules Entries

3 million

Workflow Instances

1.5 million

Gate Registrations

550k

Groups

450k

People

200k

Note: The requests & workflow instance counts are higher than I would expect in a deployment as I have not pruned this database down for completed objects.

Data file size

SQL File

Size

FIMService SQL Data (mdf)

450 GB

FIMService SQL Log (ldf)

80 GB

Note:The log file of course will grow over time & need to maintained via regular transaction backups.

Extending FIM Timeouts

2010-02-02T10:20:00Z

When building out your deployment there may be cases where you want to extend the default timeouts used in the system. A key scenario of this might be if you create a separate Administrators portal for your FIM Admins. This could be used for running large reporting queries or making configuration changes like adding\removing large sets.

If you find that the operation you are attempting to complete takes more than 60s then to resolve this you will need to extend the various timeouts in the system. Below is a breakdown of the various timeouts that you can configure in the system.

Configuration File Locations

Sync - %ProgramFiles%\Microsoft Forefront Identity Manager\2010\Synchronization Service\Bin\miisserver.exe.config
Web Service - %ProgramFiles%\Microsoft Forefront Identity Manager\2010\Service\Microsoft.ResourceManagement.Service.exe.config
Password Client - %ProgramFiles%\Microsoft Forefront Identity Manager\2010\Password Reset Client Service\PwdMgmtProxy.exe.config
Portal - C:\inetpub\wwwroot\wss\VirtualDirectories\80\web.config

Timeout Attributes

dataReadTimeoutInSeconds – The read timeout used in SQL connections by the Web Service
dataWriteTimoutInSeconds – The write timeout used in SQL connections by the Web Service
timeoutInMilliseconds – The timeout used by our Web Service client when communicating witghg the Web Service

Steps

Open the configuration file (see above for configuration file locations)
Extend the SQL timeouts – This only applies to the Web Service configuration file
- <resourceManagementService externalHostName="MySERVER" dataReadTimeoutInSeconds="300" dataWriteTimeoutInSeconds="300" />
Extend the Client timeouts – This applies to the Portal. Password Client, & Sync Client
- <resourceManagementClient resourceManagementServiceBaseAddress="MySERVER" timeoutInMilliseconds="300000" />
Restart the service – Net stop & Net Start the service for each server that you modified the configuration for (FIMService, FIMSynchronizationService, & W3SVC)

Note: These timeouts can be different on different client & webservers. So for instance in my previous post I may only want to extend the timeout for my Admin Portal which Admins & Sync use. Then I would only modify it on FIMAdminPortal.contoso.com

Bonus: If you find yourself extending timeouts significantly you may find that the UI will still timeout, but your operation will still be performed. In this case there is a timeout from ASP.Net which is causing the UI to timeout. This is not required, but will prevent the UI from displaying an error.

Open your web.config
Locate the httpRuntime node
Add the executionTimeout attribute (Default is 110 seconds)

FIM 2010 RC1 – Update 3 Available

2010-02-01T08:02:56Z

The latest update & last one prior to RTM has been posted to connect.

A few highlights from the update include:

MPRs now have a MPR Type of being Set Transition or Request based
Support for SQL Server Failover Clusters
Updates to allow taking database backups with FIMService running
Improved error details in the FIM MA from the web service

FIM 2010 RC1 - Update 2 Available

2009-12-09T23:23:54Z

You can find the latest update for FIM 2010 RC1 making it’s way out via Microsoft Update now. There are many more fixes included in this update such as improvements to Sets.

Install it on your systems & try it out.

Links

Configuring FIM Client with Multiple Servers

2009-12-06T20:04:00Z

In my previous post, I covered Server Partitions which showed how to setup multiple servers in your topology. Once you have your servers setup, you will need to ensure your clients are pointed to the appropriate servers.

Below is the topology that I discussed for reference.

In this topology we want to configure our End user actions from the Client PC to use the following Service Partitions:

FIMPassword.contoso.com – for all password reset & registration Requests
FIMPortal.contoso.com – for all portal activities & Outlook integration Requests

Configuration

When installing the client you will want to do the following:

Configure the Outlook plug-in to use the User Portal “FIMPortal” & send mails to the mail account that is monitored by the FIMPortal service.
Configure the Password Client to the “FIMPassword” web service which is the location where your password reset requests will be processed.
Configure SiteLock for the ActiveX controls
As part of the security for our ActiveX control you need to specify the sites you want to SiteLock the control for. This means this control can only be used by these sites.

Note: In my above topology, the portal is not installed on FIMPassword, but only on FIMPortal for End Users so I will use that for SiteLock. This has an added implication of now Password Reset\Registration from my client machines (i.e. Windows Login) will go through FIMPassword, but the user could also initiate a password registration\reset attempt via the portal using FIMPortal.

If you are installing on a machine with IE7 installed then you will get a slightly different dialog. This will allow you to have the portal automatically added to Trusted Sites.

Note: This configuration only tells the client which web servers to use & does not prevent an end-user from accessing the portal or web services on different servers. If you want to prevent Requests from end-users to your administration instance, then you will need to do additional configuration to only allow specific users or IP addresses to make Requests.

Service Partitions - Multiple Middle Tiers, Request & Workflow Processing

2009-11-23T06:38:00Z

One of the updates found in FIM 2010 RC1 Update 1 as David points out is the ability to ensure Requests and Workflows (WF) created on a specific FIM Service Partition are executed only on the same FIM Service Partition that it was created on. FIM Service Partitions only matter if you run 2 or more FIM Services that are connected to the same FIM Service DB.

Background

To give you some context on this change, it is important to understand some background information about how Request & WF Host works in RC1.

Requests come into our system via the web service and are then handled internally via the Request Processor, if there are any Workflows generated, they are added to a WF queue. This means we operate on a polite first come, first served basis.

This is fine for end users, but when you account for administrative tasks like sync or run on policy update, which can generate hundreds of thousands of Requests, then your end user Request with its associated Workflow(s) could be at the end of a very long line before its Workflows execute. You wouldn’t want an end user trying to create a distribution group waiting behind 100k WFs kicked off from an export that you just finished from Sync.

Solution

The solution is providing the ability to setup separate middle tiers where one or more can process sync\admin Requests & the other can handle your end user Requests and WFs

In RC1 this was possible as each Web Service has it’s own WF queue and puts WFs it generates into it’s own queue.

The problem in RC1 was that when a FIMService starts it can resume any Request or WF that was previously persisted to disk (cases 1-4 below). Therefore a Request or WF from the “Admin” portal can be resumed by the “User” portal and vice versa. At this point you no longer have control of your queue size and can’t protect your “User” Requests from being queued up behind Admin Requests.

Requests and WFs save their execution state in the following cases:

Delay Activity – If the Workflow has a delay activity it persists its state to the FIMService DB while waiting for the delay to expire
Service Stopping – When the FIMService is stopped, incomplete Workflows are unloaded & persisted to the FIMService DB
Request Processing - Requests save their current state to the FIMService DB as they move through the processing pipeline
Action WF Policy Enforcement – Some Action WF policy enforcement is executed asynchronously and persists the Request and\or WF to the FIMService DB which get resumed by a FIM Service

We resolved this by introducing (in RC1 Update 1) the concept of Service Partitions. A Service Partition is just a unique way of identifying one or more servers that will share the processing of Requests and WFs in the system.

In the above diagram I have the following Service Partitions created.

FIMPortal.Contoso.com – Single Web Service instance for handing end user portal Requests
FIMPassword.Contoso.com – NLB Web Services for handling password registration\reset Requests
FIMAdminPortal.Contoso.com – For handling Sync & Admin Requests

There is also an additional consideration of which of these servers do you want to process mails from Exchange & generate Requests. These servers will need to have polling enabled, which will be shown in the next section.

Steps to Configure

Configuration of multiple middle tiers in your environment is pretty straight forward. Here are the steps to setup the above topology.

Install FIM Service & Portal on your Admin portal environment
- If you don’t want this service processing Requests made through Outlook to Exchange then ensure the polling checkbox is unchecked.
- Set the FIM Service Server address (aka externalHostname) to be the FQDN for your Admin web service (i.e. FIMAdminPortal.Contoso.com)
Configure the FIM MA
- Set the FIM MA to point to the FQDN for the WS of the admin portal
Install FIM Service & Portal for your End User Portal
- Setup this service to process mails from Exchange (assuming you are using Exchange 2007)
- Set which email account you want this service to process mails from
  - Note: Each Service Partition that has polling enabled should have it’s own email account to ensure it only processes mails from it’s own account
- Set the FIM Service Server address (aka externalHostname) to be the FQDN for your “User” web service (i.e. FIMPortal.contoso.com)
- For this & every other instance that will use the same database, choose to reuse the database.
Install FIM Service & Portal on your Password Web service machines (You can just install the FIM Service if you don’t need the portal)
- Set the FIM Service Server address (aka externalHostname) to be the FQDN for your “Password” web service (i.e. FIMPassword.Contoso.com)

Now what you have is a guarantee that FIMPortal will only ever execute Requests & WFs that were created by end users in the Portal, FIMAdminPortal will only ever execute ones from it or Sync, & FIMPassword will allow Requests & WF processing to execute on either of the 2 NLB Password boxes.

Note: To fully take advantage of this topology you will need to also need to configure your clients appropriately to send password reset Requests, Outlook plug-in mails, & Portal links to the right servers. I will cover that in my next post.

FIM 2010 RC1 – Update 1 Available

2009-11-19T02:49:00Z

You can now find FIM 2010 RC1 Update1 making its way out via Microsoft update. This is a cumulative release with many fixes across the product. Of particular interest from a performance perspective is some changes we have made to significantly improve performance around the password reset scenario.

I highly recommend you pick up RC1 Update 1 and try it out.

FIM 2010 Performance Testing – Scale & Load

2009-10-22T15:33:49Z

Scale

For the discussion of scale the first question is the number of users & groups you expect to be in your system. There are fairly easy numbers to determine with a few queries of Active Directory or whatever is your directory store of choice. As many companies currently do not allow end users to manage groups, you will need to consider first if you are going to allow your users to create & manage groups. Then if you do, how that will impact your scale as the number of groups may grow. In our case we are using 150k+ Users & 400k+ groups in our testing.

The next portion to consider is how your configuration will impact your scale. Are you going to deploy calculated groups? If so what types & how many will you use? Are you going to manage custom object types in your environment like computers?

Last there is some impact from your policy objects on the scale of your deployment. In a cross forest environment you may be provisioning FSPs into Sync. Using codeless will increase the number of EREs & DREs in your system which can be roughly estimated based on the number of SyncRules for the object type * the count of that object type.

Load

Now that you have your system configured, gotten it populated you are ready to start rolling out your deployment. How are you going to know that you have the right set of hardware in place to sustain your deployment?

In our case we started with estimating the expected usage for a given number of users for many of the key operations in the system.

How often do we expect a request to join or leave a group?
How often do we expect a user to create a static or dynamic group?
What is the breakdown of security group usage vs. distribution list usage?
How often do we expect a request to register for password reset or reset a password?
How many computers have the FIM client installed? – The client will periodically do a check for if registration is required on logon.
Will users primarily use the Portal or the Outlook client for group management operations? – This will determine how much work is done by the Mail Listener as it polls Exchange & processes mails
What is the usage pattern of your company? Do you expect peaks in the morning when users log in?

After estimating each of these, some based on hard data we have from another tool which was deployed & some we had to estimate, we were able to create what we call a load profile.

Using that we were able to leverage VSTS to create load tests to simulate our deployment. VSTS is a great tool for doing performance testing of a product, it is geared toward product development directly but could also be leveraged in testing a product deployment.

How are you going to evaluate the readiness of your deployment to meet your scale & load requirements?

FIM 2010 Performance Testing – Policy Objects

2009-10-14T00:08:43Z

Policy Objects are the heart of your deployment where you will implement the business logic needed. These will also be influenced by the scenarios (credential management, group management, user management, etc.) that you deploy. As such it is no one deployment will be exactly the same as another but to help give you a data point I would like to cover a breakdown of what we are using currently in our deployment.

RC1 Out of Box vs. Sample Deployment

	RC1 Out of Box (OOB)	Deployed
Sets	71	96
MPRs	51	98
Workflows	12	58
Sync Rules	0	80
Domain Configuration	1	14
Email Templates	13	24

So how many objects do you plan to have in your deployment? How should you think about these policy objects with relation to performance?

Things to consider:

Sets – Every operation in the system must be evaluated for it’s impact on a given set. Some are simple transitions like someone’s building has changed, but others have cascading effects such as a manager change impacting other objects indirectly
MPRs – MPRs have two primary uses granting rights, & triggering WFs. As you build these out you will find you may increase your number of sets to capture the various states you expect objects to transition in & out of. As these could then trigger WFs & additional work in the system, you will need to be aware of this.

In RC0 we found a problem with our ability to scale with the number of MPRs in the system & as such have done significant work to help improve this. For example beyond the above set of MPRs we ship OOB, we have added 400 additional MPRs. Similarly we are doing testing around other core system object types to ensure we can meet your needs in deployment.

In my next post I will cover scale & load, what scale we are currently using & some questions to think about for load.

FIM 2010 Performance Testing - Hardware

2009-10-09T20:17:53Z

My goal with this post is to give you some data on the type of hardware we are using internally for our testing, which you could use to help inform your own hardware needs. This is the hardware we are using to evaluate the product for release & that it will be able to support the workloads needed in MSIT. This hardware is not configured or meant for production, but just for testing. An example of this is that we have our hard drives configured in RAID 0 to get the most out of fewer drives in our testing, but you would likely have yours setup for redundancy. In which case you could add additional drives to get similar results.

Similar to in my previous post about our topologies, there are two basic classes of hardware we use in our testing. What I will refer to as a standard machine & a performance machine.

	Standard Machine	Performance Machine
CPU	1x4 Core Core2 Q6600 2.4 GHz	2x4 Core Xeon E5410 2.33 GHz
Memory	4 GB	32 GB
Hard Drive	Single Hard drive	8 – 136 GB 10k Hard Drives

On the performance machine we have the 8 hard drives currently allocated as follows. Your situation will be different but we have standardized our machines on a single drive configuration for both Sync & FIMService to allow us to use machines for either depending on our testing needs.

	Drive Setup	Purpose
C	1 – 136 10k Hard Drive	OS\Applications
E	1 – 136 GB 10k Hard Drive	SQL Logs (ldf files)
F	6 – 136 GB 10k Hard Drives (RAID 0)	SQL Data Files (mdf files)

In each of our performance rigs we have 2 performance machines & then as many standard machines as needed based on the topology. In our standard topology we use the standard machine for a client & other test machines for Visual Team System (VSTS) to generate our load.

The performance machines thus far have always been allocated to the SQL server as that is where the primary horsepower is needed for both Sync & FIMService. Each of these components use resources in a different manner.

Synchronization Service

Synchronization service (formally MIIS) the primary place we have observed the most bang for your buck is in the disk subsystem. Increasing your disk throughput will increase the performance of sync the most. Most of the time if you observe disk IO of Sync you will see the disk under the heaviest utilization.

Resource Utilization of Synchronization Service
Notice the white line at the top is our SQL server drive

FIM Service

FIM Service leverages SQL to do a large amount of our processing, if you ran SQL profiler on the system you would observe some very large queries being executed. SQL queries handle many of the core services of our product including query evaluation, rights enforcement, & set transitions. As such for the FIMService db, having higher CPU power will give you the best increase in performance.

Resource Utilization of FIMService

From the above you can see that occasionally we have spikes of usage in memory & the F drive which corresponds to the SQL data file drive in this rig. Furthermore, in this case I have the FIMService & SQL Server on the same machine, so we can compare CPU utilization of both processes to see where the processing power is being used most.

FIMService vs. SQL Server CPU Utilization

From above the highlighted black line is the CPU usage of SQL Server, the light blue line is the CPU usage of Microsoft.ResourceMangementService. This demonstrates at least at this point in time that the primary consumer of your CPU & where you should invest your CPU power is your SQL Server.

In my next post I will discuss policy objects & how we are thinking about them in our performance testing.

FIM 2010 Performance Testing - Topology

2009-10-06T18:15:26Z

Customers have very different deployment needs for their product, all of these driven based on their various business requirements. For ourselves we have taken a two pronged approach for testing the performance of FIM 2010 in different topologies. We typically test in what I will refer to as our Standard topology & then we do additional testing in what I will refer to as our NLB topology.

Standard Topology
This is the standard configuration we generally test in. We look for bottlenecks & may adjust this over time if we find we need to use a difference configuration.

NLB Topology
This is our more complex topology intended to give us insight into how the product performs both functionally & performance wise in a more complex deployment.

What topology are you planning to use for your deployment? Are there any specific situations you need to handle?

In my next post, I will talk about the hardware we are using in these topologies & cover what we are seeing for resource utilization & bottlenecks.

FIM 2010 Performance Testing - Introduction

2009-10-01T10:16:07Z

One of the areas I have been focused on recently has been testing the performance of our product. With the release of RC1, I thought I would start off with some insight into what you can expect with the product.

A key problem with figuring out the performance of a given product is the number of variables that impact the results you observe, this is especially around performance. Many customers have a simple question of can your product support my company of size X. While on the surface this is a simple question there are a large number of variables that play into the answer of that question. For FIM 2010 there are a couple key pieces of information needed in evaluating what that answer is.

Topology - What is the topology you plan to deploy the product in. Will SQL Server be on the same box as the FIMService? Will you be using a Network Load Balancer?
Hardware - What hardware are you running on each piece of your topology? What is CPU, Memory, Disk, Network? How are your drives configured? How is SQL configured to store your files?
Policy Objects - Policy objects are a key component of FIM 2010. These include Sets, Management Policy Rules, Schema, Workflows, Sync Rules, etc. Depending on how you configure these, there will be additional work that the product must do & that will impact your performance.
Scale - Typically scale is talked about in terms of the number of users, but in the case of FIM you also need to think about the other object types in the system depending on the solution you are deploying. How many groups & of what type? Do you have calculated groups? Do you have custom object types you are managing like computers?
Load - How do you expect your system to be used? How often do you expect someone to create a group? What type of load do you expect from Password Reset deployment? Do you expect users to use the Portal or the Outlook Add-in for Office 2007 more?

How you answer each of these questions will likely impact the performance of the product. This is a classic problem as a tester we often find a matrix of variables & then need a way to help answer some of these questions. My goal here is not to give you a definitive answer for your specific case, but instead to share information directly with you of how we have approached our testing which can then inform you in your deployment.

For our own internal testing we have worked to leverage feedback of our customers & most specifically MSIT to model a basis of how our product will be deployed and perform. From that we have been working to expand this model to then see how changes to some of these variables then impact our performance.

In my next few posts I will discuss how we have approached each of these items, both for planning our testing & our eventual deployment.

FIM 2010 RC1 Released

2009-09-30T23:52:00Z

Today FIM 2010 RC1 has been released. There have been alot of improvements since RC0 which we believe you will enjoy, so download it, try it, & let us know your feedback.

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=4bb3f16b-27f8-4c1d-922f-2c7b522d9ad6

Introduction

2009-09-30T04:04:09Z

After about 7 years at Microsoft, I have decided to join the role of bloggers. As such I should start with a quick introduction about myself.

I have been working at Microsoft over the past 7 years as a tester in what was originally Microsoft Metadirectory Services. In that time I have worked on shipping Microsoft Identity Integration Server 2003 (MIIS), Identity Lifecycle Manager 2007 (ILM) , & most recently Forefront Identity Manager 2010 (FIM). As part of FIM 2010 I have been the test lead responsible for several areas of our product including Schema, Sets, Management Policy Rules, Mail Services, Web Services, FIM MA, & Performance.

Over the course of these releases I have had the opportunity to learn from our customers & help improve the quality of our product. As a tester I have found the information gained from our customers invaluable in our ability to test & improve the quality of our product. An example of how your feedback has helped us is if you are familiar with the MIIS 2003 Resource Kit 2, included in that package are two tools. (MIIS Dynamic Help & MIIS Provisioning Assistant) I had developed based directly on feedback received by our customers.

Additionally, I have found there is a wealth of knowledge about the product gained over the course of product development as we debug, investigate & fix issues in the product. My goal is to continue that two way communication where we continue to respond to your feedback to improve the product & we also help share information on how to debug, investigate, & implement our products.

If there are question you have, or topics of interest please feel free to let me know & I will see what I can do to address these.

Object Type	Count
Requests	9 million
Expected Rules Entries	3 million
Workflow Instances	1.5 million
Gate Registrations	550k
Groups	450k
People	200k

SQL File	Size
FIMService SQL Data (mdf)	450 GB
FIMService SQL Log (ldf)	80 GB