If no one has completely solved issues surrounding the testing of Web Sites or traditional Win32 applications then you can assume that there are problems abound testing applications that connect to either other networked applications or web services.   Advancements such as Indigo will only make these types of applications more prevalent moving forward so you’re going to have to get used to it.  Having tested the Start Page in the VS 2002/2003 releases and now leading the team responsible for testing what will be a very connected help system I can offer some advice…


Know Thy Dependencies: You need to know all of the services that are required by your application and you’ll need to know the services and data stores they depend on.  It helps to make sure you have a detailed map of your service architecture both in and out of your span of control.


Control Dependencies:  Strict control must be placed on the data going back and forth to each service.  If you have a shared Schema you should put a lot of effort into making sure that this schema is bulletproof and as restrained as possible given the needs of the application.  If a service is meant to return you a set of contact information for customers that match search criteria you have to make sure you have restrictions on each part of the contact information.  The phone number should only be acceptable phone numbers, the name only valid characters, etc.  The data store needs to have checks in place to verify this is the case, the service needs to make sure that’s all it returns, and the application requesting the information needs to be sure it only uses data in the correct form. 


Your dependencies have to lock down early and stay that way.  Once you have defined the communication arrangement you have to be sure that the call remains the same throughout the expected life of the software unless you are prepared to test and deliver updates to your client app as often as a web site could change.  It would be bad if, for VS users, all of a sudden online help searches were returned in a different format than the UI expected and was unable to display. 


Think about your edge cases here… What is the max number of records that could be returned? What is the max space allotted for each field?  How long will it take this full dataset to return over a 28.8 modem?


Goals need to be set early and monitored often at every level when it comes to the service level your application expects.  If you expect a user to be able to see their search results in under X seconds the UI must be able to display returned results in a fraction of X, the data store must be able to fetch the results in Y and the service must do its job and bundle the results in Z.  Each step must be tracked both independently and with the combined round trip to ensure the proper user experience. 


“Trust No one”: So you’ve setup testing and contracts with your dependencies.  This doesn’t mean your application can assume it always gets data in the right form.  This means you’ll need a hefty set of negative tests that attempt to bring your application to a halt at each level with invalid data.  What if you get back more than the expected max?  What if the name contains invalid characters?  Can you sneak an XML tag into the results with hidden script calls? Be sure your application ignores or reports any data it’s not supposed to use.  The VS Start Page simply chooses to ignore data it doesn’t know how to display so, hopefully, a poor return set won’t result in a completely blank set.  Since it’s not a mission critical VS component we don’t mind the loss in fidelity. 


Expect the worst from your transporter: I don’t care if Jason Statham is moving your data around; he’s going to get stuck in network traffic, pulled over by your users’ proxy servers, and potentially tripped by aggressive anti-virus software.   So this means your app has to know how to handle all of these situations appropriately.  For example, the VS Start Page used the MSXML object to fetch data from online XML stores.  Unfortunately there is a bug in the MSXML object we didn’t know about that allows it to be interfered with by some proxy servers even when a standard HTTP request would work to access the data.  We could have made the call simpler and not relied on the MSXML component to avoid this.  In this case the transporter pulled over. I unfortunately have example bugs that fit each of these categories and more so make sure you have given your transporter a good work out in conditions you expect your users to encounter. 


Test with Overlap: If your dependency chain is A->B->C->D you could test D and call it a day.  I’d recommend that in addition to testing D you work on testing the various combinations as well.   You should also ensure that even if A is tested by someone else that this person tries testing through D on occasion.  It will help the A tester understand how their component is being used up the chain as well as give you another set of eyes on D.  Yes, fair trade and all, it also means the D tester should overlap and test individually down the line as well.


That’s all for now I probably missed some important things and may not have gotten specific enough.


I’d love your feedback on the content and depth of these testing entries.