One of the common tasks that .NET RIA Services developers have to undertake is testing their mid tier business logic code. Mid tier code typically uses a data access layer (DAL) like Linq to SQL or Linq to Entities for persisting data. However directly coupling the business logic code to the DAL will pose challenges for unit testing and causes tests to depend on the database. One possible solution to avoid this problem is to adopt the repository pattern and write business logic code to go against repository.
The following steps demonstrates an implementation of a domain service class using a repository to make it unit testing friendly. The example code uses the .NET RIA services walkthrough application as its starting point and is modified to use Linq To SQL as DAL.
STEP 1: Create an interface that defines the necessary methods for performing basic CRUD operations on the repository
STEP 2: Provide concrete implementation of IRepository for Linq to SQL. This is done by performing the repository operations on Linq to SQL’s DataContext as shown below.
STEP 3: Implement the DomainService class and use the repository instead directly calling into DAL layer in domain operation implementation.
The code below implements an Organization DomainService which uses an employee repository
As you can notice from above code snippet, there are few things that are different from the original walkthrough sample :
1. OrganizationDomainService doesn't derive from the LinqToEntitiesDomainService, instead it derives from the base DomainService class
2. OrganizationDomainService has been applied with LinqToSqlMetadataProviderAttribute. This adds Linq To SQL’s type descriptor to our domain service class and enables it to reference the Linq To SQL generated entities (Employee entity in this case).
3. Query/Update methods now call into Repository methods instead of directly calling DAL’s context methods.
4. PersistChangeSet method has been overridden to call the Repository’s SubmitChanges method
STEP 4: Write a factory class to create instance of the DomainService class and the repository and register it (in Global.asax.cs). This ensures that our factory method is used for domain service class creation by the framework.
Now the app is ready for execution (there is no change required to the client side code of the walkthrough project).
STEP 5: Creating unit tests
For unit testing the domain service methods, all we need to do is write a mock repository that creates an in-memory instance of employees and pass it to DomainService.
Code below shows creation of a mock employee repository.
and the unit test code would look something like this:
Completed sample source can be downloaded from here (see 'Repository for unit testing sample').
TEST TOOLS AND FRAMEWORK
The following section gives best practices for test tools and test framework. The requirement for tools and framework are more than the tests because of the scope of their use (entire team or multiple teams uses it).
Design Checklist
You should ask yourself the following questions when you are designing any part of the system.
1. Have you iterated, selecting the best of several attempts rather than the first attempt?
2. Have you tried decomposing the system in several different ways to see which way will work best?
3. Have you approached the design problem both from the top down and from the bottom up?
4. Have you prototyped risky or unfamiliar parts of the system, creating the absolute minimum amount of throwaway code needed to answer specific questions?
5. Have you spiked the system, to convince yourself that the proposed solution is likely to work?
6. Has your design been reviewed, formally or informally, by others?
7. Have you driven the design to the point that its implementation seems obvious?
8. Does the design adequately address issues that were identified and deferred at the architectural level?
9. Is the design stratified into layers?
10. Are you satisfied with the way the program has been decomposed into subsystems, packages, and classes?
11. Are you satisfied with the way the classes have been decomposed into routines?
12. Are classes designed for minimal interaction with each other?
13. Are classes and subsystems designed so that you can use them in other systems?
14. Will the program be easy to maintain?
15. Is the design lean? Are all of its parts strictly necessary?
16. Does the design use standard techniques and avoid exotic, hard-to-understand elements? 17. Overall, does the design help minimize both accidental and essential complexity?
18. Does design make it easy to maintain and extend (high coherency and loose coupling meaning ‘open to extension but closed for modifications’)
19. Are their similar tools or pieces of infrastructure already available in other place (other teams)? Avoid duplication of effort and helps each one to leverage the work done by the others. E.g. Sharing logging infrastructure and base class infrastructure between different tools
20. Does the design take into account the fact that this tool/fx could be of use by others (outside feature area) in the team.
Unit Testing:
All test tools and framework must have unit tests written with reasonable code coverage before checking in. After the first check in, the goal is to not have a drop in code coverage % for subsequent checkins.
Static analysis:
Make sure your tool/framework is fxcop clean before you check-in.
Code review:
All test tools/framework must be code reviewed by atleast one person before checking in.
Adopt different test methodologies (test styles)
Always consider applying different test styles to your area. Each style is suited for finding specific class of issues and applying different styles gives the maximum ROI for your test effort.
Some of the approaches that have been highly effective in the past are:
1) Creation of tools to dynamically generate test cases.
2) Fuzz tools that morph the input data.
A good list of different test styles is mentioned in:
http://www.kaner.com/pdfs/GoodTest.pdf
Ensuring Testability
Design for testability is an important point to be considered that greatly affects the quality of our tests. For all new features coming online test owner should have a testability discussion with the developer before feature implementation. The important point that needs to be considered here is - Can we get to all the functionality we want to test in an easy and efficient manner?
Reviewing test strategy
Feature test plan should describe the ‘test strategy’ that will be adopted and should be reviewed as part of test plan review. Primary purpose of reviewing is to share ideas and foster discussions.
Sharing ideas and information
Strive to share the following as much as possible across teams
• test strategy ideas
• test framework/tools
• Information (e.g. new feature addition) in one area that could also affect other areas.
Guidelines for testing at the right layer
Start with the component level tests first
Always start with the lower level first and then move to the higher level. API testing should be the first to be automated through Unit tests. These tests are easiest to write and most robust of all the test layers as they have the smallest surface area for false failures. Getting test access to the core logic of the component is the key to this strategy and will need some upfront work to investigate ways of hosting your component in isolation. Work with your developers to address this as part of your testability plan. Tests written at this level should provide depth testing for features.
Pros
- Very easy to write, modify and maintain tests
- Tests run faster
- Near zero timing issues
- High level of coverage
Cons
- Upfront work to enable component isolation
- Not a real world environment
Object model tests comes next
These tests are at the higher level than component level tests and are suited for automating the scenarios and integration tests for your feature.
Pros
- Easier to write, modify and maintain tests compared to UI tests
- More closer to real world environment
- Less prone to timing issues
Cons
- Need to provide testability hook (via private interface) for areas not exposed by public object model (DTE)
UI tests comes last
The complexity of interacting via the user interface for testing can quickly dwarf the complexity of verifying the logic being accessed. Furthermore, any changes to the user interface can have a devastating impact on the tests. Hence, avoid testing through UI as much as possible. UI should be tested separately and shouldn’t be used to test the core functionality of the feature (business logic). Write UI tests for testing the UI elements in a feature. Also some of the end to end scenarios should be automated using UI automation as that’s a real simulation of how the customer would be using the product/feature.Wait till UI becomes reasonably stable before starting to automate these tests.
Pros
- Closest to real world environment and real world usage of the feature
Cons
- Extremely prone to timing issues
- Usually causes maintenance nightmare
Here is the first in the series. Note that some of the terminologies used here are specific to the product we test - Visual C# (part of VisualStudio.Net).
Test strategy
Choosing the right test strategy is an important step to achieve efficient and robust test automation. The following section describes some of the important methods that teams should consider.
Layered approach to testing
Depending on the type of interaction between the tests and the system under test, our product can be divided into the following four layers. Choosing the right layer/level of testing at the right time in the product cycle is going to be the key to our team's success.
UI level
This is the level at which most users interact with the product, through the user interface by working with controls and windows.
Object model level
At this level tests interact with the product through the public object model (DTE). At this level the different components that make Visual Studio are still interacting between themselves in the same way as they normally run when the product is handled by a user.
Component level
At this level tests interact directly with the component under test. In most cases components will not be running in an environment similar to that of the first two levels and component will be hosted in isolation.
These tests usually are most robust as they the smallest surface area for false failures.
API level
At this level the tests interact part of a component. These tests are written to test functions and methods of individual pieces of a component in isolation. E.g. Unit tests
Hmm, its been a long winter in Seattle this year, which was rainy and chilly than ever before. I am back from my winter hibernation to blog :-) . I am really glad that spring is finally here.
During this time I have changed the team and I now work in DLinq test team (I used to work in C# IDE test team working on Edit and Contine feature).
One of the things we have been doing in the team is to document the test best practices based on the things we have learnt in the past. The goal is to make this a 'Test Guide' that the whole team uses for their individual testing and to provide guidelines to achieve - effective, efficient and robust test automation.
I will post the series on this in the next several weeks.
As we are getting close to releasing VS whidbey, I was thinking about my contributions to this release and one of the major item that appeared in the list was testing C# edit and contine (here after EnC). This is one of the primary features I was responsible for testing in whidbey. Then I thought of writing a short summary about how we tested this feature. Below is the result of that exercise.
Brief history
Sometime in Jan 2004 when we were half way into the whidbey product cycle, decision was made to provide Edit and Continue feature for C# in whidbey. This was based on the overwhelming feedback from customers asking for EnC for C# and that it was the #1 feature request listed in MSDN feedback.
EnC is one of those features thats very simple to use but complicated when you get into the details of its design and implementation.It touched the core areas of VS - CLR, Compiler, Editor and debugger, making testing very challenging. However our initial understanding was that since VB already had support for this (in whidbey), it was just a matter of making changes to C# compiler and C# editor to make this happen and the support in CLR and debugger already existed (and tested). Its another matter that this assumption turned out to be incorrect (atleast partially) in the end.
Intially the test team comprised of me and my collegue Daigo (he blogs in Japanese and can be found here). Santosh joined the team some time later.
Test strategy
Quality metric
We started of by coming up with the quality metric that would help us drive our testing efforct. These are the two primary attributues we wanted to target:
Stability
EnC should reliably succeed or fail. If EnC does not succeed, the user should always be able to resume the original debugging session. However, when the original debugging session is resumed, code changes will remain.
Accuracy
EnC should accurately reflect supported code changes in the new executable image, while maintaining the purity of the rest of the image. The new image should be accurately reflected in the resumed debugging session.
Pieces making up EnC
After talking to developers about the different components making up this feature and the details about their inner workings we came up with the following breakup list that made up EnC:
- Rude edit detection: Feature that detects disallowed edits during EnC. This feature is implemented by C# editor
- Local mapping: Feature that enables tracking local variables (moving, adding, removing locals) during EnC. This feature is implemented by compiler
- UI interaction: All the UI elements associated with EnC such as displaying of rude edit squiglees, readonly markers , error dialogs, etc. This feature is implemented by editor and debugger.
- IP remapping: Feature that calculates and resets the instruction pointer(IP) to the next active statement during EnC. This feature is implemented by debugger.
- Line mapping: Feature that calculates and tracks the movements of statements in the program being debugged during EnC. This feature is implemented by debugger.
Approach
Once we had the list of sub features making up the overall EnC feature, our next task was to come up with the approach to actually test these pieces individually and also test the feature as a whole. Below list kind of summarized the approach:
- Identifying the high risk areas: We focussed our initial testing on the new code added to support C# EnC. This mainly was rude edit detection, local mapping and line mapping. There were limited modifications to other pieces and was'nt very significant.
- Data driven testing: We used this methodology which is ideal when you have a lot of different data values that you wish to exercise the feature with but where the sequence of steps to execute is pretty much identical. This required some upfront investment on comeup with the test framework but once its up and running, automating the tests was extremely fast.
- Testabilty: We got our developers to provide testabiltiy hooks in the product that made testing higly productive and tests more reliable
- Reuse existing infrastructure: For the common code, we reused the already written tests (which were already automated) for VB which helped us increase our coverage quickly.
- Exploratory testing: Adoped directed exploratory testing to test the integration scenarios (ie. test EnC the way users would do). This helped us find some really good issues early on in the product cycle.
- Stress testing: We invested good amount of time to come up with stress tool and do stress testing. This mainly comprised (among other things) of determining the upper limit on the number of edits that could be made in a single EnC session. This helped us identify some criticial performance bottlenecks in our implementation which got eventually fixed.
- Effective use of Code coverage: We used code coverage regularly to identify the test holes and beef up our automation coverage
Lessons learnt
- Using diverse methods of testing ('test styles') gives best results
- Close and constant interaction with the developers helps to keep the crucial two way feedback going
- Whitebox testing is extremely effective and helps is identifying the right 'targets' for testing
- Reusing existing tools and infrastructure makes testing more productive
- Testing != Automation. Its just one of the aspects of overall testing and not an end in itself.
Cheers!
Edit and contine was one of the top features added in whidbey release for C#. Edit and continue is the ablity to modify code and contine with execution while debugging the program.
Types of edits/changes supported
There are few restrictions to the types of edits that are permitted during Edit and contine. The restrictions exist mainly due to technical reasons than anything else. The edits are primarily restricted to within method bodies. There are few changes within method bodies that are not permitted and to apply those changes you have to stop debugging and start over again. These unsupported edits are termed as 'Rude edits'. These are notified to the user by
- populating error description in the error list
- displaying squiglee and tooltip with error description at the begining of the code block which has rude edit
For a complete list of unsupported changes for C# see here .
Let me know your feedback on the unsupported changes and things you would like to see supported in the future.
Cheers!
Now that I have posted my first ever blog, I will introduce myself. My name is Vijay Upadya and I work in C# QA team. I have been in this team for the past 4.5 years (was in C++ team before the inception of C#).
In my spare time, I love watching (and playing) cricket and volleyball.
This is my first ever blog post and am excited to be part of the elite club of bloggers !!!
