With the recent announcements and availability of Visual Studio 2010 and .NET Framework 4 Beta 2 much attention is now focused on both the testing features and capabilities offered in the new range of products. With all these new features and capabilities come many new partner opportunities. I have prepared this paper to help partners understand, at a high level, the new VS 2010 testing features , and the associated partner opportunities and to dive deeper in this area than I did in my previous blog here.
For other information about Visual Studio 2010 and .NET Framework 4 visit the Microsoft Visual Studio 2010 and .NET Framework 4 Beta site to download the Beta, submit feedback, access forums, etc. On Twitter search for hashtags #VisualStudio and #VS2010
Contents
1. Background and Context
a. The Visual Studio Test vision
b. Visual Studio Ultimate 2010 and Team Foundation Server 2010
i. What is new in 2010
ii. The Visual Studio Ultimate Test framework
2. Extending Visual Studio Ultimate Test components
a. Third Party Integration Guidance
i. Supporting support Coded UI Tests and Record and Playback on third party technologies
ii. Integrate third party test tools and/or cross platform environments with the Visual Studio 2010 Test Tools
iii. Integration with Custom Diagnostics
iv. Migration and Synchronization or test data between ALM systems
v. Training and Test Methodologies
3. Summary & Next Steps
4. More Information
Background and Context
The Visual Studio Test Vision
The Visual Studio vision for Test is to provide great development tools, tightly integrated with great testing tools so customers can produce great software. VS2010 delivers tools that bring new insight, information and opportunities to software testers providing increased empowerment to contribute more fundamentally to software quality.
The Visual Studio Test vision is part of a broader vision to manage the full software development lifecycle in one place:
There is a degree of consistency in customer feedback around testing and software development:
• “My testers are spending too long testing the same thing”
• “Tooling is too complex and too expensive (time, licenses, people)”
• “It is expensive to maintain separate test teams for different platforms and technologies”
• “The developers say the reported defects are useless”
• “It is difficult to know when software is ready to ship”
• “Developers and testers work in silos and don’t communicate/speak the same language”
• “When requirements or software is changed it is difficult to know what needs to be tested”
Visual Studio (VS) 2010 intends to address these issues and more. VS2010 will deliver test software that simply works, for both the developer and the tester. The testing tools of VS2010 are designed to simplify testing and to remove obstacles between testers and developers with VS2010’s shared test management software, work items, historical debug, and screen capture. These changes can largely remove the “no repro” problem and provide a much higher fidelity of bug data back to developers to allow them to expedite fixes.
The extensibility of VS2010 Test tools will open the opportunity for partners to bring these benefits to the broader audience of X Platform users and to support test scenarios not otherwise supported by MSFT. These scenarios include testing of migrated Cobol, CICS and other host applications, LOB application testing for systems such as SAP and Oracle financials, J2EE, Java and Linux testing and certain types of functional tests. It is also possible to extend VS2010 Test tools to support other third party test tools to offer protection of customer’s investments in tests written for those tools or to allow those tools to be used in situations where MSFT test tools are not currently an option.
With these integrations customers will have a compelling option available. One well architected test infrastructure to assist both testers and developers, and span across all platforms, languages and runtimes in use. This infrastructure provides unified reporting for project progress and team performance across each of the associated teams and environments. Managers, testers and developers across all platforms should realize significant benefit in moving to these tools.
VS2010 plans to democratize testing by putting powerful, approachable tools into the hands of more testers and allow them to fundamentally impact software quality. In summary VS2010:
· Aligns testing with the lifecycle
· Creates tighter dev/test interaction
· Enables highly leveraged testing teams
· Is extensible to other platforms, applications, environments and test tools.
Partners are a key part of this vision, with the vision representing a significant opportunity for partners interested in providing X Platform, LOB testing and other specialized components. In the end customers on all platforms and systems benefit from this unified approach to Testing.
Visual Studio Ultimate 2010 and Team Foundation Server (TFS) 2010
What is new in Visual Studio 2010
Visual Studio Team System (VSTS) 2005 introduced the core Test Framework which will be carried forward into Visual Studio 2010 and Team Foundation Server 2010. With the introduction of VS 2010 and TFS 2010 many new software test capabilities are introduced, such as:
|
Team Test |
|
|
Performance Testing |
Extensible performance solution that includes hot spot identification and analysis as well as rich Excel based analysis and reporting |
|
UI Test Automation |
• UI automation framework that supports Fast Forward for Manual Testing as well as coded UI across multiple UI technologies including Web, Winform, and WPF |
|
Manual Testing |
• Lightweight, efficient tooling to support manual testing with integrated action recording as well as rich bug filing |
|
Test Case Management |
• Infrastructures to manage the testing effort and ultimately be able to answer the question “are we ready to ship?” |
|
Test Execution |
• The ability to execute tests locally and remotely, to collect detailed data during execution, enabling rich bug reporting. |
|
Lab Management |
|
|
Manage Lab Environment |
Manage multi-machine environments using virtual machines, with the ability to create, store, start/shutdown full environments, reducing the need for IT personnel being involved in such operations. |
|
Snapshot & Reproduce Environments |
Ability to take snapshots of lab environments and revert to them allowing teams to collaborate effectively in identifying and resolving bugs that are hard to recreate. |
|
Application Deployment |
Ability to automatically deploy applications in the lab environment allowing teams to effectively automate build verification tests and other activities requiring application deployments |
|
Build Integration Workflow |
Ability to create work flows for the build that enables lab environments to be automatically deployed when triggered during build |
The various testing components of VS 2010 are each targeted at different types of testers ranging from the Generalist to the Specialist tester as detailed in the following diagram:
These VS 2010 Test Components also shown in the following high level view of all the components that have a role in the testing process. There are three (3) client tools used to manipulate test information or to run test locally or directly. These client tools connect to TFS, which acts as the heart of the system. The build server works with Lab Management and Test Case Management in TFS to run automated tests as part of the build process.
To provide organizations with simplified ways to purchase and license core development technologies, MSFT is announcing a new packaging lineup and licensing options for MSDN and Visual Studio 2010.
This new approach to branding and packaging more closely aligns with other MSFT products that developers use every day, (including Windows and Office), which simplifies the purchasing process . The following diagram details what Debugging, Diagnostics, Testing and Lab Management features are available in each of the newly announced products:
As part of the broad VS 2010 launch Team Foundation Server (TFS) 2010 will begin to play a significant role in testing with the introduction of two key new services. These services are:
· Test Case Management Services, providing Test Case Management, Execution and Rich Integration
· Lab Management Services providing centralized control and management of Test labs comprising multiple physical and virtual test machines.
The Visual Studio 2010 Test Tools Framework
Microsoft® Visual Studio® 2010 Test Tools give you the testing tools to be more productive throughout your testing life cycle of planning, testing and progress tracking. The architecture of the VS 2010 Test Tools is built around the VS 2010 Test Tools Framework. This framework lets you create, manage, edit, and run tests, as well as obtain and store test results. Several test types, including Unit, Web, load, Coded UI and manual tests, in addition to the measurement of code coverage, are integrated into VS. VS Test tools can be extended to add new test types. The VS Test Tools Framework is the overall architecture and environment in which tests run. It consists of the following:
· Test Plans, Test Suites, Test Cases and Configurations. You can plan your testing effort for a team project by using test plans. You define your testing effort by creating any necessary test suites, test cases, or configurations. The configurations are used to determine which set ups you want to use to run test suites or test cases.
· With these items defined, you are ready for testing. Manual tests can be run from Microsoft Test and Lab Manager using the Test Runner. You can also run automated tests from Microsoft Test and Lab Manager as long as there is an automation associated with a test case. These test results will be associated with a test plan.
· In addition, automated tests unassociated with a plan can also be run from VS. Tests can be selected to run individually, as part of a check-in policy that uses test categories. They can also be run from Team Build and from the command line.
· For better integration between VS Test Tools and other parts of VS, test results should be stored into the TFS database. This enables users to easily generate trend and historical reports, compare different kinds of data, see the amount of as well as which bugs were found because of testing and more.
· Typically a Test Plan is used to define and manage your testing for a particular product.
· Test Plans consist of Test Suites which are collections of Test Cases. Test Cases are typically aimed at testing a particular feature or subsystem.
· When requirements, user stories, or features are ready to be tested, you can select a Test Plan or Test Case as appropriate and run your tests for each specified configuration. This approach enables you to target your testing and measure progress as tests are run and report the amount of testing that remains.
· “Test Cases” are stored as “Work Items” in the TFS Work Item store. VS 2010 supports many work item types and work items can be associated with each other using links. Links can also have assigned “types” including “Parent/Child,” “Tests/TestedBy,” “Related,” “Affects/AffectedBy.” Test Cases are related to other work items in VS 2010 TFS as shown in the following diagram.
· The following figure shows an example of a Test Case being edited in “Microsoft Test and Lab Manager” (which is included in Microsoft® Visual Studio® 2010 Ultimate & Microsoft® Visual Studio® Test Elements 2010):
· Test Cases are really just metadata, which point at Manual and/or Automated implementations. In the case of Manual Tests they consist of one or more Test Steps (Test Steps can be normal steps or shared steps). The following diagram shows the Test Runner of “Microsoft Test and Lab Manager” providing guidance, recording, and collecting feedback during a manual test. Note the Test Steps on the left (which indicate that three (3) Test Steps have passedand one has failed).
· Beyond Test Cases, most other test artifacts are stored as custom objects in the Test Management (TM) database store (a logical subset of the TFS SQL Server Database store). The Test Management database store includes Test Plans, Test Suites, Test Configurations, Test Steps (for manual tests), Test Results, Code coverage data, Test impact data, Run and result attachments & Test settings.
· A Test Management object model (TM OM) can be used to peruse, edit, or create TM database artifacts like Plans, Test Suites, Test Configurations, and Test Steps etc. This model is recommended in place of direct database access.
· TFS Catalog: Additional information is also stored in the TFS Catalog including Test environments and Test controllers.
· VS Test Types: VS 2010 testing tools provide several test types that can be used for specific software testing purposes. Custom Test Types can also be created to meet other testing needs. VS 2010 supports the following test types “out of the box”:
o Unit Tests
§ Unit tests give developers and testers a quick way to check for logic errors in the methods of classes in C#, Visual Basic .NET, and C++ projects. Unit tests allow the developer to assess the quality of their own code and also provide a security mechanism to reduce regressions when adding new functionality or changing existing functionality (such as when refactoring). An additional feature of Unit Tests is that they can be re-used within Load/Performance tests, providing broad, flexible support for application performance testing.
o Web Tests
§ A Web Performance test (previously simply called a Web test in VSTS 2005/2008) consists of a series of HTTP requests. Web tests work at the protocol layer by issuing HTTP requests. Web tests do not run JavaScript. However, you can simulate JavaScript actions at runtime by using Web test plug-ins, Web test request plug-ins, extraction rules, or coded Web tests. Web tests are used to test the functionality of Web applications and to test Web applications under load. Web tests are used both in performance tests and stress tests. You can create Web tests by recording your activities in a browser session. You can also build Web tests manually by using the Web Test Editor.
o Load Tests
§ The primary goal of a load test is to simulate many users accessing a server at the same time. When you add Web tests to a load test, you simulate multiple users opening simultaneous connections to a server and making multiple HTTP requests. You can set properties on load tests that broadly apply to the individual Web tests. When you add unit tests to a load test, you exercise the performance of non-Web based server components. An example application of a unit test under load is to test data access model components. Load tests can be used with a set of computers known as a Test Environment (aka Test Rig), which consists of agents and a controller.
o Coded UI Tests
§ Coded UI tests are automated tests of the user interface. These tests provide functional testing of the user interface and validation of user interface controls. Automated UI tests enable you to test to ensure the user interface is functioning correctly. They are quicker to run than manual tests thus can be run more frequently. A key advantage to these tests is that they use .Net languages (code is generated for C# and VB.Net) against a .Net API, thus organizations can use/re-use familiar skills and existing code to test their apps. This also results in the test team having interchangeable skills with the development team.
o Manual Tests
§ In VS 2005/2008 manual tests are usually in the form of a Word document, in VS2010 this format is deprecated and is replaced by the new Work Item based Test Case already discussed.
o Generic Tests
§ You use generic tests to wrap external programs and tests that were not originally developed for use in the Visual Studio Test Tools. After you have done this, the generic test is treated by the test engine as any other test type. That is, you can work with generic tests using the Visual Studio Test Tools windows. For example, you can run generic tests from the Test View window and the Test List Editor and you can obtain and publish results from generic tests as from other tests.
o Ordered Tests
§ An ordered test contains other tests that are meant to be run in a specified order. An ordered test appears as a single test in the Test List Editor and the Test View window. Its results appear in a single row in the Test Results window, but you can obtain individual results for each test that was run as a part of the ordered test.
· Custom Test Types
o In addition to using the built-in test types of VS Test Tools such as unit tests, load tests, coded UI and Web tests, developers can create and integrate custom test tools into the VS Test platform using the Test Type Extensibility Framework, the same framework that MSFT used to develop the built-in test types. The Test Type Extensibility Framework is intended for developers and testers who want to do either or both of the following:
§ Integrate existing test tools into the test environment
§ Create and integrate entirely new test tools that perform types of testing that are not supported by the built-in test types of Visual Studio Test Tools
o VS Test Tools also provide a more basic way to integrate custom test tools: the built-in generic test type. This test type lets you run any of your test tools that use a command line, and then captures and interprets the returned results. Even though generic tests serve as an efficient light-weight solution for integrating tests, they are better suited for simple tests and test harnesses that do not require more advanced extensibility features, such as a custom test editor or results viewer, and a run-configuration editor.
o A Custom Test Type is a DLL exposing two sets of APIs:
§ VSX package APIs providing UI and only used by the Visual Studio IDE
§ Test Execution APIs to provide test specific behavior and control of tests. These APIs are used by the test agent to control remote tests and the Visual Studio IDE to control local tests (thus local tests do not need a controller or agent)
o Custom Test Types can be created using VS. Custom Test Type definitions are stored in a custom “VSX” assembly. When queried for which services it provides, the assembly will return it’s test type. New Custom Test Types will show up in VS, and can be tied to Test Case work items. In turn, these can be included in test suites in “Microsoft Test and Lab Manager” and run with automated test runs, as long as the custom test type is installed on the machines where the test will be run. (Custom Test Types are installed on target test machines by copying the assembly that defines the custom test type into the PrivateAssembly folder of the install location of the agent).
o When developing Custom Test Types several levels of features can be optionally supported. If only basic test-tool integration is needed, only a small set of core extensibility areas are needed to create a simple test type that runs in the test environment. Such a test type can then be used in conjunction with all the features for managing tests, including the Test Manager, Test View, and Test Results windows. By implementing additional classes and a “VSX” package, you can add other features such as a custom test editor, integration with the run configuration dialog box, and a custom results viewer for your new test type. Detailed below are the ways in which you can integrate a new test type into the VS test environment:
§ Basic Test Management Windows. For tests that produce simple results, basic integration lets test users view, manage, and run tests.
§ Custom Editing Window and Results Viewer. For more advanced test types, you can display custom UI components for authoring tests and viewing their results.
§ Creating Tests that Run Under Load. You can also create tests that will run under a load test.
§ Adding Custom Run Configuration Settings. Your new test types can access the built-in run-configuration settings, and you can add custom run-configuration settings.
§ Custom Shortcut Menu Options. You can make special commands available for your new test type by adding them to shortcut menus.
o Note that “Microsoft Test and Lab Manager” does not use or install Test Type DLLs. It uses a lowest common denominator level of functionality (Start Stop….) achieved by communicating directly to the controller without need of the information and functions provided by Test Type DLLs
o For more information on Custom Test Types please see http://msdn.microsoft.com/en-us/library/bb166484(VS.100).aspx or search for “High-Level Test Platform Extensibility Areas” in MSDN
· UI Automation testing is a key new feature with VS Premium 2010. UI Automation Testing is supported for Winforms, and WPF applications that support the MSAA (Microsoft Accessibility API) or UIA (UI Automation API) APIs . (Web applications are also supported via MSAA and DOM manipulation). Support for Silverlight Testing will be added after the release of VS2010. Third parties can provide tools or libraries to extend applications based on other technologies (e.g. Flash and Java) and component vendors should update components to support MSAA or UIA. The following table shows currently planed platform support for UI Automation:
· A test configuration is a set of configuration variables that specify the correct setup required for testing an application. The configuration variables include hardware, operating system, software, and any other characteristics that are important to use when you run the tests. Each test configuration can represent an entry in your test matrix as shown in the following diagram:
· Test Environments, Controllers and Agents. You can set up a group of computers to run tests. The group consists of a single “Controller” and one or more agents. Collectively, this group is called an “Environment.”
o An agent is the part of the environment that is used to run tests and generate simulated load
o The Controller is a separate “service” process which the VS IDE or TFS uses to communicate with one or more Agents on remote machines. For example, in a load test the Controller might control Agents on hundreds of machines creating the load. The Controller is the part of the Test Environment (aka Test Rig) that is used to coordinate the Agents and collect the test results. TFS uses a Controller (or Controllers) to connect to Agents on virtual machines provisioned by Lab Manager in order to run, control and receive feedback from tests
o The test Environment and infrastructure are basically the same for the various test types supported:
§ Unit Tests
§ Load Tests
§ Coded UI Tests
§ Web Tests
o A test environment is not used when a test is run locally. Instead VS Test Tools use the Text Execution APIs in the Test Type DLL to control the test directly
· Test Agents. The Test Agents are responsible for:
o Test Execution and control including
§ Start
§ Stop
§ Step (Start and run to the end of the next Step)
o Data Collection and passing data back to source (to VS Test Tools or to TFS Lab Management Server). Examples of data that might be collected include:
§ Screen Capture
§ Capture of the state of variables defined in the Test Plan
§ Collection of Historical Debug information
§ Capture of application performance statistics
§ Capture of machine performance statistics
§ Capture of Network, Keyboard, Mouse, Touch Screen traffic
§ More…
o Test Agents install VS Test Types DLLs and Custom Test Types DLLs as add-ins to equip them with Test Type specific functionality. As noted above Test Agent will use the Test Execution APIs in the Test Type DLL and ignore the “VSIP” Package APIs in that DLL.
· Host Adapters
o A host is a test execution environment, an application process in which test code is executed, and in the case of unit tests, production code as well. VS Test Tools provide a default test host, VSTestHost.exe. By default, tests are executed in the VSTestHost.exe process.
o There are limitations on running tests using the default test host. For example, when you run unit tests in VSTestHost.exe, the production code is not executed in its real execution environment. Production code may not work when not running in its real execution environment. A typical example is that of unit tests for ASP.NET Web applications. The ASP.NET application code may need to access the ASP.NET context, which makes objects such as the HttpSessionState, HttpApplication, and HttpRequest available to the application code. Also, page-specific objects must be initialized properly so that application code that works with them can run without throwing exceptions. VSTestHost.exe cannot provide either the ASP.NET context or these runtime objects. VS 2010 Test Tools overcomes this problem by using an ASP.NET host adapter. You are able to run unit tests for ASP.NET Web applications in the IIS process or in an ASP.NET Development Server process
o There are other situations in which you want to execute tests in hosts other than the default test host. VS 2010 Test Tools provide the means by which you can develop your own “host adapters.” For more information please see http://msdn.microsoft.com/en-us/library/bb166558.aspx or search for “Creating and Using Host Adapters” on MSDN.
· Cross Platform Testing
o Today agents only exist for Windows. The Controller connects to the Agent on the target Test machine using proprietary .Net remoteing which only works for Windows machines. For cross platform testing Microsoft is considering the creation of a controller proxy as a separate service which may support cross platform testing by supporting platform agnostic protocols (perhaps SOAP or REST) so partners can support non windows agents. The controller proxy is still under consideration, partners interested in working with Microsoft to create Cross Platform testing tools using such a facility should contact us on vsipinfo@microsoft.com
In VS 2010, testing tools allow increased production throughout the testing life cycle of planning, testing and progress tracking. The testing tools are integrated with Team Foundation Server, allow you to define your testing based on the same team projects being used by others in your organization.
Extending Visual Studio 2010 Test Tools
Third Party Integration Guidance
The primary scenarios or areas of opportunity for partners to extend VS 2010 Test Tools are:
· Write plug-ins for Java, Flash, Components suites etc. to support Coded UI Tests and Record and Playback of tests on those platforms
· Integrate third party test tools and/or cross platform environments with the VS 2010 Test Tools. Required integration components include Custom Test Types, and Agents
· Integration with custom diagnostics & historical debugging.
· Migration and Synchronization or test data between ALM systems
· Training and Test Methodologies
Additional details below:
· Providing support Coded UI Tests / UI Automation Testing and Record and Playback of tests on various technologies. Partner opportunities include:
o Extending other environments and applications to offer U I Automation support for to un-supported platforms such as Flash, SAP, CICS, Cobol etc by providing plug-ins to expose those applications for automation
o Component Vendors should update components to support MSAA or UIA APIs and then test components to confirm that UI Automation Testing works with applications using those components
· Supporting third party, and/or cross platform test tools. For partners with existing Test Tools and in particular complementary test tools such as cross platform test tools we offer the following guidance regarding how best to integrate with VS 2010 Test Tools:
o Use VS 2010 Team Foundation Server Work Items and Test Management (TM) Database as much as possible to store information and results related to tests. To do this:
§ Third party test tools should store Test Cases as TFS work items of Type=Test Case. Work item types can be extended where the third party test software require more information to be stored than can be handled by the standard VS types. Third party tests will often contain executables and run time dependencies on the third party test environments which make “compatibility” between test environments impossible. The use of a common infrastructure for storage and management of tests simplifies the test environment and reduces training costs which can result in common processes and skill set requirements across test types and environments.
§ Third parties test tools should store artifacts like Test Plans, Test Suites, Test Configurations, Test Steps (for manual tests), Test Results , Code coverage data, Test impact data, Run and result attachments, and Test settings etc. in the Test Management (TM) Database. Partners should use the Test Management object model (TM OM) to create (peruse or edit) the TM database
§ The above Test artifacts should be used to enable the following scenario: the end user of “Microsoft Test and Lab Manager” ( included in Microsoft® Visual Studio® 2010 Ultimate & Microsoft® Visual Studio® Test Elements 2010) would be able to use it in nearly or exactly the same way to run a third party test as to run a MSFT test. This approach is designed to drive greater simplicity and standardization into the test process while lowering training costs and skill requirements.
o Support Automated Testing using VS Test. This means supporting test cases defined in VS 2010 Test Tools to be executed on third party test tools or platforms such as commercial or open source software test execution frameworks (functional testing, performance testing, security testing, etc.) and/or on other operating system platforms. This means
§ The creation of new test types for those target environments and
§ The creation of Third Party Test Agents.
§ For cross platform agents use of the planned controller proxy will be required to allow use of standard protocols (controller proxy is still under consideration)
In this way it is possible to support third party test tools to protect customers’ investments in tests written for those tools or to allow those tools to be used in situations where MSFT test tools are not an option.
o Support for Automated testing using TFS Test Case Management Services. This means providing the Third Party Test Types and Agents to deploy, run, monitor and control non MSFT Tests and/or non-Windows platforms from TFS Test Case Management Services. It is also means collecting results from those tests and storing them in the Test Management (TM) database. Third Party Test Types run as an extension to TFS Test Case Management Services
o Support for Automated testing using the VS Lab Manager Services of TFS. This means extending the VS Lab Management Services of TFS using Powershell scripts etc. to support the creation of virtual instances of other non-MSFT operating systems such Linux or operating systems configured to test certain classes of applications such as Mainframe applications, web applications, SAP applications etc. VS Lab Manager Services works in concert with Test Case Management Services to deploy and run the tests in question as well as to collect results of those tests.
o Support for specific technologies such as SOA, Cloud or, Runtime environments (such as flex, flash, Java, J2EE, CICS , Mainframe Emulation etc.) This means the creation of new and/or extension of existing test types that can provide test execution, result analysis and reporting capability for those technologies and provide the Third Party Test Types and Agents to deploy, run, monitor and control those tests
o Support for Packaged and LOB applications such as SAP, Dynamics or Oracle Financials. This should cover both load, performance and functional testing. Protocol testing such as SAP, SIP, etc. should also be considered
o Extensibility for newer domains. For example, functional testing for packaged apps, flex, flash, Java, etc. Performance testing for additional protocols such as SAP, SIP, etc.
o Data Diagnostics is a broad area of opportunity for partners and includes:
§ Ability to collect data (system, application diagnostics or other) on non-Windows environments, non-.NET environments or from third party test systems
§ Ability to extend existing data collectors to provide more functionality
§ The degree of integration and functionality of Test Agents is dependent upon design objectives but could range from
· No changes necessary to an existing Third Party Test Agent because integration is done by way of a Proxy, to
· Significant integration of the Test Agent to support things such as:
o Screen capture
o Capture of the state of variables defined in the Test Plan
o Collection of historical debug information
o Capture of application performance statistics
o Capture of machine performance statistics
o Capture of Network, Keyboard, Mouse, Touch Screen traffic
o Capture of application specific data from systems like SAP, Oracle Financials, Dynamics etc.
o More…
o There are also several opportunities in the areas of Test Planning, Process Integration and Reporting:
§ Partners can create, extend or customize TFS process templates to meet specific customer needs and to implement and enact specific testing methodologies
§ Partners should verify that standard TFS reports accurately reflect third party data stored in standard VS formats such as Test Plans, Cases, Steps etc. This includes reports such as the following:
§ Partners can customize and extend out of the box reporting to address customer needs
§ Custom reports are created using the TFS reporting tools (based on SQL Server Analysis Services, Reporting Services, Excel Services and SharePoint). Reports should offer reporting for any Custom data collected and stored on TFS work items beyond the data for which TFS provides standard report formats
· Migration and Synchronization of test data between ALM systems.
o Opportunities exist for partners to create tools that migrate test data from other test management systems to TCM/TFS
o There are also opportunities around the development of interoperability tools that allow for bi-directional sync of data between TFS and other test case/requirements mgmt systems
· Training and Test Methodologies
o With the new functionality in VS 2010 there is a significant opportunity for partners to develop courseware, certification, and training around test
o There is also an opportunity to take well established test methodologies and codify them as TFS Process Templates so that those methodologies can be “enacted” by the VS / TFS ALM system. This is an effective way to take methodologies that may languish as books on the shelf and bring them to live inside familiar tools.
Summary and Next Steps
With such integrations in place the combination of MSFT + Partner test products are intended to appear nearly seamless to the user. Procedures, reports and skill can be used consistently across a broad range of test types, platforms and teams and with improved communication between testers and developers on all platforms. Such a combined offering will be unique in the test market and provide significant differentiated value in the test marketplace.
Next steps depend on your situation
· Component Vendors should focus on updating component suites to support Coded UI Tests and Record and Playback of tests .
· Test Tool vendors should look the opportunities related to
o Integrating third party test tools and/or cross platform environments with the VS 2010 Test Tools.
o Migration and Synchronization or test data between ALM systems
· Software diagnostics partners should look at the opportunities related to integration with custom diagnostics & historical debugging.
· Training Companies should look at the production and delivery of training materials and courses related to the new Test tools in VS 2010
· Test and ALM Methodology companies should look at the possibility of
Partners wanting assistance with such integrations should consider joining the Visual Studio Industry Partner Program (See http://msdn.microsoft.com/vsip ) and for more information please email vsipinfo@microsoft.com .
More information
Key MSDN Links
More detailed information on testing with VS 2010:
http://msdn.microsoft.com/en-us/library/ms182409(VS.100).aspx
http://msdn.microsoft.com/en-us/library/ms182409(VS.80).aspx
http://msdn.microsoft.com/en-us/library/fda2bad5.aspx
Visual Studio SDK - Visual Studio Team Edition for Testers
http://msdn.microsoft.com/en-us/library/bb166187(VS.80).aspx
Visual Studio 2010 Test Tools Blogs
Visual Studio 2010 New Features, Extensibility Points and Partner Opportunities
http://blogs.msdn.com/terryclancy/archive/2009/05/19/visual-studio-2010-new-features-extensibility-points-and-partner-opportunities.aspx
Visual Studio Team System 2010 New Features, Extensibility Points and Partner Opportunities
http://blogs.msdn.com/terryclancy/archive/2009/05/20/visual-studio-team-system-2010-new-features-extensibility-points-and-partner-opportunities.aspx
More …..
http://blogs.msdn.com/somasegar/archive/2008/12/12/lab-management-in-vsts-2010.aspx
http://msdn.microsoft.com/en-us/teamsystem/default.aspx
Additional VS Team Tools resources/links/videos at:
http://www.teamsystemcafe.net/Resources.aspx
