Test Specification for VS Extensibility

 

Have you ever wondered what we test on VSIP & Extensibility features in the VS QA team when we release Community Tech Previews?  We run a small number of automated tests on most of our features against each new build of VS & VSIP. If the results from tests are good, we declare the build usable, Self-host, for further testing/usage.  We call this subset of tests our Acceptance tests or Nightlies.  They basically verify build quality.  This is the same set of tests we run when we release Community Tech Previews.  We have many more automated and manual tests that are run on an ongoing basis during the product cycle.  We would like to share this set of our Acceptance tests with our customers and would like to hear your thoughts.  Is this too much testing, is it good test coverage, do you see a big hole in a most commonly used feature area?  Any other feedback you would like us to evaluate and incorporate?  Is this useful information if you also had the results of these included when we post the Community Tech Previews?

 

Extensibility consists of 2 major parts, features that ships in the VS Box(namely add-ins, macros, policy, object model) and features/tools that ships in VSIP.  Note, there are many other QA teams doing testing in other feature areas, what we present below is what is applicable to the Extensibility user community. 

 

VS

 

Add-Ins

Walk through the VS Add-in wizard for all lanugages supported, verify add-in builds

Verify add-in loads successfully

Verify unload of add-in from the Add-in manager works

Removing of .add-in file (new xml registration for add-ins) updates the add-in manager and does not show the add-in.

Verify the add-in manager respects the Tools options setting for add-in “Allow Add-in Components to load”

Verify add-in with wizard option to create a Tools menu item works.

Walk through the Shared Add-in wizard for all languages supported, verify add-in builds.

 

 

Macros

View the Macro Explorer, verify default Macro project created

Verify Can add a new module and a new macro to the MyMacros project, can run the macro

Verify creation of a new Macro project and unloading of the same.

Recording of a set of actions in the VS Editor, Stop recording, verify the code emitted,

Playback of macro above results in expected behavior

Record of new VB project creation, stop recording, verify code emitted.  Record a new macro to  unload the VB project.  Verify Temporary macro code previously emitted is overwritten.  Verify new temporary macro runs as expected.

Verify a new macro behaves like a VS menu command that can be invoked from the Command Window

Add a breakpoint to the macro in the Macros IDE, verify running the macro from Macro Explorer hits the breakpoint and then run to execution in debug mode

Macros with event handlers does disable events when user clicks on Disable event handlers.  Verify no event handlers from the Macro project are executed.

Macros with event handler, enabled for running event handlers does work correctly when a macro project with event handlers is loaded.  Verify events do fire.

Verify Tools options setting for Enabling/Disabling macros works as expected from the VS IDE macro explorer.

 

Policy

Verify a policy action file can be applied to different language projects

Verify can apply and remove policy action files to Solution folders and projects within the solution folder

Verify ElementSet Exclude works when System.Windows.Forms.dll reference is excluded. Apply this policy file to a Winform application, verify a warning is shown in the Errorlist window.  Apply the same policy file to a Console Application project, verify no warning is shown

Verify a policy file having Context links when applied shows the Contexts in the Dynamic Help window and Dynamic help window is updated when the policy file is removed.

verify can apply policy with reference for System.Windows.Forms.dll defined and excluded. Verify for a windows application a warning is shown in the error list,  verify with a console app you can add the reference and a warning shows up

Verify an errorneous policy file results in an error message and does not apply policy

Verify Property constraints, ToolboxItem constraints and Command constraints work

Verify adding and applying code type policy rules work for a Winform application, where policy file has code type for winform defined and excluded.  Verify for a winform application it shows a warning and for a console application it does not

Verify with solution folders and projects, Policy conflicts and propogation are handled correctly in a solution folder hieararchy

Enum constraints work correctly

Verify the correct warnings show up in the error list window

 

Object Model

·         Events

DTE startup/shutdown event fires

TaskList,Documents, Window events fire

VC Project and ProjectItem events fire

VB,C#,J# Project and ProjectItem events fire

Build events fire

 

·         Solution, Project & Build Configuration

On the Project object – Verify you can get to CofigurationManager, CodeModel, ProjectItems, Properties, Project.Object returns VSProject object.

On the ProjectItem object – verify you can get to ContainingProject, Document object, FileCodeModel, VSProjectItem, Properties, SubProject, Save

On the ProjectItems collection – you can get the ContainingProject, can iterate through the items and AddFromTemplate, AddFromFileCopy, AddFromFile, AddFromDirectory work

On the Solution Object – can get to the Projects, TemplatePath, AddFromTemplate, FindProjectItem, Close, Remove, Saved work

Solution Build object – can get the ActiveConfiguration, iterate through solution configurations, BuildDependencies, BuildState and Solution building works

VSProject object on VB.C#, J# projects – Verify we get can get to the BuildManager, VSProjectEvents, References, Imports and  DTE Project object, PublishManager and VSProjectEvents2

Can get to the VSProjectItem from the DTE’s ProjectItem object.

 

·         DTE, Tasklist, Outputwindow, Document, Text Editor, Toolbox, Window

Verify objects returned with DTE – ActiveWindow, ActiveDocument, ExecuteCommand

Get the tasklist object, iterate taskitems, can get to the taskitem description and add a new taskitem, set the priority on the new taskitem

Get the outputwindow object, add a new pane, can get to the TextDocument object on a outputwindowpane, add an item to the errorlist window.

Can get to the Document object, can iterate through documents collection, can get to the TextDocument object, selection returned is correct

create each object in the text editor object model and verifies creation – TextPoint, TextRange, TextSelection, VirtualPoint, EditPoint, TextWidnow, TextPane, UndoContext, HTMLWindow

Can iterate through Windows collection, can get to Document, Project, Projectitem objects on a Window object, returns the toolwindow object where supported

Get the Toolbox object, add a new toolboxtabs, new toolboxitem, can delete toolboxtab

 

·         Globals

Verify can get to DTE globals, can add a new global, verify name-value pair returned.

 

·         Code Model

Test CodeNameSpace methods and properties on C++, C#,  Projects – AddClass, AddDelegate,AddEnum,AddStruct,AddInterface,AddNameSpace, DocComment

Test FileCodemodel & FileCodeModel2 methods and properties for VB project – AddClass, AddNamespace, AddStruct, AddEnum, AddVariable, AddFunction

 

 

 

VSIP

 

Setup:

Verify VSIP SDK can be setup successfully after installing VS. 

      Verify the Experimental hive is correctly setup after installing VSIP

 

 

Interop Assemblies

A Primary Interop Assembly is a unique assembly that contains type definitions (as metadata) of types implemented with COM. VSIP Primary Interop Assemblies refer to the managed definitions for the Visual Studio Integration Partner (VSIP) interfaces that clients can use to write packages using managed code (C#, VB.NET and Managed C++) targeting Whidbey IDE.

 

·         Type Verification. Verify number of types and signatures.

Where a type can be an interface, struct or enum. The comparisons are first made between the source (a type library) and the corresponding target (an interop assembly) and then they are compared against last known good interop assembly. The second comparison is a conservative test to track, any changes or new types added, to the IA. There are some types in type libraries that are purposely omitted from the IAs.

 

PIA nightly test verifies following scenario for each IA that shipped in VSIP Everett Extra’s (as we call them baseline IAs)  plus the new IAs that are part of VSWhidbey VSIP

1. interface number in IA matches the number in the corresponding type library.

2. enum number in IA matches the number in the corresponding type library.

3. struct number in IA matches the number in the corresponding type library.

4. interface signatures in IA matches the signature in the corresponding type library.

 

Besides, test stores type information of the last known good  (LKG) IA build and verifies 4 scenarios above against LKG IA. Any unexpected change from LKG is logged as a bug.

 

·         Round-trip testing between Managed and Unmanaged code

Calling Unmanaged code from a Managed package and vice-versa for 4-5 different Interfaces

 

Wizards

·         Package Wizard:  For both the languages supported for the package wizard (C# and C++):

Create new Integration package with ToolWindow, MenuItem and Editor options checked.

Verify that the project gets built without errors

Start Exp shell and verify package loads

 

·         Language Wizard:  Test that the wizard operates correctly on the default settings (using MyC) with symbols loaded from the default lexer & parser files

Verify that service.cpp & service.h were created

Verify that the g_languageProperties table was generated according to the specifications given in the wizard

Verify that the tokenInfoTable was generated according to the specifications given in the wizard

Verify that the lexer & parser files were added to the project

Verify that the project compiles

Verify that the language exists in the registry & has the correct settings in all registry keys

Verify that the default features work in the Exp environment

 

Samples  For each Managed and Unmanaged sample:

Verify that the sample builds without errors.

Verify that the sample registers without errors.

Verify that the Environment loads the sample without errors under both the product and experimental hive.

Verify some very basic functionality of each sample.

 

Managed Package Framework

Verify correct operation of RegPkg on a Whidbey package

Verified basic functionality of the OleMenuCommandService class

Verify OleMenuCommand class

Verify the GetService method in the Package class

Verify that all attributes are registering properly with RegPkg

Verify that the Package class functionality is available to derived packages

 

Package Loading:  This is Visual Studio’s package load behavior testing with various switches.

Verify that internal package loads correctly while in safemode

verify that the external package is not skipped after having failed to load and being selected not to skip.

verify that the external package is skipped after having failed to load and being selected to skip.

verify that one previously skipped package is reset when using the /resetskippkg flag

Verify that /novsip causes invalid PLK packages to fail silently to load

Verify that /novsip does not cause an internal package to fail to load

Verify that rerunning devenv w/o the /novsip flag puts the invalid PLK pkgs back the way they were

Verify that /novsip does not cause a package with a valid PLK to fail to load

 

 

You can send us feedback at vsextend@microsoft.com