Michael Taute's Blog

VSTS 2010 Load Test Feature – Multiple Loadtest Plug-ins

2009-06-10T00:22:47Z

If you haven’t yet heard, Visual Studio Team System 2010 has reached Beta 1 and is available for public download. (You can find a copy of it here). Over the next few days my colleagues and I will be writing up information about some of the most anticipated features for the load testing portion of the product. If you haven’t yet checked out Ed Glas’s listing of the newest web and load test features, I recommend taking a look. As we write up more of these blog entries, he will link to them directly from his list.

In this post, I’m going to describe how (and why) to use multiple load test plug-ins with your load tests. If you want to run code at specific times during the loadtest, the best way to do that is to write a loadtest plug-in. Prior to Visual Studio 2010 this functionality was available, but you were limited to one plug-in per loadtest. This can be less than ideal if you want to create reusable plug-in code since all of the code had to be crammed into the same plug-in, regardless of whether it was logically similar to do so. In Visual Studio 2010 we’ve made this quite simple.

1) Add your test plug-ins to your project and compile the project.

2) After creating your loadtest, right click on the test node in the loadtest editor, and choose “Add Load Test Plug-in”

3) Select the plug-in you wish to add. From here, you can view its properties and set the initial values you wish for them to have at run time. Note that you can expose as many properties as you want from your plug-ins; just make them public, settable, and of a base type such as Integer, Boolean, or String.

4) Click “Ok”, and the newly added Load Test plug-in will be shown in the editor tree.

5) Repeat the process to add as many plug-ins as your loadtest will need.

6) If you’re worried about the order the plug-ins will execute, just click and drag the plug-ins to rearrange them. They will run in the order you have listed.

Diagnosing and fixing Web Test recorder bar issues.

2007-11-09T22:20:00Z

Since VSTS 2005 shipped there have been a variety of different questions raised by customers over the years. One topic which comes up somewhat frequently is a disabled or nonexistent web test recorder bar when recording web tests. Since it can be frustrating to diagnose and fix these problems, I’ve compiled a list of the most common reasons for this to happen and potential fixes for each.

If you know of something that fits the “My web test recorder bar isn’t working” criteria and isn’t on the list, please let me know and I’ll try to add it.

VSTS 2005 : Any OS : Recorder bar is not displaying when recording a webtest.

Fix: Upgrade to VSTS 2005 SP1. There are a few bugs preventing the webtest recorder bar from displaying under certain circumstances that were fixed in SP1. If you're having trouble in VS 2005 and haven't yet upgraded to SP1, that's your first step.

VSTS 2005 : Vista : Recorder bar displays, but the controls are disabled.

Fix: Webtest recording using VS 2005 on Vista only works when running in elevated mode. [When launching VS 2005, right click the icon and choose to run as administrator]

VSTS 2005, 2008 : Windows 2003 / Vista : Recorder bar comes up, but the controls are disabled.

Fix : the web test recorder bar does not work with IE enhanced security enabled. (VSTS 2008 will actually display an error message telling you this) IE enhanced security can be removed from within the Control panel / Add Remove Programs / Windows Setup

VSTS 2008 : Vista, Win7 (64 bit) : Recorder bar does not appear when recording a new webtest

Fix: Vista caches the list of explorer bars you have available and the recorder bar was not included in your list. The fix is to force Windows to rebuild that cache. To do this, first make sure you have all Internet Explorer instances shut down, then open the 32 bit registry editor and delete the following keys:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Discardable\PostSetup\Component Categories\{00021493-0000-0000-C000-000000000046}
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\Discardable\PostSetup\Component Categories\{00021494-0000-0000-C000-000000000046}

[Note: by default, the 32 bit registry editor is located in %WINDIR%\SysWow64\RegEdt32.exe]

The next time you boot Internet Explorer, your explorer bar cache will be rebuilt and the recorder bar should be available.

VSTS 2005, 2008 : Any OS : Recorder bar does not appear when recording a new webtest

Fix: Make sure that you don’t have browser add-ons disabled. Specifically, the web test recorder helper add-on needs to be enabled. For Internet Explorer 7, you can find this setting in Tools / Internet options / Program [tab] / Manage add-ons. From there, verify that Microsoft Web Test Recorder Helper is enabled.

VSTS 2008 : Any OS (IE 7) : When recording, a new web browser opens up and the new browser doesn’t have a recorder bar.

Fix: The reason for this is that by default, about:Blank is displayed in the browser when you start recording. This website is (by default) in the internet security zone (protected mode: on). When you open a website in a different zone (without protected mode) it will be handled by a new instance of internet explorer.

The fix is to do one of 2 things: a) Move about:blank into the zone where you’re attempting to record from or b) Change the default web page which is brought up when you start recording to something in the same zone as you’re trying to record. You can accomplish this by setting the following registry key

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\WebLoadTest]

"StartingLocation"="http://server/website"

VSTS 2005, 2008, 2010 Beta 1 : Windows 7 : When launching the recorder, the recorder bar does not appear

Fix: The Windows 7 security model is slightly different than previous versions of windows. (go figure :) To get the recorder to show up, you'll have to run Visual Studio in elevated mode.

VSTS 2005, 2008, 2010 : IE 8 : While recording, the recorder bar will sometimes just stop working and tell me it wasn't launched from within Visual Studio

The issue here is yet again, security. In this case, when you cross the zones defined in IE. When you cross between different zones (by default, these are trusted sites, intranet and internet), Internet Explorer will actually kill your existing IE process and launch up a new one. This new IE process was, as the error message says, not launched by Visual Studio and hence the recorder cannot work.

Fix: There are multiple fixes for this problem, any one of which *should* work equally well.

1) Turn off "Protected Mode" in Internet Explorer. (Tools | Options | Security - There should be a checkbox to enable / disable protected mode) You'll need to reboot IE.

2) Change the initial URL which Visual Studio loads when recording a webtest to one in the same zone as you wish to record. The regkey to do this is below. Note that "9.0" here corresponds to the Visual Studio version you have installed. (2005 = 8.0, 2008 = 9.0, 2010 = 10.0)

[HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\WebLoadTest]

"StartingLocation"="http://server/website"

3) Run in elevated mode (I'm pretty sure that running in elevated mode will automatically turn off protected mode, so this is akin to fix #1 above.)

New Web Test Features in Visual Studio Team System for Software Testers 2008

2007-06-21T22:50:00Z

Visual Studio 2008 Beta 2 is just around the corner and here is a summary of a few of the upcoming features you'll see:

Support for initialization and termination tests per virtual user.

In the past it's been difficult to manage operations which only have to be performed once at the beginning and/or end of a test. Things like login, checkout, creating a user, etc needed to be managed creatively to get to work. Visual Studio 2008 has first class support for initialization and termination tests, which, as their name implies are run at the beginning and ending of a virtual user sesson.

Adding them is easy. Just bring up the test mix editor (shown below), select the checkboxes at the bottom, and choose which test you want to be your initialization and / or termination test.

[Test mix editor]

Once you've selected your init / terminate tests, the webtest editor shows them as part of your scenario:

Multiple Web Test plug-ins and request level plug-ins:

With multiple Web Test plug-ins, componentizing your code is much easier in VSTS 2008. You're not forced to cram all of the code you might need into a single plug-in so you're free to mix and match plug-ins across multiple projects without the extra overhead. Additionally, you can also expose public properties from your plug-ins to make them even more useful on a per-test basis. Here's a screenshot of what adding a Web Test plug-in looks like now:

True request level plug-ins are now a reality as well. Just like Web Test plug-ins, you can have multiple request level plug-ins directly attached to a specific request rather than having to filter requests inside the plug-in. Of course, the request plug-in infrastructure from VSTS 2005 was not removed and your existing plug-ins will run without modification. In fact, they still provide a great way to run code after every request without adding a request plug-in to each node in the editor. Here's a look at the new Web Test editor with multiple plug-ins added:

Lastly - worried about the order your plug-ins will run? They run top-down in the order you see them in the UI!

Managing goal-based load using multiple perfcounter thresholds

2006-12-16T01:12:00Z

As promised, here is a sample plug-in that I wrote a few months ago which people here may find useful. The purpose of this plug-in is to expand the functionality of goal-based load which ships with Visual Studio team test to allow you to have your load step up or down based on any number of performance counters.

For the plug-in to work, you need to have a few things set on your loadtest.

Thresholds:

For each perfcounter you intend to track, you need to set the warning and critical threshold values on that perfcounter.

Context variables:

MinLoad: <The Minimum amount of load you wish to apply, regardless of perfcounter thresholds.>

MaxLoad: <The maximum amount of load you wish to apply, regardless of perfcounter thresholds.>

LoadStep: <The amount of load to step when increasing or decreasing.>

LoadChangeInterval: <How often you wish to change the load (in seconds and if necessary)>

PerfCounter<#>: A path to the performance counter in the form: <machine>\<category>\<instance>\<counter> For example: COMPUTER_NAME\Processor\_Total\% Processor Time

Note that you can have as many perfcounters as you wish to track, however you need to make sure they are sequential in number. (in other words, if you have PerfCounter1, PerfCounter2, and PerfCounter4 - PerfCounter4 will not be tracked.)

Plug-in Code:

using System;
using System.Collections;
using Microsoft.VisualStudio.TestTools.LoadTesting;

/*
* VariableLoadPlugin - Useful for using multiple performance counters with goal
* based load profiles.
*
* Usage: Assign the following context parameters to your loadtest
* MinLoad - The minimum amount of load to generate.
* MaxLoad - The maximum amount of load to generate.
* LoadStep - The amount of load increment / decrement by when attempting
*            to reach your target thresholds.
* LoadChangeInterval - The number of seconds between load changes
* Perfcounter# - A performance counter you wish tracked. The value should
*              be of the form: <computer>\<category>\<instance>\<counter>
*              For example: Perfcounter1 : \MYCOMPUTER\Processor\_Total\% Processor Time
*
* Notes:
*
* 1) You can have as many perfcounters as you want, they just need to have sequential
* context values. (e.g. Perfcounter1, Perfcounter2, Perfcounter3, etc)
* 2) The plugin relies on Warning and Critical ThresholdExceeded events to fire in order
* to adjust the load. That means you need to set Warning and Critical threshold values
* as part of your loadtest for the perfcounters you want tracked.
* 3) There is a bug in the RTM version of VSTT in which ThresholdExceeded events do
* not fire reliably when a unit test is part of the loadtest mix. For this reason, it's
* recommented that you don't use unit tests with this plugin.
*/

namespace LoadtestPlugins
{
public class VariableLoadPlugin : ILoadTestPlugin
{
    private ThresholdTracker m_thresholdTracker;
    private int m_minLoad;
    private int m_maxLoad;
    private int m_loadStep;
    private LoadTest m_loadTest;

    /*
     * The loadtest plugin initialize event - Used to parse the user's
     * data out of the context object and set up the ThresholdTracker object
     * as necessary.
     */
    public void Initialize(LoadTest loadTest)
    {
      int contextIndex = 0;
      int intervalLength = 0;
      string currentCounter;

m_loadTest = loadTest;

      /*
       * Assign MinLoad / MaxLoad / LoadStep and Interval based on context variables.
       * If they don't exist, assign defaults of 1, 1000, 10, and 20 respectively.
       */
      AssignContextValueIfAvailable("MinLoad", ref m_minLoad, 1);
      AssignContextValueIfAvailable("MaxLoad", ref m_maxLoad, 1000);
      AssignContextValueIfAvailable("LoadStep", ref m_loadStep, 10);
      AssignContextValueIfAvailable("LoadChangeInterval", ref intervalLength, 20);

      if (null == m_thresholdTracker)
      {
        m_thresholdTracker = new ThresholdTracker();

        currentCounter = String.Format("PerfCounter{0}", ++contextIndex);
        while (m_loadTest.Context.ContainsKey(currentCounter))
        {
          // Add a lowercase version of the perfcounter into the ThresholdTracker
          m_thresholdTracker.AddTrackedPerformanceCounter(m_loadTest.Context[currentCounter].ToString().ToLower());
          currentCounter = String.Format("PerfCounter{0}", ++contextIndex);
        }

// Allow the thresholdTracker to start tracking perfcounter data.
m_thresholdTracker.StartTracking(intervalLength);

        // Attach the event handlers as necessary.
        m_loadTest.ThresholdExceeded += new EventHandler<ThresholdExceededEventArgs>(LoadTest_ThresholdExceeded);
        m_loadTest.Heartbeat += new EventHandler<HeartbeatEventArgs>(LoadTest_Heartbeat);
      }
    }

    /*
     * The ThresholdExceeded event, executed by the loadtest engine when a perfcounter's
     * threshold (warning or critical) is violated.
     * This method passes that data into the ThresholdTracker object for consideration.
     */
    void LoadTest_ThresholdExceeded(object sender, ThresholdExceededEventArgs e)
    {
      string counterName;

      // Counters with no instance have a placeholder for the instance name
      if ("systemdiagnosticsperfcounterlibsingleinstance" == e.CounterValue.InstanceName)
      {
        counterName = String.Format("{0}\\{1}\\{2}", e.CounterValue.MachineName,
                                                     e.CounterValue.CategoryName,
                                                     e.CounterValue.CounterName);
      }
      else
      {
        counterName = String.Format("{0}\\{1}\\{2}\\{3}", e.CounterValue.MachineName,
                                                          e.CounterValue.CategoryName,
                                                          e.CounterValue.InstanceName,
                                                          e.CounterValue.CounterName);
      }

m_thresholdTracker.ReportThresholdViolation(counterName.ToLower(), e.ThresholdResult);
}

    /*
     * The loadtest heartbeat event, executed once per second.
     * This method will check to see if the ThresholdTracker object recommends
     * a load change and perform the change.
     */
    void LoadTest_Heartbeat(object sender, HeartbeatEventArgs e)
    {
      int newLoad;

      if (m_thresholdTracker.IsStarted)
      {
        switch (m_thresholdTracker.CurrentLoadDirection)
        {
          case ThresholdTracker.LoadDirection.None:
            break;
          case ThresholdTracker.LoadDirection.Decrease:
            foreach (LoadTestScenario scenario in m_loadTest.Scenarios)
            {
              newLoad = scenario.CurrentLoad - m_loadStep;
              scenario.CurrentLoad = (newLoad >= m_minLoad) ? newLoad : m_minLoad;
            }
            break;
          case ThresholdTracker.LoadDirection.Increase:
            foreach (LoadTestScenario scenario in m_loadTest.Scenarios)
            {
              newLoad = scenario.CurrentLoad + m_loadStep;
              scenario.CurrentLoad = (newLoad <= m_maxLoad) ? newLoad : m_maxLoad;
            }
            break;
        }
      }
    }

    /*
     * Helper method for grabbing values from the context data or assigning
     * a default value if needed.
     * Returns true if the value was found, false if the default value was used.
     */
    private bool AssignContextValueIfAvailable(string contextKeyName, ref int target, int defaultValue)
    {
      if (m_loadTest.Context.ContainsKey(contextKeyName))
      {
        target = Convert.ToInt32(m_loadTest.Context[contextKeyName]);
        return true;
      }
      target = defaultValue;
      return false;
    }
}

public class ThresholdTracker
{
    public enum LoadDirection
    {
      Increase = 0,
      Decrease = 1,
      None = 2,
    }

    private int m_intervalLength;
    private ArrayList m_counterList;
    private DateTime m_nextUpdateTime;
    private LoadDirection m_loadDirection;

    public ThresholdTracker()
    {
      m_intervalLength = -1;
      m_counterList = new ArrayList();
      m_loadDirection = LoadDirection.None;
    }

    /*
     * Method for adding new performance counters into the tracking array.
     * Note that this does not guarantee that counter thresholds have been set,
     * only that that the perfcounter will be tracked if they are.
     */
    public bool AddTrackedPerformanceCounter(string perfCounter)
    {
      if(null != perfCounter && !m_counterList.Contains(perfCounter))
      {
        m_counterList.Add(perfCounter);
        return true;
      }
      return false;
    }

    /*
     * Once a perfcounter has hit a threshold (be it critical or warning),
     * we track it here to determine if load should be affected or not.
     */
    public bool ReportThresholdViolation(string counter, ThresholdRuleResult thresholdResult)
    {
      if (IsStarted && m_counterList.Contains(counter))
      {
        switch (thresholdResult)
        {
          case ThresholdRuleResult.OK:
            // Do nothing in this case, as we default to increasing load anyway.
            break;

          case ThresholdRuleResult.Warning:
            /*
             * Comment this section out if you want to bounce off of the upper end
             * of your highest threshold, only decreasing load when you hit your
             * critical edge. (This will allow for more perfcounters to sit
             * between their warning and critical ranges, but will cause load to
             * be changed at every interval)
             * Leaving this section uncommented will keep your load at the low end
             * of your highest threshold (In other words, you'll stop increasing
             * load as soon as your highest tracked perfcounter hits its warning
             * threshold)
             */
            if (LoadDirection.Increase == m_loadDirection)
            {
              m_loadDirection = LoadDirection.None;
            }
            break;
          case ThresholdRuleResult.Critical:
            // Nothing to do here but decrease the load if we've violated a
            // critical threshold.
            m_loadDirection = LoadDirection.Decrease;
            break;
        }
        // The perfcounter was being tracked.
        return true;
      }
      // Perfcounter wasn't tracked anyway.
      return false;
    }

    /*
     * A property which provides a recommendation on whether to increase
     * or decrease the current load on the server in order to hit the
     * target perfcounter thresholds
     */
    public LoadDirection CurrentLoadDirection
    {
      get
      {
        if (!IsStarted)
        {
          throw new ApplicationException("Threshold tracker has not been started");
        }
        // No load changes if the interval time hasn't passed.
        if (m_nextUpdateTime > DateTime.Now)
        {
          return LoadDirection.None;
        }

        // The interval time has passed, let's change the load (if necessary)
        m_nextUpdateTime = DateTime.Now.AddSeconds(m_intervalLength);

        // Grab a copy of the current value so we can return it.
        LoadDirection tempDirection = m_loadDirection;

        /*
         * Default to increasing load next time. (This will change if a
         * threshold is violated.)
         */
        m_loadDirection = LoadDirection.Increase;

        return tempDirection;
      }
    }

    /*
     * Readonly property for quick assessment on whether perfcounters are
     * being tracked or not.
     */
    public bool IsStarted
    {
      get
      {
        return (-1 != m_intervalLength);
      }
    }

    /*
     * Allows this object to start tracking perfcounter data at specified
     * intervals
     */
    public void StartTracking(int intervalLength)
    {
      //Use the default of 20 seconds if they fail to specify a valid interval.
      m_intervalLength = (intervalLength > 0) ? intervalLength : 20;
    }

    /*
     * Forces the class to stop keeping track of perfcounter threshold violations
     */
    public void StopTracking()
    {
      if (!IsStarted)
      {
        throw new ApplicationException("Threshold tracker has not been started");
      }
      m_intervalLength = -1;
    }
}
}

A new beginning.

2006-12-16T00:13:00Z

Hi all,

I recently joined the Visual Studio Team Test group as a Software Design Engineer in Test. After spending the last 9 years working on various portions of Office, the developer organization is a breath of fresh air.

From time to time I will update this blog with neat little tidbits that I've picked up or perhaps code samples for plug-ins that I've written and think that people might find useful. If you have questions regarding my blog posts or the content therein, feel free to comment here directly and I'll get back to you. However, if you have a general question regarding VSTS the best place to go for an answer is the VSTS forum page, located here: http://forums.microsoft.com/msdn/showforum.aspx?forumid=19.

--Mike