Testing Spot

Help! Visual Studio is not displaying real-time graphs during the execution of a load test.

2013-04-29T22:18:06Z

Here is a very peculiar, interesting, and obscure issue/behavior in Visual Studio that you will rarely find. But if you do, this post will save you time!

Symptoms

When running a Load Test in Visual Studio the real time graphs do not display any information (see image below)
Changing the graph (e.g. from Key Indicators to System Under Test) does not solve the problem
The tables and other parts of the UI do show data
When loading the results after the tests finished, the graphs show the correct data

Root cause of the problem

The time in the Visual Studio machine was incorrect.

Solution

1. Ensure that both machines have the “Automatically adjust for clock Daylight Saving Time” in the Time Zone either checked or unchecked.

2. Select the correct time zone and time for both machines

3. Restart Visual Studio and connect to the test run

Important: Even though the display time may be the same, ensure that the actual time (difference from UTC) is correct. Keep reading for the details.

Details

In my case the Visual Studio client machine was not automatically adjusting for daylight savings. At one moment I noticed that the time in the controller and the client was not the same, I changed the time in the client to match the controller and it still did not work *&#*^$# (because even though the “displayed time” was the same, the difference from UTC time was different). A few hours later a coworker and I restarted the machine and noticed that the time was out of synch again. That is when we decided to look at the daylight saving option, checked the option, synched the time, and solved the problem!

A List of Tool and Sites We Like To Use

2013-02-21T15:13:00Z

One question the team gets asked a lot is "What tools/sites do you use to get things done in the labs?" Well, the answer varies from person to person, and I know there are a lot of lists available, but I figured one more list is not a bad thing. This is by no means an exhaustive list, but it is one that we can quickly give to customers. Maybe you will find some useful information here yourself.

I'd also like to point out that we often get asked for "that one tool that will just solve my issues" Regrettably, you will not find that tool in this list, because... I haven't written it yet. Nor has anyone else. In our world, we need to apply a full toolset and a lot of work. We use tools to make the work easier, but do not expect them to solve everything. There's always one more tool that could be useful. That is why these lists are so important to us.

Perf Tools
• PerfView: performance-analysis tool that helps isolate CPU and memory-related performance issues http://www.microsoft.com/en-us/download/details.aspx?id=28567
• Performance Analysis of Logs: analyzes perfmon logs and generates reports on issues http://pal.codeplex.com/
• SysInternals: All kinds of system utilities can be found here http://technet.microsoft.com/en-US/sysinternals
• Debugging Tools for Windows: The core debug toolset from Microsoft for debugging outside of Visual Studio http://msdn.microsoft.com/en-us/windows/hardware/gg463009.aspx
• Psscor4: Managed-Code Debugging Extension for WinDbg http://www.microsoft.com/en-us/download/details.aspx?id=21255
• Debug Diagnostic Tool: Tool for automatically generating and analyzing dumps http://www.microsoft.com/en-us/download/details.aspx?id=26798
• CLR Profiler for .NET Framework 4 - http://www.microsoft.com/en-us/download/details.aspx?id=16273
• NP Profiler: NP .NET Profiler - http://www.microsoft.com/en-us/download/details.aspx?id=35370
• Visual Round Trip Analyzer: Analyzes the entire web page trip for perf items http://www.microsoft.com/en-us/download/details.aspx?id=21462

Utilities
• Log Parser 2.2: Powerful and fast parsing of all kinds of logs http://www.microsoft.com/en-us/download/details.aspx?id=24659
• LinqPad: http://www.linqpad.net/
• ReguLazy: regex syntax helper http://osherove.com/storage/tools/Regulazy.103.zip - main site: http://osherove.com/tools/
• Notepad++: Free text editor with syntax highlighting, plugins, etc. very fast and powerful http://notepad-plus-plus.org/
• BareTail: real-time file viewer with syntax highlighting http://baremetalsoft.com/baretail/
• Text Crawler: multi-file search/replace/extract/process/etc. http://www.digitalvolcano.co.uk/content/textcrawler

Docs
• Visual Studio Performance Testing Quick Reference Guide: http://vsptqrg.codeplex.com/
• Improving .NET Application Performance and Scalability: http://www.microsoft.com/en-us/download/details.aspx?id=11711
• WCF Service Trace Viewer for Troubleshooting: http://msdn.microsoft.com/en-us/library/aa751795.aspx
• Biztalk Server Performance Optimization Guide: http://msdn.microsoft.com/en-us/library/cc558617(BTS.10).aspx

Lists
• Scott Hanselman’s Ultimate tool list: http://www.hanselman.com/blog/ScottHanselmans2011UltimateDeveloperAndPowerUsersToolListForWindows.aspx
• SoftwareQaTest list of tools (very comprehensive): http://www.softwareqatest.com/qatweb1.html?keepThis=true&TB_iframe=true&height=500&width=750

Blogs
• Load Test Analyzer Overview (very in-depth about Visual Studio results) http://msdn.microsoft.com/en-us/library/ms404677.aspx• Mark Russinovich's Blog - http://blogs.technet.com/b/markrussinovich/
• Clint Huffman's Windows Performance Analysis Blog: http://blogs.technet.com/b/clinth/
• Mike Lagase's Exchange Performance Analysis Blog: http://blogs.technet.com/mikelag/archive/2008/08/20/performance-troubleshooting-using-the-pal-tool.aspx
• Get a Handle on Windows Performance Analysis: http://windowsitpro.com/Windows/Articles/ArticleID/101162/pg/2/2.html
• If broken it is, fix it you should: Very good in depth debug training and info http://blogs.msdn.com/b/tess/
• Write Faster Code with VS 2010 Profiler: http://blogs.msdn.com/b/profiler/archive/2009/06/10/write-faster-code-with-vs-2010-profiler.aspx

Channel 9
• Great Site for all kinds of programming and debugging videos: http://channel9.msdn.com/
• http://channel9.msdn.com/search?term=testing
• http://channel9.msdn.com/Series/-NET-Debugging-Stater-Kit-for-the-Production-Environment

UAG Service Pack 3

2013-02-20T18:53:00Z

If you haven’t heard by now, Service Pack 3 for Unified Access Gateway (UAG) was released yesterday. In addition to many fixes, this eagerly awaited service pack includes:

Added Support for new clients

Windows 8 and Windows RT
Forefront UAG 2010 SP3 supports Windows 8 and Windows RT client computers using Internet Explorer 10 and Internet Explorer 10 on the desktop. In addition, SP3 also supports the following client applications running on Windows 8 and Windows RT:

Mail: the built-in Mail application on Windows 8 / Windows RT computers can be used to connect to Exchange servers published through UAG 2010 SP
Remote Desktop: UAG 2010 SP3 adds support for the Remote Desktop Connection (RDC) 8.0 client that is running on Windows 8

Windows Phone 8
SP3 for UAG 2010 adds support for mobile phone devices running the Windows Phone 8 Operating System

Added client applications such as Office 2013 and Remote Desktop Client 8.0 (Win 7 SP1)
Application publishing of SharePoint 2013 and Exchange 2013

We’ve been testing this in the lab for a while now and our favorite is the Windows 8 support (especially since we’re all using Windows 8).

So if you’re running UAG, you’ll definitely want to get Service Pack 3.

You can also find out more information on what’s new.

Newest Servers in the Labs

2013-02-16T14:22:32Z

Our team, Testing Services and Labs, help customers test, analyze and remediate IT solutions. Not only do we do this by providing some very talented and experienced consultants, who can work with you onsite, but we also have three, great lab facilities across the U.S.

Located in Charlotte (NC), Dallas (TX) and Issaquah (WA), these facilities have some of the latest and greatest hardware for real-world testing. This allows you to perform testing on enterprise-class hardware that is dedicated to you during the testing (not a shared environment), where you have complete control (even console access) to the bare metal. We help customers do all sorts of testing such as application scalability testing (something we’ve been doing for a very long time), migration testing (focusing on the backend), disaster recovery testing and more. We’ve even helped customers spin up reasonable facsimiles of private cloud solutions, O365 SharePoint, CRM and the like.

Staffed with top notch infrastructure consultants, these facilities were designed from the ground up specifically for testing purposes. Not only do the facilities themselves reflect this, but the processes and procedures our infrastructure consultants use are designed (and continually refined) to make your testing happen smoothly and efficiently. For example, think back of the last time you had to make a change to a testing environment that you’ve worked in (add an additional web server, make a storage configuration change, or add a server to a SAN fabric). Was it difficult? Did it happen quickly? These are all examples of change that can happen in minutes at our facilities (not days).

Since we attempt to help customers test proactively and forward looking, it is important for us to keep our inventory fairly current and up-to-date. With that said, we just received our newest shipment of equipment recently. Have a gander at the group of servers we just racked:

Dell PowerEdge R910
4 processors, 10 cores each, total: 40 cores
8GB dimms, 8 dimms per memory module, 8 memory modules, total memory: 512GB
Local storage: 16 drives @ 300GB, 15k RPM
Multiple connectivity options including 10 GB Ethernet and 8 GB HBA for SAN connectivity

How to automatically run a load test as part of a build

2013-01-22T22:33:00Z

Prerequisites

· A machine (VM or physical) to host the test rig components: test controller and agent (from now on this machine will be referred as CONTROLLER).

· A SQL database to hold the load test results (from now on this machine will be referred as SQL).

· An account with admin rights in CONTROLLER (from now this account will be referred as the CONTROLLER ACCOUNT).

· A visual studio solution with a load test.

· The installer for the visual studio agents (this includes installer for the Test Controller and the Test Agent)

Configuration of the test rig

Detailed information on how to do this can be found in this MSDN article. The following steps are a walkthrough summary to configure a one machine rig (test controller and test agent on the same machine):

1. Install the test controller service on the CONTROLLER machine

2. Configure the controller service to running load tests

a. Open Test Controller configuration tool
b. Enter the CONTROLLER ACCOUNT username and password
c. In the “Load Testing” section, select the “Configure test controller for load testing” checkmark
d. Enter the name of the SQL machine
e. Click on apply settings.

3. Install the test agent service on the CONTROLLER machine

4. Configure the test agent to run load tests, register the agent with the controller

a. Open Test Agent configuration tool

b. Enter the CONTROLLER ACCOUNT user name and password

c. Select the “Register with Test Controller” checkmark

d. Enter the name of CONTROLLER machine followed by a colon and the port of the controller service (default of 6901).

Configuration of Build machine

5. Install Visual Studio Ultimate on the build machine.

6. Ensure the Build service identity has administrator rights in the CONTROLLER machine

Configuration of Visual Studio solution

7. Open the solution that contains the load test

8. Right click at the solution level and select “Add-> new item”. Add a new test settings file

9. Open the test setting file, go to the “Roles” section (left hand side) and in the “Test Execution Method” select “Remote Execution.

10. In the “Controller” text box type the name of the CONTROLLER machine

11. Apply changes, exit, and check in the solution changes.

Configuration of Build Definition

12. Open the build definition, and navigate to the “Process” section

13. In the "Build Process Parameters” expand the “Basic” step, expand “Automated Tests”,

14. Open the “Test Assembly” configuration dialog

a. Select “MSTest.exe Runner” as the Test Runner

b. In the “Test assembly file specification” text box enter the name and location of the load test to execute relative to the drop location (e.g. if your load test is not nested in a folder, then you would only type <nameofloadtest>.loadtest, if the test is nested in a folder then you would type <foldername>\<loadtestname>.loadtest)

i. Note: Keep in mind that, unlike Unit tests and Coded UI tests, load test are not contained within an assembly. Therefore you should NOT select the output assembly of the test project in this field. Instead you need to point the runner to the .loadtest file

c. In the “Test Settings file” section click on the browse button and select the test settings file that you created in the previous section (if you cannot find it, make sure you checked in the file).

15. Save the build

Now you are ready to run the build and have the load test automatically executed.

Naturally occurring concurrency

2012-12-17T22:25:00Z

I needed to predict the level of concurrency that will naturally occur given a level of traffic and a time frame. For example:

A test case takes 3 minutes to run (including think time). 20 iterations need to run in 1 hour but be random, just as traffic to a web site is. Straight math says EXACTLY 20 cases of 3 minutes each can execute consecutively with zero concurrency. However we intuitively know that such a distribution is extremely unlikely. Given random starting times, there will certainly be SOME amount of concurrency but how much?

Here’s the model that I came up with and I welcome any feedback.

Since there are (20) 3 minute time slots in 60 minutes, randomly draw a number between 1 and 20 for each iteration. Count the results by slot. Any slot that has more than one test = concurrency.

I plugged this into Excel and looked at sample sizes of up to 3000 and the answer is ~27% of the time.

See below for a small sample.

Randomized slot for a test iteration to start																					Tests Per Slot																					27%	38%	36%
Slots	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20		1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	Multiple	Single	None
20	20	19	7	16	11	5	14	5	19	15	8	15	1	10	13	13	6	18	9	9		1	0	0	0	2	1	1	1	2	1	1	0	2	1	2	1	0	1	2	1	25%	50%	25%
	9	14	5	9	12	17	11	10	2	7	1	11	18	11	19	18	6	13	18	20		1	1	0	0	1	1	1	0	2	1	3	1	1	1	0	0	1	3	1	1	15%	60%	25%
	11	12	5	4	14	20	13	10	11	6	12	5	7	19	3	16	4	3	5	16		0	0	2	2	3	1	1	0	0	1	2	2	1	1	0	2	0	0	1	1	30%	35%	35%
	15	20	2	18	13	9	16	9	11	13	19	12	7	18	3	20	19	10	20	16		0	1	1	0	0	0	1	0	2	1	1	1	2	0	1	2	0	2	2	3	30%	35%	35%
	7	10	8	12	18	11	14	4	18	4	17	7	15	3	5	6	4	4	12	15		0	0	1	4	1	1	2	1	0	1	1	2	0	1	2	0	1	2	0	0	25%	40%	35%
	12	11	20	20	10	1	11	18	10	7	15	9	14	20	18	15	2	20	15	11		1	1	0	0	0	0	1	0	1	2	3	1	0	1	3	0	0	2	0	4	25%	30%	45%
	14	12	4	5	8	16	20	19	13	7	16	16	6	10	12	14	7	15	17	10		0	0	0	1	1	1	2	1	0	2	0	2	1	2	1	3	1	0	1	1	25%	45%	30%
	14	17	11	5	15	14	7	18	8	12	2	3	7	7	2	13	5	7	7	14		0	2	1	0	2	0	5	1	0	0	1	1	1	3	1	0	1	1	0	0	20%	40%	40%

I'm not a mathematician and this may not hold true under different circumstances but it was a useful way of illustrating the concept with my client.

Submit bugs / feedback -- Part 2

2012-12-17T18:48:19Z

This is the second in a series discussing ways to send information to the appropriate Microsoft product team.

This post will illustrate how to use an add-in to allow submissions from within Visual Studio.

Within Visual Studio

Click Report a Bug & you get the Connect page. It directs you to an add-in that you can install.

Download and install it.

After you do, you’ll get this screen.

Once you login with your Connect account, a wizard will start.

Follow along and submit screen shots, repro files, etc.

Submitting bugs / feedback -- Part 1

2012-12-17T18:26:00Z

Connect is Microsoft’s external bug reporting system. This series will demonstrate two different ways to submit information.

The first is via the web site. The second is within Visual Studio 2012 and requires an add-in to be installed.

Using the Connect website

Go here and sign up. You have to use a Live ID (now called a Microsoft ID; I use a personal Hotmail account).

http://connect.microsoft.com/intro/

Once Connect has been activated for that account, you get a dashboard when you log in:

Go to the home page to see the products that you can submit bugs/feedback against:

Use Search to get to the right product of pick it from the list.

Select the Visual Studio area.

Click the appropriate button.

Load test virtual user licenses for Visual Studio 2012

2012-12-17T17:46:00Z

This isn't new news but I hope will server as a reminder as test teams convert their Visual Studio 2010 testing infrastructure over to 2012: you no longer need virtual user licenses for load testing!

http://msdn.microsoft.com/en-us/library/dd293540.aspx

Controlling the Outcome of a Request and the Web Test

2012-09-25T15:54:00Z

Controlling the Outcome of a Test Request and the Web Test

It is annoying when a “404 – Not Found” response comes back from a request to retrieve an image, .css, .js or some other non-critical file. There’s no harm done to the test (aside from those bytes not transferring), but it flags that request as a failure and marks the whole test as a failure. Further, if the web test that was executing has the property “Stop on error” set to true, the rest of the requests will not execute at all. This will lead to test results that do not really represent the actual behavior of the web site being tested.

While the missing file(s) should be added to the site or fixed in the application’s code, there are times when this is not easy to do. For those cases, I’ve created some sample code that allows you to change the way Visual Studio handles and reports errors. You can override individual request results and/or set the PASS/FAIL outcome for the whole test.

Introduction

Changing the outcome of a test request should be done with caution. You should try to target the response that you want to override as specifically as possible. For example, use a plugin with logic to only override 404 responses instead of changing the outcome no matter what response code is returned. This way all other response codes, some of which represent valid errors, will be treated normally. Even rules that only trigger on a particular response code cannot distinguish between actual failures -- such as a 404 because the web server no longer is able to find the requested page -- and the invalid ones that you want to suppress like a 404 due to a missing image file.

NOTE – while overriding a request’s outcome property results in the desired outcome being displayed, Visual Studio has still recorded a failure and will set the outcome of the test to FAIL unless it, too, is explicitly set.

Not All Requests Are Recorded

It is important to note that the only outcome that can be evaluated is that of the request to which the plugin is attached. The screenshot below shows a simple test with one request to a SharePoint site. That request has the “Exclude 404 – Not Found” request plugin added to it.

Any calls embedded in that HTML page that were not recorded by Visual Studio (are not listed in the web test) cannot be examined. Running the above test yields an example.

This illustrates two important concepts:

The response code for the request issues by the test is “200 OK” even though an associated request failed.
None of the files listed below the request appear in the web test because they were not picked up by the recorder. Since they don’t exist in the test, you can’t place a plugin on the officeLogo.png call to trap the 404 response.

The result is that a rule looking for a “404 – Not Found” response will fail to find one because the status that will be examined will be “200 – OK”. In this situation, you cannot use a narrowly targeted rule but instead must set the outcome without using any other logic. Another option would be to re-record the test using a web proxy like Fiddler which should list every call.

Changing the Outcome of a Test Request

WARNING -- changing the outcome of the test should only be done when you are certain that all reported errors are not relevant! Without this confirmation, legitimate request failures may not be noticed.

All examples were created as part of a common name space with the following references as shown below. This code will not be duplicated for each method.

usingSystem;

using Microsoft.VisualStudio.TestTools.WebTesting;

using System.ComponentModel;

// Assembly marked as compliant. (http://msdn.microsoft.com/en-us/library/bhc3fa7f.aspx)

[assembly: CLSCompliant(true)]

namespace SkipErrors

{

…

}

Example 1 – Bare Minimum

The key operational code for all of the following plugins is “e.Request.Outcome = “ true or false. Thus a request plugin could be as simple as this, which sets the outcome to true.

public class SetOutcome : WebTestRequestPlugin

{

public override void PostRequest(object sender, PostRequestEventArgs e)

{

e.Request.Outcome = true;

}

To make the plugin more user-friendly, fields for selecting the outcome and adding comments to the test result have been added. Likewise the Description, DisplayName, and DefaultValue attributes provide helpful information to the user. Finally, the comment and outcome assignment code has been added to another class and method so that it can be called by other plugins.

Example 2 – Designating an Outcome

The user can directly set the outcome of a request regardless of the value returned by the test engine.

[Description("Directly set the outcome of a REQUEST. Use the test-level plug-in to set the TEST result.")]

[DisplayName("Set Request Outcome")]

public class SetOutcomeDirectly : WebTestRequestPlugin

{

[Description("Choose the desired outcome. This will override the test-generated value.")]

[DisplayName("Desired Outcome")]

[DefaultValue(Outcome.Pass)]

public Outcome DesiredOutcome { get; set; }

/// Auto field generation( "public string Comment { get; set; }" ) can't be used because Comment cannot be passed by reference as AddMsgAndSetOurcome requires.

string _Comment = string.Empty;

[Description("The text that will be included in the test result.")]

[DisplayName("Comment")]

[DefaultValue("Response failed. Outcome set to Pass.")]

public string Comment

{

get { return _Comment; }

set { _Comment = value; }

}

public override void PostRequest(object sender, PostRequestEventArgs e)

{

CoreCode.AddMsgAndSetOutcome(e, ref _Comment, DesiredOutcome);

}

public static class CoreCode

{

public static void AddMsgAndSetOutcome(PostRequestEventArgs e, ref string Msg, Outcome Result = Outcome.Pass)

{

if (string.IsNullOrEmpty(Msg) == false)

{

e.WebTest.AddCommentToResult(Msg);

}

e.Request.Outcome = Result;

}

Example 3 – Overriding a 404

Building on the previous example, this plugin changes the outcome of any request that receives a “404 – Not Found” response from FAIL to PASS and adds a comment to the test result indicating this occurred.

[Description("Disallow '404 - Not Found' response from being reported as an error and set the outcome of the request to Pass. Use the test-level plug-in to set the test result.")]

[DisplayName("Exclude 404 - Not Found")]

public class FilterNotFound_RPI : WebTestRequestPlugin

{

[Description("The text that will be included in the test result.")]

[DisplayName("Comment")]

[DefaultValue("404 found. Outcome set to Pass.")]

public string Comment404 { get; set; }

public override void PostRequest(object sender, PostRequestEventArgs e)

{

CoreCode.NotFound(e, Comment404);

}

(In the Core Code class)

public static bool NotFound(PostRequestEventArgs e, string Msg, Outcome Result = Outcome.Pass)

{

if (e.Response.StatusCode == System.Net.HttpStatusCode.NotFound)

{

AddMsgAndSetOutcome(e, ref Msg, Result);

return true;

}

else

{

return false;

}

Example 4 – Trapping by Response Code

In this example, the user can specify the HTTP status code to look for:

[Description("This plug-in allows any HTTP status code to be trapped and the request's outcome set to Pass. Status codes can be found at http://msdn.microsoft.com/en-us/library/aa383887.aspx.")]

[DisplayName("Filter Response Status Code")]

public class FilterAnyResponse_RPI : WebTestRequestPlugin

{

[Description("Choose the HTTP status code to match. If found, the request's outcome is overridden to Pass. 0 = Disable HTTP code status matching.")]

[DisplayName("Response Code")]

public int StatusCode { get; set; }

[Description("The text that will be included in the test result.")]

[DisplayName("Comment")]

[DefaultValue("Code matched. Outcome set to Pass.")]

public string CommentHTTPCode { get; set; }

public override void PostRequest(object sender, PostRequestEventArgs e)

{

CoreCode.ResponseTypeMatch(e, StatusCode, StatusCode.ToString() + "-" + CommentHTTPCode);

}

(In the Core Code class)

public static bool ResponseTypeMatch(PostRequestEventArgs e, int sc, string Msg, Outcome Result = Outcome.Pass)

{

/// A return value of false let's the caller skip unnecessary operations since there was no match.

if (e.Response.StatusCode == System.Net.HttpStatusCode.NotFound)

{

AddMsgAndSetOutcome(e, ref Msg, Result);

return true;

}

else

{

return false;

}

Changing the Test Outcome

As mentioned above, only the outcome of the request was changed, not the outcome of the test. Since the test will still reflect the fact there was a failure, its result needs to be changed as well. Here’s how to do that:

[Description("This plug-in allows for the final test outcome value to be chosen. This is helpful when the reported 'errors' are known and accepted conditions and do not reflect the true status of the whole test.")]

[DisplayName("Set Test Outcome Value")]

public class OverrideTestResult : WebTestPlugin

{

[Description("Choose the desired outcome. This will override the test-generated value.")]

[DisplayName("Desired Outcome")]

public Outcome DesiredOutcome { get; set; }

[Description("The text that will be included in the test result.")]

[DisplayName("Comment")]

[DefaultValue("Tester choose the final test outcome.")]

public string Comment { get; set; }

public override void PreWebTest(object sender, PreWebTestEventArgs e)

{

/// Adds a comment to the top of the test. Generally this is to inform the reader that the results of some requests or the test has been modified.

/// For example: "Tester has overridden items marked as errors and final test outcome is PASS."

if (string.IsNullOrEmpty(Comment) == false)

{

e.WebTest.AddCommentToResult(Comment);

}

public override void PostWebTest(object sender, PostWebTestEventArgs e)

{

e.WebTest.Outcome = DesiredOutcome;

}