Controlling the Outcome of a Test Request and the Web Test

It is annoying when a “404 – Not Found” response comes back from a request to retrieve an image, .css, .js or some other non-critical file. There’s no harm done to the test (aside from those bytes not transferring), but it flags that request as a failure and marks the whole test as a failure. Further, if the web test that was executing has the property “Stop on error” set to true, the rest of the requests will not execute at all.  This will lead to test results that do not really represent the actual behavior of the web site being tested.

While the missing file(s) should be added to the site or fixed in the application’s code, there are times when this is not easy to do. For those cases, I’ve created some sample code that allows you to change the way Visual Studio handles and reports errors. You can override individual request results and/or set the PASS/FAIL outcome for the whole test.


Introduction


Changing the outcome of a test request should be done with caution. You should try to target the response that you want to override as specifically as possible. For example, use a plugin with logic to only override 404 responses instead of changing the outcome no matter what response code is returned. This way all other response codes, some of which represent valid errors, will be treated normally. Even rules that only trigger on a particular response code cannot distinguish between actual failures -- such as a 404 because the web server no longer is able to find the requested page -- and the invalid ones that you want to suppress like a 404 due to a missing image file.


NOTE – while overriding a request’s outcome property results in the desired outcome being displayed, Visual Studio has still recorded a failure and will set the outcome of the test to FAIL unless it, too, is explicitly set.


Not All Requests Are Recorded


It is important to note that the only outcome that can be evaluated is that of the request to which the plugin is attached. The screenshot below shows a simple test with one request to a SharePoint site. That request has the “Exclude 404 – Not Found” request plugin added to it.

 

Any calls embedded in that HTML page that were not recorded by Visual Studio (are not listed in the web test) cannot be examined. Running the above test yields an example.

 

This illustrates two important concepts:

  • The response code for the request issues by the test is “200 OK” even though an associated request failed.
  • None of the files listed below the request appear in the web test because they were not picked up by the recorder. Since they don’t exist in the test, you can’t place a plugin on the officeLogo.png call to trap the 404 response.

The result is that a rule looking for a “404 – Not Found” response will fail to find one because the status that will be examined will be “200 – OK”. In this situation, you cannot use a narrowly targeted rule but instead must set the outcome without using any other logic. Another option would be to re-record the test using a web proxy like Fiddler which should list every call.


Changing the Outcome of a Test Request


WARNING -- changing the outcome of the test should only be done when you are certain that all reported errors are not relevant! Without this confirmation, legitimate request failures may not be noticed.

 

All examples were created as part of a common name space with the following references as shown below. This code will not be duplicated for each method.

usingSystem;

using Microsoft.VisualStudio.TestTools.WebTesting;

using System.ComponentModel;

 

// Assembly marked as compliant. (http://msdn.microsoft.com/en-us/library/bhc3fa7f.aspx)

[assembly: CLSCompliant(true)]

namespace SkipErrors

{

}

Example 1 – Bare Minimum

The key operational code for all of the following plugins is “e.Request.Outcome = “ true or false. Thus a request plugin could be as simple as this, which sets the outcome to true.

 

public class SetOutcome : WebTestRequestPlugin

{

    public override void PostRequest(object sender, PostRequestEventArgs e)

    {

            e.Request.Outcome = true;

    }

}

 

To make the plugin more user-friendly, fields for selecting the outcome and adding comments to the test result have been added. Likewise the Description, DisplayName, and DefaultValue attributes provide helpful information to the user. Finally, the comment and outcome assignment code has been added to another class and method so that it can be called by other plugins.

Example 2 – Designating an Outcome

The user can directly set the outcome of a request regardless of the value returned by the test engine.

 

[Description("Directly set the outcome of a REQUEST. Use the test-level plug-in to set the TEST result.")]

[DisplayName("Set Request Outcome")]

public class SetOutcomeDirectly : WebTestRequestPlugin

{

    [Description("Choose the desired outcome. This will override the test-generated value.")]

    [DisplayName("Desired Outcome")]

    [DefaultValue(Outcome.Pass)]

    public Outcome DesiredOutcome { get; set; }

 

    /// Auto field generation( "public string Comment { get; set; }" ) can't be used because Comment cannot be passed by reference as AddMsgAndSetOurcome requires.

    string _Comment = string.Empty;

 

    [Description("The text that will be included in the test result.")]

    [DisplayName("Comment")]

    [DefaultValue("Response failed. Outcome set to Pass.")]

    public string Comment

    {

        get { return _Comment; }

        set { _Comment = value; }

    }

 

    public override void PostRequest(object sender, PostRequestEventArgs e)

    {

        CoreCode.AddMsgAndSetOutcome(e, ref _Comment, DesiredOutcome);

    }

}

 

public static class CoreCode

{

    public static void AddMsgAndSetOutcome(PostRequestEventArgs e, ref string Msg, Outcome Result = Outcome.Pass)

    {

        if (string.IsNullOrEmpty(Msg) == false)

        {

            e.WebTest.AddCommentToResult(Msg);

        }

        e.Request.Outcome = Result;

    }

}

Example 3 – Overriding a 404

Building on the previous example, this plugin changes the outcome of any request that receives a “404 – Not Found” response from FAIL to PASS and adds a comment to the test result indicating this occurred.

 

[Description("Disallow '404 - Not Found' response from being reported as an error and set the outcome of the request to Pass. Use the test-level plug-in to set the test result.")]

[DisplayName("Exclude 404 - Not Found")]

public class FilterNotFound_RPI : WebTestRequestPlugin

{

    [Description("The text that will be included in the test result.")]

    [DisplayName("Comment")]

    [DefaultValue("404 found. Outcome set to Pass.")]

    public string Comment404 { get; set; }

    public override void PostRequest(object sender, PostRequestEventArgs e)

    {

        CoreCode.NotFound(e, Comment404);

    }

}

(In the Core Code class)

public static bool NotFound(PostRequestEventArgs e, string Msg, Outcome Result = Outcome.Pass)

{

    if (e.Response.StatusCode == System.Net.HttpStatusCode.NotFound)

    {

        AddMsgAndSetOutcome(e, ref Msg, Result);

        return true;

    }

    else

    {

        return false;

    }

}

Example 4 – Trapping by Response Code

In this example, the user can specify the HTTP status code to look for:

 

[Description("This plug-in allows any HTTP status code to be trapped and the request's outcome set to Pass. Status codes can be found at http://msdn.microsoft.com/en-us/library/aa383887.aspx.")]

[DisplayName("Filter Response Status Code")]

public class FilterAnyResponse_RPI : WebTestRequestPlugin

{

    [Description("Choose the HTTP status code to match. If found, the request's outcome is overridden to Pass. 0 = Disable HTTP code status matching.")]

    [DisplayName("Response Code")]

    public int StatusCode { get; set; }

    [Description("The text that will be included in the test result.")]

    [DisplayName("Comment")]

    [DefaultValue("Code matched. Outcome set to Pass.")]

    public string CommentHTTPCode { get; set; }

    public override void PostRequest(object sender, PostRequestEventArgs e)

    {

        CoreCode.ResponseTypeMatch(e, StatusCode, StatusCode.ToString() + "-" + CommentHTTPCode);

    }

}

(In the Core Code class)

public static bool ResponseTypeMatch(PostRequestEventArgs e, int sc, string Msg, Outcome Result = Outcome.Pass)

{

    /// A return value of false let's the caller skip unnecessary operations since there was no match.

    if (e.Response.StatusCode == System.Net.HttpStatusCode.NotFound)

    {

        AddMsgAndSetOutcome(e, ref Msg, Result);

        return true;

    }

    else

    {

        return false;

    }

}

Changing the Test Outcome

As mentioned above, only the outcome of the request was changed, not the outcome of the test. Since the test will still reflect the fact there was a failure, its result needs to be changed as well. Here’s how to do that:

    [Description("This plug-in allows for the final test outcome value to be chosen. This is helpful when the reported 'errors' are known and accepted conditions and do not reflect the true status of the whole test.")]

    [DisplayName("Set Test Outcome Value")]

    public class OverrideTestResult : WebTestPlugin

    {

        [Description("Choose the desired outcome. This will override the test-generated value.")]

        [DisplayName("Desired Outcome")]

        public Outcome DesiredOutcome { get; set; }

        [Description("The text that will be included in the test result.")]

        [DisplayName("Comment")]

        [DefaultValue("Tester choose the final test outcome.")]

        public string Comment { get; set; }

        public override void PreWebTest(object sender, PreWebTestEventArgs e)

        {

            /// Adds a comment to the top of the test. Generally this is to inform the reader that the results of some requests or the test has been modified.

            /// For example: "Tester has overridden items marked as errors and final test outcome is PASS."

            if (string.IsNullOrEmpty(Comment) == false)

            {

                e.WebTest.AddCommentToResult(Comment);

            }

        }

        public override void PostWebTest(object sender, PostWebTestEventArgs e)

        {

            e.WebTest.Outcome = DesiredOutcome;

        }

    }