I haven't posted a lot in the last few weeks, and there's a succinct reason for that.  In fact, I can name it in three words:  Full Test Pass.

The concept is close to what it sounds like.  During the earlier stages of product planning, the test team for ASP.NET and the web development profile for Visual Studio used the time to plan its test coverage.  In total, we wrote descriptions of over 40,000 test cases that we wanted to run.  In the span of about four weeks ending earlier this month, we took the time to acutally execute nearly all of them, many for the first time.

Don't take that to mean that we whittled away all the hours between whiles drinking beverages and playing foosball -- we are professionals, after all.  We used the time to automate a large subset of our tests, including our acceptance tests (tests that we run on each daily build of the product to get a rough sense of its quality and catch really the big bugs).  Automation infrastructure -- the libraries and components our testcases use to simplify automation and manage the process of reporting results to our testcase management database (named Mad Dog, if you can believe it) -- also took up a lot of time.  

But 40,000 is a big number, and in that number were a lot of scenarios that we just hadn't tried yet.  For example, one of the features I own is Control Designers, which is the mechanism a custom control developer uses to control the appearance and behavior of her control in Visual Studio's web design environment.  As part of this, I test several new host services that are exposed by Visual Studio which the authors of Control Designers can use to simplify their work.  One of these is an interface with the name IWebFormsBuilderUIService, which exposes methods for creating common UI dialogs, among them a URL editor.  In Visual Basic .NET, the code a control author would need to write in order to use this would be something like this:

Imports System.ComponentModel
Imports System.ComponentModel.Design
Imports System.Web.UI.Design
Imports System.Web.UI
Imports System

<Designer(GetType(ExampleControlDesigner)), . . . > Public Class ExampleControl
    Inherits System.Web.UI.WebControls.WebControl

    Dim _url As String

    <Bindable(True), Category("Appearance"), DefaultValue("")> Property URL() As String
        Get
            Return _url
        End Get

        Set(ByVal Value As String)
            _url = Value
        End Set
    End Property

    . . .
End Class

Public Class ExampleControlDesigner
    Inherits ControlDesigner

    Protected Overrides Sub OnClick(ByVal e As DesignerRegionEventArgs)
        Dim iwfbuis as IWebFormsBuilderUIService
        Dim ctl As ExampleControl

        Mybase.OnClick(e)
        ctl = Component
        iwfbuis = GetService(GetType(IWebFormsBuilderUIService))
        try
            ctl.URL = iwfbuis.BuildUrl(Nothing, ctl.URL, RootDesigner.DocumentUrl, "Please Choose a URL", "", UrlBuilderOptions.None)
        catch ex as exception
            Msgbox(ex.Message)
        end try
        UpdateDesignTimeHtml()
    End Sub

    . . .
End Class

The above code would bring up Visual Studio's URL dialog when the user clicked on the control, and set the control's URL property to the user's selection.

I don't have any acceptance-level tests for IWebFormsBuilderUIService.BuildUrl, because even if this method were completely busted beyond usability, it wouldn't especially hurt the other testers on our team or the other teams that depend on ASP.NET.  However, I do have plenty of regular functional tests for the feature that I plan to automate, and the plan for this area looks like this:

Testcase Breakout for BuildUrl

Hierarchy: All Testcases -> Web.NET -> Page & Controls -> Page Framework -> Designers -> Control Designers Extensibility -> Host Services -> IWebFormsBuilderUIService

BuildUrl

ClayCo

Area Description Host-implemented method for bringing up its own URI-picking dialog.

BuildUrl01 - Call and (from Venus) cancel the dialog

Pri 2, Coding

1) Verify initialURL is returned.

BuildUrl02 - Call (from ControlDesigner) and enter (from Venus) an absolute URL

Pri 2, Coding

Abstract: All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios: 1) Use http://localhost/<currentapp>, verify relative conversion 2) Call same with UrlBuilderOptions set to None, verify no conversion 3) Use http://localhost/<currentapp>/, verify relative conversion 4) Use http://localhost/<currentapp>/Page2.aspx, verify relative conversion 5) Use https://localhost/AnotherApp/Page.aspx, verify relative conversion 6) Use ftp://ftp.is.co.za/rfc/rfc1808.txt 7) Use gopher://spinaltap.micro.umn.edu/00/Weather/California/Los%20Angeles 8) Use mailto:clayco@microsoft.com 9) Use news:comp.infosystems.www.servers.unix 10) Use telnet://melvyl.ucop.edu/ 11) Use http://localhost/a/./g.aspx, verify relative conversion 12) Call same with UrlBuilderOptions set to None, verify no conversion 13) Use http://localhost/a/b/../g.aspx, verify relative conversion 14) Call same with UrlBuilderOptions set to None, verify no conversion 15) Use http://localhost/test., verify relative conversion 16) Call same with UrlBuilderOptions set to None, verify no conversion 17) Use http://localhost/.aspx, verify relative conversion 18) Call same with UrlBuilderOptions set to None, verify no conversion 19) Use http://localhost/..aspx, verify relative conversion 20) Call same with UrlBuilderOptions set to None, verify no conversion 21) Use 256-character URI 22) Use string containing Arabic Letter Khah and U+4E2E (catch).

BuildUrl03 - Call (from ControlDesigner) and enter (from Venus) a relative URL

Pri 2, Coding

Abstract: All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios: 1) Use //localhost/<currentapp>/Page2.aspx, verify an app-relative URL is returned 2) Call same with UrlBuilderOptions set to None, verify absolute conversion 3) Use /<currentapp>/Page2.aspx, verify an app-relative URL is returned 4) Call same with UrlBuilderOptions set to None, verify absolute conversion 5) Use Page2.aspx 6) Call same with UrlBuilderOptions set to None, verify absolute conversion 7) Use //www.yahoo.com:80/, verify an absolute URL is returned 8) Use /./Page2.aspx, verify an app-relative URL is returned 9) Call same with UrlBuilderOptions set to None, verify absolute conversion 10) Use /../Page2.aspx 11) Call same with UrlBuilderOptions set to None, verify absolute conversion 12) Use http.aspx 13) Call same with UrlBuilderOptions set to None, verify absolute conversion 14) Use test. 15) Call same with UrlBuilderOptions set to None, verify absolute conversion 16) Use .aspx 17) Call same with UrlBuilderOptions set to None, verify absolute conversion 18) Use test.. 19) Call same with UrlBuilderOptions set to None, verify absolute conversion 20) Use ..aspx 21) Call same with UrlBuilderOptions set to None, verify absolute conversion 22) Use ./../test.aspx, verify an app-relative URL is returned 23) Call same with UrlBuilderOptions set to None, verify absolute conversion 24) Use ./SubDir/., verify an app-relative URL is returned 25) Call same with UrlBuilderOptions set to None, verify absolute conversion 26) Use SubDir/./Page2.aspx 27) Call same with UrlBuilderOptions set to None, verify absolute conversion 28) Use SubDir/../Page2.aspx 29) Call same with UrlBuilderOptions set to None, verify absolute conversion 30) Use Test;x=1/./Page2.aspx 31) Call same with UrlBuilderOptions set to None, verify absolute conversion 32) Use Test;x=1/../Page2.aspx 33) Call same with UrlBuilderOptions set to None, verify absolute conversion 34) Use ., verify an app-relative URL is returned 35) Call same with UrlBuilderOptions set to None, verify absolute conversion 36) Use ./, verify an app-relative URL is returned 37) Call same with UrlBuilderOptions set to None, verify absolute conversion 38) Use .. 39) Call same with UrlBuilderOptions set to None, verify absolute conversion 40) Use ../ 41) Call same with UrlBuilderOptions set to None, verify absolute conversion 42) Use ../Page2.aspx 43) Call same with UrlBuilderOptions set to None, verify absolute conversion 44) Use ../.. 45) Call same with UrlBuilderOptions set to None, verify absolute conversion 46) Use ../../ 47) Call same with UrlBuilderOptions set to None, verify absolute conversion 48) Use ../../Page2.aspx 49) Call same with UrlBuilderOptions set to None, verify absolute conversion 50) Use 256-character URI 51) Use string containing Arabic Letter Ghain and U+3A2E (lift up).

BuildUrl04 - InitialUrl paramerter scenarios

Pri 2, Coding

1) Use current web application folder's absolute URL, verify UI in Venus 2) Use current web application folder's relative URL, verify UI in Venus 3) Use current Page's URL, verify UI in Venus 4) Use image file URL, verify UI in Venus 5) Use absolute path to virtual sibling directory, verify UI in Venus 6) Use non-existent absolute path to sibling directory, verify UI in Venus 7) Use absolute path to virtual parent directory, verify UI in Venus 8) Use absolute path to virtual child directory, verify UI in Venus 9) Use relative path to virtual sibling directory, verify UI in Venus 10) Use relative path to virtual parent directory, verify UI in Venus 11) Use relative path to virtual child directory, verify UI in Venus 12) Use http://, ftp://, and file:// URLs to test machine, verify UI in Venus 13) Use URL to another machine, verify UI in Venus 14) Use http://www.microsoft.com/, verify error

BuildUrl05 - BaseUrl parameter scenarios

Pri 2, Coding

Abstract: All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios: 1) Use current web application folder's absolute URL, verify relative URL calculation 2) Use current web application folder's relative URL, verify relative URL calculation 3) Use current Page's URL, verify relative URL calculation 4) Use image file URL, verify relative URL calculation 5) Use absolute path to virtual sibling directory, verify relative URL calculation 6) Use absolute path to virtual parent directory, verify relative URL calculation 7) Use absolute path to virtual child directory, verify relative URL calculation 8) Use relative path to virtual sibling directory, verify relative URL calculation 9) Use relative path to virtual parent directory, verify relative URL calculation 10) Use relative path to virtual child directory, verify relative URL calculation 11) Use http://, ftp://, and file:// URLs to the same machine, verify absolute URLs are used 12) Use URL to another machine, verify relative and absolute paths on local and remote machines. 13) Use http://www.microsoft.com/, verify absolute paths get used. 14) Use String.Empty, verify absolute paths get used. 15) Use current web application folder's absolute URL when options is None, verify absolute URL 16) Use current web application folder's relative URL when options is None, verify absolute URL 17) Use current Page's URL when options is None, verify absolute URL 18) Use image file URL when options is None when options is None, verify absolute URL 19) Use absolute path to virtual sibling directory when options is None, verify absolute URL 20) Use absolute path to virtual parent directory when options is None, verify absolute URL 21) Use absolute path to virtual child directory when options is None, verify absolute URL 22) Use relative path to virtual sibling directory when options is None, verify absolute URL 23) Use relative path to virtual parent directory when options is None, verify absolute URL 24) Use relative path to virtual child directory when options is None, verify absolute URL

BuildUrl06 - Caption parameters

Pri 2, Coding

Abstract: Perform standard string tests and verify in Venus

Scenarios: 1) S1) Verify default property value 2) S2) Verify attribute value is not in HTML view 3) S3) Set to all numbers 4) S4) Set attribute to (nothing) in HTML view 5) S5) Set to 1 character 6) S6) Set attribute to space in HTML view 7) S7) Set attribute without quotes in HTML view 8) S8) Set property to DblQuote-space-DblQuote in Design view 9) S9) Set property to Quote-space-Quote in Design view 10) S10) set property to: DblQuote-Quote-&-Quote-DblQuote; verify 11) S11) set attribute to: 3DblQuote-'''abc'''-3DblQuote; verify 12) S12) set attribute to: DblQuote-@#$%-CrLf-GreaterThan-)(*^-DblQuote; verify 13) S13) Set property to "dadadada" in prop grid 14) S14) Set to long string 15) S15) Set to a random intl string and verify 16) S16) Set to an Intl string of problem characters 17) S17) Set to a random unicode string

BuildUrl07 - Filter parameter scenarios

Pri 2, Coding

Abstract: Use Standard String tests and verify expected value in Venus

Scenarios: 1) S1) Verify default property value 2) S2) Verify attribute value is not in HTML view 3) S3) Set to all numbers 4) S4) Set attribute to (nothing) in HTML view 5) S5) Set to 1 character 6) S6) Set attribute to space in HTML view 7) S7) Set attribute without quotes in HTML view 8) S8) Set property to DblQuote-space-DblQuote in Design view 9) S9) Set property to Quote-space-Quote in Design view 10) S10) set property to: DblQuote-Quote-&-Quote-DblQuote; verify 11) S11) set attribute to: 3DblQuote-'''abc'''-3DblQuote; verify 12) S12) set attribute to: DblQuote-@#$%-CrLf-GreaterThan-)(*^-DblQuote; verify 13) S13) Set property to "dadadada" in prop grid 14) S14) Set to long string 15) S15) Set to a random intl string and verify 16) S16) Set to an Intl string of problem characters 17) S17) Set to a random unicode string

---

Testcase Summary Info:

	Design:		Pending Scenario Review:
	Coding:	7	Pending Code Review:
	Complete:
			Total:	7

This is typical -- every feature area has a lot more tests defined for it than get will ever be run on any given build.  We rely on testers using the builds for ad-hoc testing and writing automation to find bugs, and rely on automated tests to catch regressions in later builds.  We also use full test passes like this to get a deeper sense of the product's quality at a few key intervals (such as earlier this month -- we wanted to catch isses in advance of the first large-scale Beta release of the product and get a sense of how much more work is needed to stabilize the product's quality before we ship).

And we did find bugs.  In the above test plan, the very first test case failed during the full test pass.  If a user clicked “Cancel” in the URL dialog, an empty string was returned by the method, rather than the initialURL parameter value.  This has already been fixed, by the way.

In response to my first post, Scott asked a very topical question:

Not sure if this falls in your basket or not, but I would be interest in any tips or tricks for memory profiling ASP.NET applications.

Well, Scott, I'm pleased that you're interested in this subject, because it's a very good way to introduce how we're testing the performance of ASP.NET under stress.  For a grounding on the subject, I encourage anyone interested in this subject to start by reading this.  It's an article written by Thomas Marquardt, a developer (and former tester) on our team who did a lot of work creating the test infrastructure we're using today.  It should also handily answer your question.

The other reason why this I want to point you to this article is that the source code for one of the test tools used for stress automation, QQQ.exe, can be downloaded there.  QQQ is a tool that monitors performance counters for the ASP.NET worker process and automatically attaches a debugger if any of a few different critical thresholds are reached.  We use this with the Web Application Stress Tool (which, in the long tradition of giving things names that will freak out everybody in marketing, is also called “Homer”) to simulate the process of multiple users making large quantities of requests to the server and catching the problems that arise.  These tools, combined with some debugging tools, scripts for automatically re-imaging the test machines and launching stress, and (naturally) the test pages themselves, are the way in which we stress-test the product.

The tests themselves cover a pretty broad swath of scenarios.  As you'd expect, we have tests simulating high traffic in a production environment with all managed code, but we also have tests for pages that inter-operate with COM components and call data methods.  We also check how ASP.NET handles some more esoteric scenarios.  What happens if  the user changes machine.config every few minutes, causing the application to restart?  What if there's a memory leak in the user's applications?  What if the worker process crashes?  The process model, of course, should gracefully recover from these hiccups.  In all of these scenarios, the pages we use for stress testing are also functional tests that we use for regular verification of the product, so we can get as close to simulating real-life workloads and traffic as possible.

I used to debug stress, so a lot of this comes from memory (it's like riding a bicycle), but special thanks go to (fellow tester) Vladimir Kritchko for pointing me to the latest stress infrastructure documentation.