Full Test Pass - ClayCo's WebLog - Site Home

I haven't posted a lot in the last few weeks, and there's a succinct reason for that. In fact, I can name it in three words: Full Test Pass.

The concept is close to what it sounds like. During the earlier stages of product planning, the test team for ASP.NET and the web development profile for Visual Studio used the time to plan its test coverage. In total, we wrote descriptions of over 40,000 test cases that we wanted to run. In the span of about four weeks ending earlier this month, we took the time to acutally execute nearly all of them, many for the first time.

Don't take that to mean that we whittled away all the hours between whiles drinking beverages and playing foosball -- we are professionals, after all. We used the time to automate a large subset of our tests, including our acceptance tests (tests that we run on each daily build of the product to get a rough sense of its quality and catch really the big bugs). Automation infrastructure -- the libraries and components our testcases use to simplify automation and manage the process of reporting results to our testcase management database (named Mad Dog, if you can believe it) -- also took up a lot of time.

But 40,000 is a big number, and in that number were a lot of scenarios that we just hadn't tried yet. For example, one of the features I own is Control Designers, which is the mechanism a custom control developer uses to control the appearance and behavior of her control in Visual Studio's web design environment. As part of this, I test several new host services that are exposed by Visual Studio which the authors of Control Designers can use to simplify their work. One of these is an interface with the name IWebFormsBuilderUIService, which exposes methods for creating common UI dialogs, among them a URL editor. In Visual Basic .NET, the code a control author would need to write in order to use this would be something like this:

Imports System.ComponentModel
Imports System.ComponentModel.Design
Imports System.Web.UI.Design
Imports System.Web.UI
Imports System

<Designer(GetType(ExampleControlDesigner)), . . . > Public Class ExampleControl
    Inherits System.Web.UI.WebControls.WebControl

    Dim _url As String

    <Bindable(True), Category("Appearance"), DefaultValue("")> Property URL() As String
        Get
            Return _url
        End Get

        Set(ByVal Value As String)
            _url = Value
        End Set
    End Property

    . . .
End Class

Public Class ExampleControlDesigner
    Inherits ControlDesigner

    Protected Overrides Sub OnClick(ByVal e As DesignerRegionEventArgs)
        Dim iwfbuis as IWebFormsBuilderUIService
        Dim ctl As ExampleControl

        Mybase.OnClick(e)
        ctl = Component
        iwfbuis = GetService(GetType(IWebFormsBuilderUIService))
        try
            ctl.URL = iwfbuis.BuildUrl(Nothing, ctl.URL, RootDesigner.DocumentUrl, "Please Choose a URL", "", UrlBuilderOptions.None)
        catch ex as exception
            Msgbox(ex.Message)
        end try
        UpdateDesignTimeHtml()
    End Sub

    . . .
End Class

The above code would bring up Visual Studio's URL dialog when the user clicked on the control, and set the control's URL property to the user's selection.

I don't have any acceptance-level tests for IWebFormsBuilderUIService.BuildUrl, because even if this method were completely busted beyond usability, it wouldn't especially hurt the other testers on our team or the other teams that depend on ASP.NET. However, I do have plenty of regular functional tests for the feature that I plan to automate, and the plan for this area looks like this:

Testcase Breakout for BuildUrl

Hierarchy: All Testcases -> Web.NET -> Page & Controls -> Page Framework -> Designers -> Control Designers Extensibility -> Host Services -> IWebFormsBuilderUIService

BuildUrl ClayCo

Area Description

Host-implemented method for bringing up its own URI-picking dialog.

BuildUrl01 - Call and (from Venus) cancel the dialog Pri 2, Coding

1) Verify initialURL is returned.

BuildUrl02 - Call (from ControlDesigner) and enter (from Venus) an absolute URL Pri 2, Coding

Abstract:
All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios:
1) Use http://localhost/<currentapp>, verify relative conversion
2) Call same with UrlBuilderOptions set to None, verify no conversion
3) Use http://localhost/<currentapp>/, verify relative conversion
4) Use http://localhost/<currentapp>/Page2.aspx, verify relative conversion
5) Use https://localhost/AnotherApp/Page.aspx, verify relative conversion
6) Use ftp://ftp.is.co.za/rfc/rfc1808.txt
7) Use gopher://spinaltap.micro.umn.edu/00/Weather/California/Los%20Angeles
8) Use mailto:clayco@microsoft.com
9) Use news:comp.infosystems.www.servers.unix
10) Use telnet://melvyl.ucop.edu/
11) Use http://localhost/a/./g.aspx, verify relative conversion
12) Call same with UrlBuilderOptions set to None, verify no conversion
13) Use http://localhost/a/b/../g.aspx, verify relative conversion
14) Call same with UrlBuilderOptions set to None, verify no conversion
15) Use http://localhost/test., verify relative conversion
16) Call same with UrlBuilderOptions set to None, verify no conversion
17) Use http://localhost/.aspx, verify relative conversion
18) Call same with UrlBuilderOptions set to None, verify no conversion
19) Use http://localhost/..aspx, verify relative conversion
20) Call same with UrlBuilderOptions set to None, verify no conversion
21) Use 256-character URI
22) Use string containing Arabic Letter Khah and U+4E2E (catch).

BuildUrl03 - Call (from ControlDesigner) and enter (from Venus) a relative URL Pri 2, Coding

Abstract:
All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios:
1) Use //localhost/<currentapp>/Page2.aspx, verify an app-relative URL is returned
2) Call same with UrlBuilderOptions set to None, verify absolute conversion
3) Use /<currentapp>/Page2.aspx, verify an app-relative URL is returned
4) Call same with UrlBuilderOptions set to None, verify absolute conversion
5) Use Page2.aspx
6) Call same with UrlBuilderOptions set to None, verify absolute conversion
7) Use //www.yahoo.com:80/, verify an absolute URL is returned
8) Use /./Page2.aspx, verify an app-relative URL is returned
9) Call same with UrlBuilderOptions set to None, verify absolute conversion
10) Use /../Page2.aspx
11) Call same with UrlBuilderOptions set to None, verify absolute conversion
12) Use http.aspx
13) Call same with UrlBuilderOptions set to None, verify absolute conversion
14) Use test.
15) Call same with UrlBuilderOptions set to None, verify absolute conversion
16) Use .aspx
17) Call same with UrlBuilderOptions set to None, verify absolute conversion
18) Use test..
19) Call same with UrlBuilderOptions set to None, verify absolute conversion
20) Use ..aspx
21) Call same with UrlBuilderOptions set to None, verify absolute conversion
22) Use ./../test.aspx, verify an app-relative URL is returned
23) Call same with UrlBuilderOptions set to None, verify absolute conversion
24) Use ./SubDir/., verify an app-relative URL is returned
25) Call same with UrlBuilderOptions set to None, verify absolute conversion
26) Use SubDir/./Page2.aspx
27) Call same with UrlBuilderOptions set to None, verify absolute conversion
28) Use SubDir/../Page2.aspx
29) Call same with UrlBuilderOptions set to None, verify absolute conversion
30) Use Test;x=1/./Page2.aspx
31) Call same with UrlBuilderOptions set to None, verify absolute conversion
32) Use Test;x=1/../Page2.aspx
33) Call same with UrlBuilderOptions set to None, verify absolute conversion
34) Use ., verify an app-relative URL is returned
35) Call same with UrlBuilderOptions set to None, verify absolute conversion
36) Use ./, verify an app-relative URL is returned
37) Call same with UrlBuilderOptions set to None, verify absolute conversion
38) Use ..
39) Call same with UrlBuilderOptions set to None, verify absolute conversion
40) Use ../
41) Call same with UrlBuilderOptions set to None, verify absolute conversion
42) Use ../Page2.aspx
43) Call same with UrlBuilderOptions set to None, verify absolute conversion
44) Use ../..
45) Call same with UrlBuilderOptions set to None, verify absolute conversion
46) Use ../../
47) Call same with UrlBuilderOptions set to None, verify absolute conversion
48) Use ../../Page2.aspx
49) Call same with UrlBuilderOptions set to None, verify absolute conversion
50) Use 256-character URI
51) Use string containing Arabic Letter Ghain and U+3A2E (lift up).

BuildUrl04 - InitialUrl paramerter scenarios Pri 2, Coding

1) Use current web application folder's absolute URL, verify UI in Venus
2) Use current web application folder's relative URL, verify UI in Venus
3) Use current Page's URL, verify UI in Venus
4) Use image file URL, verify UI in Venus
5) Use absolute path to virtual sibling directory, verify UI in Venus
6) Use non-existent absolute path to sibling directory, verify UI in Venus
7) Use absolute path to virtual parent directory, verify UI in Venus
8) Use absolute path to virtual child directory, verify UI in Venus
9) Use relative path to virtual sibling directory, verify UI in Venus
10) Use relative path to virtual parent directory, verify UI in Venus
11) Use relative path to virtual child directory, verify UI in Venus
12) Use http://, ftp://, and file:// URLs to test machine, verify UI in Venus
13) Use URL to another machine, verify UI in Venus
14) Use http://www.microsoft.com/, verify error

BuildUrl05 - BaseUrl parameter scenarios Pri 2, Coding

Abstract:
All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios:
1) Use current web application folder's absolute URL, verify relative URL calculation
2) Use current web application folder's relative URL, verify relative URL calculation
3) Use current Page's URL, verify relative URL calculation
4) Use image file URL, verify relative URL calculation
5) Use absolute path to virtual sibling directory, verify relative URL calculation
6) Use absolute path to virtual parent directory, verify relative URL calculation
7) Use absolute path to virtual child directory, verify relative URL calculation
8) Use relative path to virtual sibling directory, verify relative URL calculation
9) Use relative path to virtual parent directory, verify relative URL calculation
10) Use relative path to virtual child directory, verify relative URL calculation
11) Use http://, ftp://, and file:// URLs to the same machine, verify absolute URLs are used
12) Use URL to another machine, verify relative and absolute paths on local and remote machines.
13) Use http://www.microsoft.com/, verify absolute paths get used.
14) Use String.Empty, verify absolute paths get used.
15) Use current web application folder's absolute URL when options is None, verify absolute URL
16) Use current web application folder's relative URL when options is None, verify absolute URL
17) Use current Page's URL when options is None, verify absolute URL
18) Use image file URL when options is None when options is None, verify absolute URL
19) Use absolute path to virtual sibling directory when options is None, verify absolute URL
20) Use absolute path to virtual parent directory when options is None, verify absolute URL
21) Use absolute path to virtual child directory when options is None, verify absolute URL
22) Use relative path to virtual sibling directory when options is None, verify absolute URL
23) Use relative path to virtual parent directory when options is None, verify absolute URL
24) Use relative path to virtual child directory when options is None, verify absolute URL

BuildUrl06 - Caption parameters Pri 2, Coding

Abstract:
Perform standard string tests and verify in Venus

Scenarios:
1) S1) Verify default property value
2) S2) Verify attribute value is not in HTML view
3) S3) Set to all numbers
4) S4) Set attribute to (nothing) in HTML view
5) S5) Set to 1 character
6) S6) Set attribute to space in HTML view
7) S7) Set attribute without quotes in HTML view
8) S8) Set property to DblQuote-space-DblQuote in Design view
9) S9) Set property to Quote-space-Quote in Design view
10) S10) set property to: DblQuote-Quote-&-Quote-DblQuote; verify
11) S11) set attribute to: 3DblQuote-'''abc'''-3DblQuote; verify
12) S12) set attribute to: DblQuote-@#$%-CrLf-GreaterThan-)(*^-DblQuote; verify
13) S13) Set property to "dadadada" in prop grid
14) S14) Set to long string
15) S15) Set to a random intl string and verify
16) S16) Set to an Intl string of problem characters
17) S17) Set to a random unicode string

BuildUrl07 - Filter parameter scenarios Pri 2, Coding

Abstract:
Use Standard String tests and verify expected value in Venus

Scenarios:
1) S1) Verify default property value
2) S2) Verify attribute value is not in HTML view
3) S3) Set to all numbers
4) S4) Set attribute to (nothing) in HTML view
5) S5) Set to 1 character
6) S6) Set attribute to space in HTML view
7) S7) Set attribute without quotes in HTML view
8) S8) Set property to DblQuote-space-DblQuote in Design view
9) S9) Set property to Quote-space-Quote in Design view
10) S10) set property to: DblQuote-Quote-&-Quote-DblQuote; verify
11) S11) set attribute to: 3DblQuote-'''abc'''-3DblQuote; verify
12) S12) set attribute to: DblQuote-@#$%-CrLf-GreaterThan-)(*^-DblQuote; verify
13) S13) Set property to "dadadada" in prop grid
14) S14) Set to long string
15) S15) Set to a random intl string and verify
16) S16) Set to an Intl string of problem characters
17) S17) Set to a random unicode string

Testcase Summary Info:

Design: Pending Scenario Review:

Coding: 7 Pending Code Review:

Complete:

Total: 7

This is typical -- every feature area has a lot more tests defined for it than get will ever be run on any given build. We rely on testers using the builds for ad-hoc testing and writing automation to find bugs, and rely on automated tests to catch regressions in later builds. We also use full test passes like this to get a deeper sense of the product's quality at a few key intervals (such as earlier this month -- we wanted to catch isses in advance of the first large-scale Beta release of the product and get a sense of how much more work is needed to stabilize the product's quality before we ship).

And we did find bugs. In the above test plan, the very first test case failed during the full test pass. If a user clicked “Cancel” in the URL dialog, an empty string was returned by the method, rather than the initialURL parameter value. This has already been fixed, by the way.