ClayCo's WebLog

Full Test Pass

ClayCo — Wed, 21 Apr 2004 00:41:00 GMT

I haven't posted a lot in the last few weeks, and there's a succinct reason for that. In fact, I can name it in three words: Full Test Pass.

The concept is close to what it sounds like. During the earlier stages of product planning, the test team for ASP.NET and the web development profile for Visual Studio used the time to plan its test coverage. In total, we wrote descriptions of over 40,000 test cases that we wanted to run. In the span of about four weeks ending earlier this month, we took the time to acutally execute nearly all of them, many for the first time.

Don't take that to mean that we whittled away all the hours between whiles drinking beverages and playing foosball -- we are professionals, after all. We used the time to automate a large subset of our tests, including our acceptance tests (tests that we run on each daily build of the product to get a rough sense of its quality and catch really the big bugs). Automation infrastructure -- the libraries and components our testcases use to simplify automation and manage the process of reporting results to our testcase management database (named Mad Dog, if you can believe it) -- also took up a lot of time.

But 40,000 is a big number, and in that number were a lot of scenarios that we just hadn't tried yet. For example, one of the features I own is Control Designers, which is the mechanism a custom control developer uses to control the appearance and behavior of her control in Visual Studio's web design environment. As part of this, I test several new host services that are exposed by Visual Studio which the authors of Control Designers can use to simplify their work. One of these is an interface with the name IWebFormsBuilderUIService, which exposes methods for creating common UI dialogs, among them a URL editor. In Visual Basic .NET, the code a control author would need to write in order to use this would be something like this:

Imports System.ComponentModel
Imports System.ComponentModel.Design
Imports System.Web.UI.Design
Imports System.Web.UI
Imports System

<Designer(GetType(ExampleControlDesigner)), . . . > Public Class ExampleControl
    Inherits System.Web.UI.WebControls.WebControl

    Dim _url As String

    <Bindable(True), Category("Appearance"), DefaultValue("")> Property URL() As String
        Get
            Return _url
        End Get

        Set(ByVal Value As String)
            _url = Value
        End Set
    End Property

    . . .
End Class

Public Class ExampleControlDesigner
    Inherits ControlDesigner

    Protected Overrides Sub OnClick(ByVal e As DesignerRegionEventArgs)
        Dim iwfbuis as IWebFormsBuilderUIService
        Dim ctl As ExampleControl

        Mybase.OnClick(e)
        ctl = Component
        iwfbuis = GetService(GetType(IWebFormsBuilderUIService))
        try
            ctl.URL = iwfbuis.BuildUrl(Nothing, ctl.URL, RootDesigner.DocumentUrl, "Please Choose a URL", "", UrlBuilderOptions.None)
        catch ex as exception
            Msgbox(ex.Message)
        end try
        UpdateDesignTimeHtml()
    End Sub

    . . .
End Class

The above code would bring up Visual Studio's URL dialog when the user clicked on the control, and set the control's URL property to the user's selection.

I don't have any acceptance-level tests for IWebFormsBuilderUIService.BuildUrl, because even if this method were completely busted beyond usability, it wouldn't especially hurt the other testers on our team or the other teams that depend on ASP.NET. However, I do have plenty of regular functional tests for the feature that I plan to automate, and the plan for this area looks like this:

Testcase Breakout for BuildUrl

Hierarchy: All Testcases -> Web.NET -> Page & Controls -> Page Framework -> Designers -> Control Designers Extensibility -> Host Services -> IWebFormsBuilderUIService

BuildUrl ClayCo

Area Description

Host-implemented method for bringing up its own URI-picking dialog.

BuildUrl01 - Call and (from Venus) cancel the dialog Pri 2, Coding

1) Verify initialURL is returned.

BuildUrl02 - Call (from ControlDesigner) and enter (from Venus) an absolute URL Pri 2, Coding

Abstract:
All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios:
1) Use http://localhost/<currentapp>, verify relative conversion
2) Call same with UrlBuilderOptions set to None, verify no conversion
3) Use http://localhost/<currentapp>/, verify relative conversion
4) Use http://localhost/<currentapp>/Page2.aspx, verify relative conversion
5) Use https://localhost/AnotherApp/Page.aspx, verify relative conversion
6) Use ftp://ftp.is.co.za/rfc/rfc1808.txt
7) Use gopher://spinaltap.micro.umn.edu/00/Weather/California/Los%20Angeles
8) Use mailto:clayco@microsoft.com
9) Use news:comp.infosystems.www.servers.unix
10) Use telnet://melvyl.ucop.edu/
11) Use http://localhost/a/./g.aspx, verify relative conversion
12) Call same with UrlBuilderOptions set to None, verify no conversion
13) Use http://localhost/a/b/../g.aspx, verify relative conversion
14) Call same with UrlBuilderOptions set to None, verify no conversion
15) Use http://localhost/test., verify relative conversion
16) Call same with UrlBuilderOptions set to None, verify no conversion
17) Use http://localhost/.aspx, verify relative conversion
18) Call same with UrlBuilderOptions set to None, verify no conversion
19) Use http://localhost/..aspx, verify relative conversion
20) Call same with UrlBuilderOptions set to None, verify no conversion
21) Use 256-character URI
22) Use string containing Arabic Letter Khah and U+4E2E (catch).

BuildUrl03 - Call (from ControlDesigner) and enter (from Venus) a relative URL Pri 2, Coding

Abstract:
All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios:
1) Use //localhost/<currentapp>/Page2.aspx, verify an app-relative URL is returned
2) Call same with UrlBuilderOptions set to None, verify absolute conversion
3) Use /<currentapp>/Page2.aspx, verify an app-relative URL is returned
4) Call same with UrlBuilderOptions set to None, verify absolute conversion
5) Use Page2.aspx
6) Call same with UrlBuilderOptions set to None, verify absolute conversion
7) Use //www.yahoo.com:80/, verify an absolute URL is returned
8) Use /./Page2.aspx, verify an app-relative URL is returned
9) Call same with UrlBuilderOptions set to None, verify absolute conversion
10) Use /../Page2.aspx
11) Call same with UrlBuilderOptions set to None, verify absolute conversion
12) Use http.aspx
13) Call same with UrlBuilderOptions set to None, verify absolute conversion
14) Use test.
15) Call same with UrlBuilderOptions set to None, verify absolute conversion
16) Use .aspx
17) Call same with UrlBuilderOptions set to None, verify absolute conversion
18) Use test..
19) Call same with UrlBuilderOptions set to None, verify absolute conversion
20) Use ..aspx
21) Call same with UrlBuilderOptions set to None, verify absolute conversion
22) Use ./../test.aspx, verify an app-relative URL is returned
23) Call same with UrlBuilderOptions set to None, verify absolute conversion
24) Use ./SubDir/., verify an app-relative URL is returned
25) Call same with UrlBuilderOptions set to None, verify absolute conversion
26) Use SubDir/./Page2.aspx
27) Call same with UrlBuilderOptions set to None, verify absolute conversion
28) Use SubDir/../Page2.aspx
29) Call same with UrlBuilderOptions set to None, verify absolute conversion
30) Use Test;x=1/./Page2.aspx
31) Call same with UrlBuilderOptions set to None, verify absolute conversion
32) Use Test;x=1/../Page2.aspx
33) Call same with UrlBuilderOptions set to None, verify absolute conversion
34) Use ., verify an app-relative URL is returned
35) Call same with UrlBuilderOptions set to None, verify absolute conversion
36) Use ./, verify an app-relative URL is returned
37) Call same with UrlBuilderOptions set to None, verify absolute conversion
38) Use ..
39) Call same with UrlBuilderOptions set to None, verify absolute conversion
40) Use ../
41) Call same with UrlBuilderOptions set to None, verify absolute conversion
42) Use ../Page2.aspx
43) Call same with UrlBuilderOptions set to None, verify absolute conversion
44) Use ../..
45) Call same with UrlBuilderOptions set to None, verify absolute conversion
46) Use ../../
47) Call same with UrlBuilderOptions set to None, verify absolute conversion
48) Use ../../Page2.aspx
49) Call same with UrlBuilderOptions set to None, verify absolute conversion
50) Use 256-character URI
51) Use string containing Arabic Letter Ghain and U+3A2E (lift up).

BuildUrl04 - InitialUrl paramerter scenarios Pri 2, Coding

1) Use current web application folder's absolute URL, verify UI in Venus
2) Use current web application folder's relative URL, verify UI in Venus
3) Use current Page's URL, verify UI in Venus
4) Use image file URL, verify UI in Venus
5) Use absolute path to virtual sibling directory, verify UI in Venus
6) Use non-existent absolute path to sibling directory, verify UI in Venus
7) Use absolute path to virtual parent directory, verify UI in Venus
8) Use absolute path to virtual child directory, verify UI in Venus
9) Use relative path to virtual sibling directory, verify UI in Venus
10) Use relative path to virtual parent directory, verify UI in Venus
11) Use relative path to virtual child directory, verify UI in Venus
12) Use http://, ftp://, and file:// URLs to test machine, verify UI in Venus
13) Use URL to another machine, verify UI in Venus
14) Use http://www.microsoft.com/, verify error

BuildUrl05 - BaseUrl parameter scenarios Pri 2, Coding

Abstract:
All scenarios assume UrlBuilderOptions is set to NoAbsolute unless stated otherwise.

Scenarios:
1) Use current web application folder's absolute URL, verify relative URL calculation
2) Use current web application folder's relative URL, verify relative URL calculation
3) Use current Page's URL, verify relative URL calculation
4) Use image file URL, verify relative URL calculation
5) Use absolute path to virtual sibling directory, verify relative URL calculation
6) Use absolute path to virtual parent directory, verify relative URL calculation
7) Use absolute path to virtual child directory, verify relative URL calculation
8) Use relative path to virtual sibling directory, verify relative URL calculation
9) Use relative path to virtual parent directory, verify relative URL calculation
10) Use relative path to virtual child directory, verify relative URL calculation
11) Use http://, ftp://, and file:// URLs to the same machine, verify absolute URLs are used
12) Use URL to another machine, verify relative and absolute paths on local and remote machines.
13) Use http://www.microsoft.com/, verify absolute paths get used.
14) Use String.Empty, verify absolute paths get used.
15) Use current web application folder's absolute URL when options is None, verify absolute URL
16) Use current web application folder's relative URL when options is None, verify absolute URL
17) Use current Page's URL when options is None, verify absolute URL
18) Use image file URL when options is None when options is None, verify absolute URL
19) Use absolute path to virtual sibling directory when options is None, verify absolute URL
20) Use absolute path to virtual parent directory when options is None, verify absolute URL
21) Use absolute path to virtual child directory when options is None, verify absolute URL
22) Use relative path to virtual sibling directory when options is None, verify absolute URL
23) Use relative path to virtual parent directory when options is None, verify absolute URL
24) Use relative path to virtual child directory when options is None, verify absolute URL

BuildUrl06 - Caption parameters Pri 2, Coding

Abstract:
Perform standard string tests and verify in Venus

Scenarios:
1) S1) Verify default property value
2) S2) Verify attribute value is not in HTML view
3) S3) Set to all numbers
4) S4) Set attribute to (nothing) in HTML view
5) S5) Set to 1 character
6) S6) Set attribute to space in HTML view
7) S7) Set attribute without quotes in HTML view
8) S8) Set property to DblQuote-space-DblQuote in Design view
9) S9) Set property to Quote-space-Quote in Design view
10) S10) set property to: DblQuote-Quote-&-Quote-DblQuote; verify
11) S11) set attribute to: 3DblQuote-'''abc'''-3DblQuote; verify
12) S12) set attribute to: DblQuote-@#$%-CrLf-GreaterThan-)(*^-DblQuote; verify
13) S13) Set property to "dadadada" in prop grid
14) S14) Set to long string
15) S15) Set to a random intl string and verify
16) S16) Set to an Intl string of problem characters
17) S17) Set to a random unicode string

BuildUrl07 - Filter parameter scenarios Pri 2, Coding

Abstract:
Use Standard String tests and verify expected value in Venus

Scenarios:
1) S1) Verify default property value
2) S2) Verify attribute value is not in HTML view
3) S3) Set to all numbers
4) S4) Set attribute to (nothing) in HTML view
5) S5) Set to 1 character
6) S6) Set attribute to space in HTML view
7) S7) Set attribute without quotes in HTML view
8) S8) Set property to DblQuote-space-DblQuote in Design view
9) S9) Set property to Quote-space-Quote in Design view
10) S10) set property to: DblQuote-Quote-&-Quote-DblQuote; verify
11) S11) set attribute to: 3DblQuote-'''abc'''-3DblQuote; verify
12) S12) set attribute to: DblQuote-@#$%-CrLf-GreaterThan-)(*^-DblQuote; verify
13) S13) Set property to "dadadada" in prop grid
14) S14) Set to long string
15) S15) Set to a random intl string and verify
16) S16) Set to an Intl string of problem characters
17) S17) Set to a random unicode string

Testcase Summary Info:

Design: Pending Scenario Review:

Coding: 7 Pending Code Review:

Complete:

Total: 7

This is typical -- every feature area has a lot more tests defined for it than get will ever be run on any given build. We rely on testers using the builds for ad-hoc testing and writing automation to find bugs, and rely on automated tests to catch regressions in later builds. We also use full test passes like this to get a deeper sense of the product's quality at a few key intervals (such as earlier this month -- we wanted to catch isses in advance of the first large-scale Beta release of the product and get a sense of how much more work is needed to stabilize the product's quality before we ship).

And we did find bugs. In the above test plan, the very first test case failed during the full test pass. If a user clicked “Cancel” in the URL dialog, an empty string was returned by the method, rather than the initialURL parameter value. This has already been fixed, by the way.

A New Language Tag?

ClayCo — Fri, 02 Apr 2004 01:22:00 GMT

Those of you who know about RFC 3066 know that it's the standard which describes the tags that you should use to identify the language of an HTML or XML document's contents. For example, in XHTML you'd code something like:

. . .
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="fr" >
. . .
</html>

to tell the world that your page is in French. RFC 3066 (albeit with a few differences) is also the syntax of the culture names used by System.Globalization.CultureInfo.

What you may not have known is that there's a way to request new tags in case you need to extend the existing language tagging mechanism. This can be very useful if you need to indicate a language variety that isn't easily described with a language-and-country combination. On the other hand, well, I'll let today's request speak for itself:

LANGUAGE TAG REGISTRATION FORM

Name of requester          : Jon Hanna
E-mail address of requester: jon [at] hackcraft.net
Tag to be registered       : en-Spam-porn

English name of language   : Pornographic Spam English

Native name of language (transcribed into ASCII):
Pr0n Freespeech ohbsehkzq!

Reference to published description of the language (book or article):
Your deleted mail folder

Any other relevant information:
The relationship between this dialect and other forms of English is
tenuous and perhaps more a matter of historical accident than any
stronger linguistic relationship. Some would argue that it is a
separate language, and some that it isn't a language at all but
illiterate gibberish, however considering it to be a dialect of English
is perhaps the most balanced position.

Interestingly, it is the probably the only dialect of any language that
can be encoded entirely in US-ASCII without doing any (further) violence
to the language. Conversely, it is also probably the only language that
can be inscribed with the use of a diacritical mark on every single
character, even spaces.

The dialect appears to have very complicated punctuation rules, so
complicated that there is strong evidence that no one fully understands
them. The exclamation mark (U+0021, and no doubt U+203C once they
discover it) appears not to be a punctuation character within the
dialect, but a symbol of some sort of religious or superstitious
fascination within the user community. The anthropologist might refer
to this as a "fetish", though he or she would be wise not to lead any
member of the user community to believe that he or she had any interest
in fetishes.

Despite widespread belief to the contrary it appears that some people
really do have this as their first, and indeed only, language. These
people would be to be pitied, if we could have the luxury of doing so
without actually coming into contact with them or their dialect.

The pun is considered the highest form of expression within the dialect,
although no user of the dialect has ever demonstrated any talent at
creating them.

Special Treatment -

Traditionally the dialect is encoded with whatever charset is believed
most likely to confuse a filter. Adoption of the UCS is increasingly
common amongst the user community as it provides a source of yet more
characters that look a bit like other characters.

Semantic software should not attempt to differentiate between definite
and indefinite articles or should treat them specially. These are not
used as they are in other dialects of English where they specify a
particular or a general example of the associated noun, but rather to
reflect a self-deluding belief in universality of a fetish item as the
user community has a strong religious belief that we are all equally
pathetic perverts to themselves. This religious belief contains a
strong emphasis on proselytising for which the user community eagerly
adapt new communication technologies including those produced by the
IETF.

In rich-text formats in which the text can be coloured there is a
strong preference towards red.

I'll readily admit that this is probably the most articulate request for a new language tag in a long time, and that reading it left me with only one question: Arê yØu`l00ki.ng 4_ä gõod t|mé?

For those of you wondering how serious this request could be, consider today's date. For everyone else, please remember that you have two weeks (starting today) to comment on the request.

Performance, Stress, and other fancy stuff

ClayCo — Tue, 02 Mar 2004 03:01:00 GMT

In response to my first post, Scott asked a very topical question:

Not sure if this falls in your basket or not, but I would be interest in any tips or tricks for memory profiling ASP.NET applications.

Well, Scott, I'm pleased that you're interested in this subject, because it's a very good way to introduce how we're testing the performance of ASP.NET under stress. For a grounding on the subject, I encourage anyone interested in this subject to start by reading this. It's an article written by Thomas Marquardt, a developer (and former tester) on our team who did a lot of work creating the test infrastructure we're using today. It should also handily answer your question.

The other reason why this I want to point you to this article is that the source code for one of the test tools used for stress automation, QQQ.exe, can be downloaded there. QQQ is a tool that monitors performance counters for the ASP.NET worker process and automatically attaches a debugger if any of a few different critical thresholds are reached. We use this with the Web Application Stress Tool (which, in the long tradition of giving things names that will freak out everybody in marketing, is also called “Homer”) to simulate the process of multiple users making large quantities of requests to the server and catching the problems that arise. These tools, combined with some debugging tools, scripts for automatically re-imaging the test machines and launching stress, and (naturally) the test pages themselves, are the way in which we stress-test the product.

The tests themselves cover a pretty broad swath of scenarios. As you'd expect, we have tests simulating high traffic in a production environment with all managed code, but we also have tests for pages that inter-operate with COM components and call data methods. We also check how ASP.NET handles some more esoteric scenarios. What happens if the user changes machine.config every few minutes, causing the application to restart? What if there's a memory leak in the user's applications? What if the worker process crashes? The process model, of course, should gracefully recover from these hiccups. In all of these scenarios, the pages we use for stress testing are also functional tests that we use for regular verification of the product, so we can get as close to simulating real-life workloads and traffic as possible.

I used to debug stress, so a lot of this comes from memory (it's like riding a bicycle), but special thanks go to (fellow tester) Vladimir Kritchko for pointing me to the latest stress infrastructure documentation.

Welcome

ClayCo — Wed, 25 Feb 2004 02:23:00 GMT

I'm a software tester, and I work on ASP.NET.

Are you still reading? Interesting.

ASP.NET's test team isn't something you hear much about, and nobody who tests the product ranks among the best known team members in our product group (the organizational unit that includes testers, developers, and program managers working on the same closely-related set of products). However, the importance of the test team around here is hardly minimal. Just to give you some sense of proportion, testers make up over half of the product group's members.

I'm going to start this weblog by describing in hopefully not too much detail the way in which ASP.NET is getting tested; as I think you'll learn, this is different from the kind and level of testing you may have seen (or done) in other products. We have our reasons for testing the way we do, and maybe you can find value in hearing about how we do it.

I also want to talk about ASP.NET itself, with attention on the new features in it that I test. As soon as I'm done with my overview of how we test ASP.NET in all of its glory, I hope to share with you something useful about Smart Navigation, the ContentPager control, control designers, and the other nifty things I happen to test for a living.

Of course, your suggestions are important; I want to share with you the things you want to know about to the greatest of my ability, so don't be afraid to give me your feedback.

Design:		Pending Scenario Review:
Coding:	7	Pending Code Review:
Complete:
		Total:	7