[29 Mar 2010: An expanded and annotated version of this is available on my website at http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf.]

Accessibility

• Check the following MSAA property settings for every control on every dialog, command bar, and other UI feature: NAME, ROLE, STATE, VALUE, KBSHORTCUT, and DEFACTION
• Change values in edit boxes and verify those values are reflected in the MSAA property set
• Run in high contrast mode
• Run in large font mode
• Run with Sound Sentry
• Run with sticky keys and other mouse key settings
• Verify focus events are passed when controls receive focus
• Verify that all audio, video, and animations have supporting textual scripts, screen narration, and/or closed captioning
• Verify that text is not rendered as text, not as bitmaps
• Verify tabbing order
• Verify shortcut keys exist for each control
• Verify that each actionable color item can have its color customized (e.g., red squiggly lines)
• Verify that all flashing objects flash to the system cursor blink rate

Alerts

• Verify each alert identifies the problem first and then a solution
• Verify alert text is not unnecessarily long (i.e., more than 3 lines)
• Verify alert text is consistently worded and consistent in style
• Verify alert text is correct and appropriate for the situation
• Verify alert text contains complete sentences with correct capitalization and punctuation
• Verify alert text avoids using abbreviations and acronyms
• Verify alert text does not use an accusatory tone but rather is polite and helpful
• Verify alert text uses the product name where necessary, not "we"
• Verify the buttons work correctly
• Verify the buttons have unique access keys
• Verify alert text is localized correctly
• Verify the correct icon (Information, Warning, Critical) is displayed on the left hand side
• Verify the title bar contains the name of the product (e.g., "Microsoft Bob")
• Verify the buttons are centered below the message text

Anti-Virus

• Test with and without anti-virus software installed

Application Interoperability 

• Verify clipboard operations (i.e., cut/copy/paste) between this and other features, between this and other applications
• Verify drag and drop operations between this and other features, between this and other applications

Boundary Cases 

• Text length (e.g., of a list entry, in a field)
• Extra-long filenames
• Spaces in the text (e.g., of a list entry, field name)
• Duplicate items
• Large selection (e.g., the entire document or list)
• Giant files
• 4096 x 4096 pixel image
• Maximum number of objects (e.g., shapes, columns)
• Maximum and minimum values (e.g., for margins, values in edit boxes)

CPU

• Multiple processors
• Multiple vendors CPU's (e.g., Intel, AMD)
• Laptop configurations

Dialogs

• Verify default edit focus and default button focus is correct
• Verify tab order
• Verify dialog and terms used therein are consistent with those in the rest of the application
• Verify ninched controls are shaded
• Verify mutually exclusive controls work correctly
• Verify any samples in the dialog reflect the actual formatting in the document
• Verify all commands that launch dialogs (e.g., F12 to Save As)
• Verify every control has a short-cut letter, that all short-cuts work, and that every action can be executed via the keyboard
• Verify every accelerator is unique within the dialog
• Verify the dialog's title is correct
• Verify the menu command to show dialog is followed with "..."
• Verify the sizing and spacing of all controls matches the design standard
• Verify all dialog states, especially including more or less controls being visible due to application state or settings
• Verify the correct system controls display on the title bar, and that they work correctly
• Verify help topics are correct
• Verify dialog tips (right-click on a control) are correct
• Verify status bar text is correct
• Verify the dialog displays and functions correctly with different color, font, and high contrast settings
• Verify the dialog displays and functions correctly in high DPI mode
• Verify all bitmaps being used with the text in the dialog are correctly localized
• Verify controls respond correctly to valid and invalid input
• Verify controls respond correctly to boundary case input (e.g., correct alerts are shown)
• Verify Escape cancels the dialog
• Verify the dialog is sticky (or not, as appropriate); i.e., it displays in same position as when last dismissed
• Verify settings in the dialg are saved (or not, as appropriate) the next time the dialog is invoked
• Verify the dialog is sized correctly in relation to its controls

File Names

• With and without the complete path
• With and without the extension
• Long file names
• Short file names
• Deep directory levels
• International characters
• Invalid characters (e.g., '?')
• Invalid filenames (e.g., COM1, COM1.txt)

Hardware

• Tablet PC
• Removable Media
• Pocket PC
• Run with Power Management settings enabled and disabled

International Sufficiency 

• Look for geo-political sensitivity issues (e.g., borders may be disputed between neighboring countries)
• Run with Complex scripts and Right to Left (RTL) support
• Verify all controls function correctly with the East Asian (IME)
• Verify switching the keyboard to a different mapping works correctly
• Verify system locale, unicode, and non-active code page support functions correctly
• Verify ANSI, double-byte, and Unicode text are handled correctly for display, input, and elsewhere (e.g., double-byte characters will be split if treated as ANSI)
• Verify any language-dependant features work correctly
• Verify extended and non-standard characters are displayed and handled correctly
• Verify the executable and other core files function correctly on any code page and any supported OS language
• Verify the correct sorting order is used.
• Verify user locale or system locale are used as appropriate.

Localization 

• Look for geo-political sensitivity issues (e.g., borders may be disputed between neighboring countries)
• Verify you know which languages to test
• Verify controls in dialogs and elsewhere are aligned and sized correctly (e.g., text is not truncated)
• Verify any functionality affected by the localization process works correctly
• Verify data is ordered correctly
• Verify tab order is correct
• Verify no unlocalized strings remain
• Verify accelerators are localized
• Verify hot keys are localzied
• Verify each accelerator is unique and consistent
• Verify each hot key is unique and consistent

Menus and Command Bars 

• Verify all commands work from menus and command bars
• Verify keyboard shortcuts are correct on all platforms
• Verify built-in menu items on a custom commandbar work correctly
• Verify commands are visible and/or enabled only when appropriate
• Verify command captions are correct and consistent with similar terms used elsewhere
• Verify custom commands work when dragged onto built-in menus and toolbars
• Verify voice command works correctly
• Verify menu/command bar item context menus work correctly
• Verify status bar text is correct and not truncated

Miscellaneous -- Have you

• Completed the spec inspection has been completed
• Discussed the feature design and target users with your Program Manager
• Done real world testing (i.e., used the application the way the user will)
• Talked with your developer about the areas they think need special attention
• Brainstormed test cases with the rest of your team
• Discussed cross-feature implications with the rest of your team
• Discussed the feature/application with Product Support
• Perused bugs and other feedback from beta releases
• Considered backward compatibility problems resulting from something that has been moved or modified changing default values or breaking something in a previous version

Networking

• Verify correct handling of different levels of user access and permissions
• Verify correct handling of loss of network access
• Verify correct handling of user authentication requests
• Verify correct handling of file access via mapped drive (mapped root drive where supported)
• Verify correct handling of file access via UNC
• Verify correct handling of Print Preview when the network printer is disconnected
• Verify correct handling of long file name and long-file-name-to-short-file-name conversion correctly

Open/Save

• Open/Save from/to Sharepoint Document Library
• Context menu | Open, New, Print
• Drag-and-drop
• Double-click a shortcut on the Desktop
• Most Recently Used list
• Command line argument
• Read-only files, directories, floppies, drives
• Open from/save to over various types and speeds of network connections
• Open from/save to different types of prototols (e.g., FAT, NTFS, UNC, http://, ftp://, floppy)
• Open from/save to previous versions; roundtrip from/to previous versions
• Open/save documents created in a different language-version of the application
• Interrupted
• File/Path name (e.g., long filenames, spaces in name)
• Auto-save
• Save when no documents are dirty, one document is dirty, multiple documents are dirty
• Save when multiple documents are dirty but the user chooses to not save some of them
• Save As

Out Of Memory (OOM)/Stress Testing 

• Low memory
• Low disk space
• OOM caused via automation
• OOM caused via real world scenarios
• Minimum requirements machine
• Flaky network
• Too may users
• Working with floppies
• Aborting operations (e.g., user cancels, network cable pulled)
• Pen And Speech Input
• Speech Tips
• Handwriting Tips (non-TabletPC)
• Microsoft and non-Microsoft Input Method Editors (IMEs)
• Unaware application support enabled and disabled (Windows SP1 or later only)
• Handwriting/speech/IME on Tablet PC Windows

Performance

• Verify perf testing has been done
• Verify perf targets exist and are being met
• Verify the correct data is being benchmarked
• Verify performance with Clear Type turned on
• Verify performance optimizations actually have the intended effect
• Compare performance with previous versions
• Compare performance with similar applications

Platforms

• Run on every supported platform; at a minimum, probably, WinXP SP1, WinXP SP2, Windows Server 2003 Gold

Printing

• Verify changing orientation works properly
• Verify printing to a file works properly
• Verify collation works properly
• Verify "Number of Copies" works properly
• Verify color printing works properly
• Verify canceling printing works properly
• Verify scaling works properly
• Verify line or page numbering works properly (if supported)
• Print to both local and network printers
• Print to Adobe Acrobat to create PDF files

Registry / Configuration File Settings

• Verify settings that should modify behavior do
• Verify the appropriate settings are available in policy templates
• Verify settings roam with the user

Security 

• Consider what buffer overrun attacks might be possible
• Consider what denial of service attacks might be possible
• Consider what SQL injection attacks might be possible
• Consider what virus attacks might be possible
• Verify no NULL DACLS are created or used.
• Verify security for links and macros is sufficient and works correctly
• Verify filenames such as "..\..\..\file" are handled correctly
• Verify any temporary files are created in appropriate locations and have appropriate permissions
• Verify any user-created files have appropriate permissions
• Verify the application functions correctly under different user rights and roles
• Verify user-specified input is handled correctly (e.g., are SQL injection attacks possible)
• Verify every input is bounds-checked
• Verify user privacy is not violated (e.g., file paths are stripped, user aliases and other identifying data can be removed)

Setup - General

• Consider what problems could appear when installing as Minimum, Typical, Install on First Use, Run from Network, Custom, and Not Available.
• Verify registry items are created/modified appropriately, and that other registry items are not created or modified.
• Verify the application works correctly when portions of it have been disabled by an administrator
• Verify any files shared with another feature or application are handled correctly
• Verify installing and uninstalling the application or feature has the correct effect (which may be "nothing should change) on other features and applications
• Verify "normal" users can use the application when it is installed by an administrator
• Verify the correct files are installed when the application is installed via Install On First Use
• Verify mass deployment (e.g., via Systems Management Server) works correctly
• Verify the application works correctly on Terminal Server
• Verify the application works correctly on Virtual PC and Virtual Server

Setup - Local Caching 

• Verify the correct CABs are cached
• Verify any files shared with another feature or appliation are handled correctly by install/uninstall/reinstall
• Verify multiple SKUs share the cache correctly
• Verify deleting a CAB from the cache causes a reinstall and the CAB is recached

Setup - MSI Authoring

• Verify the MSI includes the correct components, files, and registry settings
• Verify custom actions, conditions, shortcut creation, and other special authoring works correctly
• Verify the default installation type (e.g., Typical, Minimum) is correct
• Verify the correct install states are available
• Verify the application works correctly in all possible install states

Setup - Multi-User

• Verify the application works correctly for User2 after User1 installs it
• Verify per-user features must be installed by User2 even after User1 has installed them
• Verify User2's per-user settings do not change when User1 changed them

Setup - Network Setup 

• Does your feature uninstall cleanly when installed from the network (Post-Admin install)
• Verify the correct files are installed when the application is installed via Run From Network
• Verify the application can be uninstalled

Special Modes and States 

• Safe Mode (e.g., booting with /safe to disable some settings) works correctly (if supported)
• Zoomed (e.g., 100%, 25%, 200%)
• Send To
• Sharing documents between multiple users/machines
• Cut, copy, delete
• MDI, SDI
• No file open (just app shell)
• Full screen view
• No command bars visible
• Application and document (e.g., minimized, maximized, sized really small)
• OLE
• Autosave on
• System background color changed
• Multiple selection

Undo/Redo 

• Edit/Undo, Edit/Redo
• Multi-level Undo/Redo
• Menu command text strings

Upgrade and Migration - Application Upgrade

• Verify upgrading over a previous version replaces appropriate files but no others
• Verify side-by-side installs work correctly
• Verify the correct files (and versions thereof) exist after an upgrade
• Verify default settings are correct
• Verify previously existing settings and files are modified correctly by an upgrade
• Verify the application functions correctly when the previous and/or new version are set to Run From Network
• Verify any features or applications dependent on files or functionality affected by the upgrade work correctly after the upgrade

Upgrade and Migration - OS Upgrade 

• Verify upgrading over a previous version replaces appropriate files but no others
• Verify the application works correctly after the upgrade
• Verify any features dependent on OS files or functionality work correctly after the upgrade
• Verify the application works correctly after upgrading both the application and the OS

User Assistance/Help

• Verify each error message is accurate, easily understood, and actionable
• Verify input validation error messages refer to the correct problem
• Verify the tutorial is correct (e.g., the UI it depicts matches the actual UI, the steps it takes are correct)
• Verify all help topics have been reviewed for technical accuracy
• Verify all context sensitive help is correct

Video

• Multi-monitor environments (two, three and four monitors)
• Video resolutions (1024x768 through 1600x1200)
• color depths (8-, 16-, 24-, and 32-bit)
• Color themes

Window Interactions 

• IME
• Z-ordering, especially Help on top
• Modal, modeless dialogs
• Window and dialog focus
• Window size after a restore-from-minimize, restore-from-maximize
• Window size on first launch, on subsequent launches
• Window size after a manual resize
• Multiple windows (MDI)
• Multiple instances of the application
• Arrange Vertical, Horizontal, All, Tiled, Cascade (both MDI children and application windows on the Desktop)

Windows Logo Requirements

• Run all applicable Application Verifier tests (especially check that NULL DACLs are not used or created)
• Install and run under Application Verifier with the “High Version Lie” test enabled
• Install and run with Fast User Switching enabled, while Fast User Switched
• Install and run on Windows 2000, then upgrade to Windows XP
• Install and run in a highly secure Windows environment (“%system%\security\templates\hisecwsinf)
• Run as a limited user
• Verify temp files are placed properly
• Verify System File Protection files are not overwritten
• Verify that My Documents is the initial default location for Open and Save
• Verify all UI inherits Windows XP themes
• Verify files outside of the application folder are associated with a particular file type (i.e., double-clicking does the right thing)
• Verify all executable components have a (correct) digital signature
• Verify all UI is legible and accessible at 133DPI (high DPI)

Year 2000 

• Verify dates entered with a two-digit year are interpreted as:
•  1/1/00 through 12/31/29 are interpreted as 1/1/2000 through 12/31/2029
•  1/1/30 through 12/31/99 are interpreted as 1/1/1930 through 12/31/1999
• Verify dates on leap years are correctly interpreted (e.g., 2/29/2000, 2/29/2004)
• Verify dates at least to 2035 are supported

 

[29 Mar 2010: An expanded and annotated version of this is available on my website at http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf.]