scooblog by josh ledgard

Getting a bit more specific I figured I’d post something that my team created as a helpful resource for people building test plans.   This document started as a one page guide, and turned into a five page list of interesting tests as it passed from tester to tester in the developer division.  Please don’t consider this a comprehensive list and assume we covered everything.  It was and is only meant as a starting point to help get the creative juices flowing around building your test plan.  The more people that see it the more good examples we’ve added so feel free to contribute to any gaping holes in this guide!

Thanks,

      Josh

The guide is broken out by the different resource types that a program might depend on.

File Tests: Does your code rely on files?

• Remove (rename) a file the application is dependant on (For example in Visual Studio the ToolBox depends on a .tbd file that is stored in the users/appdata dir. What would happen if we deleted that file?)
• Corrupt files – what happens if the file doesn’t have the expected structure or contents? For example, VS Start Page expects that a custom tab xml file will comply with the schema and will be correctly formatted xml, what happens if it doesn’t or instead of xml we have a simple text file or a binary file?
• Bad Data – what if the file is in the expected format, but the values of the data that we get from the file are invalid. Wrong data type, out of value bounds, containing invalid chars, etc… see string, numeric, and file path tests below.
• Does your feature expect that a file would be ASCII? What happens if instead it had a different encoding say Unicode or UTF8?
• Does the application expect that the file it reads from will not exceed a certain size? What happens when we try working with big files? While you are doing this, what if the file is 0 length?
• Try using a file that is used by or in use by another process / user
• What happens if the application depends on a file but the permissions set on the file don’t allow it to access the file? Try security vs. read/write/execute permissions. What if the file is just hidden?
• Fill the disc (use floppy for example) and do reads/writes open/close app. You can use tools to emulate disc full conditions. Canned Heat is a good tool for that. Use FileMon to identify file system access.
• Errors when accessing media – these can happen if the hard drive / floppy / cd-rom, etc… are not available or slow. Use Canned Heat to emulate the errors / delays. What happens if you can access the media at first but it becomes unavailable while the application is doing work? Just pop out the disc you are writing to or disconnect from the share you were using and see what happens.
•  Windows 2000 and above allow junction points. It’s basically a link from one point on disc to another. If your features do any type of recursive dir walking you should try creating a loop and see how your feature handles it. For example, say you map c:\myproject\linkedfolder to c:\myproject and you have a file foo.txt in c:\myproject that contains the word foo in it. Now, if you do “Find in Files” from Visual Studio starting from c:\myproject and searching for “foo” you’ll find 14 matches instead of 1… You can use linkd.exe to create junction points.

File Paths: Are file paths ever given as an input to your program?

• Invalid file name (invalid characters etc). Make sure your feature uses the OS to handle invalid file names, rather then writing its own interpretation of that code.
• File name/path longer than the allowed max path. Be aware of anytime your feature adds something to the file name specified by the user. What if the user file path was already Max length and we still added .aspx extension to the end of the path? BO and crash most probably…
• Valid Longest File Name
• File name with spaces, dots, semicolons
• Using reserved names such as COM1, AUX to test any functions that create or open files.
• The list of reserved names: CON, PRN, AUX, CLOCK$, NUL, COM1-COM9, LPT1-LPT9
• Varied filenames, using all OS allowed characters ( `output^%$#@#.,!@#$%^)(& is a valid filename)
• Does your feature depend on the name of the file it uses? Try same file name canonical representations - trailing white spaces, .\foo = foo, short file format aka 8.3 format -<6chars>~<number>.<extension> (see Canonical Representation issues from "Writing Secure Code" for many more examples)
• Paths of type - \\?\, file://?
• Try saving/opening etc… by specifying a dir path rather then a file path.
• What is the default path that your features use when the user tries to open/save/find etc… Does that default make sense? Are we changing the default according to the path the user navigated to last time?

Input from registry: Use the registry to store/retrieve information?

• Keys deleted (removed from system)
• Registry Data Type changed (delete key, create same named key with different data type.)
• Change ACL on folder that contains the key. (read only, can’t set value, etc ) For example: Remove key and then remove the right to create a key, see how application responds.
• Delete folder that contains the key.
• Data Contents changed (see API tests below)
•  Make sure no user specific data is written in the HKLM tree (and vice versa.)
• You can use RegMon to find the registry keys that your features are using.

Strings: Most apps have at least one location where strings are used as an input.

• Null String
• String without ‘/0’ (in C)
• Extended ASCII chars (e.g. Alt-0233)
• Reserved strings or sequences that require special treatment.
• Strings that mean something to the underlying code – for example, “++”, “&”, “\n” (c++)
• Special ASCII and UNICODE chars – for example CTRL characters
• Reserved Device Names – AUX, COM1
• A good table enumerating special/troublesome ASCII chars can be found on page 29 of “How to Break Software” by James Whittaker.
• Using long strings that reach the maximum limit and over the limit of any functions that have string parameters.
• White space, carriage return, tab chars
• International Strings - lots of things you can (need to) do here that I won’t get into details on. Enable INTL character sets on your machine and have at it. Most INTL strings tend to take up 30% more space than the ENU versions. Look for overlaps and poor UI in your LOC versions.
• You can sometimes get around various UI verifications by pasting the text rather then typing it.
   

Numeric Values

• Negative values
• Zero
• Nothing (in VB, where applicable)
• Least numeric value for type
• Greatest numeric value for type
• Does your numeric value input have boundaries (from 1 to 50)? If yes test those. - test at least 1, 50, 0, 51, and, of course valid inputs.
  Different Data Types (for example if the input expects int enter real number or a character)

Inputs to Web Applications

• All previous tests apply here as well
• Escape sequences not allowed or checked to ensure they do not allow something malicious to render (There are 4 ways to represent ‘\’. ‘\’ = %5C = %255C = %%35%63 = %25%35%63 )
• HTML Encoding check where applicable: ‘<’ = < = &#x3C = &60
• See http://www.w3.org/TR/REC-html40/sgml/entities.html
• See Chapter 12 of Writing Secure Code for more checks

Network Connectivity Tests

For all features that need network access. You can control the network speed/access by using tools like Canned Heat.

• What is the behavior of your features when the network is slow?
• What happens when the network is down / goes down at runtime?

Things to check for: hangs, crashes, bad error messages.

General Advice:

1. Try to get a list of all the errors that the code is supposed to handle and try causing all of them at least once.

a.     Are the invalid inputs handled correctly?

b.     Are there any cases that you think the code should be handling, but you don’t see an associated error message? Test those…

c.     Are all error messages easy to understand and descriptive?

2. Try using the default values or blank inputs (where applicable)

d.     Ask yourself whether the defaults are reasonable. Is that what the user would want to have as the default value?

3. Try strings that have special meaning to the OS, Underlying Programming Language or are special for the corresponding char set

4. Test for buffer overruns.

5. Make sure that your feature doesn’t allow code/script to be injected through input. Online features and features using SQL are especially susceptible to this kind of problems.

6. Test repeating the same input or series of inputs many times.

7. Don’t test input only as individual fields, find related inputs and test interesting combinations. For example if you had month and date fields, test February and -1, 0, 28, 29.

8. Think about all the possible outputs that some input can result in. Try to cause all of those to happen, by changing the context/environment in which the code executes. For example, when trying to read a file from disc some possible outputs are:

e.     File is not found

f. We could not read the file - File is corrupted so we can’t read it, No permissions to read the file, File is locked so can’t read

g.     Successfully read the file

h.     The program hung because the file was too big

REFERENCES / RESOURCES:

1.     Writing Secure Code

2.     How To Break Software

3.     Developing International Software

Tools:

1. Canned Heat – http://se.fit.edu/cannedheat

2. FileMon - http://www.sysinternals.com/ntw2k/source/filemon.shtml

3. RegMon - http://www.sysinternals.com/ntw2k/source/regmon.shtml