Using WLK for testing storage devices, and other thoughts on storage testing.

Disk IO test fails on disks with free space more than 2 tera bytes

bhanug — Sat, 12 Jul 2008 03:13:00 GMT

The Disk IO test fails on the disks with free space more than 2 tera bytes. It fails with an error code 0x06 and error message “The handle is invalid”. The only workaround for this problem is to reduce the disk space to less than 2TB. This test issue will be fixed in the future WLK releases.

Optical Drive Tray status Implementation

bhanug — Tue, 08 Jul 2008 04:36:00 GMT

The WLK 1.2 version of the Optical Logo test verifies if the drive reports the following sense codes properly

DESCRIPTION	SENSE KEY	ASC	ASCQ
MEDIUM NOT PRESENT	0x02	0x3A	0x00
MEDIUM NOT PRESENT – TRAY CLOSED	0x02	0x3A	0x01
MEDIUM NOT PRESENT – TRAY OPEN	0x02	0x3A	0x02

The test expects the drives to report the tray status in the sense code. The test would fail if these sense codes are not properly reported.

There is an exception to this requirement for the slot type drives. The slot type drives would not have tray so they will not be able to report tray status in the sense code. But the slot drive must report the media loading mechanism type as 0 in the Removable medium feature. The WLK 1.2 version of the optical logo test is not considering this value. But in future, the test would depend on this value to skip the tray status verification test cases for slot type drives.

The following are the loading mechanism type values from MMC5r4.

Mechanism Type	Description
000b	Caddy/Slot type loading mechanism
001b	Tray type loading mechanism
010b	Pop-up type loading mechanism
011b	Reserved
100b	Embedded changer with individually changeable discs
101b	Embedded changer using a magazine mechanism
110b – 111b	Reserved

The WLK 1.2 version of the optical logo test causes timeout failures for slot type drives, because it relies on the sense code to wait for the tray open/closed operation to complete. The slot type drives will not be able to report the tray status in the sense code. So, the test would cause a time out failure after waiting for 20 seconds. The following errata filter would cover those failures

https://winqual.microsoft.com/EC/ECDetails.aspx?id=1172

Optical Logo Test – Drive Serial Number test case

bhanug — Tue, 08 Jul 2008 04:33:00 GMT

The WLK 1.2 version of the optical logo test would need at least two identical test drives to validate if the drive reports unique serial number. If the test drive is a Read only drive then it would need two identical test drives and one writable drive. The writable drive is to burn test data for verifying the read functionality of the read only test drive.

The optical logo test verifies if the identical test drives report unique serial numbers. But there is a test issue that would cause the test case to fail. The test case failures are covered by the following erratas.

Filter ID: 1254

https://winqual.microsoft.com/EC/ECDetails.aspx?id=1167

Filter ID: 1255

https://winqual.microsoft.com/EC/ECDetails.aspx?id=1168

This test case would be fixed for the next release of the WLK.

RAID adapter design recommendations

bhanug — Wed, 02 Jul 2008 02:37:00 GMT

If you are not aware, there is Whitepaper available at WHDC, that describes how to design RAID Adapters and drivers to work well with Windows and meet the Windows Logo requirements. The White Paper can be found at

http://www.microsoft.com/whdc/device/storage/RAID_design.mspx

Optical Logo Test failures related to SATA bus data transfer size requirement

bhanug — Fri, 11 Apr 2008 00:40:00 GMT

The SATA specification does not support data transfers of odd number of bytes. The 10.3.11 section in the SATA specification 2.6 discuss about this issue. But the optical logo test in the WLK 1.2 did not considered this in validating the test cases. So, the following test cases might fail on a SATA optical device

Inquiry test case

Mode Sense test case

These test cases validate the Inquiry/Modesense SCSI command with random allocation lengths. This is to make sure the device handles the command properly with any allocation length. But the generated random allocation length might be an odd number and that data transfer size will not supported by the SATA protocol.

Serial Number test case

This test case uses the Inquiry command to verify the drive serial number. It sends the Inquiry command with allocation length of “5” to get the additional length required to get the complete Inquiry response. The Inquiry command will fail because the requested data transfer size, "5" is an odd number of bytes.

Optical logo test - Write Speed Measurement

bhanug — Wed, 09 Apr 2008 01:33:00 GMT

The following description might help you understand the test logic being used in the optical logo test to validate the drive write speed.

The test queries the device and gets the supported write speeds for the test media. Then it will try to validate the write scenario for all the reported write speeds. The test expects the calculated write speeds for each write scenario is close to the write speed set to the drive. The test allows 10% tolerance for the measured write speed. The test validates the write speed as below:

The test writes data to the media and monitors the amount of data being written. It excludes the timing for the initial 10% written data the final 10% written data. It measures the timing only for the in between 80% data and calculates the speed for that data and adds 10% tolerance to that value. So, if ‘x’ is the calculated the speed for the middle 80% data then the final calculated speed is 1.1x. The test expects this value to be greater than or equal to the speed set to the drive before beginning the write.

| ß Initial 10% excluded à | ß Write speed measurement (80%) à | ß Final 10% excluded à |

(x)

The test excludes the initial 10% of data written timing to allow the drive to prepare for writing. The final 10% data timing is excluded to allow the drive to do the proper cleanup for the data written to the media.

Storage Query Property test

bhanug — Tue, 08 Apr 2008 00:57:00 GMT

The storage query property test validates if the storage miniport driver is reporting the correct bus type and the storage controller is reporting the correct PCI class code.

The test sends the IOCTL_STORAGE_QUERY_PROPERTY with STORAGE_PROPERTY_ID set to StorageAdapterProperty/StorageDeviceProperty to get the device bus type from the storage miniport driver. The miniport driver should report the correct bus type from the following bus type enumeration values

typedef enum _STORAGE_BUS_TYPE {

BusTypeUnknown = 0x00,

BusTypeScsi = 0x01,

BusTypeAtapi = 0x02,

BusTypeAta = 0x03,

BusType1394 = 0x04,

BusTypeSsa = 0x05,

BusTypeFibre = 0x06,

BusTypeUsb = 0x07,

BusTypeRAID = 0x08,

BusTypeiSCSI = 0x09,

BusTypeSas = 0x0A,

BusTypeSata = 0x0B,

BusTypeMaxReserved = 0x7F

} STORAGE_BUS_TYPE, *PSTORAGE_BUS_TYPE;

The documentation for these bus type values can be found at: http://msdn2.microsoft.com/en-us/library/aa363465.aspx

The test gets the storage controller PCI class code from the OS generated PNP compatibility IDs and determines the controller bus type based on the PCI class code. The test verifies if the bus type reported by the miniport driver matches with the bus type for the storage controller PCI class code. The test also verifies if the user supplied device bus type in the DTM submission matches with the corresponding bus type for the controller PCI class code.

The test uses the following PCI class code table to determine the storage controller bus type

CLASS CODE	SUB CLASS CODE	PROGRAMMING INTERFACE	DESCRIPTION
0x01	0x00	0x00	SCSI CONTROLLER
0x01	0x01	XX	IDE CONTROLLER
0x01	0x02	0x00	FLOPPY CONTROLLER
0x01	0x03	0x00	IPI BUS CONTROLLER
0x01	0x04	0x00	RAID CONTROLLER
0x01	0x05	0x20/0x30	ATA CONTROLLER
0x01	0x06	0x00/0x01	SERIAL ATA CONTROLLER
0x01	0x07	0x00	SERIAL ATTACHED SCSI (SAS) COTNROLLER
0x01	0x80	0x00	OTHER MASS STORAGE CONTROLLER

The controller bus type validation testing will continue to be a key test scenario in the Storage HBA category, but it will be modified to stop checking the Programming Interface byte (IF), of the 0x0101 (IDE) and 0x0104 (RAID) types to determine the bus type.

This results in being the following required class code definitions that must be present:

IDE Type: Mass Storage Device (0x01), ATA Parallel - IDE Type (0x01), Programming Interface (Don’t Care - 0xXX )

RAID Type: Mass Storage Device (0x01), RAID Type (0x04), Programming Interface (Don’t Care - 0xXX)

However, because the PCI Conventional specification (2.3) declares two unique values for the SATA Type (Non-ATA Compatible Mode) Programming Interface, the following is required for AHCI-enabled controllers:

AHCI Type: Mass Storage Device (0x01), SATA Type (0x06), AHCI Type (0x01)

The SATA controller must report 0x0101XX in IDE compatible mode, 0x010601 in AHCI mode and 0x010600 in vendor specific implementations.

MPIO test – WLK 1.1

bhanug — Sat, 08 Mar 2008 06:55:00 GMT

The MPIO test documentation has incorrect information in the software requirements section. Here is what the documentation says:

Software Requirements:

1. Windows Server 2008

2. Software components included with the device that is being tested.

3. Unsupported Software: Third Party MPIO Solution that does not use a Microsoft MPIO DSM.

The 4^th requirement is misleading the users to assume that, the MPIO test is not applicable for logoing the third party DSMs. But this is a required test for 3^rd party DSM submissions. We reworded this software requirements section as below to avoid confusion.

Software Requirements:

1. Windows Server 2008

2. Hardware Storage RAID array that uses either the Microsoft DSM (device specific module) or a 3rd party DSM that interfaces to Microsoft MPIO core architecture

3. Software components included with the DSM or Hardware Storage RAID array being tested

You may see this change in the WLK 1.2.

Storage submission wizard page - WLK 1.1

bhanug — Sat, 08 Mar 2008 03:51:00 GMT

The storage submission wizard page in WLK 1.1 doesn’t display device bus type combo box properly. The combo box is clipped and it will appear as an edit box. The users will notice this bug for disk drive, removable storage device and medium changer submissions. If the user enters some data in that field, that data will not match with the UI data. The user will get an error message “Please complete all fields. Field:Storage_bus_type”.

The bus type or drive type controls appear as edit boxes, but internally they are combo boxes. To work around this problem the user has to select these controls and use the up/down arrow keys to choose an item from those hidden combo boxes items.

Understanding the Crashdump Test and debugging failures

Eric StJohn — Tue, 09 Oct 2007 21:23:00 GMT

The crashdump test is part of all of the Storage > Adapter or Controller program and verifies that the storage adapter driver is able to produce a crashdump. The purpose of a crashdump file is to analyze the state of the system in the event of a non-recoverable error, also known as a crash, stop error, kernel error, or bugcheck. In order to diagnose the crash the operating system creates a dump file. Microsoft Windows supports three different levels of dumps: a complete dump, a kernel dump, and a minidump. All dumps contain header information indicating the type of dump, the nature of the error, and other data added by drivers in the system (Bugcheck callback data). The complete dump contains the entire contents of physical memory on the system, the kernel dump contains only the contents of physical memory corresponding to the operating system and omits user mode applications and unallocated memory, and the minidump contains only a list of loaded modules and information about the active process and thread that caused the crash.

In the case of a crash, there is no certainty of what parts of the operating system will still be functional. The network or file system drivers may have caused the crash, for example, preventing access to file system structures to create a dump file, or network to store the file remotely. The OS handles this by using a file that it already knows exists (the pagefile) and writes directly to that file's logical block extents on disk. The dump process writes the contents of physical memory into the pagefile on the system disk (usually c:\pagefile.sys). The pagefile must be large enough to contain the dump. The largest dump is a complete dump which requires the size of physical memory (ex: 4096) plus one extra megabyte to contain the header information. The test requires that the user configure the page file to an appropriate size before execution (see http://support.microsoft.com/kb/314482). If the page file size is insufficient, the test will log the following error during the initialization phase:

(i) Verifying paging file size.
(x) Paging file size is too small for full dump purposes.
(i) Paging file size: 330989568
(i) Physical memory size: 1073094656
(i) Please configure minimum paging file size to physical RAM size + 1MB.

After some basic settings validation the test will install a driver used to crash the system and reboot the system. After the reboot the test changes the crash control settings (for full memory dump), deletes any old dump files, and crashes the system. At the point of the crash the system will display a bugcheck screen (blue screen) with details of the nature of the crash. The type of bugcheck should be MANUALLY_INITIATED_CRASH (e2). If anything else appears here it means a second bugcheck occured during the process of writing the dump file. This should be investigated by connecting a kernel debugger to the test client and debugging the storage adapter driver. After the dump file has been written the test machine should automatically reboot.

Upon boot after a crash the operating system will detect the presense of dump inforamtion in the page file and begin the process of writing a dump. This process occurs asyncrhonously while the machine is booting and even after the user has logged in (see http://support.microsoft.com/kb/886429 for details). During this process you can view it's progress by checking the size of the dump file (C:\windows\memory.dmp) or viewing the process in task manager (werfault.exe). The test will often be running at the same time as this process, trying to access the same dump file. If this occurs the following messages will appear in the log:

(i) Connecting to DumpFile: C:\Windows\MEMORY.DMP
(i) Dump file is being used by another process. HRESULT: 0x80070020
(i) Usually memory.dmp is still being written due to large RAM, retry after 5 minutes.

The test should then retry accessing the file. If the error message code is different, or if it changes (ex: 0x80070002 : ERROR_FILE_NOT_FOUND) then it means the file could not be written to disk. The first place to check for valuable debug information is the system event log. To view the event log click Start > Run > Compmgmt.msc. In the computer management window, select Computer Management\System Tools\Event Viewer\System. Browse through the list of events for an event with the source BugCheck. The most common cause for a missing dump file is insufficient free space on the disk. As noted in the KB article referenced above, the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\MachineCrash contains information about the last crash (before a reboot), including the partial dump file if one was created. This can be useful when trying to debug other missing dump issues.

Upon reboot the test examines the dump file for correctness. The test does this, just as a developer would, using the kernel debugger kd ("Debugging Tools for Windows" package). In order to analyze a dump file the debugger needs access to symbols. I won't go into details on why these are needed (more information is available here), but think of these as a dictionary for the dump. They allow the debugger to analyze the contents of memory (or a crashdump file) into individual modules (executables, libraries, drivers, etc), functions within those modules, and data structures.

For the test to work properly, it needs to provide the debugger with symbols. When it does not have proper symbols it will log warnings during the log failures during the first analysis phase of testing. The current mechanism for doing so is to have the user download public symbol packages (here) and install them on the test machine before running the test. When symbols are not installed or the symbols do not match the operating system under test, the following message may appear in the test log:

(x) Failed to load the correct symbols.
(i) Please refer to WDK documentation on how to install OS symbols.
(i) Your symbols might also be out of date, please update symbols via the symbol server: http://support.microsoft.com/kb/311503

This won't actually cause the test to fail because in some cases the dump can still be analyzed with partially matching symbols. If the test continues and more test cases fail with messages like "Error retrieving addresses of ..." or "Unable to get..." it means that the debugger cannot analyze the dump due to the missing symbols. We understand the difficulties associated with downloading the public symbol packages and ensuring they are up to date with all of the latest hotfixes on the system. One way we have found to work around a symbol is to supplement the locally installed symbol packaged with symbols cached from an internet symbol server. The test could use the symbol server directly, but it is not advised to have the test machines connected to the internet when under test. The steps for caching the symbols locally are as follows:

Ensure that you have already created a crashdump. The easiest way to do this is to run the test once and let it fail.
Ensure that you have the debugging tools installed. Again the easiest way to do this is to run the test once. It will install the tools to C:\Debuggers.
Open a command prompt (in Vista right-click and choose run as administrator).
Type in the following command, without the quotes: "c:\Debuggers\kd -z c:\Windows\MEMORY.DMP -y SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols"
This will load the dump in the kernel debugger using the remote symbol store at microsoft and the local directory C:\Symbols as the downstream store to cache the symbols.
Make sure that symbols can be found for OS files like NTOSKRNL and NTDLL. These are necessary to analyze the dump. It is ok if errors appear loading symbols for other modules like 3rd party drivers.
You should now have a prompt "0: kd>". At this prompt type the command ".reload /f" without the quotes. This command forces the debugger to load and cache all symbols for modules loaded in the dump.
You can now exit the debugger with CONTROL+B then enter.

Documentation for Preview tests released in DTM 1.0a

Eric StJohn — Tue, 08 May 2007 22:33:00 GMT

In the DTM 1.0a release we replaced SDStress with Disk Stress, Data Verification with Disk Verification, and ExtInt13 with Boot Test. We also added the Query Property test to the Storage Adapter program. The help documents for these tests did not make it into the DTM help file so I have posted them here for those of you trying out DTM 1.0a:

Boot Test

Disk Stress

Disk Verification

Storage Query Property Test

Updated version of SCSI Compliance test (1.3.6 H was 1.3.6 G was 1.3.6 F)

Eric StJohn — Tue, 05 Dec 2006 06:54:00 GMT

An updated version of the SCSI Compliance Test has been posted on https://connect.microsoft.com. You must be a member of the Storage Logo Program Beta Users program to download. You can join this program by finding it in the Available Connections section. If you do not see the program, please send me an email by clicking email on the left of the page with an explanation of why you need to join and I can grant approval.

1.3.6 F: This version of the test is not currently supported for logo submissions. You can work around this error for now by using a basic disk for your system partition. The remaining disks must be dynamic. Place all of the test partitions on the non-system (dynamic) disk(s). If the setup documentation calls for a dynamic volume with extents on more than one disk, and only one dynamic disk is available, you may downgrade this volume to a simple volume.

1.3.6 G: This version of the test is not currently supported for logo submissions. This version fixes a bug in the REPORT LUNS test case where we were unable to find LUN 0 for certain MPIO targets. You can work around this problem in previous versions of the test by ensuring that disk you run the test on is on the same Port/Bus/Target as LUN 0 on the same physical target.

1.3.6 H: This version of the test is not currently supported for logo submissions. This version fixes a bug in the INQUIRY test case where the test will crash if it encounters a VPD page 83 descriptor encoded in UTF8. If you are encountering this failure in a logo submission, please contact me for further information.

Can DVD readers report DVD+R, DVD+RW as supported profiles?

bhanug — Sat, 02 Dec 2006 02:03:00 GMT

Our interpretation is that the drive should not report a profile if it cannot recognize the blank media for that type. If the DVD reader is a capable of reading from the recorded DVD+R, DVD+RW media then it can report DVD+R feature and DVD+RW feature as supported features but not the profiles for those media types. The MMC committee also agrees that the DVD drives should not report DVD+R and DVD+RW profiles as supported profiles if they are not capable of writing to those media types.

The MMC test fails on these types of drives because the drives are not capable of recognizing the blank media. The test expects the drive to recognize the blank media if they report it as a supported profile. The MMC spec doesn’t clearly address this issue, so we are issuing erratum to cover the test failures. This erratum will be removed when this issue is addressed in the next revisions of the MMC spec.

Testing Serial ATA Hard Disk Drives and Serial ATA Optical Drives in DTM

Eric StJohn — Mon, 27 Nov 2006 23:39:00 GMT

A few issues have come up lately when testing Serial ATA devices and I wanted to share the problems testers are hitting.

SATA-IO Sig Test is failing.

Solution: Download and install the latest errata filters from http://winqual.microsoft.com/member/SubmissionWizard/LegalExemptions/filterupdates.cab. Re-save the submission CPK.
This test is designed to validate the Windows Logo Program Requirement STORAGE-001 requirement for SATA Devices:
"All SATA devices must meet the requirements of the Serial ATA: High Speed Serialized AT Attachment, Version 2.1, (Version 2.5 is recommended and will be required for new devices on June 1 2007) by obtaining a Serial ATA International Organization (SATA-IO) logo certification."
The test is not required to pass until June 1, 2007 as stated in the requirement. The filter applied to the submission will change any failing result for the test in to a pass. This filter will automatically expire on June 1, 2007, and no longer turn the test into a pass.
In order to be prepared for this test, Serial ATA hardware vendors should start testing with SATA-IO NOW. More information is available on the Serial ATA Interoperability website.
SATA-IO will give hardware vendors a digital certificate for their device once it has passed their testing.
To run the SATA-IO Sig test the user provides the certificate file by placing it (an xml file) on the client machine and then providing the path of the file to DTM by right clicking the job in device console and choosing "Add information".

SCSI Compliance is appearing!

Important: This test is not required for SATA devices, but is appearing because of a problem in the hardware setup of the test machine.
Solution: Connect your Serial ATA device to a ATA/ATAPI class host controller that correctly identifies the bus type of its devices. This is the only supported means for testing Serial ATA hard disks and optical drives.
You can tell if a Serial ATA host controller is an ATA/ATAPI class controller by looking in Device Manager. ATA/ATAPI class controllers appear under the IDE ATA/ATAPI controllers category with the following icons:

(Vista)
(XP & Server 2003)

Many controllers offer a bios configuration page for changing the mode in which they operate. If possible set the controller into IDE or AHCI mode. This may change the driver used for the controller to an ATA/ATAPI class controller driver.
In order to effectively test Serial ATA devices we must send ATA commands to the device. SCSI/RAID class host controllers have no mechanism for doing this.

Tests that use ATA commands in SATA Device logo programs:

SATA-IO Sig Test
Piton Disk Test (Hybrid Hard Drive Test)
Media Status Notification Test
Media Lock Test

In addition to providing no means for testing ATA commands, SCSI/RAID class host controllers will often incorrectly identify the bus type of the devices they expose. This is why SCSI Compliance is appearing.

SATA RAID controllers will often return BusTypeRaid for devices, even though the device is not part of a RAID array.
Many SATA controllers just return a completely incorrect bus type (IE: BusTypeScsi). This is a violation of Windows Logo Program requirements and a test is being added to prevent these devices from getting a logo.
We recommend using AHCI controllers because we know that the MSAHCI driver reports the correct bus type.

If you are having trouble finding an ACHI controller with eSATA connections you could use an adapter such as this.

Storage Presentation Video

Eric StJohn — Mon, 13 Nov 2006 22:28:00 GMT

Video: Storage Logo Testing