Proposed Quality Acceptance Tests for Windows CE Network Interface Cards

Disclaimer
This document is provided for informational purposes only and Microsoft makes no warranties, either express or implied, in this document. Information in this document, including URL and other Internet Web site references, is subject to change without notice. The entire risk of the use or the results of the use of this document remains with the user. Unless otherwise noted, the example companies, organizations, products, people and events depicted herein are fictitious and no association with any real company, organization, product, person, or event is intended or should be inferred.

Audience:
This is targeted to user or group or organization who wants to test NDIS miniport driver for Windows CE .NET devices using Microsoft® Windows® CE 5.0 Test Kit.

Note about acronyms & terminology used in this paper:
NIC acronym is used for Network Interface Card. CETK is Microsoft® Windows® CE Test Kit & the version that is referred here is Microsoft® Windows® CE 5.0 Test Kit. More information for CETK can be found out at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conWindowsCENETTestKit.asp.

Summary:
This paper outlines a group of tests that should be run on a network interface card in any Windows CE-based device to verify that it meets a certain level of quality. This paper defines the test cases and points to online help provided in the CETK for details on how to run the tests and/or how to configure the setup needed to run the tests. The tests explained below are run using Microsoft® Windows® CE 5.0 Test Kit unless otherwise specified.

The information given here is applicable to following types of NICs:
<1> Wired LAN NIC. Supported media include NdisMedium802_3, NdisMedium802_5, and NdisMediumFddi.
<2> Wireless LAN NIC (802.11)
<3> RNDIS Function
<4> NDIS Intermediate driver

The information given here is not applicable to following types of NICs:
<1> Bluetooth device
<2> FIR/IRDA device
<3> ATM device, Modems/Cable modems, DSL device
<4> Wireless WAN devices (CDMA/GPRS)


Proposed Quality Acceptance Criteria

In order to ensure that the miniport driver for a NIC qualifies as Production Quality Driver, the following tests should be run and the miniport driver should meet all the listed requirements. All of the following tests are present in the CETK shipped with Microsoft® Windows® CE 5.0.

General Pass Criteria:
All the tests cases should pass unless otherwise specified. In addition to that general criteria, there should not be any memory leaks or exceptions due to the miniport driver under test during or after the test runs. A temporary hang may occur while tests are being run but it should not be permanent/persistent and the system should fully recover once the test is over. If you encounter problems with any OS components or with the tests themselves while testing your driver, it would be greatly appreciated if you could gather up a detailed description of what tests were being run, where the failure occurred, and how the hardware/OS were configured. Please either pass this data on to Product Support Services or post it on the newsgroups microsoft.public.windowsce.testtools.

<1> NDIS One Card Test
More information can be found out at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conOneCardNetworkCardMiniportDriverTest.asp.

This test suite assesses all the functionalities of a network card that can be checked by using just one test network card. Please run through all the cases under the test suite.

By default test case 8 'Fault handling' gets skipped. If your miniport driver can be unloaded using IOCTL_NDIS_DEREGISTER_ADAPTER then you should try test case 8. On board NIC or PCI or ISA based NIC can be unloaded using IOCTL_NDIS_DEREGISTER_ADAPTER. But if NIC is PCMCIA/CF and is based on legacy PCMCIA client driver then IOCTL_NDIS_DEREGISTER_ADAPTER will not work and that test case will fail.

If your Windows CE–based device does not support automatic resume using SYSINTR_RTC_ALARM or if it does not support IOCTL_HAL_ENABLE_WAKE IOCTL then test case 11 'Suspend and Resume' will get skipped. If this test case is skipped and your Windows CE-based device supports suspend and resume operation, then you should do a manual suspend/resume to ensure that miniport driver is still functional on resume.

Pass criteria:
All the tests cases should pass.

<2> NDIS Two Card Test or NDIS Two Card Test on Two Machines
More information can be found out at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conTwoCardNetworkCardMiniportDriverTest.asp.

This test suite needs another network card (in addition to the card being tested) called 'support card' to be used as a control channel. If your Windows CE-based device can not have more than one card then you will need to run NDIS Two card test in two machines; it is same test as NDIS Two card test except the support card is on another Windows CE-based device. To repeat, you need two Windows CE-based devices to run NDIS Two Card test in Two Machines. Also if test NIC is a wireless card then you can not run NDIS Two Card Test and you will have to run NDIS Two Card Test in Two Machines. Please run through all the cases under the test suite.

For more information on NDIS Two Card test in Two Machines please refer to
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dncenet/html/ndistestTestingNDISMiniportDriversByUsingWindowsCENETTestKit.asp?frame=true.

Pass criteria:
All the tests cases should pass.
It is acceptable if 2c_MulticastReceive fails when the NIC hardware and miniport driver indicates the multicast packets up the stack that were sent to multicast addresses not set in the multicast address list of the miniport driver. If NIC hardware and its miniport driver fails to filter out unwanted multicast packets then the protocol layers like TCP/IP will filter them out. Note this isn't desirable behavior, but it will work.


<3> NDIS Two Card or Two Card in Two Machines Test for Media Sense
If your NIC is RNDIS Function or wired NIC but does not support media sense then skip this test. Otherwise you have to run this test to verify that your miniport driver (important for wireless NICs) supports media sense.
Note:Currently, you can not run this test using CETK. You have to use Microsoft® Windows® CE Platform Builder since the test needs user intervention while running the test.
Please run through all the cases under the test suite.

Setup and Configuration
This test needs the same setup and configuration as is required for normal NDIS Two Card Test or NDIS Two card Test in Two machines. For details, please refer to
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conTwoCardNetworkCardMiniportDriverTest.asp

Running the Test
This test gets run in the same fashion as normal NDIS Two card Test or NDIS Two Card Test in Two Machines except that you need to use Windows CE Platform builder to run the tests on the Windows CE-based device.

While running the tests, please pay attention to the debug output in Platform Builder. Whenever the test's debug output indicates to Disconnect, take out the network cable from the test NIC on the Windows CE-based Test device. If the NIC is wireless then remove the power from the Access Point to which both the Test NIC and Support NIC are connected. Whenever the test's debug output indicates to Connect plug the network cable back in (wired) or power on the Access Point (wireless).

Command lines to run
In CE Target Control Window of Windows CE Platfrom Builder for the device under test, run the following commands once the setup is ready.
<a> Two Card Test
s tux -o -d ndt_2cm.dll -c "-t <Test NIC instance> -s <Support NIC instance>".
<Test NIC instance>: name of instance of Test NIC on Windows CE-based device
<Support NIC instance>: name of instance of support NIC on same Windows CE-based device.
Ex: s tux -o -d ndt_2cm.dll -c "-t PCI\RTL81391 -s PCI\RTL81392".
OR
<b> Two Card Test in Two Machines
s ndtserver

s tux -o -d ndt_2cm.dll -c "-t <Test NIC instance> -s <Support NIC instance>@<IP addr".
<Test NIC instance>: name of instance of Test NIC on Test Windows CE-based device
<Support NIC instance>: name of instance of support NIC on Support Windows CE-based device.
<IP addr>: a IP addr on Support Windows CE-based device.

Ex: s tux -o -d ndt_2cm.dll -c "-t CISCO1 -s CISCO1@11.22.33.44"
Possible Issues:
If the data transfer portion of the test fails for a wireless NIC, then try re-running the test with the -msewait nn switch, where nn is the number of minutes the test should wait for the Test NIC to reconnect to the Access Point. (ex. s tux -o -d ndt_2cm.dll -c "-t CISCO1 -s CISCO1@11.22.33.44 -msewait 10"). The default wait is 1 minute.

Pass criteria:
All the tests cases should pass.
 

Summary for NDIS Tests
The NDIS One card tests and NDIS Two Card tests (or NDIS Two Card Test in Two Machines) verify the following:
<a> Miniport is functional
<b> Miniport follows NDIS Specs.
<c> Miniport can withstand stress test; sending, receiving and attending OID Queries.

For more information on how to run NDIS test on various Windows CE-based devices please refer to
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dncenet/html/ndistestTestingNDISMiniportDriversByUsingWindowsCENETTestKit.asp?frame=true


<4> Winsock Stress Tests
Although the NDIS One Card and Two Card tests fulfill most driver verification needs, Winsock stress tests can still sometimes find bugs. The following tests cases should be run using the Perf_winsock tool in the CETK
<a> Perf_winsock test for TCP send throughput
<b> Perf_winsock test for TCP recv throughput
<c> Perf_winsock test for UDP send throughput
<e> Perf_winsock test for UDP recv throughput

Edit the command line for Perf_winsock in the CETK UI exactly as follows to help you to run through all the four test cases defined above.
tux -o -d perf_winsock2 -x 1001,1002,1007,1009 -c "-s <IP addr or name> -i 4 -b 128,32768 -n 1000"

IP addr or name: the name of a desktop host running perf_winsockd2 server either from the command line or as service.

More information for perf_winsock can be found out at
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conWinsockPerformanceTest.asp

Pass criteria:
All the tests cases should pass.


<5> Perfomance Tests
The CETK has two test suites, Perf_ndis test and Perf_Winsock to measure the performance at the NDIS and Winsock levels respectively. The tests should be run in the following configuration.
Windows CE-based device (Perf client) <---- network----> Windows XP Professional SP2 (Perf server)

Perf_ndis
Run the following test cases for perf_ndis:
<a> Send throughput
Edit command line for Perf_ndis in CETK UI exactly as follows
tux -o -d perf_ndis -c "-mode send -wsock -s <name of your miniport instance>"

<b> Recv throughput
Edit command line for Perf_ndis in CETK UI exactly as follows
tux -o -d perf_ndis -c "-mode recv -wsock -s <name of your miniport instance>"

You can get the names of all the miniport instances present on your WIndows CE-based device if you edit the command line for Perf_ndis in CETK UI to be
tux -o -d perf_ndis -c "-enum"

For more information on perf_winsock, please visit:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conNDISPerformanceTest.asp

Perf_winsock
You need not run Perf_winsock test again if you have run the test as explained above in <4> Winsock Stress Test

Pass criteria:
It is difficult to set a firm criteria for performance since it mostly depends on,
<1> the platform hardware for your Windows CE-based device. E.g. CPU, CPU speed, R/W RAM, type and amount of Cache memory, etc
<2> NIC hardware
<3> Type of bus the NIC is based on. SDIO, PCMCIA, PCI etc.
<4> Type of network. Wired regular 10/100 mbps LAN, 802.11b WiFi network etc.
<5> Miniport driver
<6> Bus driver and/or any driver in the stack and platform code.
So right now you should run through above defined perf tests and see if the performance numbers meet your expectations for a given set of conditions for your Windows CE-based device.

If you notice the performance is lower than you expect, you can use a Monte Carlo profiler during a perf_ndis run to determine what components are taking up the majority of the transfer time.

For more details on Monte Carlo profiling, please visit:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/wcedebug5/html/wce50conMonteCarloProfiling.asp

Author: Harish Rane