Funny, It Worked Last Time : Performance

Event Tracing for Windows (ETW) -- Part 2

ryanmy — Fri, 10 Jun 2005 00:42:00 GMT

    So, there were two major groups of comments on the last post, and I'll try to address them.

    The first was a question about managed support for ETW. I talked to the ETW team, and the current state is that there is no official managed interface for ETW. Being a standard Win32 API, it is posisble to PInvoke the functions involved, and several internal teams have written their own managed wrappers around ETW. This isn't expected to change up through Whidbey (Visual Studio 2005); for Orcas (the version of VS after that!) an official managed interface is on the table. The second is one of backwards compatibility -- ETW is only available in Win2K and later OSes. Users will expect that software works similarly on all OSes; thus, if you want to support 9x-era OSes, you have to write your own logging code anyways. So, if you are already putting in old-style logging, why use ETW? I'll try to answer that with this entry.

    One of the gains of ETW is that it's fast; you can spit out thousands of events per second while using relatively little CPU, far faster than you can fprintf() a string to disk. The biggest gain, though, is combining multiple sources -- including information outside your own process. And the most notable external source is the kernel.

    The XP and Codename Longhorn kernels are extremely extensive providers, and can be enabled to log any or all of these to a log, and we publish decoding information for:

Hardware configuration events -- notes on the system's CPUs, hard drives, NICs, video card, and ACPI power states
Disk-level I/O -- every I/O on the system, including IRP flags, operation time in microseconds, number of bytes, and target disk
File-level I/O -- every access to every file on the system (including information to tie it to the disk I/Os above)
Image layouts -- filenames, locations in memory, and PIDs for every image in the system
Page faults -- pointers to instructions and pages whenever a fault occurs (including COWs, demand-zero faults, hard page faults, transition faults, and guard pages)
Network I/O -- all TCP and UDP actions, including connects/accepts, transmits (and retransmits!), recieves, etc.
Registry I/O -- all Registry key/value creation/deletions/changes, registry flushes, etc.
Process and thread info -- all creations/deletions of processes and threads

    And Codename Longhorn adds even more events -- most notably, the ability to trace extremely fine-grained high-frequency events such as individual context switches, interrupts (ISRs and DPCs), etc.

    (FYI, since all of the above information is exceedingly detailed, you can only enable the kernel provider if you have Administrator privileges, are part of the Performance Log Users group, or a service running as LocalSystem, LocalService, or NetworkService.)

    Thus, ETW can be used as an effective debugging tool. By allowing ETW to pull from, sort, and combine events from multiple providers, you can get a powerful log of everything the system was doing, probably the most accurate log available (save for running the entire OS in a debugger). It's an incredible tool for noticing "hey, things act strangely when X, Y, and Z, but not W, are happening" at a system level, as well as a code level, and it takes far less time than getting symbols and attaching a debugger/profiler to the system. And it's all available to devs -- and even to users, given a generic ETW tool such as tracelog in Server 2003!

    Even if you don't specifically use the kernel as an information source, ETW's ability to combine providers is useful for mixing and matching information from multiple DLLs, EXEs, etc. in a system. ETW events are timestamped by the kernel to extremely high resolution (RDTSC on stable machines, converted to microsecond intervals; MM timers on others) and are automatically sorted at process time, so you don't have to write or parse plaintext date/time formats.

    The goal I'm driving at is that when you have more than one DLL or EXE providing information, individually implementing logging for each component means that you usually need a third app to read in the logs from each component and combine them into a single log with coherent event ordering, and this can be difficult -- especially if you have to tie it to some event. ETW allows you to automate all that, and it's exceedingly efficient at it as well. Even if you are only personally maintaining one component, ETW can log very quickly, and it can be shipped in retail builds -- and if you publish the structures of some or all of the events you provide, you can give valuable information to your consumers and to future devs without ever needing to work with them.

    Next entry, I'll start discussing how providers are written, starting with thread structures and common ways of publishing event structs.

Event Tracing for Windows (ETW)

ryanmy — Sat, 28 May 2005 05:13:00 GMT

    A lot of work in performance tuning is organizational. There's only so much work one can do with a profiler and a single module. A good example is the Registry -- we can attach profilers to the Registry access routines and optimize them until they run as smooth as silk, but performance will still be impacted if you do thousands of Registry accesses per second. For many problems, the cause is systemic: several components in a chain of command that are individually well-tuned, but didn't expect to call each other in a huge chain. A good example of that is DirectShow -- no matter how skillfully crafted an individual filter is, if the mean path in a DirectShow graph is ten filters deep (with memory management between each one for passing buffers of audio or video around), latency is going to be high.

    More often than not, the best solution is simply logging. Log when filters are instantiated or connected, log when Registry accesses are made, etc.. You want to mark high-level concepts, and try to get a picture for what's going on with the system as a whole. This works fine if you only have one application that has to log... but more often that not, these systemic problems have hundreds of files involved, most of which aren't coded by you! If every programmer performs their logging in a different way, it can be a nightmare to combine all those logs together, mixing different types of timestamps and different methods of delivery, and get a single ordered log of what happened over time. Of course, that's exactly what we need... and that's where Event Tracing for Windows, or ETW for short, comes in.

    ETW is, at its core, a unified system for one-way packetized I/O managed by the Windows kernel, built for logging. Every use of ETW has three participants in it -- the controller, the provider, and the consumer:

A provider is an module (DLL/EXE) doing something worth logging. Most of the time, it runs without logging; it can, however, be "enabled" by a controller, at which point it recieves a handle from the kernel and starts logging "events" to that handle. An event is an arbitrary struct (binary block) of data, the only condition being that it start with a 48-byte header. This header contains a timestamp and identifying information.
A controller controls the actual act of logging. The controller can ask the kernel to start a logging session, creating a handle and specifying that the kernel should take any events delivered to that handle and save them to a file. (That file is usually on a hard drive, although we occasionally save them to RAM drives to ensure minimal interference.) The controller can also enable and disable logging by providers, passing them a handle to log to.
A consumer reads events out of a file created by a logging session and parses them. (It is also technically possible to have a consumer directly attach to a logging session's handle and retrive events in real-time, but this is rare.)

So, why use this system over your own homebrew system?

Uniformity. If you're debugging systemic problems involving multiple components, and all the involved components use ETW, you can have them all deliver their information to a single log file with uniform, steady timestamps, and write a single application that parses them all.
Speed. ETW is extremely fast for providers to use, since all the I/O is handled by the kernel instead of by your module. It typically takes only 1500-2000 cycles, depending on settings, to deliver an event and return to your code. One can easily deliver thousands of events per second even on ancient machines. We've achieved 20,000 events per second while only using 5% CPU load on a P3 500MHz! (Yes, we have machines that old in our perf testing labs -- not everyone who uses Longhorn will be using a modern machine!)
Consistency. With fprintf() or other homebrew systems, logging tends to be very slow and intrusive and is thus usually compiled in. With ETW, logging is extremely fast; furthermore, since logging is turned on by a controller and is usually off by default, you can actually leave the ETW events in final shipping code! If problems are found in the field, send the tester an app that starts a trace and turns on the provider, then read it later. Many, many components in Longhorn will ship as ETW providers.
Reliability. ETW isn't a new thing -- it's actually been in the OS and actively used since Win2K, and has been constantly refined since then. Furthermore, ETW is available in both user-mode apps and kernel components. (The latter access it through a MJ_SYSTEM_CONTROL IRP.) This leads to...
OS cooperation. The Windows kernel can provide many highly useful events via ETW for diagnosing performance problems. Find out when and where disk I/Os, registry accesses, hard faults, and other performance problems happen! More on this later...

I'll start discussing the actual APIs in the next entry -- those whose curiosity has been piqued can jump into the MSDN documentation, which is not very good IMO but better than nothing.

Misinformation and the The Prefetch Flag

ryanmy — Thu, 26 May 2005 02:27:00 GMT

Hello! I haven't updated this blog in a while; work and other events have conspired to keep me from writing. Also, blogs.msdn.com moved internally from .Text to Telligent Community Server, and my CSS markup was an unfortunate casualty of the move, so I'm working on redesigning the blog's visual appearance. More entries will be coming eventually. :)

In the meantime, I want to defuse a long-standing controversy -- the /prefetch flag.

With modern computing, the absolute worst thing you can ever do for performance is having to touch the hard drive -- or any non-memory storage for that matter. The fastest hard drives on earth are still horridly slow compared to a PC's main memory; even with solid state drives, in order to access the drive, one has to jump into system code and drivers, and this will push your own program's code out of the CPU's L2 cache. (This is called a locality loss.) There's two typical reasons one has to touch the disk -- the first is when the application requests it explicitly (Word asks the OS to load blog.doc into memory), and the other is a "hard fault" -- when the application tries to use memory that has been paged out to disk via "virtual memory" and needs to be paged back in.

Now, imagine that a DVD player program always starts playback by loading a DLL to decode MPEG-2 video. Wouldn't it be nice if we could attempt to pre-load the MPEG-2 DLL whenever we loaded the DVD player's EXE? That way, when it tries to run code on that DLL, one doesn't have to hard fault and go to disk for it! This is what a prefetcher does: it tracks what code pages are used by an application, and the next time that application loads, it loads those pages in advance as soon as it's got some idle time. A prefetcher was added to Windows in XP, and is vastly improved in Windows Longhorn.

XP systems have a Prefetch directory underneath the windows root directory, full of .pf files -- these are lists of pages to load. The file names are generated from hashing the EXE to load -- whenever you load the EXE, we hash, see if there's a matching (exename)-(hash).pf file in the prefetch directory, and if so we load those pages. (If it doesn't exist, we track what pages it loads, create that file, and pick a handful of them to save to it.) So, first off, it is a bad idea to periodically clean out that folder as some tech sites suggest. For one thing, XP will just re-create that data anyways; secondly, it trims the files anyways if there's ever more than 128 of them so that it doesn't needlessly consume space. So not only is deleting the directory totally unnecessary, but you're also putting a temporary dent in your PC's performance.

Secondly, one can specify a /prefetch:# flag when launching an app. Many people have noticed that auto-generated shortcuts to Windows Media Player do this, and the number varies depending on what it does. For example, the shortcut used by the shell when you double-click a WMV file to play it has one prefetch number; the auto-run shortcut to play or rip music that appears when you insert a music CD have other numbers. Some sites have guessed that this switch turns on prefetching, and suggest that you add that to every executable you care about -- this has appeared on so many, many, many sites to be urban legend. Other sites write this off as garbage and guess that it's a switch specific to Media Player, guessing from references to prefetching in the Windows driver subsystem. Both guesses are incorrect.

The /prefetch:# flag is looked at by the OS when we create the process -- however, it has one (and only one) purpose. We add the passed number to the hash. Why? WMP is a multipurpose application and may do many different things. The DLLs and code that it touches will be very different when playing a WMV than when playing a DVD, or when ripping a CD, or when listening to a Shoutcast stream, or any of the other things that WMP can do. If we only had one hash for WMP, then the prefetch would only be correct for one such use. Having incorrect prefetch data would not be a fatal error -- it'd just load pages into memory that'd never get used, and then get swapped back out to disk as soon as possible. Still, it's counterproductive. By specifying a /prefetch:# flag with a different number for each "mode" that WMP can do, each mode gets its own separate hash file, and thus we properly prefetch. (This behavior isn't specific to WMP -- it does the same for any app.)

This flag is looked at when we create the first thread in the process, but it is not removed by CreateProcess from the command line, so any app that chokes on unrecognized command line parameters will not work with it. This is why so many people notice that Kazaa and other apps crash or otherwise refuse to start when it's added. Of course, WMP knows that it may be there, and just silently ignores its existence.

I suspect that the "add /prefetch:1 to make rocket go now" urban legend will never die, though. I know that at least one major company ships products with it in their shortcuts, without ever asking us... just for good measure, I guess. :-P All it does is change your hash number -- the OS is doing exactly the same thing it did before, and just saving the prefetch pages to a different file.

(ATTENTION: This is merely an informative article; this information is completely unsupported, and the functionality may change or disappear entirely in future versions of Windows or service packs. Furthermore, it is merely a hint for the XP prefetcher, and it may choose to ignore it if it wishes.)