April, 2012

  • The Old New Thing

    There's the kernel, and there's kernel mode - confusing historical terminology


    A few weeks ago, I mentioned that the kernel folks decided not to expose bonus bytes to applications. Some people were confused by this statement, not for what it said, but for what it implied. "Wait, you're telling me that the heap is implemented in kernel mode?"

    Let's turn the clock back to 1983.

    The core components of Windows fell into three categories:

    • The window manager, known as user, because it handled the user interface.
    • The graphics engine, known as GDI, short for Graphics Device Interface.
    • File I/O, the scheduler, memory management, and other low-level bits, known as kernel.

    Windows 1.0 ran on the 8086, which had no concept of CPU modes or memory protection or any stuff we take for granted nowadays. Everything ran in a single mode, and since there was only one mode, it didn't have a name.

    Although future versions of Windows distinguished between kernel mode and user mode (in the CPU mode sense), the old terminology stuck around. The "kernel" was anything related to file I/O, the scheduler, memory management, and other low-level operations, even if they were implemented in user mode.

    For a time, there was an effort to use the term "base" to refer to all of these low-level operations and thereby avoid the confusing term "kernel." As you can tell, the attempt was largely unsuccessful. People continued to call low-level stuff "kernel" out of habit.

  • The Old New Thing

    Why do we need IsDialogMessage at all?


    alv wonders why we need the IsDialogMessage function at all. "All its activity could take place inside the window procedure of the modeless dialog itself", since when it doesn't have focus, it shouldn't be responding to messages anyway.

    Sure, that works great if the modeless dialog has focus. But it almost never does. What has focus is a control inside the modeless dialog. And in that case, the modeless dialog never sees the message, since the rule is that keyboard messages go to the window with focus. And that ain't the modeless dialog box.

    Consider, for example, a message box with OK and Cancel buttons. Focus defaults to the OK button. You hit the TAB key to move to the Cancel button. The TAB message goes to the OK button because it is the control with keyboard focus. The OK button says, "I don't know what TAB means. I guess I'll just beep." (The button control doesn't know whether it's part of a dialog box or not; it just worries about being a button.)

    The IsDialogMessage function is in the message loop so that it can intercept the TAB message before it reaches the OK button. At this point, IsDialogMessage can step in and say "Hang on, I'll take care of this" and use the TAB key to perform dialog box navigation. (Of course, as we know, it first checks with the control that stealing the message is okay.)

    Some progamming frameworks capture the concept of "stealing messages" with names like PreTranslateMessage and event routing. But the basic idea is the same: Letting somebody other than the target of a window message get a chance to intercept the message and do something special with it.

  • The Old New Thing

    You already know the answer since you do it yourself


    A customer was writing a program that performed virtual drag/drop. They were using the IStream technique but found that many applications don't support drag/drop of virtual content. They support only CF_HDROP. What's more, often these applications query for CF_HDROP on DragEnter not because they want to access the file, but just because they want to get the file names (for example, because they want to put up the no-entry cursor if the file types are not ones the application supports).

    Given that we want to be able to drop content onto applications which do not support drag/drop of virtual content, we have the problem of knowing exactly when to generate the content into a temporary file. If we generate the content too soon, then we may end up going to a lot of effort of creating a file that won't actually be used; if we generate it too late, then the application will try to open the file and find that it isn't there. When is the correct moment to generate the content? Is there some set of rules by which applications which do not support virtual drag/drop indicate whether they are obtaining the CF_HDROP just to see the names of the files, and whether they are obtaining it because they want to go open the files?

    If you think about it, you already know the answer to this question because you already do it yourself when you write code that operates the client side of the drag/drop contract. When you write your program that accepts CF_HDROP content, do you use any special signal to tell the data object, "Hey, I'm asking for CF_HDROP just because I want the file names, but I promise not to try to open the files yet"? No, you don't, so why would you expect any other application to? Even if there were rules surrounding this signaling protocol, the fact that they are widely ignored (because even you yourself ignore them) means that you can't rely on the client performing them anyway.

    The rule for CF_HDROP is that at the moment you offer CF_HDROP content, the files must already exist. The CF_HDROP clipboard format was created by File Manager in Windows 3.1, and File Manager did not support virtual content. The only thing it knew how to drag and drop was files, and the only thing it could drag was files that already existed.

    In a sense, the rules around CF_HDROP were not so much codified by rule as codified by circumstance. Since the only things you could drag were files that already existed, those became the de facto rules for CF_HDROP.


    Note that there is a little you can do: If an application calls QueryGetData for CF_HDROP, that does not force you to create the file content yet, because QueryGetData is just a yes/no query as to whether you could produce CF_HDROP if asked. You're not being asked to do so, so you can just say "Why yes, I have files" even though you don't yet. It's only at the GetData that you have to return a list of file names and therefore must create the files.

    Most applications are kind enough to use QueryGetData when they are not interested in the files yet but only in the potential for files, but there are some which use GetData on the DragEnter, and those applications will force you to commit to creating the content even if the user ultimately abandons the operation without dropping. (You could try deferring the creation until your IDropSource::QueryContinueDrag returns DRAGDROP_S_DROP, but programs which sniff the file contents during DragEnter will not find the file there, and they probably won't like that. It also doesn't work if the transfer is done via copy/paste rather than drag/drop—and you need copy/paste support for accessibility—since your drop source is not active during a copy/paste.)

    Exercise: How do you know when it's safe to delete the temporary files?

  • The Old New Thing

    What is the real maximum length of a DNS name?


    The maximum length of a DNS name is 255 octets. This is spelled out in RFC 1035 section 2.3.4. A customer didn't understand why the DnsValidateName was rejecting the following string:

    (63 letters).(63 letters).(63 letters).(62 letters)

    The documentation says

    Returns ERROR_INVALID_NAME if the DNS name

    • Is longer than 255 octets.
    • Contains a label longer than 63 octets.
    • ... other criteria not relevant here...

    The length of the domain name passed in is 63+1+63+1+63+1+62=254 characters, just under the length limit of 255. Why is it rejecting this name that is under the limit?

    Because the limit isn't the number of characters; it's the number of octets.

    Section 3.3 says that a domain-name is represented as a series of labels, and is terminated by a label of length zero. (The label of length zero represents the root label.) A label consists of a length octet followed by that number of octets representing the name itself. Therefore, the domain name www.microsoft.com is encoded as follows:

    3 'w' 'w' 'w' 9 'm' 'i' 'c' 'r' 'o' 's' 'o' 'f' 't' 3 'c' 'o' 'm' 0

    Technically, www.microsoft.com is shorthand for www.microsoft.com. with a trailing period, and the trailing zero byte encodes that implied period.

    If you sit down and do the math, you'll see that the the readable maximum length of an ASCII DNS name is 253 characters: You don't encode the dots, but you do encode the length bytes, so they cancel out, except for the length byte of the first label and the length byte of the root label, for an additional cost of two bytes. (On the off chance that you explicitly specified the root label, don't count it towards the 253-character limit.)

    If you use UTF-8 encoding, then the maximum length is harder to describe since UTF-8 is a variable-length encoding.

  • The Old New Thing

    I know that an overlapped file handle requires an lpOverlapped, but why does it (sometimes) work if I omit it?


    A customer observed that the formal requirements for the Read­File function specify that if the handle was opened with FILE_FLAG_OVERLAPPED, then the lpOverlapped parameter is mandatory. But the customer observed that in practice, passing NULL results in strange behavior. Sometimes the call succeeds, and sometimes it even returns (horrors!) valid data. (Actually the more horrifying case is where the call succeeds and returns bogus data!)

    Now sure, you violated one of the requirements for the function, so the behavior is undefined. But why doesn't Read­File just flat-out fail if you call it incorrectly?

    The answer is that the Read­File function doesn't know whether you're calling it correctly.

    The Read­File function doesn't know whether the handle you passed was opened for overlapped or synchronous access. It just trusts that you're calling it correctly and builds an asynchronous call to pass into the kernel. If you passed a synchronous handle, well, it just issues the I/O request into the kernel anyway, and you get what you get.

    This quirk traces its history all the way back to the Microsoft Windows NT OS/2 Design Workbook. As originally designed, Windows NT had a fully asynchronous kernel. There was no such thing as a blocking read. If you wanted a blocking read, you had to issue an asynchronous read (the only kind available), and then block on it.

    As it turns out, developers vastly prefer synchronous reads. Writing asynchronous code is hard. So the kernel folks relented and said, "Okay, we'll have a way for you to specify at creation time whether you want a handle to be synchronous or asynchronous. And since lazy people prefer synchronous I/O, we'll make synchronous I/O the default, so that lazy people can keep being lazy."

    The Read­File function is a wrapper around the underlying Nt­Read­File function. If you pass an lpOverlapped, then it takes the OVERLAPPED structure apart so it can pass the pieces as an Io­Status­Block and a Byte­Offset. (And if you don't pass an lpOverlapped, then it needs to create temporary buffers on the stack.) All this translation takes place without the Read­File function actually knowing whether the handle you passed is asynchronous or synchronous; that information isn't available to the Read­File function. It's relying on you, the caller, to pass the parameters correctly.

    As it happens, the Nt­Read­File function does detect that you are trying to perform synchronous I/O on an asynchronous handle and fails with STATUS_INVALID_PARAMETER (which the Read­File function turns into ERROR_INVALID_PARAMETER), so you know that something went wrong.

    Unless you are a pipe or mailslot.

    For some reason, if you attempt to issue synchronous I/O on an asynchronous handle to a pipe or mailslot, the I/O subsystem says, "Sure, whatever." I suspect this is somehow related to the confusing no-wait model for pipes.

    Long before this point, the basic ground rules for programming kicked in. "Pointers are not NULL unless explicitly permitted otherwise," and the documentation clearly forbids passing NULL for asynchronous handles. The behavior that results from passing invalid parameters is undefined, so you shouldn't be surprised that the results are erratic.

  • The Old New Thing

    Microspeak: scoped to


    The Merriam-Webster dictionary gives as the meaning of scope as a verb to look at for evaluation, as in "to scope out the competition." But that's not how we use it at Microsoft.

    Here are some fake citations:

    The Widgets pop-up shows the available widgets scoped to the current selection.
    The results of the search are scoped to the current folder.
    Workflows can be scoped to containers, content types (scopeable to containers, sites, collections, servers, or enterprises), or combinations of these.

    Okay, that last one wasn't fake. You can tell it's not fake because it is extra confusing.

    To be scoped to something is to be limited to or filtered to that thing, or things which apply to that thing. In other words limited in scope to. Here's an attempt to translate those citations into English:

    The Widgets pop-up shows the available widgets which apply to the current selection.
    The results of the search are filtered to those in the current folder.

    I'm not going to try to translate that last one on there. It has this sort of Escherian feeling to it. "Workflows can be scoped to A, or to B (which can be scoped to A), or combinations of A and B (which means combinations of A, and B, and A's within B's?)"

    Another sense of the verb phase scoped to is altered in scope. Usually, the change is to reduce the scope to meet external constraints:

    The proposal has been scoped to meet our December release.

    To emphasize that the scope has been narrowed, you add the adverb down: "scoped down to."

    Translation: "The original proposal was too broad and could not be accomplished within the required time allotted, so we reduced its scope to the point where the parts that remain can be accomplished in time."

    On the other hand, sometimes scope expands.

    Based on customer feedback, the search results have been scoped to include archived data as well.

    And finally, a citation from an official Microsoft job description:

    Oversees the structuring of consulting engagements to ensure they are properly scoped to meet the customer's requirements profitably.
  • The Old New Thing

    There's the interface contract, and there are the implementations of the interface contract


    Ivo wants to know whether it is legal to use NULL as the icon parameters to IExtractIcon::Extract. The documentation says that the parameters are optional, but some shell folder implementations treat them as mandatory.

    Yes, the parameters are technically optional, but it's also the case that many people mess up their implementation of the interface and treat them as mandatory, either by crashing on a null pointer or by returning E_INVALIDARG. Since IExtractIcon is an extension interface, you are pretty much at the mercy of all the implementations of that extension.

    Welcome to the land of application compatibility, where you have to incorporate workarounds for other people's bugs. In this case, it means always passing non-null pointers for the output icons if you want to get anything meaningful back, even if that means asking for more than you really need and then throwing part of it away.

    Ivo's second question was whether there is a performance benefit of asking just for the icon you want, or whether it's almost as fast to get both.

    If you ask for just one of the icons, then the icon extractor doesn't need to go extract it, which saves you a small amount of disk access (or a large amount if you're asking for the monster 256×256 icon). But given that compatibility forces you to ask for both anyway, the answer doesn't help you any. Given that there are drivers who run red lights, you could say that, in theory, "It is more efficient to cross the street as soon as the light turns green," but in practice, you'd be better served to look for traffic before stepping out into the roadway.

    You'd be right, but you'd be dead right.

  • The Old New Thing

    Registration-free COM the old-fashioned way: The car mp3 player


    Windows XP introduced Registration-Free COM, permitting you to place your COM object registrations in a manifest rather than in the registry. Very handy when you want to support xcopy deployment or running directly off a thumb drive. And very much in keeping with the principle of not using a global solution for a local problem. (If you need your COM object to be used from other programs, then global registration is not unreasonable, but if the only client is another part of your program, then you have a local problem that should employ a local solution.)

    Here are some articles on the subject:

    Before manifest-based COM object registration was possible, you had to do things the old-school way. Mind you, old-school registration-free COM is not a very useful skill any more, falling into the same category as knowing how to churn butter or use a typewriter eraser, but since when did that ever stop me from writing about something?

    One old-school way is to call Dll­Get­Class­Object directly. This works only if you control both sides of the equation (the client and the server) because it's now your responsibility to ensure that both sides agree on the threading model. You won't have the actual COM libraries sitting in between watching out for scenarios that require marshalling, for example.

    For a toy project of mine, I used a different approach. My little project was an mp3 player for my laptop. Now, sure, we already have tons of mp3-playing apps out there, but mine was different: Mine was designed to be used on long driving trips. (This was back in the days when long driving trips were part of my life. I don't drive that much any more.)

    Here was the plan: I connected the line-out from the laptop into my car sound system, so that the music came out my car speakers. Meanwhile, all input to the program came from the mouse. Specifically, it came from the mouse buttons and the wheel. The mouse itself never moved. The idea was that I could hook up a mouse to the laptop, put the laptop on the passenger seat, and leave the mouse on the center console. I would then use the mouse buttons and wheel to navigate my music collection. I forget exactly what I ended up doing, but it was something like

    • Left button click = select current item
    • Right button click = go up one level
    • Rotate wheel = scroll through current directory

    Now remember, this program was designed to be used while driving, which means both eyes on the road. (This was long before hands-free driving laws came on the scene.) Therefore, the program provided feedback not by displaying items on the screen but by using speech synthesis to read the names of the files and directories out loud. Finding a song to play, therefore, consisted of turning the wheel and listening as the laptop read out the name of the album, then when I found the one I wanted, I would click the left mouse button, and then I would use the wheel to scroll through the songs, and when I heard the title of the one I wanted, I clicked the left mouse button again, and boom, the song started playing.

    I added some heuristics to the code so if consecutive tracks began with the same words (which happens often with classical music, think Symphony #5 in c minor, first movement followed by Symphony #5 in c minor, second movement) it spoke only the changes.

    While the song was playing, the mouse buttons served as playback controls. I think it went something like this:

    • Left button click = pause / play
    • Right button click = exit and choose another song
    • Rotate wheel = rewind / fast-forward ten seconds
    • Press middle mouse button and rotate wheel = previous/next track

    (Exercise: Why didn't I need a volume control?)

    The easiest programming language for this was a Web page. I created a host program that simply created a Web browser control. The host program told the Web browser control to navigate to my carplay.html file, and boom, I now had an in-car playback system. I could use things like File­System­Object to navigate the file system and the Windows Media Player control to do the playback. Now, this story takes place so many years ago that Internet Explorer didn't support the mouse wheel yet, so the host program also converted wheel messages into fake keyboard activity (wheel motion was converted into the up and down arrows) so that the Web page could respond to them.

    Once nice thing about this whole set-up is that I had the HTML DOM at my disposal. My program spewed diagnostic output to the screen like crazy, but who cares? The end user isn't looking at the screen. Therefore, the entire Web page is free real estate for debugging.

    The only thing missing was the speech synthesizer.

    There was no ActiveX control for speech synthesis, so I wrote one of my own. I let SAPI do the heavy lifting; my ActiveX control was simply some glue that let a Web page pass a string to be spoken. (Or pass a null string to shut up.) I wanted my program to be xcopy-deployable (USB thumb drives didn't exist back then) so I looked for a registration-free technique. The Dll­Get­Class­Object technique wouldn't work since I didn't control how Internet Explorer created COM objects; it always called Co­Create­Instance.

    The technique I used was Co­Register­Class­Object. I created a class factory for my object and explicitly registered it before creating the Web browser control. That way, when the Web page asked for my object, lo and behold, there it was: In memory, not in the registry.

    That was a really long-winded story with a punch line that tells you how to do something you don't need to do any more because there are easier ways of doing it nowadays. I wouldn't be surprised if you wanted a refund.

    The real punch line: I spent far more time writing the program than I ever did using it.

  • The Old New Thing

    You can use an OVERLAPPED structure with synchronous I/O, too


    Even if you didn't open a file with FILE_FLAG_OVERLAPPED, you can still use the OVERLAPPED structure when you issue reads and writes. Mind you, the I/O will still complete synchronously, but you can take advantage of the other stuff that OVERLAPPED has to offer.

    Specifically, you can take advantage of the Offset and OffsetHigh members to issue the I/O against a file location different from the current file pointer. (This is a file pointer in the sense of Set­File­Pointer and not in the sense of the C runtime FILE*.) If your program does a lot of reads and writes to random locations in a file, using the synchronous OVERLAPPED structure saves you a call to Set­File­Pointer at each I/O.

    Let's illustrate this by writing some code to walk through a file format that contains a lot of offsets to other parts of the file: The ICO file format. First, the old-fashioned way:

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <pshpack1.h>
    struct ICONDIRHEADER {
        WORD idReserved;
        WORD idType;
        WORD idCount;
    struct ICONDIRENTRY {
        BYTE bWidth;
        BYTE bHeight;
        BYTE bColorCount;
        BYTE  bReserved;
        WORD  wPlanes;
        WORD  wBitCount;
        DWORD dwBytesInRes;
        DWORD dwImageOffset;
    #include <poppack.h>
    BOOL ReadBufferAt(__in HANDLE hFile,
        __out_bcount(cbBuffer) void *pvBuffer,
        DWORD cbBuffer,
        DWORD64 offset)
     DWORD cbRead;
     li.QuadPart = offset;
     return SetFilePointerEx(hFile, li, nullptr, FILE_BEGIN) &&
            ReadFile(hFile, pvBuffer, cbBuffer, &cbRead, nullptr) &&
            cbBuffer == cbRead;
    int __cdecl wmain(int argc, wchar_t **argv)
     HANDLE hFile = CreateFile(argv[1], GENERIC_READ,
      nullptr, OPEN_EXISTING, 0, nullptr);
     if (hFile != INVALID_HANDLE_VALUE) {
      if (ReadBufferAt(hFile, &hdr, sizeof(hdr), 0) &&
          hdr.idReserved == 0 && hdr.idType == 1) {
       for (UINT uiIcon = 0; uiIcon < hdr.idCount; uiIcon++) {
        ICONDIRENTRY entry;
        if (ReadBufferAt(hFile, &entry, sizeof(entry),
                         sizeof(hdr) + uiIcon * sizeof(entry))) {
         void *pvData = LocalAlloc(LMEM_FIXED, entry.dwBytesInRes);
         if (pvData) {
          if (ReadBufferAt(hFile, pvData,
                           entry.dwBytesInRes, entry.dwImageOffset)) {
           // process one image in the icon
     return 0;

    Run this program with the name of an icon file on the command line, and nothing interesting happens because the program doesn't generate any output. But if you step through it, you can see that we start by reading the ICON­DIR­HEADER to verify that it's an icon and determine the number of images. We then loop through the images: For each one, we read the ICON­DIR­ENTRY (specifying the explicit file offset), then read the image data (again, specifying the explicit file offset).

    We use the Read­Buffer­At function to read data from the file. For each read, we first call Set­File­Pointer to position the file pointer at the byte we want to read, then call Read­File to read it.

    Let's change this program to take advantage of our newfound knowledge:

    BOOL ReadBufferAt(__in HANDLE hFile,
        __out_bcount(cbBuffer) void *pvBuffer,
        DWORD cbBuffer,
        DWORD64 offset)
     OVERLAPPED o = { 0 };
     o.Offset = static_cast<DWORD>(offset);
     o.OffsetHigh = static_cast<DWORD>(offset >> 32);
     DWORD cbRead;
     return ReadFile(hFile, pvBuffer, cbBuffer, &cbRead, &o) &&
            cbBuffer == cbRead;

    We merge the Set­File­Pointer call into the Read­File by specifying the desired byte offset in the optional OVERLAPPED structure. The I/O will still complete synchronously (since we opened the handle synchronously), but we saved ourselves the hassle of having to call two functions when it could be done with just one.

  • The Old New Thing

    I totally presented to an executive the wrong way


    Some time ago, Gray Knowlton wrote an article on how to present to an executive.

    As you might have guessed, I've done it completely the wrong way.

    Many years ago, I was part of a group presenting to a senior-level executive. I was the one who wrote the document establishing the background for the topic and laying out the various options with their pros and cons. I wasn't the one doing the actual presenting, but I was asked to attend anyway, just in case the senior executive had a question that the presenters couldn't answer.

    For the duration of the meeting, I sat in the back and knitted.

    As it turns out, I wasn't needed. But at least I got some knitting done.

    (I was commended for the quality of the document I had written, so don't think I was just blowing off the issue entirely.)

Page 2 of 3 (22 items) 123