February, 2009

  • The Old New Thing

    What the various registry data types mean is different from how they are handled

    • 55 Comments

    Although you can tag your registry data with any of a variety of types, such as REG_DWORD or REG_BINARY or REG_EXPAND_SZ. What do these mean, really?

    Well, that depends on what you mean by mean, specifically, who is doing the interpreting.

    At the bottom, the data stored in the registry are opaque chunks of data. The registry itself doesn't care if you lie and write two bytes of data to something you tagged as REG_DWORD. (Try it!) The type is just another user-defined piece of metadata. The registry dutifully remembers the two bytes you stored, and when the next person comes by asking for the data, those two bytes come out, along with the type REG_DWORD. Garbage in, garbage out. The registry doesn't care that what you wrote doesn't many any sense any more than the NTFS file system driver doesn't care that you wrote an invalid XML document to the file config.xml. Its job is just to remember what you wrote and produce it later upon request.

    There is one place where the registry does pay attention to the type, and that's when you use one of the types that involve strings. If you use the RegQueryValueA function to read data which is tagged with one of the string types (such as REG_SZ), then the registry code will read the raw data from its database, and then call WideCharToMultiByte to convert it to ANSI. But that's the extent of its assistance.

    Just as the registry doesn't care whether you really wrote four bytes when you claimed to be writing a REG_DWORD, is also doesn't care whether the various string types actually are of the form they claim to be. If you forget to include the null terminator in your byte count when you write the data to the registry, then the null terminator will not be stored to the registry, and the next person to read from it will not read back a null terminator.

    This simplicity in design pushes the responsibility onto the code that uses the registry. If you read a registry value and the data is tagged with the REG_EXPAND_SZ type, then it's up to you to expand it if that's what you want to do. The REG_EXPAND_SZ value is just part of the secret handshake between the code that wrote the data and the code that is reading it, a secret handshake which is well-understood by convention. After all, if RegQueryValueEx automatically expanded the value, then how could you read the original unexpanded value?

    Windows Vista added a new function RegGetValue which tries to take care of most of the cumbersome parts of reading registry values. You can tell it what data types you are expecting (and it will fail if the data is of an incompatible type), and it coerces the data to match its putative type. For example, it auto-expands REG_EXPAND_SZ data, and if a blob of registry data marked REG_SZ is missing a null terminator, RegGetValue will add one for you. Better late than never.

  • The Old New Thing

    Foreground activation permission is like love: You can't steal it, it has to be given to you

    • 48 Comments

    This is the blog entry that acted as the inspiration for the last topic in my 200 PDC talk.

    When somebody launches a second copy of your single-instance program, you usually want the second copy to send its command line to the first instance (and deal with the current directory somehow), and then you want the first instance to come to the foreground. But a common problem people run into is that when the first instance calls SetForegroundWindow, it fails.

    The problem with this design is that as far as the window manager is concerned, what happened is that the first instance received a message and then decided to steal foreground. That message wasn't an input message, so the window manager sees no reason for the first instance to have any right to take the foreground. There is no evidence that the first instance is coming to the foreground in response to some user action.

    There are a variety of ways of addressing this problem. The easiest way is simply to have the second instance make the call to SetForegroundWindow. The second program has permission to take the foreground because you just launched it. And if a program can take the foreground, it can also give it away, in this case, by setting the first program as the foreground window.

    Another way to do this is to have the second program call the AllowSetForegroundWindow function with the process ID of the first program before it sends the magic message. The AllowSetForegroundWindow function lets a program say to the window manager, "It's okay; he's with me." And then when the first program finally gets around to calling SetForegroundWindow, the window manager says, "Oh, this guy's okay. That other program vouched for him."

    If you are transferring foreground activation to a COM server, you can use the corresponding COM function CoAllowSetForegroundWindow.

    In all cases, note that you can't give away something that's not yours. If you don't have permission to take foreground, then calling AllowSetForegroundWindow (or one of its moral equivalents) will have no effect: You just told the window manager, "It's okay; he's with me," and the window manager replied, "Who the hell are you?"

    Pre-emptive snarky comment: "There are some really sneaky people who found a way to circumvent the rules and steal foreground activation." Well yeah, and there are some really sneaky people who find ways to steal love, so there you have it. If everything is right with the world, both groups of people will eventually be found out and made to suffer for their malfeasance.

    Update (02/21): Deleted all comments that showed ways of circumventing the rules. Duh, people.

  • The Old New Thing

    Why doesn't the file system have a function that tells you the number of files in a directory?

    • 46 Comments

    There are any number of bits of information you might want to query from the file system, such as the number of files in a directory or the total size of the files in a directory. Why doesn't the file system keep track of these things?

    Well, of course, one answer is that it certainly couldn't keep track of every possible fragment of information anybody could possibly want, because that would be an infinite amount of information. But another reason is simply a restatement of the principle we learned last time: Because the file system doesn't keep track of information it doesn't need.

    The file system doesn't care how many files there are in the directory. It also doesn't care how many bytes of disk space are consumed by the files in the directory (and its subdirectories). Since it doesn't care, it doesn't bother maintaining that information, and consequently it avoids all the annoying problems that come with attempting to maintain the information.

    For example, one thing I noticed about many of the proposals for maintaining the size of a directory in the file system is that very few of them addressed the issue of hard links. Suppose a directory contains two hard links to the same underlying file. Should that file be double-counted? If a file has 200 hard links, then a change to the size of the file would require updating the size field in 200 directories, not just one as one commenter postulated. (Besides, the file size isn't kept in the directory entry anyway.)

    Another issue most people ignored was security. If you're going to keep track of the recursive directory size, you have to make sure to return values consistent with each user's permissions. If a user does not have permission to see the files in a particular directory, you'd better not include the sizes of those files in the "recursive directory size" value when that user goes asking for it. That would be an information disclosure security vulnerability. Now all of a sudden that single 64-bit value is now a complicated set of values, each with a different ACL that controls which users each value applies to. And if you change the ACL on a file, the file system would have to update the file sizes for each of the directories that contains the file, because the change in ACL may result in a file becoming visible to one user and invisible to another.

    Yet another cost many people failed to take into account is just the amount of disk I/O, particular writes, that would be required. Generating additional write I/O is a bad idea in general, particularly on media with a limited number of write cycles like USB thumb drives. One commenter did note that this metadata could not be lazy-written because a poorly-timed power outage would result in the cached value being out of sync with the actual value.

    Indeed the added cost of all the metadata writes is one of the reasons why Windows Vista no longer updates the Last Access time by default.

    Bonus chatter: My colleague Aaron Margosis points out a related topic over on the ntdebugging blog: NTFS Misreports Free Space? on the difficulties of accurate accounting, especially in the face of permissions which don't grant you total access to the drive.

  • The Old New Thing

    Smart quotes: The hidden scourge of text meant for computer consumption

    • 43 Comments

    Smart quotes—you know, those fancy quotation marks that curl “like this” ‘and this’ instead of standing up straight "like this" 'and this'—are great for text meant for humans to read. Like serifs and other typographic details, they act as subtle cues that aid in reading.

    But don't let a compiler or interpreter see them.

    In most programming languages, quotation marks have very specific meanings. They might be used to enclose the text of a string, they might be used to introduce a comment, they might even be a scope resolution operator. But in all cases, the language specification indicates that the role is played by the quotation mark U+0022 or apostrophe U+0027. From the language's point of view, the visually similar characters U+2018, U+2019, and U+02BC (among others) are completely unrelated.

    I see this often on Web sites, where somebody decided to "edit" the content to make it "look better" by curlifying the quotation marks, turning what used to be working code into a big pile of syntax errors.

    I even see it in email. Somebody encounters a crash in a component under development and connects a debugger and sends mail to the component team describing the problem and including the information on how to connect to the debugger like this:

    WinDbg –remote npipe:server=abc,pipe=def

    Or maybe like this:

    Remote.exe “abc” “def”

    And you, as a member of the team responsible for that component copy the text out of the email (to ensure there are no transcription errors) and paste it into a command line.

    C:\> Remote.exe "abc" "def"
    

    and you get the error

    Unable to connect to server ôabcö
    

    What happened? You got screwed over by smart quotes. The person who sent the email had smart quotes turned on in their email editor, and it turned "abc" into “abc”. You then got lulled into a false sense of security by the best fit behavior of WideCharToMultiByte, which says I can't represent “ and ” in the console code page, but I can map them to " which is a close visual approximation, so I'll use that instead. As a result, the value you see on the command line shows straight quotes, but that's just a façade behind which the ugly smart quotes are lurking.

    I've even seen people hoist by their own smartly-quoted petard.

    I can't seem to access a file called aaa bbb.txt. The command

    type “aaa bbb.txt”

    results in the strange error message

    The system cannot find the file specified.
    Error occurred while processing: "a.
    The system cannot find the file specified.
    Error occurred while processing: x.txt".

    Why can't I access this file?

    Somehow they managed to type smart quotes into their own command line.

    So watch out for those smart quotes. When you're sending email containing code or command lines, make sure your editor didn't "make it pretty" and in the process destroy it.

    Exercise: What is wrong with the WinDbg command line above?

    Bonus chatter: PowerShell is a notable exception to this principle, for it treats all flavors of smart quotes and smart dashes as if they were dumb quotes and dumb dashes. From what I can tell, you have this guy to thank (or blame).

  • The Old New Thing

    How does Raymond decide what to post on any particular day?

    • 35 Comments

    Occasionally somebody asks about the timing of an entry I've written and wants to know how far ahead with this blog thing I really am.

    To give you an idea of how far in advance I write my blog entries, I wrote this particular entry on February 13, 2008. Generally, the articles are published in the order I wrote them; this particular entry ended up on February 27, 2009 because that was the next available open day. If the big news topic of February 27th, 2009 happens to be related to this entry, it's just a coincidence.

    Now, with a buffer of over a year, I do have quite a bit of leeway in choosing when any particular article is published. Although articles in general just get slotted in for the next open day, I will occasionally arrange for one to come out on a thematically-related day. Sometimes the connection is blatant, like writing about time and time zones on the Fridays before Daylight Saving Time transitions, or writing about Hallowe'en on, well, Hallowe'en. More often the connection is low-key, like telling a story about the economics of parking tickets on a day when parking is free in Seattle or warning about fake guacamole on the Friday before the Super Bowl. And sometimes the connection is impossibly obscure, like taking a story that I know a friend will like and slotting it in on their birthday, anniversary, or some other day meaningful to them.

    (To answer Paul's specific question: I found the article on guacamole and said, "That would be a great article to use for the Super Bowl." Since I already had a Super Bowl article picked out for 2007, the guacamole article was slotted in for 2008.)

    There are other patterns you may have picked up on:

    • Mondays are usually spent "answering viewer mail" (i.e., taking topics from the Suggestion Box).
    • Less technical articles appear towards the beginning of the week; more technical articles appear later in the week. Tuesday in particular tends to get the funny stories.
    • There is usually one e-mail related posting per month.
    • There is usually one Microspeak posting per month.
    • Sometimes an entire week is devoted to a theme, such as the annual CLR Week.
    • An unusually short or unusually long technical post is usually balanced by an amusing non-technical post. (I have a stash of about a hundred of these "light diversions".)
    • I tend to avoid technical topics on major holidays.

    But generally, it's just a FIFO queue.

    Oh, and right now, the queue is full up through the beginning of June 2010.

  • The Old New Thing

    What does the COM Surrogate do and why does it always stop working?

    • 35 Comments

    The dllhost.exe process goes by the name COM Surrogate and the only time you're likely even to notice its existence is when it crashes and you get the message COM Surrogate has stopped working. What is this COM Surrogate and why does it keep crashing?

    The COM Surrogate is a fancy name for Sacrificial process for a COM object that is run outside of the process that requested it. Explorer uses the COM Surrogate when extracting thumbnails, for example. If you go to a folder with thumbnails enabled, Explorer will fire off a COM Surrogate and use it to compute the thumbnails for the documents in the folder. It does this because Explorer has learned not to trust thumbnail extractors; they have a poor track record for stability. Explorer has decided to absorb the performance penalty in exchange for the improved reliability resulting in moving these dodgy bits of code out of the main Explorer process. When the thumbnail extractor crashes, the crash destroys the COM Surrogate process instead of Explorer.

    In other words, the COM Surrogate is the I don't feel good about this code, so I'm going to ask COM to host it in another process. That way, if it crashes, it's the COM Surrogate sacrificial process that crashes instead of me process. And when it crashes, it just means that Explorer's worst fears were realized.

    In practice, if you get these types of crashes when browsing folders containing video or media files, the problem is most likely a flaky codec.

    Now that you know what the COM Surrogate does, you can answer this question from a customer:

    I'm trying to delete a file, but I'm told that "The action can't be completed because the file is open in COM Surrogate." What is going on?
  • The Old New Thing

    The checkbox: The mating call of the loser

    • 33 Comments

    (Cultural note: The phrase the mating call of the loser is a term of derision. I used it here to create a more provocative headline even though it's stronger than I really intended, but good writing is bold.)

    When given a choice between two architectures, some people say that you should give users a checkbox to select which one should be used. That is the ultimate cowardly answer. You can't decide between two fundamentally different approaches, and instead of picking one, you say "Let's do both!", thereby creating triple, perhaps quadruple the work compared to just choosing one or the other.

    It's like you're remodeling a book library and somebody asks you, "Should we use Dewey Decimal or Library of Congress?" Your answer, "Let's do both and let the user choose!"

    Imagine if there were a checkbox somewhere in the Control Panel that let you specify how Windows XP-styled controls were implemented. Your choices are either to require applications to link to a new UxCtrl.DLL (let's call this Method A) or to link to COMCTL32.DLL with a custom manifest (let's call this Method B). Well, it means that every component that wanted styled common controls would have to come in two versions, one that linked to UxCtrl and used the new class names in its dialog boxes and calls to CreateWindow, and one that used a manifest and continued to use the class names under their old names.

    #ifdef USE_METHODA
     hwnd = CreateWindow(TEXT("UxButton"), ...);
    #else
     hwnd = CreateWindow(TEXT("Button"), ...);
    #endif
    
    DLG_WHATEVER DIALOG 36, 44, 230, 94
    STYLE WS_POPUP | WS_CAPTION | WS_SYSMENU |
          DS_MODALFRAME | DS_SHELLFONT
    CAPTION "Whatever"
    BEGIN
    #ifdef USE_METHODA
        CONTROL         "Whatever",IDC_WHATEVER,"UxButton",
                        BS_AUTOCHECKBOX | WS_TABSTOP, 14,101,108,9
    #else
        CONTROL         "Whatever",IDC_WHATEVER,"Button",
                        BS_AUTOCHECKBOX | WS_TABSTOP, 14,101,108,9
    #endif
        ...
    

    At run time, every program would have to check this global setting and spawn off either the "Method A" binary or the "Method B" binary.

    Now you might try to pack this all into a single binary with something like this:

    if (GetSystemMetrics(SM_USEMANIFESTFORSTYLEDCONTROLS)) {
     hwnd = CreateWindow(TEXT("Button"), ...);
    } else {
     hwnd = CreateWindow(TEXT("UxButton"), ...);
    }
    ...
    
    if (GetSystemMetrics(SM_USEMANIFESTFORSTYLEDCONTROLS)) {
      DialogBox(hInstance,
                MAKEINTRESOURCE(DLG_TEMPLATE_WITH_OLDNAME_CONTROLS),
                ...
    } else {
      DialogBox(hInstance,
                MAKEINTRESOURCE(DLG_TEMPLATE_WITH_UXCONTROLS),
                ...
    }
    

    But it's not actually that simple because a lot of decisions take place even before your program starts running. For example, if your program specifies a load-time link to UXCTRL.DLL, then that DLL will get loaded before your program even runs, even if the system switch is set to use Method B. A single-binary program that tries to choose between the two methods at runtime will have to do some activation context juggling and delay-loading. Hardly a slam dunk.

    Okay, so now you have two versions of every program. And you also have to decide what should happen if somebody writes and ships a program that uses Method A exclusively, even when the system switch is set to Method B. Does everything still work within that program as if Method A were the system setting, while the rest of the system uses Method B? (If you go this route, then you've completely undermined the point of Method B. The whole point of Method B is to allow programs that rely on specific class names to continue working, but this rogue Method A program is running around using the wrong class names!)

    Now the entire operating system and application compatibility work needs to be done with the checkbox set both to Method A and to Method B, because the compatibility impact of each of the methods is quite different. Okay, that's double the work. Where is triple and quadruple?

    Well, the two different versions of the program need to be kept in sync, since you want them to behave identically. This can of course be managed with judicious use of #ifdefs or runtime branches. But you have to remember both ways of doing things and be mindful of the two method each time you modify the program. Somebody else might come in and "fix a little bug" or "add a little feature" to your program, unaware of how your program manages the shuffle of Method A versus Method B. The mental effort necessary to remember two different ways of doing the same thing plus having to expend that effort to correct mistakes in the code, that's the triple.

    The quadruple? I'm not sure, maybe the ongoing fragility of such a scheme, especially one that, at the end of the day, is a choice between two things that have no real impact on the typical end user.

    Engineering is about making tradeoffs. If you refuse to make the choice, then you're taking the cowardly route and ultimately are creating more work for your team. Instead of solving problems, you're creating them.

    All because you're too chicken to make a hard decision.

  • The Old New Thing

    A different type of writing exercise, this time in preparation for buying a house

    • 30 Comments

    One of my colleagues was overwhelmed by how many times papers need to be signed when you buy a house. A seemingly endless stack of papers. Sign and date here, initial here, initial here, now sign this, and this, and this, and sign and date here, and sign here, and initial here... By the time it's over, your arm is about to fall off.

    Some years later, my colleague was about to buy a new house and began to dread the signature-fest that would invariably ensue at the closing. Another ten-foot-tall stack of papers that needed to be signed and initialed.

    In preparation, my colleague actually did hand exercises to build up stamina and strength. I'm not sure what the exercises consisted of, but they were probably a mix of strength-building wrist exercises and some stints of extended longhand writing.

    Finally closing day came. My colleague walked into the agent's office, sat down, and prepared for the worst, only to be surprised when the stack of papers was only a dozen pages long. What happened to the ten-foot-tall stack of papers that took hours to sign and initial?

    The difference is that my colleague paid cash for the new house rather than taking out a loan. That ten-foot-tall stack of papers? Nearly all of them were related to the mortgage, not to the actual sale of the house. The paperwork associated with a house sale proper is comparatively light.

    But all was not lost. At least my colleague had pretty strong wrists now.

  • The Old New Thing

    If you get confused by a register dump, then you'll just have to accept that some of my postings won't make any sense

    • 29 Comments

    This Web site is not for beginners. I try to write for advanced programmers, and if you're not an advanced programmer, then you'll just have to accept that there will be times you are baffled by what I write. Often I dial the geek back a notch, explaining some things which should be "obvious" to an advanced programmer, such as why storing a constant pointer into a stack location from dynamically-generated code is a clear indicator of a framework thunk. But I will dial it back only so far, and eventually you may just be forced to (horrors!) do your own background research to get yourself up to speed or simply give up and see if you have better luck with the next entry.

    There are some topics I have sitting in my ideas folder which I will probably never actually post about because they are so advanced that even Don Box, COM guru extraordinaire, admitted in email to me that they're too advanced even for his super-advanced book on COM. At the PDC, talks used to be categorized as 100, 200, 300, or 400-level, mimicking the categorization of classes at most U.S. universities, with 100-level classes being introductory, and 400-classes being college senior seminar type stuff. COM weak QueryInterface would be somewhere at the graduate research level. Stuff so esoteric, nobody would actually need to know it.

    To be honest, I don't think I've written anything truly advanced in a long time. It's all been fairly intermediate for the past few years. People don't seem to mind too much, so I'll just keep it going.

  • The Old New Thing

    What are your high school language students complaining about today?

    • 29 Comments

    One of my friends is a high school language teacher, and I used to ask her, "So, what are your students complaining about today?" That was back when her school used the traditional "First we learn the personal pronouns, then we learn the present tense of regular verbs, then we learn nouns in nominative case..." grammar-based language acquisition system. That system works reasonably well for me (although I prefer to learn the major points all at once rather than in pieces), but it requires an appreciation for rules (and their exceptions) that most high school language students don't really have.

    That's why I like to ask her what her students are complaining about. One day it'll be "Why do I have to learn noun genders?", another day, it'll be "Why do we have to change the form of the verb depending on who the subject is?", and another day it'll be "Why are there two different past tenses?" These students usually fail to realize that their own native language suffers from the same (if not more) "Why?" questions. Why does English have both a simple present tense and a present tense with progressive aspect? (And why are there so many exceptions? For example, why can't you say "He is owing me five dollars"?)

    Anyway, one day, when I asked her, "What are your students complaining about today?" I was expecting another point of grammar. What I wasn't expecting was, "They want to know why they need to learn synonyms."

    I guess they figured that once they learned the word big, there would be no need to know words like large, huge, grand, massive, or vast.

    I no longer ask this question, because my friend now teaches based on the principle of comprehensible input, wherein students are exposed to the language by means more closely mimicking the natural way language is acquired: employing stories and conversations, and using contextual or visual cues to infer the meanings of unknown words and tenses instead of learning them from a vocabulary list and a rule sheet. And it's working fantastically. Now I don't have to ask her about how the students are doing: She happily volunteers it! A few years ago, she was excited about students in their sixth week of class reading a story that would have stymied second-year students under the old system. Another story that stands out for me was students who knew where to put accent marks in relatively unfamiliar words even though they were never formally taught the rules on accent marks; they just used their gut feeling.

    Most recently, it was "I formally introduced the past tense and asked one student to give me a sentence in the past tense. He said that when he was younger, he once took his dad's car for a joy ride. Well, that quickly led to a game of one-upsmanship with students fighting to tell their own stories of juvenile delinquency. And they all used the past tense." Sure, they didn't get the irregular verbs quite right, but hey, they didn't get the irregular verbs right when they were learning English for the first time, either.

    Happy birthday, Rebecca!

Page 1 of 4 (32 items) 1234