July, 2006

  • The Old New Thing

    How are DLL functions exported in 32-bit Windows?

    • 27 Comments

    The designers of 32-bit Windows didn't have to worry quite so much about squeezing everything into 256KB of memory. Since modules in Win32 are based on demand-paging, all you have to do is map the entire image into memory and then run around accessing the parts you need. There is no distinction between resident and non-resident names; the names of exported functions are just stored in the image, with a pointer (well, a relative virtual address) to the name stored in the export table.

    Unlike the 16-bit ordinal export table, the 32-bit ordinal export table is not sparse. If your DLL exports two functions, one as ordinal 10 and one as ordinal 1000, you will have a 991-entry table that consists of two actual function pointers and a lot of zeros. Thus, you should try not to have large gaps in your ordinal exports, or you will be wasting space in your DLL's export table.

    As I noted above, there is only one exported names table, so you don't have to distinguish between resident and non-resident names. The exported names table functions in the same manner as the exported names table in 16-bit Windows, mapping names to ordinals. Unlike the 16-bit named export tables, where order is irrelevant, the exported names table in 32-bit Windows is kept sorted so that a more efficient binary search can be used to locate functions.

    As with 16-bit Windows, every named function is assigned an ordinal. If the programmer didn't assign one in the module definition file, the linker will make one up for you, and as with 16-bit Windows, the value the linker makes up can vary from build to build. However, there is a major difference between the two models: Recall that named exports in 16-bit Windows were discouraged (on efficiency grounds), and as a result, every exported function was explicitly assigned an ordinal, which was the preferred way of linking to the function. On the other hand, named exports in 32-bit Windows are the norm, with no explicit ordinal assignment. This means that the ordinal for a named export is not fixed. For example, let's look at the ordinal that got assigned to the kernel32 function LocalAlloc in the early years:

    Windows NT 3.1314
    Windows NT 3.5372
    Windows 95501
    Windows NT 4.0407

    Now, some people are in the habit of reverse-engineering import libraries, probably because they can't be bothered to download the Platform SDK and get the real import libraries. The problem with generating the import library manually is that you can't tell whether the ordinal that was assigned to, say, the LoadLibrary function was assigned by the module definition file (and therefore will not change from build to build) or was just auto-generated by the linker (in which case the ordinal will change). The import library generation tools could just play it safe and use the named export, since that will work in both cases, but for some reason, they use the ordinal export instead. (This is probably a leftover from 16-bit Windows, where ordinals were preferred over names, as we saw earlier.)

    This unfortunate choice on the part of the import library generation tools to live dangerously has created compatibility problems for the DirectX team. (I don't know why DirectX got hit by this harder than other teams. Perhaps because game developers don't have the time to learn the fine details of Win32; they just want to write their game.) Since they used one of these tools, they ended up linking to DirectX functions like DirectDrawCreate by ordinal rather than by name, and then when the next version of DirectX came out and the name was assigned a different ordinal by the linker, their programs crashed pretty badly. The DirectX team had to go back to the old DLLs, write down all the ordinals that the linker randomly assigned, and explicitly assign those ordinals in the module definition files so they wouldn't move around in the future.

    There are other reasons why you cannot generate an import library from a DLL; I'll pick up those topics later when I talk about import libraries in more detail.

    Next time, forwarders.

  • The Old New Thing

    The day the peloton lost its way

    • 21 Comments

    It's the one time a year that more than fifty people in the United States actually give a hoot about bicycle racing. Yes, it's the Tour de France. (Follow the racers live via GPS!) But this story isn't about the Tour. It's about Stage 4 of the 2005 ENECO Tour.

    You can follow along with the live report. Everything was going pretty much like a typical racing stage, until time code 13:55.

    Veikkanen is now gone from the back of the lead group, but the peloton appears to have taken a wrong turn!!

    The peloton is just stopped in the road.

    nobody knows what happened, riders are getting off their bikes. Zabel and Verbrugghe are now negotatiting with the race leadership. Verbrugghe gestures that the peloton must follow him and that he'll just lead them back unto the race parcours. He's a local guy, and he probably knows the way. The peloton now over small roads that aren't part of the parcours with Verbrugghe leading the way.

    It's almost a Monty Python episode! Riders are everywhere!

    That's right. The peloton got lost.

    But keep reading. The story gets more and more bizarre. The police get involved to detain the three lead riders! Here's a news report that is less stream-of-consciousness.

    I remember in last year's Tour, a rider whose name I recognized was disqualified for being too far behind the stage winner. Apparently, the rules require all riders to be within a certain amount of time of the leader; I guess the idea is to make the domestiques at least put up the appearance of trying to win. I wondered if there ever was a Tour stage where the winner had such a huge lead that the entire peloton was disqualfied. What a strange situation that would be!

    It turns out that it actually did happen. I lost the link, however, but maybe somebody can find it for me. There was one stage some years ago where the stage winner had such an enormous lead on the peloton that according to the rules, the entire peloton would have been disqualified! However, the race rules also contain a clause granting the race director the authority to cancel the "automatic disqualification" if doing so would be to the detriment of the race itself. Obviously, this was one of those times.

  • The Old New Thing

    How were DLL functions imported in 16-bit Windows?

    • 16 Comments

    Last time, we looked at the way functions were exported from 16-bit DLLs. Today, we'll look at how they were imported.

    When each segment is loaded into memory, the raw contents are loaded from disk, and then relocation fixups are applied. A fixup for an imported function consists of the name of the target DLL, the target function (either a name or ordinal), and the position of the first location in the segment where the fixup needs to be applied. All imported addresses are far addresses since they reside in another segment. (If they were in the same segment, then they would be in the same DLL, so you wouldn't be importing it!) On 16-bit Windows, a far address is four bytes (a two-byte selector and a two-byte offset), and since the target address is not known when the DLL is generated, those four bytes are just placeholders, waiting to be filled in with the actual target address when the import is resolved. And it is those placeholder bytes that serve double duty.

    All the calls within a segment that import the same function are chained in a linked list, where the relocation record points to the first entry. The items in the linked list? The four-byte placeholders. And the "next" pointer in the linked list? The placeholder itself! For example, suppose we have a segment that requires two fixups for the function GetPrivateProfileInt, which happens to be kernel function 127. The relocation table entry would say "This segment needs function 127 from KERNEL; start at offset 01D1". The on-disk copy of the segment might go something like this:

    ...
    01D09A
    01D1FE
    01D201
    01D300
    01D000
    ...
    01FD9A
    01FEFF
    01FFFF
    020000
    020100
    ...

    To apply the fixup, we first call GetProcAddress to get the address of kernel function 127. Then we go to the first fixup location (0x01D1), write the address there, then look at the value we overwrote. That value was 0x01FE, so we now go to offset 0x01FE and write the address there, too. The value we overwrote was 0xFFFF, which marks the end of the fixup chain.

    But what if the call to GetProcAddress fails? (Say, there is no such function 127 in KERNEL.) Then instead of writing the address of the target function, the loader wrote the address of a function that displayed the "Call to Undefined Dynalink" fatal error dialog.

    Okay, that's a quick introduction to how functions are imported and exported on 16-bit Windows. Next time, we'll look at the transition to 32-bit Windows and the design decisions that went into the new model.

  • The Old New Thing

    How were DLL functions exported in 16-bit Windows?

    • 20 Comments

    The whole point of dynamic link libraries (DLLs) is that the linkage is dynamic. Whereas statically-linked libraries are built into the final product, a module that uses a dynamically-linked library merely says, "I would like function X from Y.DLL, please." This technique has advantages and disadvantages. One advantage is more efficient use of storage, since there is only one copy of Y.DLL in memory rather than a separate copy bound into each module. Another advantage is that an update to Y.DLL can be made without having to re-compile all the programs that used it. On the other hand, the ability to swap in functionality automatically is also one of the main disadvantages of dynamic link libraries, because one program can change a DLL that has cascade effects on other clients of that DLL.

    Anyway, let's start with how 16-bit Windows managed imports and exports. After that, we'll see how things changed during the switch to 32-bit Windows, and then we'll take a look at the compiler-specific dllimport declaration specifier. (I already discussed dllexport earlier.)

    A 16-bit DLL has not one but three export tables. (Things are actually more complicated than I describe them here, but I'm going to skip over the nitpicky details just to keep everyone's heads from exploding.) The most important table is a sparse array of functions, indexed by a 1-based integer (the "ordinal"). It is this function table that is the master list of all exported functions. If you request a function by ordinal, the ordinal is looked up in this table. The table is physically rather complicated due to the sparseness, but logically, it looks like this:

    Ordinal Addressother goo
    102:0014...
    204:0000...
    502:02C8...

    The first column in the table is the ordinal of the function, and the second function describes where the function can be found. (Notice that there is no function 3 or 4 in this DLL.)

    Things get interesting when you want to export a function by name. The exported names table is a list of function names with their associated ordinal equivalents. For example, a section of the exported names table for the 16-bit window manager (USER) went like this:

    ...
    ClipCursor16
    GetCursorPos17
    SetCapture18
    ...

    If somebody asks for the address of the function ClipCursor, the exported names table is consulted, the value 16 is retrieved, and the function at position 16 in the ordinal export table is returned. Although you can't see it here, there was no requirement that the names in the exported names table be in any particular order, or that every ordinal have a corresponding name.

    Wait, did I say the exported names table? I'm sorry, that was an oversimplification. There are actually two exported names tables, the resident names table and the non-resident names table. As their names suggest, the names in the resident names table remain in memory as long as the DLL is loaded, whereas the names in the non-resident names table are loaded into memory only when somebody calls GetProcAddress (or one of its moral equivalents). This distinction is a reflection of the extremely tight memory constraints that Windows had to run within back in those days. For example, the window manager (USER) has over six hundred export functions; if all the exported names were kept resident, that would be over ten kilobytes of data. You'd be wasting four percent of the memory of your 256KB machine remembering things you don't need most of the time.

    The large size of the table for exported function names meant that only functions that are passed to GetProcAddress with high frequency deserve to be placed in the resident names table. For most DLLs, no function falls into this category, and the resident names table is empty. (Head-exploding details deleted for sanity's sake.)

    Since obtaining a function by name is so expensive (requiring the non-resident names table to be loaded from disk so it can be searched), all functions exported by operating system DLLs are exported both by name and by ordinal, with the ordinal taking precedence in the import library table. Obtaining a procedure address by ordinal avoids the name tables entirely.

    Notice that every named function has a corresponding ordinal. If you do not assign an ordinal to your named function in your module definition file, the linker will make one up for you. (However, the value that it makes up need not be the same from build to build.) This situation did not occur in practice, for as we noted above, everybody explicitly assigned an ordinal to their exports and put that ordinal in the import library in order to avoid the huge cost of a name-based function lookup.

    That's a quick look at how functions were exported in 16-bit Windows. Next time, we'll look at how they are imported.

  • The Old New Thing

    Glass houses are great places to throw stones

    • 105 Comments

    Whenever I write an article explaining that programs should avoid doing X, I can confidently rely on a comment saying, "Well, Microsoft Product Q does this!" as if to say, "Gotcha, you hypocrite!"

    But they're saying "gotcha" to the wrong person. Because, and I'm sure it's a shock to many people to read this, I did not personally write every line of software Microsoft ever produced. (And even if I did write it, I may have written it as a younger developer, before I learned about said rule. Because, and I'm sure this is also a shock to many people, I was once a beginner, too.)

    If you find a Microsoft product breaking a rule, then go complain to that product team. Complaining to me won't accomplish anything. I don't have access to their source code, and even if I did, I certainly don't have permission to go in and make changes to their code, nor do I have the time to go in and learn how their product works and figure out the right place to make the fix. Furthermore, and I don't know if you all can handle three shocking revelations in one article, product teams do not send me every line of code for review.

    Indeed, one of the reasons I write here about things programs should or shouldn't do is because I myself will see a Microsoft product breaking a rule! By discussing the problem here rather than in an internal mailing list, the information gets out to everybody. And maybe, just maybe, the product team will read the entry and say, "Oops, I think we do that." Because (shocking revelation number four) not all Microsoft programmers are seasoned experts in Win32 user-interface programming.

    (Articles where I was consciously tapping my colleagues on the head include my discussion of CallMsgFilter, the long and sad story of the Shell Folders key, and reminding you to pass unhandled messages to DefWindowProc. In fact, for every "do/don't do this" article, I'd say odds are good that with enough searching, you can find a Microsoft product that breaks the rule. And when you do, complain to that product team. Even the difference between the tray and the notification area was in part a response to all the other groups that perpetuate the misuse of the terminology.)

    So when I write something like, "Applications shouldn't do this", go ahead and insert the phrase "and this means all applications, including those published by Microsoft." When I write, "some annoying programs", go ahead and insert the phrase, "which might even include programs published by Microsoft". I'm not going to insert those phrases into every sentence I write. I'm assuming you're smart enough to realize that general statements apply to everyone regardless of who signs their paychecks.

    Of course, if the consensus of my readership is that I shouldn't tell you not to do things until every last Microsoft product has been scoured to ensure that none of them violate that rule either, then I can abide by that decision. I'll just stop posting those tips here and keep them on the internal mailing lists. It's much less work for me.

  • The Old New Thing

    Win $5000 every summer for life (some restrictions on your life apply)

    • 17 Comments

    Back in 2003, M&M offered a chance to win $5000 every summer for life, but if you looked more carefully, the offer actually read, "Win $5000 Every Summer For Life*", and the asterisk at the bottom read, "Maximum 50 years". That fine print was filled with strange stuff. For example,

    3. Sponsor responsible only for delivery of prize; not responsible for prize utility, quality or otherwise.
    ...
    10. Sponsor: M&M/Mars, High Street, Hackettstown, NJ 07840.

    One of the prizes was approximately ten pounds of M&Ms. The logical conclusion: "M&M is not responsible for the quality of M&Ms."

    I was reminded of this by the recent flap over how hard it was for one person to cancel his AOL account. There was an AOL contest some years back that offered a chance to "Win free AOL dial-up service for life!", and it too had limited your life to 50 years in the fine print. That one would actually be fun, though. Imagine, in the year 2052, AOL will still have to keep one modem up and running just for this contest winner.

  • The Old New Thing

    Pidls and monikers do roughly the same thing, just backwards

    • 11 Comments

    When operating with the Windows shell, you will almost certainly find yourself at some point working with a pointer to an item ID list, known also as a "pidl" (rhymes with "middle"). On the other hand, when working with OLE you may find yourself having do deal with monikers. In a sense, they both do the same thing. They let you refer to some sort of object inside a namespace (to which you can bind or otherwise operate on), they have a hierarchical structure, you can persist them, and so on.

    Why, then, did the Windows shell team invent pidls when monikers do the same thing?

    The fundamental difference between pidls and monikers is not in what you can do with them, but rather in their "bias". A moniker is a COM object. This means that its primary existence is in the form of a COM interface (IMoniker), which means that the code behind the object is loaded and ready. You can convert it to a persistence format, but that's not how monikers really spend their lives. If you have an array of monikers, all the code behind those objects has been loaded and initialized.

    A pidl, on the other hand, spends most of its life in its persistence format. Only when you bind to it does a live COM object come out. Consider, for example, the case where you enumerated the contents of a shell folder. This produces a pidl for each item in the folder, but producing and retaining that pidl doesn't require that the code for each item be loaded and initialized. A folder with a thousand items produces a thousand little chunks of data, not a thousand COM objects. Consider, for example, a folder that contains a dozen Excel spreadsheets. If you enumerate the contents of the folder, you don't want Excel to fire up and load each of those spreadsheets to give you a dozen live, running spreadsheet objects. At this stage, you are just talking about that spreadsheet file. It's not until the user double-clicks on other otherwise tries to activate that spreadsheet that you want Excel to start up.

    Since the shell spends nearly all its time talking about things and comparatively rarely talks to them, an object that spends most of its time "dead" was more appropriate. You might say that the difference between pidls and monikers is a matter of life and death.

  • The Old New Thing

    News flash: Big houses have bigger heating bills

    • 18 Comments

    Big houses have bigger heating bills. Thanks, Associated Press! I learn something new every day.

    (Previous amazing fact.)

  • The Old New Thing

    Not everybody with a non-Windows partition type is a geek

    • 79 Comments

    In the discussions following why Windows setup lays down a new boot sector, some commenters suggested that Windows setup could detect the presence of a non-Windows partition as a sign that the machine onto which the operating system is being installed belongs to a geek. In that way, the typical consumer would be spared from having to deal with a confusing geeky dialog box that they don't know how to answer.

    The problem with this plan is that not everybody with a non-Windows partition type is necessarily a geek. Many OEM machines ship with a hard drive split into two partitions, one formatted for Windows and the second a small non-Windows partition to be used during system diagnostics and recovery. The presence of this small non-Windows partition is typically not well-known, and it comes into play only when you boot from the manufacturer's "system recovery CD".

    The upshot of this is that if Windows setup took the "anybody with a non-Windows partition must be a geek" approach, it would end up tagging an awful lot of people as geeks who really aren't.

    Now, you might say, "Well, only geeks install the operating system anyway. Normal people typically buy a computer with the operating system pre-installed. The fact that they are running Windows setup proves that they're a geek in the first place. Therefore, Windows setup should be optimized for geeks." Indeed, the premise of this argument—that only geeks run Windows setup—is true, but only once you've reached steady state. In the months immediately following the release of a new version of Windows, everybody is installing the operating system, geeks and non-geeks alike. (There is also an influx of non-geek people installing Windows every Christmas.) Magazine reviewers are writing boatloads of articles on the new operating system, and the initial setup experience is the very first thing they notice about Windows. It had better be smooth and painless.

  • The Old New Thing

    Wenn Ausländer Deutsch phonetisch singen

    • 18 Comments

    When foreigners sing German phonetically, the results are kind of painful. Listen, if you dare, to Johnny Cash sing "Wer kennt den Weg? (I Walk the Line)" [WMV] [Real]. If you can't get enough, you can grab the lyrics and sing along.

    Listening to the recording brought back painful memories of my high school German class, where there were those students who simply couldn't lose their thick American accents. It has been twenty years since I last studied German formally, and I'm certain that my own German pronunciation has picked up a Swedish accent in the meantime. My vocabulary has also deteriorated (on that page I get the gender of the word "Person" wrong), although my grammar at least has thankfully held steady. (Well, except when I lose my senses entirely and forget how to decline adjectival nouns.)

    If Johnny Cash isn't your thing, you can go to the article and click to listen to David Bowie sing Space Oddity in Italian, or the Beatles sing Get Back in mixed German and French...

Page 3 of 4 (39 items) 1234