• The Old New Thing

    The importance of passing the WT_EXECUTELONGFUNCTION flag to QueueUserWorkItem


    One of the flags to the QueueUserWorkItem function is WT_EXECUTELONGFUNCTION. The documentation for that flag reads

    The callback function can perform a long wait. This flag helps the system to decide if it should create a new thread.

    As noted in the documentation, the thread pool uses this flag to decide whether it should create a new thread or wait for an existing work item to finish. If all the current thread pool threads are busy running work items and there is another work item to dispatch, it will tend to wait for one of the existing work items to complete if they are "short", because the expectation is that some work item will finish quickly and its thread will become available to run a new work item. On the other hand, if the work items are marked WT_EXECUTELONGFUNCTION, then the thread pool knows that waiting for the running work item to complete is not going to be very productive, so it is more likely to create a new thread.

    If you fail to mark a long work item with the WT_EXECUTELONGFUNCTION flag, then the thread pool ends up waiting for that work item to complete, when it really should be kicking off a new thread. Eventually, the thread pool gets impatient and figures out that you lied to it, and it creates a new thread anyway. But it often takes a while before the thread pool realizes that it's been waiting in vain.

    Let's illustrate this with a simple console program.

    #include <windows.h>
    #include <stdio.h>
    DWORD g_dwLastTick;
    void CALLBACK Tick(void *, BOOLEAN)
     DWORD dwTick = GetTickCount();
     printf("%5d\n", dwTick - g_dwLastTick);
    DWORD CALLBACK Clog(void *)
     return 0;
    int __cdecl
    main(int argc, char* argv[])
     g_dwLastTick = GetTickCount();
     switch (argc) {
     case 2: QueueUserWorkItem(Clog, NULL, 0); break;
     case 3: QueueUserWorkItem(Clog, NULL, WT_EXECUTELONGFUNCTION); break;
     HANDLE hTimer;
     CreateTimerQueueTimer(&hTimer, NULL, Tick, NULL, 250, 250, 0);
     return 0;

    This program creates a periodic thread pool work item that fires every 250ms, and which merely prints how much time has elapsed since the timer was started. As a baseline, run the program with no parameters, and observe that the callbacks occur at roughly 250ms intervals, as expected.


    Next, run the program with a single command line parameter. This causes the "case 2" to be taken, where the "Clog" work item is queued. The "Clog" does what its names does: It clogs up the work item queue by taking a long time (four seconds) to complete. Notice that the first callback doesn't occur for a whole second.


    That's because we queued the "Clog" work item without the WT_EXECUTELONGFUNCTION flag. In other words, we told the thread pool, "Oh, don't worry about this guy, he'll be finished soon." The thread pool wanted to run the Tick event, and since the Clog work item was marked as "fast", the thread pool decided to wait for it and recycle its thread rather than create a new one. After about a second, the thread pool got impatient and spun up a new thread to service the now-long-overdue Tick events.

    Notice that as soon as the first Tick event was processed, three more were fired in rapid succession. That's because the thread pool realized that it had fallen four events behind (thanks to the clog) and had to fire the next three immediately just to clear its backlog. The fifth and subsequent events fire roughly on time because the thread pool has figured out that the Clog really is a clog and should be treated as a long-running event.

    Finally, run the program with two command line parameters. This causes the "case 3" to be taken, where we queue up the Clog but also pass the WT_EXECUTELONGFUNCTION flag.


    Notice that with this hint, the thread pool no longer gets fooled by the Clog and knows to spin up a new thread to handle the Tick events.

    Moral of the story: If you're going to go wading into the thread pool, make sure you play friendly with other kids and let the thread pool know ahead of time whether you're going to take a long time. This allows the thread pool to keep the number of worker threads low (thus reaping the benefits of thread pooling) while still creating enough threads to keep the events flowing smoothly.

    Exercise: What are the consequences to the thread pool if you create a thread pool timer whose callback takes longer to complete than its timer period?

  • The Old New Thing

    News flash: Going for a walk on a nice day makes you feel better


    Scientists continue to amaze us with their research breakthroughs, such as a discovery late last year that going for a walk on a nice day makes you feel better. What would we do without science?

  • The Old New Thing

    FindFirstFile is not a SQL query


    The FindFirstFile function is not a SQL query. It's a very simple directory enumerator. There is a slightly fancier version called FindFirstFileEx, but even that function doesn't add much at present beyond filtering for directories or devices.

    You don't get to pass it sort criteria like or "return the files/directories sorted smallest file first", or "return the files/directories in reverse order of creation" The only filtering you can provide are the wildcard pattern and (if you use FindFirstFileEx) a directory filter. The wildcard language is very limited as well; it can't express queries like "files whose extension are either .exe or .dll" or "all files whose extension is .c plus all directories regardless of extension". You also can't ask it questions like, "Tell me the total size of the files in this directory", for as we saw earlier, this question is underspecified.

    If you want to do any of those advanced queries, you'll have to code it up yourself. Or as Hippie Tim is fond of saying, "Start typing!"

  • The Old New Thing

    The Northwest Mahler Festival performs Mahler's Second Symphony ("Resurrection")


    Last night I attended the Northwest Mahler Festival's performance of Mahler's Second Symphony (The Resurrection). The concert opened with Copland's El Salón México and Barber's Prayers of Kierkegaard. [Typo fixed 12:30pm]

    The Copland was kind of shaky, in a way that I couldn't quite put a finger on. The wind balance seemed a bit off, and it somehow didn't seem to "come together". By contrast, my knowledge of the Barber was zero, so they could've pulled out kazoos and I wouldn't've know that something was amiss.

    The Mahler demands quite a bit from both the woodwind and brass sections, but I was relieved to find that the tricky problem of getting them to play friendly appeared to be nonexistent. The Mahler "came together". (Well, duh, this is the Northwest Mahler Festival, after all.) I was so at ease with it that I started to ignore the occasional technical error...

    Performances of Mahler symphonies have a significant visual component. It's always a blast to see the clarinets playing "Schalltrichter auf", and for the Second, I was oddly fascinated by the rute. (I think my favorite Mahler percussion instrument is the "large hammer striking a wooden block" from the Sixth Symphony. When you see the percussionist raise that huge mallet, you know it's coming... and when the blow finally comes, it sends shock waves through your body.)

    Anyway, there's no real point to this entry. Just babbling about a symphony concert that I found very satisfying.

  • The Old New Thing

    Why does FindFirstFile find short names?


    The FindFirstFile function matches both the short and long names. This can produce somewhat surprising results. For example, if you ask for "*.htm", this also gives you the file "x.html" since its short name is "X~1.HTM".

    Why does it bother matching short names? Shouldn't it match only long names? After all, only old 16-bit programs use short names.

    But that's the problem: 16-bit programs use short names.

    Through a process known as generic thunks, a 16-bit program can load a 32-bit DLL and call into it. Windows 95 and the Windows 16-bit emulation layer in Windows NT rely heavily on generic thunks so that they don't have to write two versions of everything. Instead, the 16-bit version just thunks up to the 32-bit version.

    Note, however, that this would mean that 32-bit DLLs would see two different views of the file system, depending on whether they are hosted from a 16-bit process or a 32-bit process.

    "Then make the FindFirstFile function check to see who its caller is and change its behavior accordingly," doesn't fly because you can't trust the return address.

    Even if this problem were solved, you would still have the problem of 16/32 interop across the process boundary.

    For example, suppose a 16-bit program calls WinExec("notepad X~1.HTM"). The 32-bit Notepad program had better open the file X~1.HTM even though it's a short name. What's more, a common way to get properties of a file such as its last access time is to call FindFirstFile with the file name, since the WIN32_FIND_DATA structure returns that information as part of the find data. (Note: GetFileAttributesEx is a better choice, but that function is comparatively new.) If the FindFirstFile function did not work for short file names, then the above trick would fail for short names passed across the 16/32 boundary.

    As another example, suppose the DLL saves the file name in a location external to the process, say a configuration file, the registry, or a shared memory block. If a 16-bit program program calls into this DLL, it would pass short names, whereas if a 32-bit program calls into the DLL, it would pass long names. If the file system functions returned only long names for 32-bit programs, then the copy of the DLL running in a 32-bit program would not be able to read the data written by the DLL running in a 16-bit program.

  • The Old New Thing

    What is the deal with the ES_OEMCONVERT flag?


    The ES_OEMCONVERT edit control style is a holdover from 16-bit Windows. This ancient MSDN article from the Windows 3.1 SDK describes the flag thus:

    ES_OEMCONVERT causes text entered into the edit control to be converted from ANSI to OEM and then back to ANSI. This ensures proper character conversion when the application calls the AnsiToOem function to convert a Windows string in the edit control to OEM characters. ES_OEMCONVERT is most useful for edit controls that contain filenames.

    Set the wayback machine to, well, January 31, 1992, the date of the article.

    At this time, the predominant Windows platform was Windows 3.0. Windows 3.1 was still a few months away from release, and Windows NT 3.1 was over a year away. The predominant file system was 16-bit FAT, and the relevant feature of FAT of this era for the purpose of this discussion is that file names were stored on disk in the OEM character set. (We discussed the history behind the schism between the OEM and ANSI code pages in an earlier article.)

    Since GUI programs used the ANSI character set, but file names were stored in the OEM character set, the only characters that could be used in file names from GUI programs were those that exist in both character sets. If a character existed in the ANSI character set but not the OEM character set, then there would be no way of using it as a file name; and if a character existed in the OEM character set but not the ANSI character set, the GUI program couldn't manipulate it.

    The ES_OEMCONVERT flag on a edit control ensures that only characters that exist in both the ANSI and OEM character sets are used, hence the remark "ES_OEMCONVERT is most useful for edit controls that contain filenames".

    Fast-forward to today.

    All the popular Windows file systems support Unicode file names and have for ten years. There is no longer a data loss converting from the ANSI character set to the character set used by the file system. Therefore, there is no need to filter out any characters to forestall the user typing a character that will be lost during the conversion to a file name. In other words, the ES_OEMCONVERT flag is pointless today. It's a leftover from the days before Unicode.

    Indeed, if you use this flag, you make your program worse, not better, because it unnecessarily restricts the set of characters that the user will be allowed to use in file names. A user running the US-English version of Windows would not be allowed to enter Chinese characters as a file name, for example, even though the file system is perfectly capable of creating files whose names contain those characters.

  • The Old New Thing

    Watching the game of "Telephone" play out on the Internet


    Let's see if I can get this straight.

    First, Chris Pirillo says (timecode 37:59) he's not entirely pleased with the word "podcast" in Episode 11 of This Week in Tech. The Seattle-PI then reports that the sentiment is shared with "several Microsoft employees" who have coined the word "blogcast" to replace it. Next, c|net picks up the story and says that the word "podcast" is a "faux-pas" on Microsoft campus. [Typo fixed: 9am]

    In this manner, a remark by someone who isn't even a Microsoft employee becomes, through rumor, speculation, and wild extrapolation, a word-ban at Microsoft.

    Pretty neat trick.

  • The Old New Thing

    If InitCommonControls doesn't do anything, why do you have to call it?


    One of the problems beginners run into when they start using shell common controls is that they forget to call the InitCommonControls function. But if you were to disassemble the InitCommonControls function itself, you'll see that it, like the FlushInstructionCache function, doesn't actually do anything.

    Then why do you need to call it?

    As with FlushInstructionCache, what's important is not what it performs, but just the fact that you called it.

    Recall that merely listing a lib file in your dependencies doesn't actually cause your program to be bound to the corresponding DLL. You have to call a function in that DLL in order for there to be an import entry for that DLL. And InitCommonControls is that function.

    Without the InitCommonControls function, a program that wants to use the shell common controls library would otherwise have no reference to COMCTL32.DLL in its import table. This means that when the program loads, COMCTL32.DLL is not loaded and therefore is not initialized. Which means that it doesn't register its window classes. Which means that your call to the CreateWindow function fails because the window class has not been registered.

    That's why you have to call a function that does nothing. It's for your own good.

    (Of course, there's the new InitCommonControlsEx function that lets you specify which classes you would like to be registered. Only the classic Windows 95 classes are registered when COMCTL32.DLL loads. For everything else you have to ask for it explicitly.)

  • The Old New Thing

    The apocryphal history of file system tunnelling


    One of the file system features you may find yourself surprised by is tunneling, wherein the creation timestamp and short/long names of a file are taken from a file that existed in the directory previously. In other words, if you delete some file "File with long name.txt" and then create a new file with the same name, that new file will have the same short name and the same creation time as the original file. You can read this KB article for details on what operations are sensitive to tunnelling.

    Why does tunneling exist at all?

    When you use a program to edit an existing file, then save it, you expect the original creation timestamp to be preserved, since you're editing a file, not creating a new one. But internally, many programs save a file by performing a combination of save, delete, and rename operations (such as the ones listed in the linked article), and without tunneling, the creation time of the file would seem to change even though from the end user's point of view, no file got created.

    As another example of the importance of tunneling, consider that file "File with long name.txt", whose short name is say "FILEWI~1.TXT". You load this file into a program that is not long-filename-aware and save it. It deletes the old "FILEWI~1.TXT" and creates a new one with the same name. Without tunnelling, the associated long name of the file would be lost. Instead of a friendly long name, the file name got corrupted into this thing with squiggly marks. Not good.

    But where did the name "tunneling" come from?

    From quantum mechanics.

    Consider the following analogy: You have two holes in the ground, and a particle is in the first hole (A) and doesn't have enough energy to get out. It only has enough energy to get as high as the dotted line.

      A   B  

    You get distracted for a little while, maybe watch the Super Bowl halftime show, and when you come back, the particle somehow is now in hole B. This is impossible in classical mechanics, but thanks to the wacky world of quantum mechanics, it is not only possible, but actually happens. The phenomenon is known as tunneling, because it's as if the particle "dug a tunnel" between the two holes, thereby allowing it to get from one hole to another without ever going above the dotted line.

    In the case of file system tunneling, it is information that appears to violate the laws of classical mechanics. The information was destroyed (by deleting or renaming the file), yet somehow managed to reconstruct itself on the other side of a temporal barrier.

    The developer who was responsible for implementing tunneling on Windows 95 got kind of carried away with the quantum mechanics analogy: The fragments of information about recently-deleted or recently-renamed files are kept in data structures called "quarks".

  • The Old New Thing

    When Marketing edits your PDC talk description


    A few years ago, I told a story of how Marketing messed up a bunch of PDC slides by "helpfully" expanding acronyms... into the wrong phrases. Today I got to see Marketing's handiwork again, as they edited my talk description. (Oh, and psst, Marketing folks, you might want to link to the full list of PDC sessions from your Conference Tracks and Sessions page. Unless, of course, y'know, you don't want people to know about it.)

    For one thing, they stuck my name into the description of the talk, thereby drawing attention to me rather than putting the focus on the actual talk topic. Because I'm not there to be me. I'm there to give a talk. If I were just there to be me, the title would be "Raymond Chen reads the newspaper for an hour while listening to music on his headphones."

    (That's why I don't do interviews. Interviews are about the interviewee, and I don't want to talk about me. People should care about the technology, not the people behind it.)

    They also trimmed my topic list but stopped before the punch line.

    ... asynchronous input queues, the hazards of attaching thread input, and other tricks and traps ...

    The punch line was "... and how it happens without your knowledge." After all, you don't care about the fine details of a feature you don't use. The point is that it's happening behind your back so you'd better know about it because you're using it whether you realize it or not.

    They also took out the reference to finger puppets.

Page 376 of 457 (4,570 items) «374375376377378»