Holy cow, I wrote a book!
A customer said that their program's I/O pattern is to open a file
and then every so often write about
of data into the file.
They are currently using the
flags to open a file,
and they wanted to know what else they could do
to make their writes go even faster.
Um, for one thing, you stop passing those two flags!
Those two flags in combination basically mean
"Give me the slowest possible I/O performance!"
because they force all I/O to go through to the physical media
flag will be a big help.
This allows the hardware disk cache to do its normal job
completing the I/O immediately
and performing the physical I/O lazily
(perhaps in an optimized order based on subsequent writes).
A 100KB write is a small enough write that your I/O time on
rotational media will be dominated by the seek time.
It'll take five to ten milliseconds to move the head into position
and only one millisecond to write out the data.
You're wasting 80% or more of your time just preparing for the write.
Much better would be to issue the I/O without the
so that the entire 100KB I/O request goes into the hard drive
(It will fit quite easily, since the on-board cache for
today's hard drives will be 8 megabytes or larger.)
Your WriteFile will complete immediately,
and the commit to physical storage will occur
while your program is busy doing computation.
If the writes truly are sporadic (as the customer claims),
the I/O buffer will be flushed out by the time the next round
of application I/O begins.
FILE_FLAG_NO_BUFFERING flag will also
because that allows the operating system disk cache to get involved.
If the application reads back from the file, the read can be satisfied
from the disk cache, avoiding the physical I/O entirely.
As a side note, the
is largely ineffective nowadays,
because SATA drivers ignore the flush request.
The file system doesn't know that the driver is lying to it,
so it will still do all the work on the assumption that the
write-through request worked, even though we know that the extra work
is ultimately pointless.
For example, NTFS will issue metadata writes with a flush
to ensure that the data on the physical media is consistent.
But if the driver is ignoring flush requests, all this extra work
accomplishes nothing aside from wasting I/O bandwidth.
NTFS thinks that the data on the drive is physically
consistent, but it isn't.
The result is that a poorly-timed power outage (or device removal)
can result in metadata corruption that takes a chkdsk
Now, it may be that the customer's program is using the
flags for a specific purpose unrelated to performance,
so you can't just go walking in and ripping them out
without understanding why they were there.
But if they added the flags thinking that it would make
the program run faster,
then they were operating under a false assumption.
sets the background color to COLOR_WINDOW
by setting the class background brush as follows:
wc.hbrBackground = (HBRUSH)(COLOR_WINDOW + 1);
What's with the +1?
Okay, first of all, let's backtrack a bit.
The real first question is,
"What's the deal with taking an integer (COLOR_WINDOW)
and casting it to a HBRUSH and expecting anything sane
The window manager wants to provide multiple ways of setting the class
The first three cases are easy:
If you don't want automatic background drawing,
the hollow brush.
If you want custom background drawing,
then pass NULL as the brush.
And if you want background drawing with a specific brush,
then pass that brush.
It's the last case that is weird.
Now, if RegisterClass were being invented today,
we would satisfy the last requirement
"If you want the background to be a system color,
then use a system color brush like this:
wc.hbrBackground = GetSysColorBrush(COLOR_WINDOW);
But just as
NASA couldn't use the Space Shuttle to rescue the Apollo 13 astronauts,
RegisterClass function couldn't use
GetSysColorBrush for class brushes:
At the time
RegisterClass was designed,
system color brushes had not yet been invented yet.
In fact, they won't have been invented for over a decade.
Therefore, RegisterClass had to find some
way of smuggling an integer inside a pointer,
and the traditional way of doing this is to say that
certain numerically-small pointer values are actually integers
We've seen this with
the HINSTANCE returned by
the MAKEINTATOM macro,
and with the second parameter to the
(There are plenty of other examples.)
The naïve solution would therefore be to say,
"Well, if you want a system color to be used as the brush color,
then just cast the COLOR_XXX value to
and the RegisterClass function will recognize it
as a smuggled integer and treat it as a color code rather than an
And then you run into a problem:
The numeric value of
COLOR_SCROLLBAR is zero.
Casting this to a HBRUSH would result in a
but a NULL brush already means something else:
Don't draw any background at all.
To avoid this conflict, the RegisterClass
function artificially adds 1 to the system color number
so that none of its smuggled integers will be mistaken for
If you don't specify a sort order, then the DIR command
lists the files in the order that the files are returned by the
Um, okay, but that just pushes the question to the next level:
What order does
FindFirstFile return files?
The order in which
FindFirstFile returns files in unspecified.
It is left to the file system driver to return the files in whatever
order it finds most convenient.
Now we're digging into implementation details.
For example, the classic FAT file system simply returns the names
in the order they appear on disk,
and when a file is created, it is merely assigned the first available
slot in the directory.
Slots become available when files are deleted,
and if no slots are available, then a new slot is created at the
Modern FAT (is that an oxymoron?)
with long file names is more complicated
because it needs to find a
sequence of contiguous entries
large enough to hold the name of the file.
There used to be (maybe there still are) some low-level disk
management utilities that would go in and
your directory entries.
The NTFS file system internally maintains directory entries
in a B-tree structure, which means that the most convenient way
of enumerating the directory contents is in B-tree order,
which if you cover one eye and promise not to focus too closely
looks approximately alphabetical for US-English.
(It's not very alphabetical for most other languages,
and it falls apart once you add characters with diacritics
or anything outside of the Latin alphabet, and that includes spaces
The ISO 9660 file system (used by CD-ROMs) requires that directory
entries be lexicographical sorted by ASCII code point.
Pretty much everybody has abandoned the base ISO 9660 file system
and uses one of its many extensions, such as Joliet or UDF,
so you have that additional wrinkle to deal with.
If you are talking to a network file system, then the file system
on the other end of the network cable could be anything at all,
so who knows what its rules are (if it even has rules).
When people ask this question, it's usually in the context of
a media-playing device which plays media from a CD-ROM or USB thumb drive
in the raw native file order.
But they don't ask this question right out;
they ask some side question that they think will solve their problem,
but they don't come out and say what their problem is.
So let's solve the problem in context:
If the storage medium is a CD-ROM or an NTFS-formatted USB thumb drive,
then the files will be enumerated in sort-of-alphabetical order,
so you can give your files names like
000 First track.mp3,
001 Next track.mp3, and so on.
000 First track.mp3
001 Next track.mp3
If the storage medium is a FAT-formatted USB thumb drive,
then the files will be enumerated in a complex order based on the
order in which files are created and deleted and the lengths of their
But the easy way out is simply to remove all the files from a directory
then move file files into the directory in the order you want them
That way, the first available slot is the one at the end of the directory,
so the file entry gets appended.
Of course, none of this behavior is contractual.
NTFS would be completely within its rights to,
for example, return entries in reverse
alphabetical order on odd-numbered days.
Therefore, you shouldn't write a program that relies on any particular
order of enumeration.
(Or even that the order of enumeration is consistent between two runs!)
My niece (two years old at the time) was put in the corner
as punishment for some sort of misdeed.
At the expiration of her punishment,
her grandfather returned and asked her,
"你乖唔乖?" (Are you going to be nice?)
She cheerfully replied,
"仲未乖!" (Still naughty!)
In an unrelated incident,
one of my
honorary nieces was being similarly punished.
She told her aunt who was passing nearby,
"In a little while, my daddy is going to ask me if I'm sorry.
I'm not really sorry, but I'm going to say that I am."
Suppose you have a program that is part of your workflow,
and it has the annoying habit of showing a message box
when it is finished.
You want to automate this workflow,
and part of that automation is dismissing the message box.
Let's start by writing the annoying program:
int WINAPI WinMain(
HINSTANCE hinst, HINSTANCE hinstPrev,
LPSTR lpCmdLine, int nCmdShow)
MessageBox(nullptr, GetTickCount() % 1000 < 800 ?
"Succeeded!" : "Failed!", "Annoying", MB_OK);
This annoying program pretends to do work for a little while,
and then displays a message box saying whether or not it succeeded.
(Let's say it succeeds 80% of the time.)
Our Little Program will automate this task and respond based on
whether the operation succeeded.
This is just a small extension of our previous program which
logs the contents of every message box,
except we are paying attention only to one specific message box.
To the rescue: UI Automation.
public static void Main(string args)
int processId = 0;
bool succeeded = false;
var resultReady = new ManualResetEvent(false);
(sender, e) =>
var element = sender as AutomationElement;
AutomationElement.ProcessIdProperty) != processId)
var text = element.FindFirst(TreeScope.Children,
if (text != null && text.Current.Name == "Succeeded!")
succeeded = true;
var okButton = element.FindFirst(TreeScope.Children,
var invokePattern = okButton.GetCurrentPattern(
InvokePattern.Pattern) as InvokePattern;
// Start the annoying process
Process p = Process.Start("annoying.exe");
processId = p.Id;
// Wait for the result
Most of this program you've seen before.
We register an automation event handler for new window
creation that ignores windows that belong to processes
we don't care about.
That keeps us from being faked out by windows that happen
to be created while our annoying task is running.
For simplicity's sake, I've removed other sanity checks
which verify that the window which appeared is the one
we actually care about.
In real life, we might check the window class or the window title.
I left that out because it's not really relevant to the story.
Once we think we have the window we want,
we suck out the text so we can see whether the message was
a success or failure message.
If it was a success,
we dismiss the dialog box by pushing the OK button;
otherwise we leave the error message on the screen so the
user can see what happened.
we signal the main thread that we have a result.
After registering the event handler, we run the annoying process,
tell the event handler the process ID,
and wait for the signal.
Once the signal arrives,
we see whether it declared the operation a success,
and if so, we
proceed to the next step of, um, say,
launching the calculator.
(I just picked something arbitrary.)
Suppose you want your program to behave differently depending on whether
it is launched from the Start menu,
or by clicking the pinned icon on the taskbar,
or by Scheduled Task,
or from a service,
How can a program detect and distinguish these scenarios?
The answer is you don't.
And you shouldn't try.
Instead of trying to guess how your program was executed,
you should have the launcher tell you how they are
executing your program.
You do this by registering a different command line
for each of the scenarios,
and then checking for that command line in the program.
(We saw a variation of this
a little while ago.)
For example, you could have your Start menu shortcut
contain one command line parameter,
give the taskbar pinned shortcut a different command line parameter,
register yet another command line parameter with the task scheduler,
and have the service launch the program with a still different command
They all run the same program,
but the command line parameter lets the program know what context
it is being run in and alter its behavior accordingly.
It's like creating multiple email addresses that all map to the same inbox.
Many email services let you take an email address and
insert a plus sign followed by anything else you like
before the at-sign,
and it'll all get delivered to the same inbox.
The thing after the plus-sign is ignored for delivery purposes,
but you can use it to help organize your inbox,
so you know that the message sent to
firstname.lastname@example.org is related to your fantasy baseball team,
email@example.com is something about your frequent flier account.
One thing you shouldn't do is try to guess, however.
Programs that magically change their behavior based on details
of the environment lead to problems that are very difficult to debug.
Given this discussion, perhaps you can provide guidance to this customer:
How can my DLL detect that it is running inside a service?
Somebody with a rude name wonders
what the SEE_MASK_UNICODE flag does.
It does nothing.
The flag was introduced when porting the Windows 95 shell
to Windows NT.
further back in history than I have permission to access the Windows
source code history database,
but I can guess how it got introduced.
One of the things that the porting team had to do was make
Unicode versions of all the ANSI functions that Windows 95 created.
Sometimes this was done by
creating separate A and W versions of a function.
Sometimes this was done by
having separate A and W versions of an interface.
adding additional fields to the A version of a structure
with a flag that says whether the ANSI or Unicode members should be used.
My guess is that the porting team initially decided to make
ShellExecuteEx use that third model,
with Unicode strings,
and the mask specified whether the caller preferred you to
use the ANSI strings or the Unicode strings.
Presumably they decided to change course and switch to
But when they switched from one model to the other,
they left that flag behind,
probably with the intention of removing it once all
existing callers had been updated to stop passing the flag,
they never managed to get around to it.
So the flag is just sitting in the header file
even though nobody pays any attention to it.
If you have a string like
and you render it on an Arabic system,
you might get
The leading digits are rendered as Arabic-Indic digits, but the trailing
digits are rendered as European digits.
What's going on here?
This is a feature known as
contextual digit substitution.
You can specify whether European digits are replaced with native
equivalents by going to the Region
control panel (formerly known as Regional and Language Options),
clicking on the Formats tab,
going to Additional settings
(formerly known as Customize this format),
and looking at the
options under Use native digits.
The three options there correspond to
the three values for LOCAL_IDIGITSUBSTITUTION.
Programmatically, you can override the user preference
(if you know that you are in a special case, like an IP address)
following the instructions in MSDN.
As a last resort,
you can stick a Unicode NODS (U+206F) at the beginning of the
string to force European digits, or a
Unicode NADS (U+206E)
to force national digits.
What's the point of contextual digit substitution anyway?
Suppose you have the string
"there are 3 items remaining."
(Let's say that all text in lowercase is in Arabic.)
You want this 3 to be rendered in Arabic-Indic digits
because it is part of an Arabic sentence.
On the other hand, if you have the string
"that's a really nice BMW 350."
you want the 350 to be in European digits since it is part
of the brand name "BMW 350".
Contextual digit substitution chooses whether to use Arabic-Indic
digits or European digits by matching them to the characters that
immediately precede them.
(And if no character precedes them, then it uses the ambient language.)
One of my former colleagues on the Windows kernel team
wasn't afraid to make changes all across the system when necessary.
If the engineering team decided to upgrade to a new version of the
my colleague was the one who
gave it a test-drive on the entire Windows source code,
and fixed all the warnings and errors that kick up
as well as ensuring that it passed the build verification tests
before updating the compiler in the official toolset.
my colleague also ran around being a superhero,
writing tools that needed to be written,
fixing tools that were broken,
Since the effect on the Windows project was so far-reaching,
everybody on the team knew this person,
or at least recognized the name,
and as a result,
my colleage ended up receiving
a lot of email about all different parts of Windows,
be they bug reports, requests for help using a particular component,
And when the question was about something outside my colleague's
sphere of responsibility,
the message was forwarded to the correct people with a simple introduction:
To: XYZ-owners, Y
Subject: Problem with XYZ
Subject: Problem with XYZ
Blah blah blah blah
To: XYZ-owners, Y
Subject: Problem with XYZ
Subject: Problem with XYZ
Blah blah blah blah
I've used this technique a few times,
but it's been a while.
I should start using it again.
At least one of you has come out and said that you post your
complaints here with the expectation that the complaints will
be forwarded to the appropriate team.
This expectation is false.
No such forwarding occurs.
This Web site is not a complaint desk.
My brother-in-law enjoys greeting his nieces when they come over
to visit by throwing them into the air and asking,
"叫聲我?" (Who am I?)
The nieces happily reply,
He then tosses them up into the air a second time and says,
And the nieces happily shout,
One time, my wife was talking with her brother at a normal
and his niece came into the room and said to my wife,
Uncle can't hear you!)
Frank's suggestion below, changed the niece's outburst from
The incident occurred many years ago, and I cannot remember
exactly what was said, so I'll go with what's funnier.