August, 2013

  • The Old New Thing

    Can an x64 function repurpose parameter home space as general scratch space?


    We saw some time ago that the x64 calling convention in Windows reserves space for the register parameters on the stack, in case the called function wants to spill them. But can the called function use the memory for other purposes, too?

    You sort of already know the answer to this question. Consider this function:

    void testfunction(int a)
     a = 42;

    How would a naïve compiler generate code for this function?

        sub rsp, 8 ;; realign the stack
        ;; spill all register parameters into home locations
        mov [rsp+0x10], rcx
        mov [rsp+0x18], rdx
        mov [rsp+0x20], r8
        mov [rsp+0x28], r9
        ;; a = 42
        mov [rsp+0x10], 42
        ;; return
        add rsp, 8 ;; clean up local frame

    Observe that after spilling the register parameters into their home locations onto the stack, the function modified the local variable, which updated the value in the home location.

    Since a function can arbitrarily modify a parameter, you can see that a function is therefore allowed to arbitrarily modify a parameter's home location. At which point you can see that an optimizing compiler might choose an arbitrary value completely unrelated to the parameter.

    Our test function has only one parameter. What about the other three home registers?

    The caller is responsible for allocating space for parameters to the callee, and must always allocate sufficient space for the 4 register parameters, even if the callee doesn't have that many parameters.

    A function can therefore treat those 32 bytes as bonus free play. The rationale behind those 32 bytes is that it gives you a place to spill your inbound register parameters so that they will be adjacent to the stack-based parameters. (We saw how the naïve compiler took advantage of this by not trying to be clever in its function prologue and simply spilling all register parameters whether it needs them or not.)

    Nevertheless, you are free to use them for whatever purpose you like, and if you're looking at heavily-optimized code, you'll probably find that the compiler found all sorts of clever things it can do with them. For example, a common trick is to use them to save the nonvolatile registers that the function locally uses to hold the corresponding parameter!

    (Did this article look familiar? Turns out I covered this article a few years ago, but I'm senile and accidentally repeated a topic. And since I put so much effort into writing it, I'm going to make you suffer through it, even though it's a repeat. Hey, television programs repeat during the summer.)

  • The Old New Thing

    How can I write to a file only as long as nobody's looking?


    A customer wanted to know how to detect that the user has opened Notepad to view a particular file. They had come up with method based on polling and sniffing the Notepad title bar, but they found that it consumed a lot of CPU. (They hadn't noticed yet that it doesn't localize, and that it can trigger false positives since Notepad shows only the file name and not the full path in the title bar.)

    Okay, let's step back and make sure we understand the problem, because this sounds like the sort of thing where the customer is looking for an answer rather than a solution. After all, why Notepad? What about some other text editor? What about Explorer's preview pane?

    The scenario here is that the program is generating some sort of output file, and they want the program to exit if somebody double-clicks the output file in an attempt to open it in Notepad. The customer wasn't specifically interested in Notepad, but since that was the most common case, that's all that they were really interested in.

    One option is to open the file in FILE_SHARE_READ mode. This allows other applications to open the file for reading even while you are writing. If, on the other hand, you expect the user to try to edit the output file and save the result, then they will either encounter a sharing violation (if you opened in deny-write mode) or overwrite the changes that the generator program had made in the meantime.

    The customer said that they were fine with the program just exiting if somebody tried to look at the output file while it was being generated. In that case, they could use an oplock, specifically a Read-Write oplock. The Read-Write oplock grants you exclusive access to the file, but notifies you if somebody else tries to access the same file, so that you can close your handle and let the other application gain access.

  • The Old New Thing

    The format of data and custom resources


    Continuing the highly-sporadic series of Win32 resource formats, today we'll look at the format of RT_DATA resources, which are declared in resource files as RCDATA. Also the format of custom resources, which are declared in resource files by just giving the custom resource name or ordinal as the second word on the declaration.

    The format is very simple: It's just raw binary data. If you ask for a 16-bit integer followed by an ANSI string, then you get a 16-bit integer followed by an ANSI string.

    If you ask for the data to be inserted from a file, then the contents of the file become the resource data. No processing is done on the data. You get what you get.

    In particular (and this is what catches some people), if your file is a text file, the resource compiler will just treat your text file as binary data and dump it raw into the resources. It doesn't know that it's a text file; it's just a file. This means that if you want a null terminator at the end of your text file, you need to put a null terminator at the end of the text file. Nobody's going to magically do it for you.

    (The Size­Of­Resource function may come in handy here.)

  • The Old New Thing

    How can I find out which process and user is modifying a file?


    When troubleshooting a problem, you may discover that a file is being modified that shouldn't, and you figure out would be nice if there were some way of finding out which process is modifying the file (so you can get it to stop).

    Enter the security auditing system.

    Every securable object has an associated system access control list (SACL) which controls what audit events are raised when a request is made to access the object. You can say, for example, "Log an event in the security event log if somebody tries to open this file for writing but is denied access," or "Log an event in the security event log if somebody successfully creates a new file in this directory."

    Here's how it works. Let's say that you want to access successful requests from any user to write to a particular file.

    View the Properties of the file, go to the Security tab, and click Advanced, then go to the Auditing tab and elevate to administrator if necessary.

    Next, click Add. What happens next depends on what version of Windows you're using, since the interface varies slightly (but the basic idea remains the same). When asked for the security principal, set the Location to the local computer and use the object name Everyone.

    Older vesions of Windows will give you a grid of options. Look for the row corresponding to the operation you want to audit and check the box under Successful if you want to audit successful accesses or the box under Failed to audit failed accesses. (Or check both to audit both successful and failed accesses.) Repeat for each access you want to audit. In our case, we would check the Create files / write data and Create folders / append data boxes under the Successful column.

    Newer versions of Windows break the grid up into two questions. The first is whether you want to audit Success, Fail, or All access. In our case, we want to audit Success. The next question is what type of access you want to audit, and in our case we would check Write. (Or for finer control, click Show advanced permissions and check Create files / write data and Create folders / append data.)

    OK your way out of all the dialog boxes to save the changes.

    All right, let's take this thing out for a spin. Open the file in Notepad, make some change, and then Save them. Now open the Event Viewer and go to the Security event log.

    And... no log entry.

    That's because I forgot a step: You have to enable object access auditing.

    Open the Control Panel and look in the Administrative Tools folder. From there, you can run the Local Security Policy snap-in. If you are a command line nerd, you can run secpol.msc.

    Under Local Policies, Audit Policy set the Audit object access policy to enable global auditing of successful or failed accesses, depending on what you need.

    Okay, let's try it again. Modify the file and save it. Now go back to the security event viewer and you'll see audit success events in your log. Again, depending on what version of Windows you're using, the successful audit event will appear differently. For example, older versions of Windows might show

    Event Type: Success Audit
    Event Source: Security
    Event Category: Object Access
    Event ID: 567
    Date: ...
    Time: ...
    User: ...
    Computer: ...
    Object Access Attempt:
    Object Server: Security
    Handle ID: 208
    Object Type: File
    Process ID: 1964
    Image File Name: C:\WINDOWS\system32\notepad.exe
    Access Mask: WriteData (or AddFile)
    AppendData (or AddSubdirectory or CreatePipeInstance)

    whereas newer versions might show

    Keywords: Audit Success
    Date and Time: ...
    Source: Microsoft Windows security auditing
    Event ID: 4663
    Task Category: File System
    An attempt was made to access an object.
    Security ID: computer\user
    Account Name: user
    Account Domain: computer
    Logon ID: 0x27ADB
    Object Server: Security
    Object Type Name: File
    Object Name: C:\test.txt
    Handle ID: 0x15c
    Resource Attributes: S:AI
    Process Information:
    Process ID: 0xdb0
    Process Name: C:\Windows\System32\notepad.exe
    Access Request Information:
    Accesses: WriteData (or AddFile)
    AppendData (or AddSubdirectory or CreatePipeInstance)
    Access Mask: 0x6

    Either way, you can see which process obtained write access to the file, running as what user, at what time.

    Newer versions of Windows include a bit more information in the event log entry to make it easier to find the access request you're looking for as well as chase the access further. (For example, from the Logon ID, you can figure out which logon session modified the file.)

    This feature has been around since the beginning of Windows NT, but it seems that very few people know about it. Whenver I point it out to people, they say, "Hey, that's cool. How long has that feature been there?"

    Now you too can look smart.

  • The Old New Thing

    The challenge of improvisational historical comedy


    I dreamed that one of my friends signed me up for some sort of combination Punk'd/Thank God You're Here improv show without my knowledge, and I had to explain how her husband had unwittingly coined some nonspecific Internet meme.

  • The Old New Thing

    Why doesn't the "Automatically move pointer to the default button in a dialog box" work for nonstandard dialog boxes, and how do I add it to my own nonstandard dialog boxes?


    The Mouse control panel has a setting called Automatically move pointer to the default button in a dialog box, known informally as Snap to default button or simply Snap To. You may have discovered that it doesn't work for all dialog boxes. Why not?

    The Snap To feature is implemented by the dialog manager. When the window is shown and the setting is enabled, it will center the pointer on the default button. If your application does not use the dialog manager but instead creates its own custom dialog-like windows, then naturally the code in the standard dialog manager will not run.

    If you want your nonstandard dialog box to support the Snap To feature, you get to implement it yourself.

    Here's a Little Program that creates a window with a default pushbutton inside it, and which centers the mouse on the button when the window is shown. Start with our scratch program and make these changes:

    POINT GetRectCenter(LPCRECT prc)
      POINT pt = {
        prc->left + (prc->right - prc->left) / 2,
        prc->top + (prc->bottom - prc->top) / 2
      return pt;

    The Get­Rect­Center helper function calculates center of a rectangle.

    BOOL OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)
                   TEXT("Button 1"),
                   0, 0, 200, 50,
      return TRUE;

    When our main window is created, we put a default button inside it.

    void OnShowWindow(HWND hwnd, BOOL fShow, UINT status)
      if (fShow && status == 0)
        PostMessage(hwnd, WM_CHECKSNAPDEFBUTTON, 0, 0);
    void OnCheckSnapDefButton(HWND hwnd)
      BOOL fSnapToDefButton;
      if (SystemParametersInfo(SPI_GETSNAPTODEFBUTTON, 0,
                               &fSnapToDefButton, 0) &&
                               fSnapToDefButton &&
                               hwnd == GetForegroundWindow())
        RECT rcButton;
        GetWindowRect(GetDlgItem(hwnd, 1), &rcButton);
        POINT ptCenter = GetRectCenter(&rcButton);
        SetCursorPos(ptCenter.x, ptCenter.y);

    When the window is shown, we post a message to check the Snap To setting after things have settled down. Once things settle down, we check the Snap To setting, and if it's enabled, and if our window is still the foreground window, then we center the cursor on our button.

    It's important to check that our window is still the foreground window, because it would be rude to move the cursor to our button even if we opened in the background.

    That's why we need to post a message to perform the check later. The WM_SHOW­WINDOW message is sent early in the Show­Window calculations, before the activation actually changes. If we performed the check then, the answer would always be, "No, you're not the foreground window," and we would always back off.

    Naturally, we need to hook up our new messages.

      HANDLE_MSG(hwnd, WM_SHOWWINDOW, OnShowWindow);
      case WM_CHECKSNAPDEFBUTTON: OnCheckSnapDefButton(hwnd); break;

    And there you have it, a program that honors the Automatically move pointer to the default button in a dialog box setting in its custom nonstandard dialog.

    Exercise: What assumptions are made about the rectangle by the Get­Rect­Center function? How do they differ from the assumptions made by this alternate version:

    POINT GetRectCenter(LPCRECT prc)
      POINT pt = {
        (prc->left + prc->right) / 2,
        (prc->top + prc->bottom) / 2
      return pt;
  • The Old New Thing

    If I attach a file to an existing completion port, do I have to close the completion port handle a second time?


    There are two ways of calling the Create­IO­Completion­Port function. You can pass a null pointer as the Existing­Completion­Port parameter, indicating that you would like to create a brand new completion port, associated with the file handle you passed (if you passed one). Or you can pass the handle of an existing completion port, and the file handle you passed will be associated with that port, in addition to whatever other file handles are already associated with that port.

    In both cases, the return value on success is a handle to the I/O completion port, either the brand new one or the existing one. The question from a customer was, "In the case where I am associating a file handle to an existing completion port, do I need to close the completion port handle a second time?" In other words, is there a handle leak in this code:

    BOOL CCompletionPort::Attach(HANDLE h, ULONG_PTR key)
     // you are required to have created the completion port first
     assert(m_hPort != nullptr);
     HANDLE hPort = CreateIoCompletionPort(h, m_hPort, key, 0);
     // line missing? if (hPort != nullptr) CloseHandle(hPort);
     return hPort != nullptr;

    No, there is no handle leak. If you ask to associate a file with an existing completion port, the same handle you passed as the Existing­Completion­Port parameter is returned back on success. There is no new obligation to close the handle a second time because kernel handles are not reference-counted. Closing a kernel handle twice is always an error, because the first close closes the handle, and the second close attempts to use a handle that is already closed. There is no reference count on handles.¹

    ¹ There is a reference count on kernel objects, but handles are not reference-counted. The standard way to increment the reference count on a kernel object is... to create a new handle to it!

  • The Old New Thing

    All I/O on a synchronous file handle is serialized; that's why it's called a synchronous file handle


    File handles are synchronous by default. If you want asynchronous file handles, you need to pass the FILE_FLAG_OVERLAPPED flag when you create the handle. And all operations on a synchronous file handle are serialized.

    You'd think this was a simple and obvious rule, but "Someone" finds it "very surprising that operations can block which only handle file metadata."

    Imagine if synchronous file handles were not serialized for metadata operations. First of all, it means that the documentation for synchronous file handles suddenly got a lot more complicated. "Some operations on synchronous file handles are serialized, but not others. Specifically, operations on file contents are synchronized, but operations on file metadata are not synchronized."

    Now things get weird.

    For example, the size of a file is metadata. Allowing file size operations to bypass serialization means that if you issue a Write­File operation, and then on another thread call Get­File­Size, you can get a size that is neither the old size nor the new size. Maybe that doesn't bother you.

    Okay, how about the Device­Io­Control function? Does that operate on file contents or file metadata? Your initial reaction might be that Device­Io­Control is obviously a metadata operation. For example, the object ID you get from FSCTL_GET_OBJECT_ID has nothing to do with the file contents. On the other hand, some I/O control codes do affect file contents. FSCTL_SET_ZERO_DATA zeroes out chunks of a sparse file, FSCTL_FILE_LEVEL_TRIM tells the underlying storage that it may (but is not required to) throw away the current contents of a section of a file. Since some I/O control codes affect file contents and some don't, you would have to say that the Device­Io­Control function on a synchronous file handle is sometimes serialized and sometimes not. It's not very reassuring to read documentation that goes something like "If the file handle is a synchronous file handle, the I/O control operation might be serialized, or it might not."

    Recall that all I/O in the kernel internally follows the asynchronous programming model. Synchronous file handles are a convenience provided by the kernel which converts operations on synchronous handles into the equivalent asynchronous operation, followed by a wait for the operation to complete; and all of these operations are serialized.

    Since drivers are allowed to make up custom I/O control codes, the I/O subsystem cannot know for certain whether a particular control code affects file contents or not. It wouldn't know whether any particular control code issued on a synchronous file handle should be converted to a synchronous operation or allowed to proceed asynchronously.

    So now you're in an even more confused situation, where the kernel doesn't even know whether it should serialize a control operation or not. What would the documentation say now? "Actually, the kernel doesn't know whether the operation should be serialized, so it just flips a coin. Heads, the operation is serialized. Tails, it isn't. Do you feel lucky, punk?"

    So the kernel chooses a very simple algorithm: All operations are serialized. It doesn't care if an obvious file contents operation, a subtle file contents operation, a hidden file contents operation, or not a file contents operation at all.

    Sometimes the best way to solve a problem is to stop trying to be clever and just focus on predictability and reducing complexity.

    If you want to perform an operation that is not serialized, you can just create a second handle to the same file and perform the operation on the second handle. (The Re­Open­File function may come in handy.) Whether a file handle is synchronous or asynchronous is a property of the handle, not of the underlying file. There can be five handles to the same file, some synchronous and some asynchronous. Operations on the synchronous handles are serialized with respect to the file handle; operations on the asynchronous handles are not.

    And with the decision to take the simple approach and serialize all accesses to a synchronous file handle, the kernel can wait on the hFile itself to determine when the I/O has completed, thereby saving it from having to create a temporary event for every single I/O operation.

  • The Old New Thing

    Adding a confirmation dialog to every drag/drop operation does not solve the problem


    A customer wanted to know how to enable a confirmation dialog whenever the user inadvertently perform a drag/drop operation in Explorer to move files within a volume.

    For example, if I have an S drive mapped to \\server\share, I would like to display the confirmation dialog when users inadvertently drag and drop a file or folder within the S drive.

    Okay, first of all, the problem statement doesn't match the question. But that's good, because the question was misguided.

    The question was "How do I add a confirmation dialog to every drag/drop operation?" But the problem statement was "We have users who inadvertently drag and drop files."

    So they didn't actually want the confirmation on every operation; they only wanted it on the accidental ones.

    Of course, the computer can't read your mind, so it doesn't know what you were thinking when you performed a drag/drop operation in order to determine whether the operation was intentional or accidental. The customer asked for a dialog box to appear after every operation just in case it was inadvertent. But since most actions are intentional, the result is that users get confirmation dialog after confirmation dialog. Soon, dialog fatigue sets in, and users just get into the habit of ignoring them.

    But let's go back to that inadvertent adjective. The way to reduce the likelihood of accidental drag/drop operations is to adjust the drag sensitivity so that the mouse must move a "significant" distance before a drag/drop operation is initiated. What constitites a "significant" distance is something you can choose based on your own experiments.

  • The Old New Thing

    Microspeak: The train


    Work on Windows occurs in several different branches of the source code, and changes in one branch propagate to other branches. The goal is to push changes toward the trunk on a regular cadence. I will illustrate with an extremely simplified example because the complexities aren't really important. Consider this branch structure:

    A B C D
    A1 A2 B1 B2 C1 C2 D1 D2

    Suppose for the sake of simplicity that odd-numbered leaf branches push their changes to the first-level branches on Mondays and Wednesdays, and even-numbered leaf branches push their changes on Tuesdays and Thursdays. Furthermore, once a week (A on Monday, B on Tuesday, etc, with Friday as a buffer day) each first-level branch pushes its changes to the trunk. It does this before accepting changes from the child branch that same day, so that every change spends at least one day baking in a first-level branch.

    This means that if you work in A1 and you want a change to land in the trunk by next Monday, you need to commit it to A1 by Wednesday. Otherwise, you will miss the train.

    The metaphor here is that code changes are cargo (known in the lingo as payload) which are transported by an imaginary train to the next station (branch). If you miss the train, then your change sits on the platform and waits for the next train.

    The imaginary train schedule looks like this:

    Day Train Departure
    Monday A → Trunk 4pm
    A1 → A 5pm
    B1 → B 5pm
    C1 → C 5pm
    D1 → D 5pm
    Tuesday B → Trunk 4pm
    A2 → A 5pm
    B2 → B 5pm
    C2 → C 5pm
    D2 → D 5pm
    Wednesday C → Trunk 4pm
    A1 → A 5pm
    B1 → B 5pm
    C1 → C 5pm
    D1 → D 5pm
    Thursday D → Trunk 4pm
    A2 → A 5pm
    B2 → B 5pm
    C2 → C 5pm
    D2 → D 5pm
    Friday No trains

    (Note that nobody actually refers to this as the train schedule. We call it the RI schedule, where RI is pronounced as two letters and not like the word rye.)

    Suppose that there's a problem in B1 on Monday, and the branch managers decide to delay pushing their changes to B until they understand the problem and have a fix for it. If they cannot get this done before 4pm Tuesday, then the scheduled RI from B to the trunk will take place without any payload from B1. B1 is said to have missed the RI train. Unless they take special steps, their changes won't appear in the trunk until the following Tuesday at the earliest.

    There are few things that B1 could do when they find themselves in this situation.

    They could ask that the RI from B to the trunk be delayed until after they figure out the problem and push the fix from B1 into B. This is like asking for a train to be held at the station so that passengers on an inbound train can make their connection.

    Maybe there's a bad bug that's in the trunk, and B1 has a fix for it. They could push just the one bug fix from B1 into B, and let that fix get pushed to the trunk on Tuesday. I guess this is like kicking everybody off the train back onto the platform except for the one fix you want to take.

    B1 could negotiate a deal with C. "Hi, we have a fix that needs to go the trunk, but our branch is in kind of bad shape. Could you take it to the trunk for us?" The fix is pushed from B1 to C on Tuesday, and then it catches the train from C to the trunk on Wednesday. This is sort of like going to a different station.

    They could ask for permission to make a direct change to the trunk, bypassing all the trains that normally take changes to the trunk. I'm running out of metaphors here. Maybe this is "taking the plane"? No wait, we actually have lingo for this too. It's called fast-track, sometimes written as fasttrack without the hyphen.

    The train terminology can get more specific, with trains named after the release they are destined for. For example, the C1 branch manager might say, "The last beta train has left C1. If you still have a beta bug and you can get your fix signed off before Wednesday, contact the C branch team; we may be able to get you onto the last beta train out of C. Otherwise, you need to work with the beta release team. See the XYZ Web site for further information."

    Here's some sample text from the trunk management team which incorporates a lot of the lingo. Practice it and you too can sound like a hip insider.

    Today, we took the scheduled RI from A and two fast-track fixes, one from the B team (for bug 3141) and one from the C team (for bug 2718)
Page 1 of 3 (29 items) 123