• The Old New Thing

    You can use a file as a synchronization object, too

    • 35 Comments

    A customer was looking for a synchronization object that had the following properties:

    • Can be placed in a memory-mapped file.
    • Can be used by multiple processes simultaneously. Bonus if it can even be used by different machines simultaneously.
    • Does not leak resources if the file is deleted.

    It turns out there is already a synchronization object for this, and you've been staring at it the whole time: The file.

    File locking is a very old feature that most people consider old and busted because it's just one of those dorky things designed for those clunky database systems that use tape drives like they have in the movies. While that may be true, it's still useful.

    The idea behind file locking is that every byte of a file can be a synchronization object. The intended pattern is that a database program indicates its intention to access a section of a file by locking it, and this prevents other processes from accessing that same section of the file. This allows the database program to update the file without race conditions. When the database program is finished with that section of the file, it unlocks it.

    One interesting bit of trivia about file locking is that you can lock bytes that don't even exist. It is legal to lock bytes beyond the end of the file. This is handy in the database case if you want to extend the file. You can lock the bytes you intend to add, so that nobody else can extend the file at the same time.

    The usage pattern for byte-granular file locks maps very well to the customer's requirements. The synchronization object is... the file itself. And you put it in the file by simply choosing a byte to use as the lock target. (And the byte can even be imaginary.) And if you delete the file, the lock disappears with it.

    Note that the byte you choose as your lock target need not be dedicated for use as a lock target. You can completely ignore the contents of the file and simply agree to use byte zero as the lock target. You just have to understand that when the byte is locked, only the owner of the lock can access it via the Read­File and Write­File family of functions. (Reading or writing a byte that is locked by somebody else will fail with ERROR_LOCK_VIOLATION. Note that access via memory-mapping is not subject to file locking, which neatly lines up with the customer's first requirement.)

    To avoid the problem with locking an actual byte, you can choose imaginary bytes at ridiculously huge offsets purely for locking. Since those bytes don't exist, you won't interfere with other code that tries to read and write them. For example, you might agree to lock byte 0xFFFFFFFF`FFFFFFFF, on the assumption that the file will never become four exabytes in size.

    File locking supports the reader/writer lock model: You can claim a lock for shared access (read) or for exclusive access (write).

    The basic Lock­File function is a subset of the more general Lock­File­Ex function, so let's look at the general function.

    To lock a portion of a file, you call Lock­File­Ex with the range you want to lock, the style of lock (shared or exclusive), and how you want failed locks to be handled. To release the lock, you pass the same range to Unlock­File­Ex. Note that ranges cannot be chopped up or recombined. If you lock bytes 0–10 and 11–19 with separate calls, then you must unlock them with separate matching calls; you can't make a single bulk call to unlock bytes 0–19, nor can you do a partial unlock of bytes 0–5.

    Most of the mechanics of locking are straightforward, except for the "how you want failed locks to be handled" part. If you specify LOCKFILE_FAIL_IMMEDIATELY and the lock attempt fails, then the call simply fails with ERROR_LOCK_VIOLATION and that's the end of it. It's up to you to retry the operation if that's what you want.

    On the other hand, if you do not specify LOCKFILE_FAIL_IMMEDIATELY, and the lock attempt fails, then the behavior depends on whether the handle is synchronous or asynchronous. If synchronous, then the call blocks until the lock is acquired. If asynchronous, then the call returns immediately with ERROR_IO_PENDING, and the I/O completes when the lock is acquired.

    The documentation in MSDN on how lock failures are handled is a bit confusing, thanks to tortured sentence structure like "X behaves like Y if Z unless Q." Here is the behavior of lock failures in table form:

    If Lock­File­Ex fails Handle type
    Asynchronous Synchronous
    LOCKFILE_FAIL_IMMEDIATELY specified Returns FALSE immediately.
    Error code is ERROR_LOCK_VIOLATION.
    LOCKFILE_FAIL_IMMEDIATELY not specified Returns FALSE immediately.
    Error code is ERROR_IO_PENDING.
    I/O completes when lock is acquired.
    Blocks until lock is acquired, returns TRUE.

    Here's a little test app that exercises all the options. Run the program with two command line options. The first is the name of the file you want to lock, and the second is a string describing what kind of lock you want. Pass zero or more of the following letters:

    • "o" to open an overlapped (asynchronous) handle; otherwise, it will be opened non-overlapped (synchronous).
    • "e" to lock exclusively; otherwise, it will be locked shared
    • "f" to fail immediately; otherwise, it will wait

    For example, you would pass "ef" to open a synchronous handle and request an exclusive lock that fails immediately if it cannot be acquired. If you want all the defaults, then pass "" as the options.

    #include <windows.h>
    #include <stdio.h>
    #include <tchar.h>
    
    int __cdecl _tmain(int argc, TCHAR **argv)
    {
     // Ensure correct number of command line arguments
     if (argc < 3) return 0;
    
     // Get the options
     DWORD dwFileFlags = 0;
     DWORD dwLockFlags = 0;
     for (PTSTR p = argv[2]; *p; p++) {
      if (*p == L'o') dwFileFlags |= FILE_FLAG_OVERLAPPED;
      if (*p == L'e') dwLockFlags |= LOCKFILE_EXCLUSIVE_LOCK;
      if (*p == L'f') dwLockFlags |= LOCKFILE_FAIL_IMMEDIATELY;
     }
    
     // Open the file
     _tprintf(TEXT("Opening the file '%s' as %s\n"), argv[1],
              (dwFileFlags & FILE_FLAG_OVERLAPPED) ?
              TEXT("asynchronous") : TEXT("synchronous"));
     HANDLE h = CreateFile(argv[1], GENERIC_READ,
                    FILE_SHARE_READ | FILE_SHARE_WRITE,
                    NULL, OPEN_EXISTING,
                    FILE_ATTRIBUTE_NORMAL | dwFileFlags, NULL);
     if (h == INVALID_HANDLE_VALUE) {
      _tprintf(TEXT("Open failed, error = %d\n"), GetLastError());
      return 0;
     }
    
     // Set the starting position in the OVERLAPPED structure
     OVERLAPPED o = { 0 };
     o.Offset = 0; // we lock on byte zero
    
     // Say what kind of lock we want
     if (dwLockFlags & LOCKFILE_EXCLUSIVE_LOCK) {
      _tprintf(TEXT("Requesting exclusive lock\n"));
     } else {
      _tprintf(TEXT("Requesting shared lock\n"));
     }
    
     // Say whether we're going to wait to acquire
     if (dwLockFlags & LOCKFILE_FAIL_IMMEDIATELY) {
      _tprintf(TEXT("Requesting immediate failure\n"));
     } else if (dwFileFlags & FILE_FLAG_OVERLAPPED) {
      _tprintf(TEXT("Requesting notification on lock acquisition\n"));
      // The event that will be signaled when the lock is acquired
      // error checking deleted for expository purposes
      o.hEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
     } else {
      _tprintf(TEXT("Call will block until lock is acquired\n"));
     }
    
     // Okay, here we go.
     _tprintf(TEXT("Attempting lock\n"));
     BOOL fRc = LockFileEx(h, dwLockFlags, 0, 1, 0, &o);
    
     // If the lock failed, remember why.
     DWORD dwError = fRc ? ERROR_SUCCESS : GetLastError();
     _tprintf(TEXT("Wait %s, error code %d\n"),
              fRc ? TEXT("succeeded") : TEXT("failed"), dwError);
    
     if (fRc) {
      _tprintf(TEXT("Lock acquired immediately\n"));
     } else if (dwError == ERROR_IO_PENDING) {
      _tprintf(TEXT("Waiting for lock\n"));
      WaitForSingleObject(o.hEvent, INFINITE);
      fRc = TRUE; // lock has been acquired
     }
    
     // If we got the lock, then hold the lock until the
     // user releases it.
     if (fRc) {
      _tprintf(TEXT("Hit Enter to unlock\n"));
      getchar();
      UnlockFileEx(h, 0, 1, 0, &o);
     }
    
     // Clean up
     if (o.hEvent) CloseHandle(o.hEvent);
     CloseHandle(h);
     return 0;
    }
    

    When you run this program, it will try to acquire the lock in the manner requested, and if the lock is successfully acquired, it will wait for you press Enter, then it will release the lock.

    You naturally need to run multiple copies of this program to see how the flags interact. (If you run only one copy, then it will always succeed.)

    Exercise: What changes would you make if you wanted to wait at most 5 seconds to acquire the lock? (Hint.)

  • The Old New Thing

    Aha, I have found a flaw in the logic to detect whether my program is running on 64-bit Windows

    • 60 Comments

    Some time ago, I described how to detect programmatically whether you are running on 64-bit Windows, and one of the steps of the algorithm was "If you are a 64-bit program, then you are running on 64-bit Windows, because 32-bit Windows cannot run 64-bit programs."

    Every so often, somebody will claim that they found a flaw in this logic: "This algorithm may work today, but it assumes that the only version of Windows that can run 64-bit applications is 64-bit Windows. What if a future non-64-bit version of version of Windows runs 64-bit applications? Then your algorithm will incorrectly say that it is running on 64-bit Windows!"

    Yeah, but so what?

    Suppose you detect that the program is running on this hypothetical version of Windows that is not natively 64-bit but still runs 64-bit applications. What will your program do differently? How can you reason about the feature set and compatibility requirements of something that hasn't been invented yet?

    This is another case of If you don't know what you're going to do with the answer to a question, then there's not much point in asking it.

    In this specific case, you should just continue about your normal business and let the emulation layer of the hypothetical future version of Windows do its job of giving you a 64-bit sky with 64-bit birds in the 64-bit trees.

  • The Old New Thing

    Why does the timestamp of a file increase by up to 2 seconds when I copy it to a USB thumb drive?

    • 24 Comments

    We saw some time ago that the FAT file system records timestamps in local time to only two-second resolution. This means that copying a file to a FAT-formatted device (typically a floppy drive or a USB thumb drive) can increase the timestamp by up two seconds. And even after the file is copied, the timestamp is not stable. The timestamp changes depending on the time zone employed by the computer that accesses the drive. In particular, if you are in a part of the world which changes clocks during the summer, then the timestamp on the file moves by an hour every spring and then moves in the opposite direction every autumn. (Because you change time zones twice a year.)

    Okay, but why does the timestamp always increase to the nearest two-second interval? Why not round to the nearest two-second interval? That way, the timestamp change is at most one second.

    Because rounding to the nearest interval means that the file might go backward in time, and that creates its own problems. (Causality can be such a drag.)

    For example, suppose you regularly back up files from your NTFS-formatted C: drive to your USB thumb drive mounted as drive F: by typing

    xcopy /D C:\Files\* F:\Files\*
    

    If the timestamps rounded to the nearest two-second interval, then half the files on average will have a timestamp on the USB thumb drive older than the files on the C: drive. This means that if you perform the command a second time, approximately half of the files will be copied again. To the user, it looks like the xcopy command never finishes the job, because each time you tell it "Perform an incremental backup" it always finds something to copy. It never says, "All files up to date, you can go home now."

    To avoid this infinite loop, the convention is always to round up, so that the copy of a file is never older than the original.

  • The Old New Thing

    Who wrote the text for the Ctrl+Alt+Del dialog in Windows 3.1?

    • 42 Comments

    One of the differences between standard-mode Windows and enhanced-mode Windows was what happened when you hit Ctrl+Alt+Del. Since 16-bit Windows applications are co-operatively multi-tasked, it is easy to determine whether the system is responding, and if not, it is also easy to identify the application which is responsible. In that case, Windows gave you options to close the non-responsive application, restart the computer, or cancel.

    During this time period, Steve Ballmer was head of the Systems Division, and he paid a visit to the Windows team to see what they were up to, as is the wont of many executives.¹ When they showed him the Ctrl+Alt+Del feature, he nodded thoughtfully and added, "This is nice, but I don't like the text of the message. It doesn't sound right to me."

    "Okay, Steve. If you think you can do a better job, then go for it." Unlike some other executive, Steve took up the challenge, and a few days later, he emailed what he thought the Ctrl+Alt+Del screen should say.

    The text he came up with was actually quite good, and it went into the product pretty much word for word.



    Contoso Deluxe Music Composer


      This Windows application has stopped responding to the system.

      *  Press ESC to cancel and return to Windows.
      *  Press ENTER to close this application that is not responding.
         You will lose any unsaved information in this application.
      *  Press CTRL+ALT+DEL again to restart your computer. You will
         lose any unsaved information in all applications.


    Note to journalists: This is the Ctrl+Alt+Del dialog, not the blue screen of death. Thank you for paying attention.

    ¹ It occurred to me only as I wrote up this entry that people took the phrase Right on top of my notepad from the earlier story literally: There was a chair, the chair had a notepad on its seat, Bill sat in the chair (on top of the notepad). That interpretation never occurred to me. From the description in the previous paragraph, it was apparent to me that the notepad was on a desk, and Bill's choice of seat blocked access to the notepad. (I.e., the manager would have to reach around Bill to get the notepad.) The person telling the story is not a native English speaker, so there may have been a preposition translation issue.

  • The Old New Thing

    It's a trap! Employment documents that require you to violate company policy

    • 20 Comments

    One of my colleagues had a previous job involving tuning spam filters and removing objectionable content. Before he could start, he was told that he had to sign a special release. The form said basically, "I understand that my job may require me to see pornography or other objectionable material, and I promise not to sue."

    He asked, "So where is the part that says I'm not going to be fired for doing that?"

    "What do you mean?"

    He explained, "This document protects the company from me. But where is the part that protects me from the company?"

    "I don't know what you're talking about."

    He spelled it out: "Company policy says that watching pornography at work is grounds for termination. This document does not actually say that it's okay for me to do so if it is done in the course of my job duties."

    "Look, you can either sign the release form or not, but you can't work until you sign it."

    My colleague sighed as he signed the form. "Whatever. Nevermind."

  • The Old New Thing

    How can I detect that a user's SID has changed and recover their old data?

    • 9 Comments

    A customer maintained a database which recorded information per user. The information in the database is keyed by the user's SID. This works out great most of the time, but there are cases in which a user's SID can change.

    "Wait, I thought SIDs don't change."

    While it's true that SIDs don't change, it is also true that the SID associated with a user can change. Since SIDs encode the domain to which they belong, a user which moves from one domain to another within an organization, will need to be assigned a new SID.

    But wait, does that mean that the user lost access to all their stuff? After all, all their stuff was marked "Owned by X\UserName" but the user's SID is now Y\UserName.

    No, the user doesn't lose access to their stuff thanks to SID history, and if you move users around a lot, the SID history can get quite large.

    A token for a user contains not only their current identity but also all of their earlier identities. That is what permits Y\UserName to continue to access things that was marked "Owned by X\UserName": The token for Y\UserName includes an entry that says, "Oh, I used to be X\UserName."

    The customer's database can take advantage of the SID history to match up users with their former selves. Our customer was lucky in that their database recorded only users who had logged into the local machine, so that list is typically pretty small. The simplest solution for this particular customer is just to go through all the users in the database, and for each one, see if the current user has that database user in their SID history. And the easy way to do that is to make the security system do the work for you: To see if the current user has user X in their SID history, create a security descriptor that grants access only to user X, then call Access­Check to see if the current user can access it. If so, then that means that the current user was at one point in the past known as X.

    (If you have a large database where iterating over all users is impractical, you can ask for the current user's SID-History attribute and walk through the previous identities manually.)

  • The Old New Thing

    Taking advantage of the fact that the handle returned when you create a kernel synchronization object has full access regardless of the actual ACL

    • 6 Comments

    A customer wanted some help deciding what security attributes to place on an event object intended to be used by multiple security contexts.

    We have two processes, call them A and B, running in different security contexts. I have an event that process A creates and shares with process B. The only thing process A does with the event is signal it, and the only thing process B does with the event is wait on it. Our question is what ACLs you recommend for the event. For now, we're using O:BAD:(A;;GR;;;WD)(A;;GA;;;LS)(A;;GA;;;BA). (In case it matters, process A is usually running as a service with Local System privileges, though for testing purposes it may be running as local administrator. Process B runs as a service with Local Service privileges.)

    For those who don't speak SDDL, that weird line noise is shorthand for

    • Owner: Builtin Administrators
    • DACL:
      • Allow Generic Read to Everyone (aka World).
      • Allow Generic All to Local Service.
      • Allow Generic All to Builtin Administrators.

    Given the requirements, there is no need to grant Everyone any access at all, so we can delete the (A;;GR;;;WD) ACE.

    Since process B needs only to wait on the object, granting it Generic All access is far too broad. That would allow process B to signal the event or even change its ACL! To wait on an object, all you need is Synchronize, so the second ACE can be tightened to (A;;0x00100000;;;LS). (There is no shorthand for Synchronize, so we use its hex value.)

    The intention of the third ACE is to allow process A to signal the event, but for that all it needs is EVENT_MODIFY_STATE, not Generic All. But we can do better: We can delete the ACE entirely.

    "But Mister Wizard, if you delete the third ACE, then process A won't be able to signal the event!"

    Ah yes it can, thanks to a special feature of the Create­Event function:

    The handle returned by Create­Event has the EVENT_ALL_ACCESS access right.

    If you created the event, you get full access to the event regardless of what the ACLs on the event would normally say.

    Therefore, the event can be ACL'd with simply O:BAD:(A;;0x00100000;;;LS). When process A creates the event, it needs to hold on tight to that event handle, since that is the process's only way of setting the event! (If it loses the handle, it won't be able to get it back because the attempt to reacquire the handle will be blocked by the ACL.)

    Here's a quick program that demonstrates the behavior.

    #include <windows.h>
    #include <sddl.h>
    #include <tchar.h>
    
    // This is a demonstration, so there is no error checking
    // and we leak memory.
    
    int __cdecl _tmain(int, TCHAR **)
    {
     ULONG cb;
     SECURITY_ATTRIBUTES sa = { sizeof(sa), NULL, FALSE };
    
     // Create a security descriptor that grants access to no one.
     ConvertStringSecurityDescriptorToSecurityDescriptor(TEXT("D:"),
        SDDL_REVISION_1, &sa.lpSecurityDescriptor, &cb);
    
     // Create a handle with that security descriptor
     HANDLE h = CreateEvent(&sa, TRUE, TRUE,
                 TEXT("NobodyCanAccessMeButMe"));
    
     // Even though nobody has access to the object, we can still
     // signal it using the handle returned by CreateEvent.
     SetEvent(h); // succeeds
    
     // But nobody else can obtain the handle via the object name.
     HANDLE h2 = OpenEvent(EVENT_MODIFY_STATE, FALSE,
                           TEXT("NobodyCanAccessMeButMe")); // fails
    
     return 0;
    }
    

    The customer wrote back, "This worked perfectly. Thanks!"

    For bonus points, you can be even more specific and grant Synchronize access only to process B's service SID (NT SERVICE\Service­Name) rather than to all local services.

  • The Old New Thing

    Where does the Installed Updates control panel get the install date from?

    • 12 Comments

    A corporate customer wanted to know where the Installed Updates control panel gets the Installed On information from, because they found that the values were always set to the current date regardless of when the update was actually installed.

    The algorithm goes roughly like this:

    First, ask MSI what date the update was installed by calling Msi­Get­PatchInfo­Ex and asking for the INSTALL­PROPERTY_INSTALL­DATE.

    If that doesn't work, then go to the registry key under Software\Microsoft\Windows\Current­Version\Uninstall\Unique­Id and look for a value called (surprise) Install­Date. (Note that 32-bit updates on 64-bit machines will be redirected into a Wow­64­32­Node key.)

    If that still doesn't work, then it's time to guess: Windows XP uses the last-modified date of the directory that contains the uninstaller. Windows Vista and higher use the last-modified date of the Software\Microsoft\Windows\Current­Version\Uninstall\Unique­Id registry key. (Again, possibly with a Wow­64­32­Node stuck in there.)

    Bonus chatter: Interestingly, the customer didn't phrase the problem like that. The customer said, "The first time a user logs on each day, the install date changes to the current date. Subsequent logons the same day do not change the date. But the first logon the next day changes the date again. What's so special about the first logon of each day?" What's so special about the first logon of each day is that it's a new day! I suspect that the the date is updated on every logon. It's just that they don't notice the change because the date is the same.

  • The Old New Thing

    It's time we face reality, my friends: We're not rocket scientists

    • 13 Comments

    During the development of Windows 95, it was common for team members to pay visits to other teams to touch base and let them know what's been happening on the Windows 95 side of the project.

    It was during one of these informal visits that the one of my colleagues reported that he saw that one of the members of the partner team had a Gary Larson cartoon from The Far Side depicting a group of scientists studying a multi-stage rocket ship they just assembled, but the stages are connected all crooked. One of the scientists says, "It's time we face reality, my friends. … We're not exactly rocket scientists."

    The comic was "enhanced" a bit by the partner team. They added a sign on the wall of the laboratory that says Windows 95 Development, and the stages of the rocket are alternately labeled 16-bit and 32-bit. The graffiti were clearly poking fun at Windows 95's attempt to straddle the 16-bit and 32-bit worlds.

    The Windows 95 team knew how to take a joke, and for a time, they adopted "Hey, we're not rocket scientists" as a catch phrase.

    Following up on that article from 2010: Microsoft ran a free seminar on Windows 95 development for Macintosh programmers at the 1995 MacWorld Expo. Upon successful completion, participants received T-shirts with the slogan "Windows 95 sucks less."

  • The Old New Thing

    How do I read the "Double-click to open an item (single-click to select)" setting in Folder Options?

    • 21 Comments

    Today's Little Program reports whether the Double-click to open an item (single-click to select) option is selected in the Folder Options dialog. A customer wanted to know how to do this, presumably so that their program would respect the setting and adjust its user interface to match.

    #include <windows.h>
    #include <shlobj.h>
    #include <stdio.h>
    
    BOOL IsDoubleClickToOpenEnabled()
    {
     SHELLFLAGSTATE sfs;
     SHGetSettings(&sfs, SSF_DOUBLECLICKINWEBVIEW);
     return sfs.fDoubleClickInWebView;
    }
    
    int __cdecl main(int, char**)
    {
     printf("Double-click is %s\n",
            IsDoubleClickToOpenEnabled() ? "enabled" : "disabled");
     return 0;
    }
    

    The flag and member name is kind of weird. The ability to single-click to open an item was introduced as part of the Windows Desktop Update which came with Internet Explorer 4. This update made Explorer more Web-like, with single-click to activate and underlines appearing on hover. (This was back in the day when making everything more Web-like was the new hotness.)

    The internal name of this Explorer feature was WebView, and that name is reflected in the flag and structure.

Page 7 of 436 (4,357 items) «56789»