February, 2011

  • The Old New Thing

    iPhone pricing as economic experiment

    • 42 Comments

    Back in 2005, Slate's Tim Harford wondered why Microsoft didn't raise the introductory price of Xbox 360 game consoles. With the price set at $300, lines were long and shortages were many. Harford's readers came up with their own theories for resisting the laws of supply and demand and holding to a fixed price.

    The Xbox 360 is hardly unique in this respect. When there's a hot product, manufacturers hold to the original price and let the lines grow, the shortages fester, and the customers get more frustrated. Think Tickle Me Elmo or Cabbage Patch Kids. Even though from an economic-theoretical standpoint, a product that has sold out with unmet demand is a product whose price was set too low.

    With the iPhone, Apple unwittingly ran the experiment that Harford proposed. There were lines, but by some reports, the lines weren't all that bad. After the initial demand subsided, Apple did what the economists say they should have done: They lowered the price. And the people who bought the phones at the higher price complained (forcing Apple to offer a store credit) and one of them even sued. Slate's Daniel Gross opines on the lessons learned.

  • The Old New Thing

    Why does SHGetKnownFolderPath return E_FAIL for a known folder?

    • 38 Comments

    A customer reported having problems with the SH­Get­Known­Folder­Path function. I've left in the red herrings.

    On Windows 7, I'm trying to retrieve the Internet folder with the following code:

    if (OpenProcessToken(GetCurrentProcess(), TOKEN_ALL_ACCESS, &hToken))
    {
     HRESULT hr = SHGetKnownFolderPath(FOLDERID_InternetFolder,
                                  KF_FLAG_DONT_VERIFY, hToken, &pszPath);
     ...
    }
    

    The call always fails with E_FAIL. What am I doing wrong? I tried passing NULL as the token, but that didn't help.

    The reason the call fails has nothing to do with Windows 7 or the token. The call fails because FOLDERID_Internet­Folder is a virtual folder—there is no path in the first place!

    The reason is that the folder you are requesting is a virtual folder (KF_CATEGORY_VIRTUAL). Virtual folders don't exist in the file system, so they don't have a path. SH­Get­Known­Folder­Item should work.

    The customer appears to have misinterpreted this response in a way I wasn't expecting (but which sadly I've seen before):

    I added the KF_CATEGORY_VIRTUAL flag, but I still get the same error back.

    if (OpenProcessToken(GetCurrentProcess(), TOKEN_ALL_ACCESS, &hToken))
    {
     HRESULT hr = SHGetKnownFolderPath(FOLDERID_InternetFolder,
                                  KF_FLAG_DONT_VERIFY | KF_CATEGORY_VIRTUAL,
                                  hToken, &pszPath);
     ...
    }
    

    Um, no, that makes no sense at all. KF_CATEGORY_VIRTUAL is a KF_CATEGORY value, but the second parameter to SH­GetKnown­Folder­Path is a KNOWN_FOLDER_FLAG. You can't just combine unrelated values like that. It's like adding 3 grams to 12 meters.

    And second, the KF_CATEGORY_VIRTUAL enumeration isn't something that you pass in to "override" anything. The point is that FOLDERID_Internet­Folder is a virtual folder: It has no path, so if you try to ask for its path, you'll just get an error back because the thing you're looking for simply doesn't exist.

    I never did figure out what this customer was trying to do. Maybe they figured, since they can't download the Internet, they could at least try to do a Find­First­File on it.

  • The Old New Thing

    What is the difference between a directory and a folder?

    • 37 Comments

    Windows 95 introduced Windows Explorer and along with it the term folder. What is the relationship between folders and directories?

    Some people believe that Windows 95 renamed directories to folders, but it's actually more than that.

    Windows Explorer lets you view folders, which are containers in the shell namespace. Directories are one type of folder, namely, folders which correspond to file system locations. There are other types of folders, such as Control Panel or Network Neighborhood or Printers. These other types of folders represent objects in the shell namespace which do not correspond to files. In common usage, the term virtual folder has been applied to refer to folders which are not directories. In other words, we have this Euler diagram:

    Folders

    Directories
    Virtual folders = Folders − Directories

    In general, code which manipulates the shell namespace should operate on folders and items, not directories and files, so as not to tie themselves to a particular storage medium. For example, code which limits itself to files won't be able to navigate into a Zip file, since the contents of a Zip file are exposed in the form of a virtual folder.

    Update: The Web server "helpfully" closed some tags prematurely. Apparently it doesn't want you to nest tables. Replaced nested table with DIV.

  • The Old New Thing

    If you want to use GUIDs to identify your files, then nobody's stopping you

    • 37 Comments

    Igor Levicki proposes solving the problem of file extensions by using a GUID instead of a file name to identify a file.

    You can do this already. Every file on an NTFS volume has an object identifier which is formally 16-byte buffer, but let's just call it a GUID. By default a file doesn't have an object identifier, but you can ask for one to be created with FSCTL_CREATE_OR_GET_OBJECT_ID, which will retrieve the existing object identifier associated with a file, or create one if there isn't one already. If you are a control freak, you can use FSCTL_SET_OBJECT_ID to specify the GUID you want to use as the object identifier. (The call fails if the file already has an object identifier.) And of course there is FSCTL_GET_OBJECT_ID to retrieve the object identifier, if any.

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <stdio.h>
    #include <tchar.h>
    #include <ole2.h>
    #include <winioctl.h>
    
    int __cdecl _tmain(int argc, PTSTR *argv)
    {
     HANDLE h = CreateFile(argv[1], 0,
                     FILE_SHARE_READ | FILE_SHARE_WRITE |
                     FILE_SHARE_DELETE, NULL,
                     OPEN_EXISTING, 0, NULL);
     if (h != INVALID_HANDLE_VALUE) {
      FILE_OBJECTID_BUFFER buf;
      DWORD cbOut;
      if (DeviceIoControl(h, FSCTL_CREATE_OR_GET_OBJECT_ID,
                     NULL, 0, &buf, sizeof(buf),
                     &cbOut, NULL)) {
        GUID guid;
        CopyMemory(&guid, &buf.ObjectId, sizeof(GUID));
        WCHAR szGuid[39];
        StringFromGUID2(guid, szGuid, 39);
        _tprintf(_T("GUID is %ws\n"), szGuid);
      }
      CloseHandle(h);
     }
     return 0;
    }
    

    This program takes a file or directory name as its sole parameter and prints the associated object identifier.

    Big deal, now we have a GUID associated with each file.

    The other half is, of course, using this GUID to open the file:

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <stdio.h>
    #include <tchar.h>
    #include <ole2.h>
    
    int __cdecl _tmain(int argc, PTSTR *argv)
    {
     HANDLE hRoot = CreateFile(_T("C:\\"), 0,
                     FILE_SHARE_READ | FILE_SHARE_WRITE |
                     FILE_SHARE_DELETE, NULL,
                     OPEN_EXISTING,
                     FILE_FLAG_BACKUP_SEMANTICS, NULL);
     if (hRoot != INVALID_HANDLE_VALUE) {
      FILE_ID_DESCRIPTOR desc;
      desc.dwSize = sizeof(desc);
      desc.Type = ObjectIdType;
      if (SUCCEEDED(CLSIDFromString(argv[1], &desc.ObjectId))) {
       HANDLE h = OpenFileById(hRoot, &desc, GENERIC_READ,
                     FILE_SHARE_READ | FILE_SHARE_WRITE |
                     FILE_SHARE_DELETE, NULL, 0);
       if (h != INVALID_HANDLE_VALUE) {
        BYTE b;
        DWORD cb;
        if (ReadFile(h, &b, 1, &cb, NULL)) {
         _tprintf(_T("First byte of file is 0x%02x\n"), b);
        }
        CloseHandle(h);
       }
      }
      CloseHandle(hRoot);
     }
     return 0;
    }
    

    To open a file by its GUID, you first need to open something—anything—on the volume the file resides on. Doesn't matter what you open; the only reason for having this handle is so that OpenFileById knows which volume you're talking about. In our little test program, we use the C: drive, which means that the file search will take place on the C: drive.

    Next, you fill in the FILE_ID_DESCRIPTOR, saying that you want to open the file by its object identifier, and then it's off to the races with OpenFileById. Just as a proof of concept, we read and print the first byte of the file that was opened as a result.

    Notice that the file you open by its object identifier does not have to be in the current directory. It can be anywhere on the C: drive. As long as you have the GUID for a file, you can open it no matter where it is on the drive.

    You can run these two programs just to enjoy the thrill of opening a file by its GUID. Notice that once you get the GUID for a file, you can move it anywhere on the drive, and OpenFileById will still open it.

    (And if you want to get rid of those pesky drive letters, you can use the volume GUID instead. Now every file is identified by a pair of GUIDs: the volume GUID and the object identifier.)

    So Igor's dream world where all files are referenced by GUID already exists. Why isn't everybody switching over to this utopia of GUID-based file identification?

    You probably know the answer already: Because people prefer to name things with something mnemonic rather than a GUID. Imagine a file open dialog in this dream world. "Enter the GUID of the file you wish to open, or click Browse to see the GUIDs of all the files on this volume so you can pick from a list." How long would this dialog survive?

    For today, you don't have to call me Raymond. You can call me {7ecf65a0-4b78-5f9b-e77c-8770091c0100}, or "91c" for short.

    (And I've totally ignored the fact that using GUIDs to identify files does nothing to solve the problem of trying to figure out what program should be used to open a particular file.)

    Bonus chatter: You can also open files by their file identifer, which is a volume-specific 64-bit value. But I chose to use the GUID both for the extra challenge, and just to show that Igor's dream world already exists.

  • The Old New Thing

    How do specify that a shortcut should not be promoted as newly-installed on the Start menu?

    • 32 Comments

    Windows XP employed a number of heuristics to determine which Start menu shortcuts should be promoted when an application is newly-installed. But what if those heuristics end up guessing wrong?

    You can set the System.App­User­Model.Exclude­From­Show­In­New­Install property to VARIANT_TRUE to tell the Start menu, "I am not the primary entry point for the program; I'm a secondary shortcut, like a help file."

    #include <windows.h>
    #include <tchar.h>
    #include <shlobj.h>
    #include <atlbase.h>
    
    // class CCoInitialize incorporated here by reference
    
    int __cdecl _tmain(int argc, TCHAR **argv)
    {
     // error checking elided for expository purposes
     CCoInitialize init;
     CComPtr<IShellLink> spsl;
     spsl.CoCreateInstance(CLSID_ShellLink);
     spsl->SetPath(TEXT("C:\\Program Files\\LitWare\\LWUpdate.exe"));
     PROPVARIANT pvar;
     pvar.vt = VT_BOOL;
     pvar.boolVal = VARIANT_TRUE;
     CComQIPtr<IPropertyStore>(spsl)->SetValue(PKEY_AppUserModel_ExcludeFromShowInNewInstall, pvar);
     CComQIPtr<IPersistFile>(spsl)->Save(L"LitWare Update.lnk", TRUE);
     return 0;
    }
    
  • The Old New Thing

    Ready... cancel... wait for it! (part 1)

    • 31 Comments

    One of the cardinal rules of the OVERLAPPED structure is the OVERLAPPED structure must remain valid until the I/O completes. The reason is that the OVERLAPPED structure is manipulated by address rather than by value.

    The word complete here has a specific technical meaning. It doesn't mean "must remain valid until you are no longer interested in the result of the I/O." It means that the structure must remain valid until the I/O subsystem has signaled that the I/O operation is finally over, that there is nothing left to do, it has passed on: You have an ex-I/O operation.

    Note that an I/O operation can complete successfully, or it can complete unsuccessfully. Completion is not the same as success.

    A common mistake when performing overlapped I/O is issuing a cancel and immediately freeing the OVERLAPPED structure. For example:

    // this code is wrong
     HANDLE h = ...; // handle to file opened as FILE_FLAG_OVERLAPPED
     OVERLAPPED o;
     BYTE buffer[1024];
     InitializeOverlapped(&o); // creates the event etc
     if (ReadFile(h, buffer, sizeof(buffer), NULL, &o) ||
         GetLastError() == ERROR_IO_PENDING) {
      if (WaitForSingleObject(o.hEvent, 1000) != WAIT_OBJECT_0) {
       // took longer than 1 second - cancel it and give up
       CancelIo(h);
       return WAIT_TIMEOUT;
      }
      ... use the results ...
     }
     ...
    

    The bug here is that after calling Cancel­Io, the function returns without waiting for the Read­File to complete. Returning from the function implicitly frees the automatic variable o. When the Read­File finally completes, the I/O system is now writing to stack memory that has been freed and is probably being reused by another function. The result is impossible to debug: First of all, it's a race condition between your code and the I/O subsystem, and breaking into the debugger doesn't stop the I/O subsystem. If you step through the code, you don't see the corruption, because the I/O completes while you're broken into the debugger.

    Here's what happens when the program is run outside the debugger:

    ReadFile I/O begins
    WaitForSingleObject I/O still in progress
    WaitForSingleObject times out
    CancelIo I/O cancellation submitted to device driver
    return
    Device driver was busy reading from the hard drive
    Device driver receives the cancellation
    Device driver abandons the rest of the read operation
    Device driver reports that I/O has been canceled
    I/O subsystem writes STATUS_CANCELED to OVERLAPPED structure
    I/O subsystem queues the completion function (if applicable)
    I/O subsystem signals the completion event (if applicable)
    I/O operation is now complete

    When the I/O subsystem receives word from the device driver that the cancellation has completed, it performs the usual operations when an I/O operation completes: It updates the OVERLAPPED structure with the results of the I/O operation, and notifies whoever wanted to be notified that the I/O is finished.

    Notice that when it updates the OVERLAPPED structure, it's updating memory that has already been freed back to the stack, which means that it's corrupting the stack of whatever function happens to be running right now. (It's even worse if you happened to catch it while it was in the process of updating the buffer!) Since the precise timing of I/O is unpredictable, the program crashes with memory corruption that keeps changing each time it happens.

    If you try to debug the program, you get this:

    ReadFile I/O begins
    WaitForSingleObject I/O still in progress
    WaitForSingleObject times out
    Breakpoint hit on Cancel­Io statement
    Stops in debugger
    Hit F10 to step over the CancelIo call I/O cancellation submitted to device driver
    Breakpoint hit on return statement
    Stops in debugger
    Device driver was busy reading from the hard drive
    Device driver receives the cancellation
    Device driver abandons the rest of the read operation
    Device driver reports that I/O has been canceled
    I/O subsystem writes STATUS_CANCELED to OVERLAPPED structure
    I/O subsystem queues the completion function (if applicable)
    I/O subsystem signals the completion event (if applicable)
    I/O operation is now complete
    Look at the OVERLAPPED structure in the debugger
    It says STATUS_CANCELED
    Hit F5 to resume execution
    No memory corruption

    Breaking into the debugger changed the timing of the I/O operation relative to program execution. Now, the I/O completes before the function returns, and consequently there is no memory corruption. You look at the OVERLAPPED structure and say, "See? Immediately on return from the Cancel­Io function, the OVERLAPPED structure has been updated with the result, and the buffer contents are not being written to. It's safe to free them both now. Therefore, this can't be the source of my memory corruption bug."

    Except, of course, that it is.

    This is even more crazily insidious because the OVERLAPPED structure and the buffer are updated by the I/O subsystem, which means that it happens from kernel mode. This means that write breakpoints set by your debugger won't fire. Even if you manage to narrow down the corruption to "it happens somewhere in this function", your breakpoints will never see it as it happens. You're going to see that the value was good, then a little while later, the value was bad, and yet your write breakpoint never fired. You're then going to declare that the world has gone mad and seriously consider a different line of work.

    To fix this race condition, you have to delay freeing the OVERLAPPED structure and the associated buffer until the I/O is complete and anything else that's using them has also given up their claim to it.

       // took longer than 1 second - cancel it and give up
       CancelIo(h);
       WaitForSingleObject(o.hEvent, INFINITE); // added
       // Alternatively: GetOverlappedResult(h, &o, TRUE);
       return WAIT_TIMEOUT;
    

    The Wait­For­Single­Object after the Cancel­Io waits for the I/O to complete before finally returning (and implicitly freeing the OVERLAPPED structure and the buffer on the stack). Better would be to use GetOverlapped­Result with bWait = TRUE, because that also handles the case where the hEvent member of the OVERLAPPED structure is NULL.

    Exercise: If you retrieve the completion status after canceling the I/O (either by looking at the OVERLAPPED structure directly or by using GetOverlapped­Result) there's a chance that the overlapped result will be something other than STATUS_CANCELED (or ERROR_CANCELLED if you prefer Win32 error codes). Explain.

    Exercise: If this example had used Read­File­Ex, the proposed fix would be incomplete. Explain and provide a fix. Answer to come next time, and then we'll look at another version of this same principle.

  • The Old New Thing

    I am no longer impressed by your fancy new 10,000 BTU hot pot burner

    • 31 Comments

    Two years ago, we had a gathering at my house for some friends for hot pot, the traditional way of ringing in the lunar new year (which takes place today). It was actually a bit of a cross-cultural event, since the attendees came from different regions of Asia, where different traditions reign. (And the American guests just had to choose sides!)

    My house has but one portable stove for hot pot, so one of the guests brought her own unit, a unit as it turns out which was purchased specifically for the occasion, which gleamed in the light and proudly proclaimed 10,000 BTU of raw heating power. This was cause for much boasting, particularly since I didn't know the heating power of my own puny old unit, but I accepted my second-place position with grace.

    Some time later, we had a quiet family hot pot, and my old and horrifically unfashionable burner was brought out to do its tired but important job, and it was then that I found the sticker that specified its heating power.

    9,925 BTU.

    Now I am no longer impressed by my friend's 10,000 BTU burner.

  • The Old New Thing

    Sharktopus: Just when you thought it was safe to see what movies are coming out

    • 27 Comments

    Sharktopus: Half-shark. Half-octopus. All-killer.

    I am not making that up.

    The Web site is sharktopusmovie.com, presumably because sharktopus.com was already taken. I am not making that up.

    I guess they wanted to ride the coattails of Megashark vs. Giant Octopus?

    Even more disturbing discovery: Megashark vs. Giant Octopus has a sequel: Megashark versus Crocosaurus.

    One of my colleagues says that he's going to wait for the sequel: Sharktopuses on a Plane.

  • The Old New Thing

    Psychic debugging: Because of course when something doesn't work, it's because the program was sabotaged from above

    • 25 Comments

    When something stops working, you begin developing theories for why it doesn't work, and normally, you start with simple theories that involve things close to you, and only after you exhaust those possibilities do you expand your scope. Typically, you don't consider that there is a global conspiracy against you, or at least that's not usually your first theory.

    I'm trying to use the XYZ.DLL that comes with your product. I have successfully registered this DLL (as specified in the documentation) by performing a regsvr32 C:\path\to\XYZ.DLL.

    According to the documentation, I should now be able to create a Xyz.Xyz­Widgetizer object, but when I try to do so from C#, I get the exception

    Retrieving the COM class factory for component with CLSID
    {...} failed due to the following error: 80040154.
    

    I tried using the Visual Basic code sample which comes with the documentation, which contains only two lines:

    Dim oXyzWidgetizer
    Set oXyzWidgetizer = WScript.CreateObject("Xyz.XyzWidgetizer")
    

    However, it still fails with the following error:

    Microsoft (R) Windows Script Host Version 5.7
    Copyright (C) Microsoft Corporation. All rights reserved.
    C:\test.vbs(2, 1) WScript.CreateObject: Could not create object
                      named "Xyz.XyzWidgetizer".
    

    Has support for the XyzWidgetizer been silently dropped?

    Let's look at the error message more closely. Error 80040154 is REGDB_E_CLASSNOTREG: The class is not registered. Therefore, whatever regsvr32 did, it didn't register the class.

    My psychic powers tell me that you registered the 32-bit version of XYZ.DLL on a 64-bit machine.

    Registering the 32-bit DLL records the entries into the 32-bit registry (because 32-bit programs run in an emulator), and the 32-bit registry is not consulted when you try to create a COM object from a 64-bit application. Letting 64-bit applications see the registration for 32-bit DLLs doesn't actually accomplish anything because you cannot load a 32-bit DLL into a 64-bit process and vice versa—even if a 64-bit process can figure out what DLL it wants, it won't able to load it.

    It so happens that my psychic powers were correct. How did I know that the person asking the question was running the 32-bit version of XYZ on a 64-bit version of Windows? I didn't, but it was the simplest theory that fit the (extremely limited) data. And it didn't involve a global conspiracy.

  • The Old New Thing

    There is no longer any pleasure in reading the annual Microsoft injury reports

    • 22 Comments

    Microsoft is required by law to file reports on employees who have sustained injuries on the job. They are also required to post the reports in a location where employees can see them. These reports come out every year on February 1.

    Back in the old days, these reports were filled out by hand, and reading them was oddly amusing for the details. My favorite from the mid 1990's was a report on an employee who was injured on the job, and the description was simply pencil lead embedded in hand.

    Sadly, the reports are now computerized, and there isn't a place to describe the nature of each injury. It's just a bunch of numbers.

    Numbers are nice, but they don't tell a story in quite the same way.

Page 1 of 3 (28 items) 123