• The Old New Thing

    Photoshop meme: Mark Reynolds casually eating sunflower seeds

    • 3 Comments

    July 7, 2011: David Ortiz hits a home run against the Baltimore Orioles. As he rounds third base, Orioles third baseman Mark Reynolds casually finishes off a package of sunflower seeds (photo 6 in the slide show).

    An Internet meme is born.

    Follow the hilarity.

  • The Old New Thing

    What is that horrible grinding noise coming from my floppy disk drive?

    • 41 Comments

    Wait, what's a floppy disk drive?

    For those youngsters out there, floppy disks are where we stored data before the invention of the USB earring. A single floppy disk could hold up to two seconds of CD-quality audio. This may not sound like a lot, but it was in fact pretty darned awesome, because CDs hadn't been invented yet either.

    Anyway, if you had a dodgy floppy disk (say, because you decided to fold it in half), you often heard a clattering sound from the floppy disk drive as it tried to salvage what data it could from the disk. What is that sound?

    That sound is recalibration.

    The floppy disk driver software kept getting errors back from the drive saying "I can't find any good data." The driver figures, "Hm, maybe the problem is that the drive head is not positioned where I think it is." You see, floppy drives do not report the actual head position; you have to infer it by taking the number of "move towards the center" commands you have issued and subtracting the number of "move towards the edge" commands. The actual location of the drive head could differ from your calculations due to an error in the driver, or it could just be that small physical errors have accumulated over time, resulting in a significant difference between the theoretical and actual positions. (In the same way that if you tell somebody to step forward ten steps, then backward ten steps, they probably won't end up exactly where they started.)

    To get the logical and physical positions back in sync, the driver does what it can to get the drive head to a known location. It tells the hardware, "Move the drive head one step toward the edge of the disk. Okay, take another step. One more time. Actually, 80 more times." Eventually, the drive head reaches the physical maximum position, and each time the driver tells the hardware to move the head one more step outward, it just bumps against the physical boundary of the drive hardware and makes a click sound. If you issue at least as many "one more step outward" commands as there are steps from the innermost point of the disk to the edge, then the theory is that at the end of the operation, the head is in fact at track zero. At that point, you can set your internal "where is the drive head?" variable to zero and restart the original operation, this time with greater confidence that the drive head is where you think it is.

    The amount of clattering depends on where the drive head was when the operation began. If the drive head were around track 40, then the first 40 requests to move one step closer to the center would do exactly that, and then next 43 requests would make a clicking noise. On the other hand, if the drive head were closer to track zero already, then nearly all of the requests result in the drive head bumping against the physical boundary of the drive hardware, and you get a longer, noisier clicking or grinding sound.

    You can hear the recalibration at the start of this performance.

    Bonus floppy drive music.

    Bonus reading: Tim Paterson, author of DOS, discusses all those floppy disk formats.

  • The Old New Thing

    The tradition of giving cute names to unborn babies

    • 30 Comments

    Many of my friends gave names to their unborn babies. Most of them were based on various objects that were the size of the adorable little parasite¹ at the time they discovered that they were pregnant:

    • The Peanut
    • Gumdrop
    • Jellybean
    • Blueberry
    • Mr. Bean

    There were a few outliers, though.

    That last one takes a bit of explaining. Having grown tired of people asking her what she was planning on naming the baby, my friend made up an absurd name and used it with a straight face. "We're think of naming her Aubergine, if it's a girl." People would respond with a polite but confused "Oh, that's an interesting name."

    Then, still deadpan, she would add, "If it's a boy, then we're leaning toward Mad-Dog."

    That usually tipped people off that she was just messing with them.

    Related:

    ¹ A peek behind the curtain: I couldn't decide whether to write fetus or embryo, and I knew that if I picked one, then people would say that I should've picked the other, so I decided to avoid the issue entirely by writing "adorable little parasite". This is what nitpickers have turned me into.

  • The Old New Thing

    What does the executable timestamp really mean?

    • 12 Comments

    A customer was looking for some information on the executable timestamp:

    I would like my program to display the date it was linked. The IMAGE_FILE_HEADER.Time­Date­Stamp looks like what I need. Is there an easy way to retrieve this information so I don't have to parse the EXE header myself? Also, what functions exist for formatting this timestamp into something human-readable?

    The customer didn't explain why they needed this information, but presumably it was for diagnostic purposes. For example, the program might display the information in the About box to help the product support team identify which version of the program the end-user is running.

    We'll answer the questions in reverse order, and then answer a question that wasn't even asked.

    The timestamp is a Unix-style time_t timestamp; therefore, you can use the ctime function to convert it to text. If there is a particular format you like, you can use the appropriate time formatting function (though you may have to convert it first).

    If you want to retrieve this value, you can use helper functions in the imagehlp library; the one most applicable here appears to be Image­Nt­Header or even Get­Time­stamp­For­Loaded­Library.

    The unasked question is "Does this in fact give me the date and time that the image was linked?" Fortunately, I don't have to write out the answer to this question, because I answered it last year. The name timestamp is misleading. Its real purpose is to act as a signature so that the operating system can determine whether a DLL against which one set of values was precalculated matches the DLL physically on the system. A better name for it would have been UniqueId.

  • The Old New Thing

    At least it'll be easy to write up the security violation report

    • 37 Comments

    Many years ago, Microsoft instituted a new security policy at the main campus: all employees must visibly wear their identification badge, even when working in their office. As is customary with with nearly all new security policies, it was met with resistance.

    One of my colleagues was working late, and his concentration was interrupted by a member of the corporate security staff at his door.

    Sir, can I see your cardkey?

    My colleague was not in a good mood (I guess it was a nasty bug), so he curtly replied, "No. I'm busy."

    Sir, you have to show me your cardkey. It's part of the new security policy.

    "I told you, I'm busy."

    Sir, if you don't show me your cardkey, I will have to write you up.

    "Go ahead, if it'll get you out of my office."

    All right, then. What's your name?

    Without even looking from his screen, my colleague replied impatiently, "It's printed on the door."

    The policy was rescinded a few weeks later.

  • The Old New Thing

    Windows has supported multiple UI languages for over a decade, but nobody knew it

    • 30 Comments

    In the early days of Windows, there was a separate version of Windows for each language, and once you decided to install, say, the French version of Windows, you were locked into using French. You couldn't change your mind and, say, switch to German. The reason for this is that there were bits and pieces of language-dependent information stored all over the system.

    One obvious place is in file names. For example, a shortcut to the calculator program was kept at %USERPROFILE%\Start Menu\Programs\Accessories\Calculator.lnk on US-English systems, but %USERPROFILE%\Startmenü\Programme\Zubehör\Rechner.lnk on German systems. The name of the physical file system directory or file was displayed to the user as the name of the menu item. This means that if you started with an English system and simply replaced all the user interface resources with the corresponding German ones, you would still see a folder named Accessories on your Start menu, containing a shortcut named Calculator, even though they should now be displayed as Zubehör and Rechner.

    The registry was another place where language-dependent strings were stored. For example, file type descriptions were stored in plain text, which meant that if you installed an English system, then HKEY_CLASSES_ROOT\txtfile had the value Text Document, and that's the value shown to the user under the Typ column even though the user had switched the user interface resources to German.

    For Windows 2000, an effort was made to move all language-dependent content into resources so that they could be changed dynamically. If you need to store a language-dependent string anywhere, you can't store the string in plain text, because that would not survive a change in language. You have to store an indirect string and convert the indirect string to a real string at runtime, so that it mapped through the user's current user interface language. It was quite an effort identifying all the places that needed to be changed to conform to the new rules while still ensuring that the new rules were backward compatible with old code that followed the old rules.

    For example, you couldn't just say "To register a language-aware file type friendly name, write an indirect string to HKEY_CLASSES_ROOT\progid. For example, set HKEY_CLASSES_ROOT\txtfile to REG_SZ:@C:\Windows\system32\notepad.exe,-469." If you did that, then applications which retrieved file type friendly names by reading directly from HKEY_CLASSES_ROOT\progid (instead of using functions like SHGet­File­Info) would end up showing this to the user:

    Name Type Modified
    House pictures @C:\Windows\system32\zipfldr.dll,-10195 11/16/1998 4:09 PM
    notes @C:\Windows\system32\notepad.exe,-469 11/23/1998 1:52 PM
    Proposal @"C:\Program Files\Windows NT\Accessories\WORDPAD.EXE",-190 10/31/1998 10:32 AM

    instead of

    Name Type Modified
    House pictures Compressed Folder 11/16/1998 4:09 PM
    notes Text Document 11/23/1998 1:52 PM
    Proposal Rich Text Document 10/31/1998 10:32 AM

    Designing and implementing all this was a major undertaking (that's what happens when you have to retrofit something as opposed to designing it in from the beginning), and to keep the test matrix from growing quadratically in the number of supported languages, a decision was made early on to support dynamic language changes only if the starting language is English. So yes, you could have both English and Dutch resources installed, but you have to start with English and add Dutch and not the other way around.

    Mind you, the implementation in Windows 2000 was not perfect. There were still places where English strings appeared even after you switched the user interface language to Dutch or German, but things got better at each new version of Windows. Unfortunately, pretty much nobody knew about this feature, since it was marketed to large multinational corporations and not to your random everyday users who simply want to change the user interface to a language they are more comfortable with.

    For Windows 2000 and Windows XP, you still had two ways of installing Windows with a German user interface: You could either install the English version and then add the German language pack (the fancy Windows 2000 multilingual way), or you could install the fully-localized German version of Windows, just as you always did. In Windows Vista, fully-localized versions of Windows were dropped. From Windows Vista onwards, all versions of Windows consist of a base language-neutral version with a language pack installed on top.

    While it's true that access to the feature has improved in more recent versions of Windows, the feature has existed for over a decade. But of course, that doesn't stop people from claiming that it's a "new" feature. Don't let the facts get in the way of a good story.

  • The Old New Thing

    We've traced the pipe, and it's coming from inside the process!

    • 5 Comments

    We saw last time one of the deadlocks you can run into when playing with pipes. Today we'll look at another one:

    Our program runs a helper process with stdin/stdout/stderr redirected. The helper process takes input via stdin and prints the result to stdout. Sometimes we find that the Write­File from the controlling process into the stdin pipe hangs. Closer examination reveals that the helper process no longer exists. Under these conditions, should the Write­File fail, since the reader is no longer available?

    If you attempt to write to a pipe when there is nobody around to call Read­File to read the data out the other end, the call to Write­File should fail with the error ERROR_BROKEN_PIPE (known in Unix-land as EPIPE). What does it mean when the write pends? It means that there is still somebody around who can read the data out of the pipe, but the internal pipe buffer is full, so the write call waits for the reader to drain the data.

    But the helper process no longer exists. Maybe it crashed or exited prematurely. That means that there is nobody around to read the data out of the pipe. Why, then, does the call not return immediately with an error?

    Because there is still somebody around to read the data out of the pipe.

    Did you remember to close the controlling process's copy of the read end of the pipe?

    If the controlling process hasn't closed its copy of the read end of the pipe, then the pipe is correct in believing that there is still somebody around to read the data out of the pipe, namely you. You have a handle to the read end of the pipe, so the pipe manager cannot declare the pipe dead; for all it knows, you intended for the controlling process to call Read­File to read the data out of the pipe. As far as the pipe is concerned, you simply haven't gotten around to it yet, so the pipe waits patiently.

    Yes, our code calls Close­Handle on the controlling process's copy of the pipe handles. I've highlighted it below. (Error checking has been elided for simplicity.)

    // create the pipe for stdout/stderr
    CreatePipe(&hReadPipeTmp, &hWritePipeTmp, NULL, 0);
    
    // duplicate the handles with bInheritHandle=FALSE to prevent
    // them from being inherited
    DuplicateHandle(GetCurrentProcess(), hWritePipeTmp,
                    GetCurrentProcess(), &hWritePipe,
                    0, FALSE, DUPLICATE_SAME_ACCESS);
    DuplicateHandle(GetCurrentProcess(), hReadPipeTmp,
                    GetCurrentProcess(), &hReadPipe,
                    0, FALSE, DUPLICATE_SAME_ACCESS);
    
    // create the pipe for stdin
    CreatePipe(&hHelperReadPipe, &hHelperWritePipe,
               NULL, 0);
    
    // disable inheritance on on the write end of the stdin pipe
    SetHandleInformation(hHelperWritePipe, HANDLE_FLAG_INHERIT, 0);
    
    // prepare to create the process
    ... blah blah blah other stuff unrelated to handles ...
    startupInfo.hStdInput = hHelperReadPipe;
    startupInfo.hStdOutput = hWritePipeTmp;
    startupInfo.hStdError = hWritePipeTmp;
    
    CreateProcess(...);
    
    // Here is where we close the handles
    CloseHandle(hReadPipeTmp);
    CloseHandle(hWritePipeTmp);
    
    // Write the input to the helper process (hangs here sometimes)
    WriteFile(hHelperWritePipe, ...);
    

    This is another case of getting so excited about doing something that you forget to do it. (Notice how the comments to that article very quickly descend into a discussion of command line quotation marks.)

    Observe that the handles being closed are hRead­Pipe­Tmp and hWrite­Pipe­Tmp, which is a good thing to do, but neither has any effect on the Write­File. The Write­File is writing to hHelper­Write­Pipe and therefore the handle you need to close is hHelper­Read­Pipe. Since that handle is still open in the controlling process, the pipe manager will not break the pipe, because it's waiting for you to read from it.

  • The Old New Thing

    And... that cadence means it's halftime, concert-goers!

    • 23 Comments

    In college, one of my classmates (who is now the conductor of an orchestra, so I guess that whole music thing worked out for him) coined the term halftime to refer to a resounding cadence in the first half of a piece, the type of cadence that might fool an inattentive or unseasoned listener into thinking that the piece is over, when in fact it's just getting started. We're not talking about a false ending, which is comparatively easy to find, but rather a "big finish" when we're nowhere near the finish.

    Also sprach Zarathustra has a big halftime cadence, complete with a reprise of the opening fanfare, at the end of Der Genesende. This was the piece that inspired the coinage of the term.

    Rhapsody on a Theme by Paganini has a quiet halftime cadence (echoing a big Dies Irae just a few seconds earier) in variation XX.

    Schubert's Trout Quintet has a halftime cadence in the precise center of the final movement. Warning listeners about it doesn't help.

    The first movement of Mahler's Second Symphony has a massive halftime cadence. (Super sideburns version.) The piece closed Gerard Schwarz's final concert (for nitpickers: Schwarz's final regular-season concert as music director of the Seattle Symphony), and when the halftime cadence was reached, I heard scattered applause through the hall. Which told me a few things.

    • The people who attended that concert were not regular concert-goers. Another cue was that people were far more nattily-dressed than your average Seattle Symphony crowd, and we were in the third tier! (Now, I don't normally sit in the third tier, so who knows, maybe the third-tier crowd is the dress-up crowd.)
    • The people who applauded under the mistaken impression that the piece was over were in for a big disappointment, since there was another hour to go!

    Can you come up with other examples of halftime?

    False ending sidebar: Haydn is famous for false endings and other tricks. Here's a fun story about selling the false ending in Haydn's 90th.

  • The Old New Thing

    Be careful when redirecting both a process's stdin and stdout to pipes, for you can easily deadlock

    • 18 Comments

    A common problem when people create a process and redirect both stdin and stdout to pipes is that they fail to keep the pipes flowing. Once a pipe clogs, the disturbance propagates backward until everything clogs up.

    Here is a common error, in pseudocode:

    // Create pipes for stdin and stdout
    CreatePipe(&hReadStdin, &hWriteStdin, NULL, 0);
    CreatePipe(&hReadStdout, &hWriteStdout, NULL, 0);
    
    // hook them up to the process we're about to create
    startup_info.hStdOutput = hWriteStdout;
    startup_info.hStdInput = hReadStdin;
    
    // create the process
    CreateProcess(...);
    
    // write something to the process's stdin
    WriteFile(hWriteStdin, ...);
    
    // read the result from the process's stdout
    ReadFile(hReadStdout, ...);
    

    You see code like this all over the place. I want to generate some input to a program and capture the output, so I pump the input as the process's stdin and read the output from the process's stdout. What could possibly go wrong?

    This problem is well-known to unix programmers, but it seems that the knowledge hasn't migrated to Win32 programmers. (Or .NET programmers, who also encounter this problem.)

    Recall how anonymous pipes work. (Actually, this description is oversimplified, but it gets the point across.) A pipe is a marketplace for a single commodity: Bytes in the pipe. If there is somebody selling bytes (Write­File), the seller waits until there is a buyer (Read­File). If there is somebody looking to buy bytes, then the buyer waits until there is a seller.

    In other words, when somebody writes to a pipe, the call to Write­File waits until somebody issues a Read­File. Conversely, when somebody reads from a pipe, the call to Read­File waits until somebody calls Write­File. When there is a matching read and write, the bytes are transferred from the writer's buffer to the reader's buffer. If the reader asks for fewer bytes than the writer provided, then the writer continues waiting until all the bytes have been read. (On the other hand, if the writer provides fewer bytes than the reader requested, the reader is given a partial read. Yes, there's asymmetry there.)

    Okay, so where's the deadlock in the above code fragment? We write some data into one pipe (connected to a process's stdin) and then read from another pipe (connected to a process's stdout). For example, the program might take some input, do some transformation on it, and print the result to stdout. Consider:

    YouHelper
    WriteFile(stdin, "AB")
    (waits for reader)
    ReadFile(stdin, ch)
    reads A
    (still waiting since not all data read)
    encounters errors
    WriteFile(stdout, "Error: Widget unavailable\r\n")
    (waits for reader)

    And now we're deadlocked. Your process is waiting for the helper process to finish reading all the data you wrote (specifically, waiting for it to read B), and the helper process is waiting for your process to finish reading the data it wrote to its stdout (specifically, waiting for you to read the error message).

    There's a feature of pipes that can mask this problem for a long time: Buffering.

    The pipe manager might decide that when somebody offers some bytes for sale, instead of making the writer wait for a reader to arrive, the pipe manager will be a market-maker and buy the bytes himself. The writer is then unblocked and permitted to continue execution. Meanwhile, when a reader finally arrives, the request is satisfied from the stash of bytes the pipe manager had previously bought. (But the pipe manager doesn't take a 10% cut.)

    Therefore, the error case above happens to work, because the buffering has masked the problem:

    YouHelper
    WriteFile(stdin, "AB")
    pipe manager accepts the write
    ReadFile(stdout, result)
    (waits for read)
    ReadFile(stdin, ch)
    reads A
    encounters errors
    WriteFile(stdout, "Error: Widget unavailable\r\n")
    Read completes

    As long as the amount of unread data in the pipe is within the budget of the pipe manager, the deadlock is temporarily avoided. Of course, that just means it will show up later under harder-to-debug situations. (For example, if the program you are driving prints a prompt for each line of input, then the problem won't show up until you give the program a large input data set: For small data sets, all the prompts will fit in the pipe buffer, but once you hit the magic number, the program hangs because the pipe is waiting for you to drain all those prompts.)

    To avoid this problem, your program needs to keep reading from stdout while it's writing to stdin, so that neither will block the other. The easiest way to do this is to perform the two operations on separate threads.

    Next time, another common problem with pipes.

    Exercise: A customer reported that this function would sometimes hang waiting for the process to exit. Discuss.

    int RunCommand(string command, string commandParams)
    {
     var info = new ProcessStartInfo(command, commandParams);
     info.UseShellExecute = false;
     info.RedirectStandardOutput = true;
     info.RedirectStandardError = true;
     var process = Process.Start(info);
     while (!process.HasExited) Thread.Sleep(1000);
     return process.ExitCode;
    }
    

    Exercise: Based on your answer to the previous exercise, the customer responds, "I added the following code, but the problem persists." Discuss.

    int RunCommand(string command, string commandParams)
    {
     var info = new ProcessStartInfo(command, commandParams);
     info.UseShellExecute = false;
     info.RedirectStandardOutput = true;
     info.RedirectStandardError = true;
     var process = Process.Start(info);
     var reader = Process.StandardOutput;
     var results = new StringBuilder();
     string lineOut;
     while ((lineOut = reader.ReadLine()) != null) {
      results.AppendLine("STDOUT: " + lineOut);
     }
     reader = Process.StandardError;
     while ((lineOut = reader.ReadLine()) != null) {
      results.AppendLine("STDERR: " + lineOut);
     }
     while (!process.HasExited) Thread.Sleep(1000);
     return process.ExitCode;
    }
    
  • The Old New Thing

    The historical struggle over control of the Portuguese language

    • 42 Comments

    Portugal has been going through a rough patch. Its international stature has diminished over the years, its economy has always struggled to remain competitive, the government had to accept a bailout to avoid defaulting on its debt, and on top of it all, it is losing control of its own language.

    In Portugal, the latest round of Portuguese spelling reform takes effect over a six-year transition period, leaving the Portuguese dismayed that the spelling of their language is being driven by Brazil, a former colony. I sympathize with the plight of the Portuguese, although I also understand the value of consistent spelling. (The rules for the English language are established not by any central authority but rather are determined by convention.)

    I wonder if the U.K. feels the same way about its former colony.

    Bonus chatter: The Microsoft Language Portal Blog reports that Microsoft intends to phase in the spelling reform over a four-year period for Brazil-localized products. A quick glance at the Microsoft style guide for Portuguese (Portugal) says that the spelling reform has yet to take effect among the Portugal-localized version of Microsoft products.

Page 120 of 448 (4,479 items) «118119120121122»