July, 2011

  • The Old New Thing

    Be careful when redirecting both a process's stdin and stdout to pipes, for you can easily deadlock

    • 18 Comments

    A common problem when people create a process and redirect both stdin and stdout to pipes is that they fail to keep the pipes flowing. Once a pipe clogs, the disturbance propagates backward until everything clogs up.

    Here is a common error, in pseudocode:

    // Create pipes for stdin and stdout
    CreatePipe(&hReadStdin, &hWriteStdin, NULL, 0);
    CreatePipe(&hReadStdout, &hWriteStdout, NULL, 0);
    
    // hook them up to the process we're about to create
    startup_info.hStdOutput = hWriteStdout;
    startup_info.hStdInput = hReadStdin;
    
    // create the process
    CreateProcess(...);
    
    // write something to the process's stdin
    WriteFile(hWriteStdin, ...);
    
    // read the result from the process's stdout
    ReadFile(hReadStdout, ...);
    

    You see code like this all over the place. I want to generate some input to a program and capture the output, so I pump the input as the process's stdin and read the output from the process's stdout. What could possibly go wrong?

    This problem is well-known to unix programmers, but it seems that the knowledge hasn't migrated to Win32 programmers. (Or .NET programmers, who also encounter this problem.)

    Recall how anonymous pipes work. (Actually, this description is oversimplified, but it gets the point across.) A pipe is a marketplace for a single commodity: Bytes in the pipe. If there is somebody selling bytes (Write­File), the seller waits until there is a buyer (Read­File). If there is somebody looking to buy bytes, then the buyer waits until there is a seller.

    In other words, when somebody writes to a pipe, the call to Write­File waits until somebody issues a Read­File. Conversely, when somebody reads from a pipe, the call to Read­File waits until somebody calls Write­File. When there is a matching read and write, the bytes are transferred from the writer's buffer to the reader's buffer. If the reader asks for fewer bytes than the writer provided, then the writer continues waiting until all the bytes have been read. (On the other hand, if the writer provides fewer bytes than the reader requested, the reader is given a partial read. Yes, there's asymmetry there.)

    Okay, so where's the deadlock in the above code fragment? We write some data into one pipe (connected to a process's stdin) and then read from another pipe (connected to a process's stdout). For example, the program might take some input, do some transformation on it, and print the result to stdout. Consider:

    YouHelper
    WriteFile(stdin, "AB")
    (waits for reader)
    ReadFile(stdin, ch)
    reads A
    (still waiting since not all data read)
    encounters errors
    WriteFile(stdout, "Error: Widget unavailable\r\n")
    (waits for reader)

    And now we're deadlocked. Your process is waiting for the helper process to finish reading all the data you wrote (specifically, waiting for it to read B), and the helper process is waiting for your process to finish reading the data it wrote to its stdout (specifically, waiting for you to read the error message).

    There's a feature of pipes that can mask this problem for a long time: Buffering.

    The pipe manager might decide that when somebody offers some bytes for sale, instead of making the writer wait for a reader to arrive, the pipe manager will be a market-maker and buy the bytes himself. The writer is then unblocked and permitted to continue execution. Meanwhile, when a reader finally arrives, the request is satisfied from the stash of bytes the pipe manager had previously bought. (But the pipe manager doesn't take a 10% cut.)

    Therefore, the error case above happens to work, because the buffering has masked the problem:

    YouHelper
    WriteFile(stdin, "AB")
    pipe manager accepts the write
    ReadFile(stdout, result)
    (waits for read)
    ReadFile(stdin, ch)
    reads A
    encounters errors
    WriteFile(stdout, "Error: Widget unavailable\r\n")
    Read completes

    As long as the amount of unread data in the pipe is within the budget of the pipe manager, the deadlock is temporarily avoided. Of course, that just means it will show up later under harder-to-debug situations. (For example, if the program you are driving prints a prompt for each line of input, then the problem won't show up until you give the program a large input data set: For small data sets, all the prompts will fit in the pipe buffer, but once you hit the magic number, the program hangs because the pipe is waiting for you to drain all those prompts.)

    To avoid this problem, your program needs to keep reading from stdout while it's writing to stdin, so that neither will block the other. The easiest way to do this is to perform the two operations on separate threads.

    Next time, another common problem with pipes.

    Exercise: A customer reported that this function would sometimes hang waiting for the process to exit. Discuss.

    int RunCommand(string command, string commandParams)
    {
     var info = new ProcessStartInfo(command, commandParams);
     info.UseShellExecute = false;
     info.RedirectStandardOutput = true;
     info.RedirectStandardError = true;
     var process = Process.Start(info);
     while (!process.HasExited) Thread.Sleep(1000);
     return process.ExitCode;
    }
    

    Exercise: Based on your answer to the previous exercise, the customer responds, "I added the following code, but the problem persists." Discuss.

    int RunCommand(string command, string commandParams)
    {
     var info = new ProcessStartInfo(command, commandParams);
     info.UseShellExecute = false;
     info.RedirectStandardOutput = true;
     info.RedirectStandardError = true;
     var process = Process.Start(info);
     var reader = Process.StandardOutput;
     var results = new StringBuilder();
     string lineOut;
     while ((lineOut = reader.ReadLine()) != null) {
      results.AppendLine("STDOUT: " + lineOut);
     }
     reader = Process.StandardError;
     while ((lineOut = reader.ReadLine()) != null) {
      results.AppendLine("STDERR: " + lineOut);
     }
     while (!process.HasExited) Thread.Sleep(1000);
     return process.ExitCode;
    }
    
  • The Old New Thing

    The historical struggle over control of the Portuguese language

    • 42 Comments

    Portugal has been going through a rough patch. Its international stature has diminished over the years, its economy has always struggled to remain competitive, the government had to accept a bailout to avoid defaulting on its debt, and on top of it all, it is losing control of its own language.

    In Portugal, the latest round of Portuguese spelling reform takes effect over a six-year transition period, leaving the Portuguese dismayed that the spelling of their language is being driven by Brazil, a former colony. I sympathize with the plight of the Portuguese, although I also understand the value of consistent spelling. (The rules for the English language are established not by any central authority but rather are determined by convention.)

    I wonder if the U.K. feels the same way about its former colony.

    Bonus chatter: The Microsoft Language Portal Blog reports that Microsoft intends to phase in the spelling reform over a four-year period for Brazil-localized products. A quick glance at the Microsoft style guide for Portuguese (Portugal) says that the spelling reform has yet to take effect among the Portugal-localized version of Microsoft products.

  • The Old New Thing

    Looking at the problem at the wrong level: Closing a process's stdin

    • 11 Comments

    A customer was having trouble manipulating the stdin stream that was given to a process.

    How do you simulate sending Ctrl+Z to a hidden console process programmatically?

    I am using Redirect­Standard­Input and want to send the console a Ctrl+Z. I've tried sending ASCII code 26, but that doesn't work. Generate­Console­Ctrl­Event supports Ctrl+C and Ctrl+Break but not Ctrl+Z.

    Here's what I'm doing, but it doesn't work:

    ProcessStartInfo info = new ProcessStartInfo(@"...");
    info.CreateNoWindow = true;
    info.RedirectStandardError = true;
    info.RedirectStandardOutput = true;
    info.RedirectStandardInput = true;
    info.UseShellExecute = false;
    Process p = Process.Start(info);
    // 0x1A is ASCII code of Ctrl+Z but it does not work
    p.StandardInput.WriteLine("\x1A");
    

    The customer was kind enough to do more than simply ask the question. The customer set up the scenario and even provided a code fragment that illustrates the problem. Which is good, because the original question was the wrong question.

    The customer asked about simulating typing Ctrl+Z to a console, but what they actually doing was sending a character to stdin; they weren't sending it to a console. In fact, the way they created the process, there is no console at all.

    The customer confused stdin with consoles. It's true that Ctrl+Z is the convention used by console windows to indicate that stdin should be closed. But that is hardly any consolation when you took control of stdin yourself and are not using a console window to manage it.

    It's like saying, "Normally, when I want somebody to take my order, I pull into a parking space and turn on my headlights, and somebody will come out. But I can't get it to work."

    Um, that's because you pulled into your own driveway.

    Ctrl+Z is a convention used by console windows to indicate that stdin should be closed, but if you said "I am going to manage stdin myself," then you aren't using a console window, and that convention carries no weight. If you write a Ctrl+Z to the process's stdin, it will simply read a Ctrl+Z. But since you are managing stdin yourself, you can do it yourself: Just take the stream you set as the process's stdin and close it.

    Exercise: Perhaps you can answer this related question from a different customer:

    I am trying to send a Ctrl+C (SIGINT) to a process.

    CurrentProcess = new Process();
    CurrentProcess.StartInfo.FileName = "foo.exe";
    CurrentProcess.StartInfo.UseShellExecute = false;
    CurrentProcess.StartInfo.RedirectStandardInput = true;
    StandardInputWriter = CurrentProcess.StandardInput;
    char c = '\u0003';
    StandardInputWriter.Write(c);
    StandardInputWriter.Flush();
    StandardInputWriter.Close();
    

    If I launch the process from a command prompt and type Ctrl+C, it flushes its output and terminates, but when I start it from within my application and send it a Ctrl+C via the code above, the process is still running. How do I send a Ctrl+C to a process?

  • The Old New Thing

    Microspeak: Reporting through

    • 5 Comments

    I'll start with the citation from a hypothetical conversation:

    "This is being handled by Jonathan Swift."

    Who does he report through?

    "He reports up through Jane Austen's org."

    The Microspeak term report through (or report up through) comes up often in situations where people from different groups are working together. In its most literal meaning, to report through someone is to have that person as your manager, or your manager's manager, or your manager's manager's manager, etc. (Not that any of those people beyond two levels actually reads any of the reports you've written!)

    Usually, when someone asks who a person reports through, they are looking for the manager who reports to an implied reference point. The location of this reference point varies depending context.

    For example, if the discussion is in the context of coordinating efforts across all of Microsoft, then the implied reference point is Steve Ballmer. Somebody who works on Windows reports through Steve Sinofsky, whereas someone who works on Office reports through Kurt DelBene.

    On the other hand, if the discussion is in the context of coordinating efforts within Windows, then the implied reference point is Steve Sinofsky. The answer to "Who does he report through" would be the manager who directly reports to Steve Sinofsky.

    Bonus Microspeak: Org is a common short form of the word organization, meaning "group of people who all report (directly or indirectly) to the same person."

  • The Old New Thing

    A handful of trips through the time machine

    • 20 Comments

    A few trips through the time machine:

    In the Internet Explorer time machine video, I was struck by the remark, "Appearance-wise, very little had changed [in Internet Explorer 4] since IE3. Not much changed in terms of functionality, either." In fact, Internet Explorer 4 was probably the most significant revision of Internet Explorer in its history, because that's the version that completely replaced the old layout engine with a new one code-named Trident, the layout engine that continues to power Internet Explorer today. Another case of "When you change the insides, nobody notices."

  • The Old New Thing

    The list of heaps returned by GetProcessHeaps is valid when it returns, but who knows what happens later

    • 5 Comments

    A customer had a problem involving heap corruption.

    In our code, we call Get­Process­Heaps and then for each heap, we call Heap­Set­Information to enable the low fragmentation heap. However, the application crashes due to an invalid heap handle.

    HANDLE heaps[1025];
    DWORD nHeaps = GetProcessHeaps(heaps, 1024);
    for (DWORD i = 0; i < nHeaps; i++) {
     ULONG HeapFragValue = HEAP_LFH;
     HeapSetInformation(heaps[i], HeapCompatibilityInformation,
                        &HeapFragValue, sizeof(HeapFragValue));
    }
    

    My question is, why do we need to allocate an array of size 1025 even though we pass 1024 to Get­Process­Heaps?

    Ha, faked you out with that question, eh? (It sure faked me out.)

    It's not clear why the code under-reports the buffer size to Get­Process­Heaps. So let's ignore the customer's stated question and move on to the more obvious question: Why does this code crash due to an invalid heap handle?

    Well, for one thing, the code mishandles the case where there are more than 1024 heaps in the process. But as it happens, the value returned by Get­Process­Heaps was well below 1024, so that wasn't the reason for the crash.

    Unlike kernel objects, heaps are just chunks of user-mode-managed memory. A heap handle is not reference-counted. (Think about it: If it were, how would you release the reference count? There is no Heap­Close­Handle function.) When you destroy a heap, all existing handles to that heap become invalid.

    The consequence of this is that there is a race condition inherent in the use of the Get­Process­Heaps function: Even though the list of heaps is correct when it is returned to you, another thread might sneak in and destroy one of those heaps out from under you.

    This didn't explain the reported crash, however. "We execute this code during application startup, before we create any worker threads, so there should be no race condition."

    While it may be true that the code is executed before the program calls Create­Thread, a study of the crash dump reveals that some sneaky DLLs had paid a visit to the process and had even unloaded themselves!

    0:001> lm
    start    end        module name
    75b10000 75be8000   kernel32   (deferred)
    77040000 7715e000   ntdll      (deferred)
    ...
    
    Unloaded modules:
    775e0000 775e6000   NSI.dll 
    76080000 760ad000   WS2_32.dll
    71380000 713a2000   COEC23~1.DLL
    

    "Well, that explains how a heap could have been destroyed from behind our back. That COEC23~1.DLL probably created a private heap and destroyed it when it was unloaded. But how did that DLL get injected into our process in the first place?"

    Given the presence of some networking DLLs, the customer guessed that COEC23~1.DLL was injected by network firewall security software, but given that these were Windows Error Reporting crash dumps, there was no way to get information from the user's machine about how that COEC23~1.DLL ended up loaded in the process, and then spontaneously unloaded.

    Even though we weren't able to find the root cause, we were still able to make some suggestions to avoid the crash.

    Instead of trying to convert every heap to a low fragmentation heap, just convert the process heap. The process heap remains valid for the lifetime of the process, so you won't see it destroyed out from under you. (Or if you do, then you have bigger problems than a crash in Heap­Set­Information.)

    In fact, you can remove the code entirely when running on Windows Vista or higher, because all heaps default to the low fragmentation heap starting in Windows Vista.

    Running around and changing settings on heaps you didn't create is not a good idea. Somebody else owns that heap; who knows what they're going to do with it?

    Okay, so if Get­Process­Heaps is so fragile, why does it even exist?

    Well, it's not really intended for general use. It exists primarily for diagnostics. You might be chasing down a memory corruption bug, so you sprinkle into your code some calls to a helper function that calls Get­Process­Heaps to get all the heaps and then calls Heap­Validate on each one to check for corruption. Or maybe you're chasing down a memory leak in a particular scenario, so you have a function which calls Get­Process­Heaps and Heap­Walk once before entering the scenario, and then again after the scenario completes, and then compares the results looking for leaks. In both cases, you're using the facility for debugging and diagnostic purposes. If there's a race condition that destroys a heap while you're studying it, you'll just throw away the results of that run and try again.

    Bonus chatter: While writing up this story, I went back and did some more Web searching for that mysterious COEC23~1.DLL. (Tracking it down is hard because all you really know about the file name is that it begins with "CO"; the rest is a hashed short file name.) And I found it: It's not an antivirus program. It's one of those "desktop enhancement" programs that injects itself into every process with the assistance of App­Init_DLLs, or as I prefer to call it Deadlock_Or_Crash_Randomly_DLLs. (You may have noticed that I anonymized the program as "CO", short for Contoso, a fictional company used throughout Microsoft literature.)

Page 3 of 3 (26 items) 123