• The Old New Thing

    Intentionally making the suggestion look nothing like any scripting language, yet understandable enough to get the point across

    • 7 Comments

    In an internal peer-to-peer discussion list for an internal tool I'll call Program Q, somebody asked,

    How can I query the number of free frobs in every table in my table repository?

    I suggested that they could use the command

    q query-property "*::frobs-free"
    

    taking advantage of the fact that in Program Q, you can specify a wildcard for the table name to query across all tables.

    Thanks, this looks promising, but my repository has a huge number of tables, so the q query-property command refuses to expand the * wildcard that much. I can get around this by issuing 26 queries, one for each letter of the alphabet:

    q query-property "a*::frobs-free"
    q query-property "b*::frobs-free"
    q query-property "c*::frobs-free"
    ...
    q query-property "z*::frobs-free"
    

    Is there a better way to do this?

    I replied with some pseudocode.

    (
      from table in `q list-tables`
      select table + "::frobs-free"
    ) | q query-property @-
    

    (The @ means that it should take the list of properties from a file, and we give - as the file name, meaning standard input. Not that it's important because I completely made this up.)

    A colleague of mine noted that I provided just enough syntax to explain the algorithm clearly, but in a form that cannot be executed in any scripting language, so the user understands that it is just an algorithm that needs to be massaged into something that will actually run.

    It's a neat trick when it works. But when it fails, it fails spectacularly. Fortunately, in this case, it worked.

    Bonus chatter: For all I know, that's valid PowerShell.

  • The Old New Thing

    Trying out all the different ways of recognizing different types of timestamps from quite a long way away

    • 12 Comments

    Today's Little Program takes a 64-bit integer and tries to interpret it in all the various timestamp formats. This comes in handy when you have extracted a timestamp from a crash dump and want to see it in a friendly format.

    using System;
    
    class Program
    {
     static void TryFormat(string format, Func<DateTime> func)
     {
      try
      {
       DateTime d = func();
       Console.WriteLine("{0} {1}", format, d);
      }
      catch (ArgumentException)
      {
       Console.WriteLine("{0} - invalid", format);
      }
     }
    

    The Try­Format method executes the passed-in function inside a try/catch block. If the function executes successfully, then we print the result. If it raises an argument exception, then we declare the value as invalid.

     static DateTime DateTimeFromDosDateTime(long value)
     {
      if ((ulong)value > 0x00000000FFFFFFFF) {
       throw new ArgumentOutOfRangeException();
      }
      int intValue = (int)value;
      int year = (intValue >> 25) & 127;
      int month = (intValue >> 21) & 15;
      int day = (intValue >> 16) & 31;
      int hour = (intValue >> 11) & 31;
      int minute = (intValue >> 5) & 63;
      int second = (intValue << 1) & 63;
      return new DateTime(1980 + year, month, day, hour, minute, second);
     }
    

    The Date­Time­From­Dos­Date­Time function treats the 64-bit value as a 32-bit date/time stamp in MS-DOS format. Assuming the value fits in a 32-bit integer, we extract the bitfields corresponding to the year, month, day, hour, minute, and second, and construct a Date­Time from it.

     public static void Main(string[] args)
     {
      if (args.Length < 1) return;
    
      long value = ParseLongSomehow(args[0]);
    
      Console.WriteLine("Timestamp {0} (0x{0:X}) could mean", value);
    
      TryFormat("Unix time",
        () => DateTime.FromFileTimeUtc(10000000 * value + 116444736000000000));
      TryFormat("UTC FILETIME",
        () => DateTime.FromFileTimeUtc(value));
      TryFormat("Local FILETIME",
        () => DateTime.FromFileTime(value));
      TryFormat("UTC DateTime",
        () => new DateTime(value, DateTimeKind.Utc));
      TryFormat("Local DateTime",
        () => new DateTime(value, DateTimeKind.Local));
      TryFormat("Binary DateTime",
        () => DateTime.FromBinary(value));
      TryFormat("MS-DOS Date/Time",
        () => DateTimeFromDosDateTime(value));
      TryFormat("OLE Automation Date/Time",
        () => DateTime.FromOADate(BitConverter.Int64BitsToDouble(value)));
     }
    }
    

    Once we have parsed out the command line, we pump the value through all the different conversion functions. Most of them are natively supported by the Date­Time structure, but we had to create a few of them manually.

  • The Old New Thing

    Why does the BackupWrite function take a pointer to a modifiable buffer when it shouldn't be modifying the buffer?

    • 15 Comments

    The Backup­Write function takes a non-const pointer to the buffer to be written to the file being restored. Will it actually modify the buffer? Assuming it doesn't, why wasn't it declared const? It would be much more convenient if it took a const pointer to the buffer, so that people with const buffers didn't have to const_cast every time they called the function. Would changing the parameter from non-const to const create any compatibility problems?

    Okay, let's take the questions in order.

    Will it actually modify the buffer? No.

    Why wasn't it declared const? My colleague Aaron Margosis explained that the function dates back to Windows NT 3.1, when const-correctness was rarely considered. A lot of functions from that area (particularly in the kernel) suffer from the same problem. For example, the computer name passed to the Reg­Connect­Registry function is a non-const pointer even though the function never reads from it.

    Last question: Can the parameter be changed from non-const to const without breaking compatibility?

    It would not cause problems from a binary compatibility standpoint, because a const pointer and a non-const pointer take the same physical form in Win32. However, it breaks source code compatiblity. Consider the following code fragment:

    BOOL WINAPI TestModeBackupWrite(
      HANDLE hFile,
      LPBYTE lpBuffer,
      DWORD nNumberOfBytesToWrite,
      LPDWORD lpNumberOfBytesWritten,
      BOOL bAbort,
      BOOL bProcessSecurity,
      LPVOID *lpContext)
    {
     ... simulate a BackupWrite ...
     return TRUE;
    }
    
    BOOL (WINAPI *BACKUPWRITEPROC)(HANDLE, LPBYTE, DWORD,
                     LPDWORD, BOOL, BOOL, LPVOID *);
    BACKUPWRITEPROC TestableBackupWrite;
    
    void SetTestMode(bool testing)
    {
     if (testing) {
      TestableBackupWrite = TestModeBackupWrite;
     } else {
      TestableBackupWrite = BackupWrite;
     }
    }
    
    

    The idea here is that the program can be run in test mode, say to do a simulated restore. (You see this sort of thing a lot with DVD-burning software.) The program uses Testable­Backup­Write whenever it wants to write to a file being restored from backup. In test mode, Testable­Backup­Write points to the Test­Mode­Backup­Write function; in normal mode, it points to the Backup­Write function.

    If the second parameter were changed from LPBYTE to const BYTE *, then the above code would hit a compiler error.

    Mind you, maybe it's worth breaking some source code in order to get better const-correctness, but for now, the cost/benefit tradeoff biases toward leaving things alone.

  • The Old New Thing

    Is a SID with zero subauthorities a valid SID? It depends whom you ask

    • 6 Comments

    Here's an interesting table.

    Function Is Sub­Authority­Count=0 valid?
    IsValidSid Yes
    Convert­Sid­To­String­Sid Yes
    ConvertString­­Sid­To­Sid No

    That last entry creates the unfortunate situation where a SID with no subauthorities can be converted to a string, but cannot be converted back.

    If it's any consolation, SIDs with no subauthorities aren't encountered in normal usage, so if you ever accidentally reject one of these, it's not going to inconvenience anyone.

    Oh, and the answer to the question at the top: Yes, a SID with zero subauthorities is technically valid. It's a degenerate case that's not very interesting, but it is technically valid.

  • The Old New Thing

    How secure are GUIDs? Can I use it to generate passwords?

    • 29 Comments

    GUIDs are globally unique identifiers, and given their rather incomprehensible presentation, some people may be tempted to use them (or parts of them) as passwords. For example, one person wondered whether it was okay to use the first eight characters of a GUID as a temporary account password.

    This is a really bad idea. GUIDs are designed for uniqueness, not for security.

    For example, we saw that substrings of GUIDs are not unique. For example, in the classic v1 algorithm, the first part of the GUID is a timestamp. Timestamps are a great technique for helping to build uniqueness, but they are horrifically insecure because, well, duh, the current time is hardly a secret!

    Using the first eight characters of a v1 GUID as a password is equivalent to using the current time as a password. And given that the current time isn't a secret, you made things an awful lot easier for the bad guys. All they need to know is what time you generated the GUID and they can get the first eight characters. Even if they don't know the exact time, if they can narrow it down to a range of a few minutes, they can brute-force the remaining options.

    Even if you use one of the other GUID generation algorithms, that doesn't solve the problem, because the purpose of a GUID is to be globally unique, not to be unguessable.

    Besides, the first eight characters of a GUID carry only 32 bits of entropy, because they come from a very limited alphabet: They can be a digit, or a letter from A to F. That's it.

    If you want something cryptographically secure, then use a cryptographically-secure random number. Don't use something that "looks random to me." And use more entropy than 32 bits.

    (Yes, this is basically a rehash of an earlier article, but I wanted to address directly the matter of using GUIDs as passwords.)

  • The Old New Thing

    2015 mid-year link clearance

    • 11 Comments

    The semi-annual link clearance is here!

  • The Old New Thing

    Microspeak: Stretch goal

    • 10 Comments

    Recall that Microspeak is not merely for jargon exclusive to Microsoft, but also for jargon that you need to know.

    A project will set some goals, which are things it intends to accomplish. It may also set stretch goals, which are things it hopes to acomplish. If you fail to achieve a stretch goal, your project is still a success, but if you make it, your project is even more awesome. A stretch goal could be unrelated to an existing goal.

    Goals for this release

    • Support dynamic widget recolorization.

    Stretch goals for this release

    • Improve throughput by 10% compared to previous version.

    Non-goals for this release

    • Support offline mode.

    But more often, it takes the form of a higher level of achievement for an existing goal:

    Goal: 40% of the programs will land actual spend within 10% of estimate. Stretch goal: 60%.

    Setting a stretch goal is tricky. You want to set it just at the edge of achievability. If you make it unrealistic, then nobody will take it seriously.

    You can think of a stretch goal as an "extra credit" assignment. You won't be penalized for missing it, but making it will earn you kudos.

  • The Old New Thing

    Parsing a string as a 64-bit integer, somehow

    • 17 Comments

    Today's Little Program takes a string and tries to parse it as a 64-bit integer in formats that a programmer would likely encounter.

    Here's a first stab:

    using System;
    using System.Globalization;
    
    class Program
    {
     static long ParseLongSomehow(string s)
     {
      if (s.StartsWith("0x", StringComparison.OrdinalIgnoreCase)) {
       return long.Parse(s.Substring(2), NumberStyles.HexNumber);
      } else {
       return long.Parse(s);
      }
     }
    
     public static void Main(string[] args)
     {
      long value = ParseLongSomehow(args[0]);
      Console.WriteLine(value);
      Console.WriteLine("0x{0:X}", value);
     }
    }
    

    If the string begins with 0x, then we treat the rest of the argument as a hex value; otherwise, we treat it as a decimal value.

    Unfortunately, this doesn't work if the input is 9223372036854775808, which is the value of 1 << 63, a value that is representable as a 64-bit unsigned value but not a 64-bit signed value.

    Our problem statement was pretty vague, so let's write a functional specification. It helps to know what problem you're solving before you start to solve it. Otherwise, you're just flailing around writing code before you have a plan. When I tried to solve this problem, I flailed around a bit until I realized that I didn't have a spec.

    What formats would a programmer be likely to encounter as the string representation of a 64-bit integer?

    • 0x1234: 64-bit number in hex format, case-insensitive. The value can range from 0 to UInt64.MaxValue.
    • 12345: 64-bit unsigned number in decimal format. The value can range from 0 to UInt64.MaxValue.
    • -12345: 64-bit signed number in decimal format. The value can range from Int64.MinValue to Int64.MaxValue.
    • Other formats may be permitted, but you need to support at least the above.

    Writing down exactly what I was doing and what I wasn't doing was the part that solved my flailing. I had been worrying about things like -0x12345 and -9223372036854775809 and 9999999999999999999, even though those numbers would not be something a programmer would be likely to encounter.

    From the specification we can develop our algorithm.

    • If the string begins with 0x, then parse what's left as an unsigned 64-bit hexadecimal number.
    • If the string begins with a minus sign, then parse it as a 64-bit signed number in decimal format.
    • If the string does not begin with a minus sign, then parse it as a 64-bit unsigned number in decimal format.

    And that is pretty easy to implement.

     static long ParseLongSomehow(string s)
     {
      if (s.StartsWith("0x", StringComparison.OrdinalIgnoreCase)) {
       return long.Parse(s.Substring(2), NumberStyles.HexNumber);
      } else if (s[0] == '-') {
       return long.Parse(s);
      } else {
       return (long)ulong.Parse(s);
      }
     }
    

    Note that we are a little sloppy with our treatment of whitespace. We accept leading and trailing spaces on decimal values, and allow trailing spaces on hex values (and even allow spaces between the 0x and the first hex digit). That's okay, because the spec allows us to accept formats beyond the ones listed.

    Now, for bonus points, let's revise the functional specification a little bit, specifically by adding another case:

    • 0x12`3456789A: 64-bit number in hex format, case-insensitive, with backtick separating the upper 32 bits from the lower 32 bits.

    This is the format used by the Windows debugger engine.

     static long ParseLongSomehow(string s)
     {
      if (s.StartsWith("0x", StringComparison.OrdinalIgnoreCase)) {
       return long.Parse(s.Substring(2).Replace("`", ""), NumberStyles.HexNumber);
      } else if (s[0] == '-') {
       return long.Parse(s);
      } else {
       return (long)ulong.Parse(s);
      }
     }
    

    We'll leave it here for now. Next time, we'll start putting some blocks together.

  • The Old New Thing

    Windows 95 and Windows NT manage the TEB on x86 differently

    • 21 Comments

    Today, a historical note of no consequence. Just a story.

    The Win32 x86 ABI specifies that the FS register holds a selector which is based at the current thread's TEB. In other words, fs:[n] is the nth byte of the TEB.

    It so happens that the two operating systems chose to manage the FS register differently.

    Windows 95 gave each TEB in the system its own selector.

    Windows NT allocated a single selector to represent the TEB, and each time the processor changed threads, the selector base was updated to match the TEB for the new thread. With this model, every thread has the same value for FS, but the selector's descriptor kept changing.

    It's as if you had a car-rental service, and one of the features of the service is that the radio remembers your presets. The instructions for setting the radio are as follows:

    • Enter the four-digit customer preferences ID printed on your receipt.
    • Your radio is now set to your preferences.

    There are two ways you could set up this system.

    Windows 95 assigns each customer a unique customer preferences ID and prints it on the receipt. When the customer enters the ID, the radio looks up the ID in a database and applies the user's radio preferences.

    Windows NT prints the same preferences ID on every receipt: 1234. Before the customer picks up the car from the rental service, the service agent sets the radio to the customer's preferences. When the customer enters the ID, the radio does nothing (aside from printing an error message if you enter anything other than 1234).

    Even though the Windows NT way creates more work for the service agent, it does solve an important problem: It lets your service scale to more than 10,000 customers, for once you have 10,001 customers, you cannot assign each of them a unique four-digit ID any more.

    Car Windows
    car processor
    customer thread
    radio preferences TEB
    customer ID selector

    By assigning a unique selector to each TEB, Windows 95 limited itself to at most 8192 threads. (In practice, the limit was much lower because selectors were used for other things, too.) This was not an issue in practice because Windows 95 would run into other problems long before you ran into the 8192-thread limit.

    But Windows NT was designed to be scalable to very large workloads, and they couldn't artificially limit themselves to a maximum of 8192 threads.

    The consequences of changing the meaning of the FS register at every thread switch means that some tricks that happened to work in Windows 95 didn't work in Windows NT. For example, in Windows 95, if you captured the value of the FS register in one thread, you could use it (in violation of the ABI) on another thread in the same process and still access the originating thread's TEB. If you tried this trick on Windows NT, you would just see your own TEB.

    In the analogy, it's as if you copied the customer preferences ID from another customer's receipt and tried to use it in your car. In a Windows NT car, you would just get your own preferences again, because every receipt has the same customer preferences ID printed on it.

    According to the Win32 ABI for x86, the only thing you can do with the FS register is dereference it to access your TEB. If you try to fiddle with its value or try to copy its value somewhere, you are off in unsupported territory, and the resulting behavior is undefined.

  • The Old New Thing

    Where is the full version of the music that plays when you start Windows 98 for the first time?

    • 23 Comments

    A customer was presumably exercising the unlimited support part of their support contract when they asked, "Where is the full song for the music that plays when you start Windows 98 for the first time? The Welcome application plays only the first 30 seconds. Can you send us the rest of the song?"

    I guess the IT department really likes that music and wants the extended dance remix for their late-night raves. (If you stick it out, there's a special appearance in the linked video of the screen I wrote.)

    The music was commissioned to be 30 seconds long. There is no extended version of the song, or at least not one that Microsoft acquired the rights to. The original composer could have written a five-minute version and sent Microsoft only the first 30 seconds, or maybe the piece is only 30 seconds long to begin with. All we know is that we have a 30-second music clip.

Page 1 of 453 (4,528 items) 12345»