• The Old New Thing

    Is it wrong to call SHFileOperation from a service? Revised

    • 18 Comments

    My initial reaction to this question was to say, "I don't know if I'd call it wrong, but I'd call it highly inadvisable."

    I'd like to revise my guidance.

    It's flat-out wrong, at least in the case where you call it while impersonating.

    The registry key HKEY_CURRENT_USER is bound to the current user at the time the key is first accessed by a process:

    The mapping between HKEY_CURRENT_USER and HKEY_USERS is per process and is established the first time the process references HKEY_CURRENT_USER. The mapping is based on the security context of the first thread to reference HKEY_CURRENT_USER. If this security context does not have a registry hive loaded in HKEY_USERS, the mapping is established with HKEY_USERS\.Default. After this mapping is established it persists, even if the security context of the thread changes.

    Emphasis mine.

    This means that if you impersonate a user, and then access HKEY_CURRENT_USER, then that binds HKEY_CURRENT_USER to the impersonated user. Even if you stop impersonating, future references to HKEY_CURRENT_USER will still refer to that user.

    This is probably not what you expected.

    The shell takes a lot of settings from the current user. If you impersonate a user and then call into the shell, your service is now using that user's settings, which is effectively an elevation of privilege: An unprivileged user is now modifying settings for a service. For example, if the user has customized the Print verb for text files, and you use Shell­Execute to invoke the print verb on a text document, you are at the mercy of whatever the user's print verb is bound to. Maybe it runs Notepad, but maybe it runs pwnz0rd.exe. You don't know.

    Similarly, the user might have a per-user registered copy hook or namespace extension, and now you just loaded a user-controlled COM object into your service.

    In both cases, this is known to insiders as hitting the jackpot.

    Okay, so what about if you call Shell­Execute or some other shell function while not impersonating? You might say, "That's okay, because the current user's registry is the service user, not the untrusted attacker user." But look at that sentence I highlighted up there. Once HKEY_CURRENT_USER get bound to a particular user, it remains bound to that user even after impersonation ends. If somebody else inadvisedly called a shell function while impersonating, and that shell function happens to be the first one to access HKEY_CURRENT_USER, then your call to a shell function while not impersonating will still use that impersonated user's registry. Congratulations, you are now running untrusted code, and you're not even impersonating any more!

    So my recommendation is don't do it. Don't call shell functions while impersonating unless the function is explicitly documented as supporting impersonation. (The only ones I'm aware of that fall into this category are functions like SHGet­Folder­Path which accept an explicit token handle.) Otherwise, you may have created (or in the case of copy hooks, definitely created) a code injection security vulnerability in your service.

  • The Old New Thing

    A library loaded via LOAD_LIBRARY_AS_DATAFILE (or similar flags) doesn't get to play in any reindeer module games

    • 22 Comments

    If you load a library with the LOAD_LIBRARY_AS_DATA­FILE flag, then it isn't really loaded in any normal sense. In fact, it's kept completely off the books.

    If you load a library with the LOAD_LIBRARY_AS_DATA­FILE, LOAD_LIBRARY_AS_DATA­FILE_EXCLUSIVE, or LOAD_LIBRARY_AS_IMAGE_RESOURCE flag (or any similar flag added in the future), then the library gets mapped into the process address space, but it is not a true module. Functions like Get­Module­Handle, Get­Module­File­Name, Enum­Process­Modules and Create­Toolhelp32­Snapshot will not see the library, because it was never entered into the database of loaded modules.

    These "load library as..." flags don't actually load the library in any meaningful sense. They just take the file and map it into memory manually without updating any module tracking databases. This functionality was overloaded into the Load­Library­Ex function, which in retrospect was probably not a good idea, because people expect Load­Library­Ex to create true modules, but these flags create pseudo-modules, a term I made up just now.

    It would have been less confusing in retrospect if the "load library as..." functionality were split into another function like Load­File­As­Pseudo­Module. Okay, that's a pretty awful name, but that's not the point. The point is to put the functionality in some function that doesn't have the word library in its name.

    Okay, so now that we see that these pseudo-modules aren't true modules, and they don't participate in any reindeer module games. So what use are they?

    Basically, the only thing you can do with a pseudo-module is access its resources with functions like Find­Resource, Load­Resource, and Enum­Resource­Types. Note that this indirectly includes functions like Load­String, and Format­Message which access resources behind the scenes.

    So maybe a better name for the function would have been Load­File­For­Resources, since that's all the pseudo-module is good for.

  • The Old New Thing

    Distinguishing between normative and positive statements to help people answer your question

    • 27 Comments

    Often, we get questions from a customer that use the word should in an ambiguous way:

    Our program creates a widget whose flux capacitor should have reverse polarity. Attached is a sample program that shows how we create the widget with Create­Widget. However, the resulting widget still has a flux capacitor with standard polarity. Can you help us?

    The phrase should have reverse polarity is ambiguous. The question could be

    We would like to create a widget whose flux capacitor has reverse polarity. Attached is a sample program that shows how to create a widget whose flux capacitor has standard polarity. How should we modify it in order to get reverse polarity?

    Or the question might be

    We would like to create a widget whose flux capacitor has reverse polarity. Attached is a sample program that attempts to do so, but the resulting widget has a flux capacitor with standard polarity. The polarity flag appears to be ignored. Are are we doing something wrong, or is this a bug in Windows?

    The first is a normative statement: "This is what we would like to happen." The second is a positive statement: "This is what is happening."

    The distinction is important because the two types of statements require very different types of responses. If have a program that does X, and you want to change it to do Y, then you're asking for help working through the Y feature, clarifying the documentation, informing you which flags you need to pass, and so on. But if you have a program that tries to do Y and fails, then you're asking for help debugging your code and possibly identifying a bug in the operating system.

    Being clear with your request means that you can avoid wasting a lot of time when the wrong set of people are called in to help you out.

    Here's another example of vague use of the word should:

    We're trying to do XYZ. We've been told that it is blocked for security reasons, but there should be a way to do this.

    In this case, it is not clear what the customer means by the phrase should be a way to do this. It could be

    We're trying to do XYZ. We've been told that it is blocked for security reasons, but we think that Windows should be changed to allow our scenario. How can we file a change request with the Windows security team to make an exception for us?

    Or the customer might be trying to say

    We're trying to do XYZ. We've been told that it is blocked for security reasons, but we think that there is a way to get the effect of XYZ without triggering the security issue. Can you help us find it?

    Note that in both cases, the customer either failed to asked a question or made some statements and asked for nonspecific advice, which is effectly the same as not asking a question. If they had remembered to ask a question, then that question would have clarified what they intended by the word should.

    Bonus chatter: A physicist classmate of mine got a chuckle out of the phrase flux capacitor because it combines two physics terms in an impressive-sounding but mostly nonsensical way.

    A capacitor is a device which stores electric potential. In the hydraulic analogy of electricity, a capacitor is a rubber diaphragm that separates two parts of a pipe, but which "stores" water flow by stretching and "discharges" the water flow by returning to its rest position.

    Flux is cross-sectional flow per unit time. Water flux is volumetric flow rate (liters per second per square meter): it measures how vigorously the water flows across a boundary. Magnetic flux measures the strength of a magnetic field.

    The combination is nonsensical because the units don't match. A capacitor stores potential, whereas flux is measured in current or magnetic field strength. But if you generalize the term capacitor to mean "a thing that stores stuff", then a flux capacitor is a device which stores a magnetic field.

    Such devices already exist today. They are called magnets.

  • The Old New Thing

    File version information does not appear in the property sheet for some files

    • 25 Comments

    A customer reported that file version information does not appear on the Details page of the property sheet which appears when you right-click the file and select Properties. They reported that the problem began in Windows 7.

    The reason that the file version information was not appearing is that the file's extension was .xyz. Older versions of Windows attempted to extract file version information for all files regardless of type. I believe it was Windows Vista that changed this behavior and extracted version information only for known file types for Win32 modules, specifically .cpl, .dll, .exe, .ocx, .rll, and .sys. If the file's extension is not on the list above, then the shell will not sniff for version information.

    If you want to register a file type as eligible for file version extraction, you can add the following registry key:

    HKEY_LOCAL_MACHINE
     \Software
      \Microsoft
        \Windows
          \CurrentVersion
            \PropertySystem
              \PropertyHandlers
                \.XYZ
                 (Default) = REG_SZ:"{66742402-F9B9-11D1-A202-0000F81FEDEE}"
    

    (Thanks in advance for complaining about this change in behavior. This always happens whenever I post in the Tips/Support category about how to deal with a bad situation. Maybe I should stop trying to explain how to deal with bad situations.)

  • The Old New Thing

    How do I enumerate drives the same way that the NET USE command does?

    • 14 Comments

    If you use the Remote Desktop Connection client to connect to another computer, you have the option of making your local drives available to the remote computer.

    A customer wanted to know how to enumerate all the drives on the local machine. The were able to get the volumes mapped to drive letters, but they also wanted to get the redirected drives injected by Terminal Services. (Mind you, these aren't volumes that are assigned drive letters, so it's not clear why they are interested in them, but whatever.)

    With the NET USE command, they see the Terminal Services volumes in Explorer, and they can be browsed via \\tsclient\d:

    Status       Local     Remote                    Network
    
    -------------------------------------------------------------------------------
                           \\TSCLIENT\D              Microsoft Terminal Services
    The command completed successfully.
    

    The customer wanted to enumerate these Terminal Services client volumes. "How does the NET USE command enumerate these volumes?"

    Let's write that program. Remember, Little Programs do little to no error checking.

    #define UNICODE
    #define _UNICODE
    #include <windows.h>
    #include <winnetwk.h>
    #include <stdio.h>
    
    void report(PCWSTR pszLabel, PCWSTR pszValue)
    {
     printf("%ls = %ls\n", pszLabel, pszValue ? pszValue : L"(null)");
    }
    
    int __cdecl main(int, char **)
    {
     HANDLE hEnum;
     WNetOpenEnum(RESOURCE_CONNECTED,
                  RESOURCETYPE_DISK,
                  0,
                  NULL,
                  &hEnum);
    
     DWORD cbBuffer = 65536;
     void *buffer = LocalAlloc(LMEM_FIXED, cbBuffer);
     LPNETRESOURCE pnr = (LPNETRESOURCE)buffer;
    
     DWORD err;
     do {
      DWORD cEntries = INFINITE;
      DWORD cb = cbBuffer;
      err = WNetEnumResource(hEnum, &cEntries, buffer, &cb);
      if (err == NO_ERROR || err == ERROR_MORE_DATA) {
       for (DWORD i = 0; i < cEntries; i++) {
        report(L"localName", pnr[i].lpLocalName);
        report(L"remoteName", pnr[i].lpRemoteName);
        report(L"provider", pnr[i].lpProvider);
        printf("\n");
       }
      }
     } while (err == ERROR_MORE_DATA);
    
     LocalFree(buffer);
     WNetCloseEnum(hEnum);
     return 0;
    }
    

    We open an enumeration for connected disks and then start enumerating out of it. The usage pattern for WNet­Enum­Resources is kind of messy, with a bunch of in/out parameters that need to get reset each time. Each time, we say "Enumerate as much as you can into this 64KB buffer" and then print what we got. If we were told, "There's still more," then we go back and ask for more.

    That's all. Nothing particularly fancy.

  • The Old New Thing

    How to view the stack of a user-mode thread when its kernel stack has been paged out

    • 4 Comments

    Suppose you have a machine that has crashed, and your investigation shows that the reason is that there is a critical section that everybody is waiting for. While waiting for that critical section, work piles up, and eventually the machine keels over. Suppose further that this crash is given to you in the form of a kernel debugger.

    In case it wasn't obvious, by "you" I mean "me".

    Okay, so the critical section that is the cause of the logjam is this one:

    1: kd> !cs CONTOSO!g_csDataLock
    -----------------------------------------
    Critical section   = 0x00007ff7f0ed2f68 (CONTOSO!g_csDataLock+0x0)
    DebugInfo          = 0x0000000022f2efd0
    LOCKED
    LockCount          = 0x5D
    WaiterWoken        = No
    OwningThread       = 0x0000000000004228
    RecursionCount     = 0x1
    LockSemaphore      = 0x17A0
    SpinCount          = 0x00000000020007cb
    

    "Great," you say. "I just need to look at thread 0x4228 to see why it is stuck.

    1: kd> !process -1 4
    PROCESS ffffe000047ae900
        SessionId: 1  Cid: 0604    Peb: 7ff74ecfa000  ParentCid: 05cc
        DirBase: 0eb07000  ObjectTable: ffffc000014c5680  HandleCount: 7003.
        Image: contoso.exe
    
            ...
            THREAD ffffe0000c136080  Cid 0604.4228  Teb: 00007ff74e94c000 Win32Thread: fffff90144edea60 WAIT
            ...
    

    Woo-hoo, there's the thread. Now I just need to switch to its context to see what it is stuck on.

    1: kd> .thread ffffe0000c136080
    Can't retrieve thread context, Win32 error 0n30
    

    Okay, that didn't work out too well. Now what?

    Even though the kernel stack is paged out, the user-mode stack may still be available.

    1: kd> !thread ffffe0000c136080
    THREAD ffffe0000c136080  Cid 0604.4228  Teb: 00007ff74e94c000
           Win32Thread: fffff90144edea60 WAIT: (UserRequest) UserMode Non-Alertable
        ffffe000077a7830  NotificationEvent
    Not impersonating
    DeviceMap                 ffffc00000e89c80
    Owning Process            ffffe000047ae900       Image:         contoso.exe
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      12735890       Ticks: 328715 (0:01:25:36.171)
    Context Switch Count      75             IdealProcessor: 2
    UserTime                  00:00:00.000
    KernelTime                00:00:00.031
    Kernel stack not resident.
    

    The limits of the user-mode stack are kept in the Teb.

    2: kd> !teb 00007ff74e94c000
    TEB at 00007ff74e94c000
        ExceptionList:        0000000000000000
        StackBase:            00000001027d0000
        StackLimit:           00000001027c2000
        SubSystemTib:         0000000000000000
        FiberData:            0000000000001e00
        ArbitraryUserPointer: 0000000000000000
        Self:                 00007ff74e94c000
        EnvironmentPointer:   0000000000000000
        ClientId:             0000000000000604 . 0000000000004228
        RpcHandle:            0000000000000000
        Tls Storage:          000000010132dfc0
        PEB Address:          00007ff74ecfa000
        LastErrorValue:       1008
        LastStatusValue:      c0000034
        Count Owned Locks:    2
        HardErrorMode:        0
    

    We now use the trick we learned some time ago where we grovel the stack of a thread without knowing where its stack pointer is.

    In this case, the groveling is made easier because we already know that everybody is waiting on the data lock. The data lock is taken in only two functions, so it's a matter ot looking for any occurrences of one of those two functions. And here it is: Data­Wrapper::Verify­Data.

    00000001`027cef88  00007ffe`f1bbac9a NTDLL!NtWaitForSingleObject+0xa
    00000001`027cef90  00000000`00006ac8
    00000001`027cef98  00007ffe`ef1bd085 KERNELBASE!WaitForSingleObjectEx+0xa5
    00000001`027cefa0  00000000`00006ac8
    00000001`027cefa8  00000000`00000000
    00000001`027cefb0  00000000`026d0000
    00000001`027cefb8  00000000`00000000
    00000001`027cefc0  00000001`01739ee0
    00000001`027cefc8  00000000`00000001
    00000001`027cefd0  00000000`00000048
    00000001`027cefd8  00000001`00000001
    00000001`027cefe0  00000000`00000000
    00000001`027cefe8  00000000`00000000
    00000001`027ceff0  00000000`00000000
    00000001`027ceff8  00000000`00000000
    00000001`027cf000  00000000`00000000
    00000001`027cf008  00000000`00000000
    00000001`027cf010  00000000`00000000
    00000001`027cf018  00007ff7`eefe6540 FABRIKAM!Lock::IsInitialized+0xfc
    00000001`027cf020  00000000`00000000
    00000001`027cf028  00000000`3b803fa0
    00000001`027cf030  00000000`00006ac8
    00000001`027cf038  00007ff7`eeff3c43 FABRIKAM!AccessRequest::WaitTimeout+0x87
    00000001`027cf040  00000000`ffffffff
    00000001`027cf048  10f513ec`6a161fb8
    00000001`027cf050  00000000`00000000
    00000001`027cf058  00000000`00006ac8
    00000001`027cf060  00000000`3b803fb8
    00000001`027cf068  00007ff7`eeff3cc0 FABRIKAM!AccessRequest::Wait+0x18
    00000001`027cf070  00000000`8007000e
    00000001`027cf078  00000000`fd416ff0
    00000001`027cf080  4fe80c4a`51236583
    00000001`027cf088  00000000`3b803fb8
    00000001`027cf090  00000000`ffffffff
    00000001`027cf098  00000000`00000000
    00000001`027cf0a0  00000000`3b803fa0
    00000001`027cf0a8  00007ff7`da8375b7 FABRIKAM!DataAccess::RequestAccess+0x93
    00000001`027cf0b0  00000000`3b803fa0
    00000001`027cf0b8  00000000`3b803fb8
    00000001`027cf0c0  4fe80c4a`51236583
    00000001`027cf0c8  10f513ec`6a161fb8
    00000001`027cf0d0  00000000`00000076
    00000001`027cf0d8  00007ff7`f07a8619 CONTOSO!Widget::SetColor+0x9
    00000001`027cf0e0  000095dc`90897985
    00000001`027cf0e8  00000000`00000110
    00000001`027cf0f0  00000000`00000000
    00000001`027cf0f8  00000000`00000000
    00000001`027cf100  00007ff7`4dd8000c
    00000001`027cf108  00007ff7`f07a85e5 CONTOSO!Widget::UpdateColor+0x39
    00000001`027cf110  00000000`ffc03f90
    00000001`027cf118  00000001`027cf1e8
    00000001`027cf120  00000000`ffc03fc8
    00000001`027cf128  00000000`00000000
    00000001`027cf130  00000000`00000000
    00000001`027cf138  00007ff7`4dd8000c
    00000001`027cf140  00007ff7`da5bc420 FABRIKAM!DataAccess::`vftable'+0x18
    00000001`027cf148  00000000`ffc03f90
    00000001`027cf150  00000001`027cf260
    00000001`027cf158  00007ff7`f0b3459c CONTOSO!DataWrapper::VerifyData+0x428
    00000001`027cf160  00000000`3b803fa0
    00000001`027cf168  00007ff7`f0ed1d30 CONTOSO!g_DataManager
    00000001`027cf170  00000000`fe196f10
    00000001`027cf188  00000000`00000001
    00000001`027cf190  00000000`00000000
    00000001`027cf198  00000001`027cf270
    00000001`027cf1a0  00000001`027cf480
    00000001`027cf1a8  00007ff7`00000001
    00000001`027cf1b0  00000001`027cf220 
    00000001`027cf1b8  00000000`00000000 
    00000001`027cf1c0  00000000`00000000 
    

    I left the red herrings in place just to make things a little more interesting.

    The Data­Wrapper::Verify­Data method enters the critical section and then calls Data­Access::Request­Access via a virtual method call:

    00007ff7`f0b3458f mov     dword ptr [rsp+28h],eax
    00007ff7`f0b34593 mov     eax,dword ptr [rsi+10h]
    00007ff7`f0b34596 mov     dword ptr [rsp+20h],eax
    00007ff7`f0b3459a call    qword ptr [rdi] ←
    

    Let's disassemble the start of Data­Access::Request­Access to see how it sets up its stack. This will help us interpret the other values in the stack dump.

    0: kd> u 00007ff7`da8375b7-93
    FABRIKAM!DataAccess::RequestAccess
    ;; function prologue
    00007ff7`da837524 mov     rax,rsp
    00007ff7`da837527 mov     qword ptr [rax+8],rbx
    00007ff7`da83752b mov     qword ptr [rax+20h],r9
    00007ff7`da83752f mov     qword ptr [rax+18h],r8
    00007ff7`da837533 mov     qword ptr [rax+10h],rdx
    00007ff7`da837537 push    rbp
    00007ff7`da837538 push    rsi
    00007ff7`da837539 push    rdi
    00007ff7`da83753a push    r12
    00007ff7`da83753c push    r13
    00007ff7`da83753e push    r14
    00007ff7`da837540 push    r15
    00007ff7`da837542 sub     rsp,70h
    ;; function body
    00007ff7`da837546 lea     rsi,[rcx+18h]
    00007ff7`da83754a mov     rdi,rcx
    00007ff7`da83754d mov     rbp,r9
    00007ff7`da837550 mov     rcx,rsi
    00007ff7`da837553 call    qword ptr [FABRIKAM!_imp_EnterCriticalSection]
    00007ff7`da837559 xor     r13b,r13b
    00007ff7`da83755c mov     ebx,8007000Eh
    ...
    00007ffe`da8375b1 call    FABRIKAM!AccessRequest::Wait
    

    We can replay the above code in our head and annotate the stack trace accordingly. On entry to the function, the stack pointer is 00000001`027cf158 (the return address). The function stashes some registers in the caller-provided spill area and it pushes others onto the stack, and then it subtracts some space for local variables as well as for outbound parameters of functions it intends to call.

    /-00000001`027cf0b0  00000000`3b803fa0
    | 00000001`027cf0b8  00000000`3b803fb8
    | 00000001`027cf0c0  4fe80c4a`51236583
    | 00000001`027cf0c8  10f513ec`6a161fb8
    | 00000001`027cf0d0  00000000`00000076
    | 00000001`027cf0d8  00007ff7`f07a8619 CONTOSO!Widget::SetColor+0x9
    | 00000001`027cf0e0  000095dc`90897985
    | 00000001`027cf0e8  00000000`00000110
    | 00000001`027cf0f0  00000000`00000000
    | 00000001`027cf0f8  00000000`00000000
    | 00000001`027cf100  00007ff7`4dd8000c
    | 00000001`027cf108  00007ff7`f07a85e5 CONTOSO!Widget::UpdateColor+0x39
    | 00000001`027cf110  00000000`ffc03f90
    \-00000001`027cf118  00000001`027cf1e8
      00000001`027cf120  00000000`ffc03fc8 // VerifyData's r15
      00000001`027cf128  00000000`00000000 // VerifyData's r14
      00000001`027cf130  00000000`00000000 // VerifyData's r13
      00000001`027cf138  00007ff7`4dd8000c // VerifyData's r12
      00000001`027cf140  00007ff7`da5bc420 FABRIKAM!DataAccess::`vftable'+0x18 // VerifyData's rdi
      00000001`027cf148  00000000`ffc03f90 // VerifyData's rsi
      00000001`027cf150  00000001`027cf260 // VerifyData's rbp
      00000001`027cf158  00007ff7`f0b3459c CONTOSO!DataWrapper::VerifyData+0x428 ← ESP is here
      00000001`027cf160  00000000`3b803fa0 // VerifyData's rbx
      00000001`027cf168  00007ff7`f0ed1d30 CONTOSO!g_DataManager // VerifyData's rdx
      00000001`027cf170  00000000`fe196f10 // VerifyData's r8
      00000001`027cf188  00000000`00000001 // VerifyData's r9
      00000001`027cf190  00000000`00000000
      00000001`027cf198  00000001`027cf270
      00000001`027cf1a0  00000001`027cf480
      00000001`027cf1a8  00007ff7`00000001
      00000001`027cf1b0  00000001`027cf220 
      00000001`027cf1b8  00000000`00000000 
      00000001`027cf1c0  00000000`00000000 
    

    The region marked in brackets is the 0x70 bytes of space for local variables and outbound parameters. Notice that some red herring function pointers are in that space. Those are probably variables that haven't been initialized yet, and the memory happened previously to have been used to hold some return addresses.

    A reassuring observation is that the rdx coming from Verify­Data is the address of CONTOSO!g_Data­Manager. That is the second function parameter (or first, if you aren't counting the hidden this) to Request­Access.

    Another reassuring observation is that that Verify­Data's rdi points into the vtable for Data­Access, since that matches the code we saw at the call point: call qword ptr [rdi].

    The mov rdi, rcx instruction in the function body tells us that the function stashed its this pointer in rdi. That's good info to keep track of, because that will let us look at the Data­Access object once we figure out what is in rdi.

    The next function on the stack is Access­Request::Wait.

    FABRIKAM!AccessRequest::Wait:
    00007ff7`eeff3ca8 sub     rsp,38h
    00007ffe`eeff3cb3 mov     dword ptr [rsp+20h],0FFFFFFFFh
    00007ffe`eeff3cbb call    FABRIKAM!AccessRequest::WaitTimeout
    00007ffe`eeff3cc0 add     rsp,38h
    00007ffe`eeff3cc4 ret
    

    This function doesn't bother saving any registers; it just reserves space for local variables and outbound parameters. From inspection, you can see that this is a simple wrapper that passes all its parameters onward to Wait­Timeout, with an INFINITE tacked onto the end, so this function has no local variables at all. Everything is just for outbound parameters.

    We can annotate some more entries in our stack trace.

    00000001`027cf070  00000000`8007000e // spill space for WaitTimeout
    00000001`027cf078  00000000`fd416ff0 // spill space for WaitTimeout
    00000001`027cf080  4fe80c4a`51236583 // spill space for WaitTimeout
    00000001`027cf088  00000000`3b803fb8 // spill space for WaitTimeout
    00000001`027cf090  00000000`ffffffff // INFINITE parameter
    00000001`027cf098  00000000`00000000 // unused
    00000001`027cf0a0  00000000`3b803fa0 // unused
    00000001`027cf0a8  00007ff7`da8375b7 FABRIKAM!DataAccess::RequestAccess+0x93
    

    The next function on the list is Access­Request::Wait­Timeout.

    FABRIKAM!AccessRequest::WaitTimeout:
    00007ff7`eeff3bbc mov     qword ptr [rsp+8],rbx
    00007ff7`eeff3bc1 mov     qword ptr [rsp+10h],rbp
    00007ff7`eeff3bc6 push    rsi
    00007ff7`eeff3bc7 sub     rsp,20h
    00007ff7`eeff3bcb mov     ebx,edx
    00007ff7`eeff3bcd mov     rsi,rcx
    00007ff7`eeff3bd0 mov     edx,0Bh
    00007ff7`eeff3bd5 mov     rcx,r8
    

    This function stashes two registers in the parameter spill space, pushes one onto the stack, and reserves another 0x20 bytes for local use (outbound parameters).

    00000001`027cf040  00000000`ffffffff // spill space for WaitForSingleObjectEx
    00000001`027cf048  10f513ec`6a161fb8 // spill space for WaitForSingleObjectEx
    00000001`027cf050  00000000`00000000 // spill space for WaitForSingleObjectEx
    00000001`027cf058  00000000`00006ac8 // spill space for WaitForSingleObjectEx
    00000001`027cf060  00000000`3b803fb8 // Wait's rsi
    00000001`027cf068  00007ff7`eeff3cc0 FABRIKAM!AccessRequest::Wait+0x18
    00000001`027cf070  00000000`8007000e // Wait's rbx
    00000001`027cf078  00000000`fd416ff0
    00000001`027cf080  4fe80c4a`51236583
    00000001`027cf088  00000000`3b803fb8
    

    Notice that the stashed rbx value is 8007000E, which conveniently lines up with the mov ebx,8007000Eh instruction in Data­Access::Request­Access. That's a bit reassuring, since it's another sign that we're on the right track.

    Next up is Wait­For­Single­Object­Ex.

    0: kd> u 00007ffe`ef1bd085 -a5
    KERNELBASE!WaitForSingleObjectEx
    00007ffe`ef1bcfe0 mov     r11,rsp
    00007ffe`ef1bcfe3 mov     qword ptr [r11+8],rbx
    00007ffe`ef1bcfe7 mov     dword ptr [r11+18h],r8d
    00007ffe`ef1bcfeb push    rsi
    00007ffe`ef1bcfec push    rdi
    00007ffe`ef1bcfed push    r14
    00007ffe`ef1bcfef sub     rsp,80h
    00007ffe`ef1bcff6 mov     ebx,r8d
    

    Incorporating this prologue into our stack annotation yields

    /-00000001`027cefa0  00000000`00006ac8 // spill space for NtWaitForSingleObject
    | 00000001`027cefa8  00000000`00000000 // spill space for NtWaitForSingleObject
    | 00000001`027cefb0  00000000`026d0000 // spill space for NtWaitForSingleObject
    | 00000001`027cefb8  00000000`00000000 // spill space for NtWaitForSingleObject
    | 00000001`027cefc0  00000001`01739ee0
    | 00000001`027cefc8  00000000`00000001
    | 00000001`027cefd0  00000000`00000048
    | 00000001`027cefd8  00000001`00000001
    | 00000001`027cefe0  00000000`00000000
    | 00000001`027cefe8  00000000`00000000
    | 00000001`027ceff0  00000000`00000000
    | 00000001`027ceff8  00000000`00000000
    | 00000001`027cf000  00000000`00000000
    | 00000001`027cf008  00000000`00000000
    | 00000001`027cf010  00000000`00000000
    \-00000001`027cf018  00007ff7`eefe6540 FABRIKAM!Lock::IsInitialized+0xfc
      00000001`027cf020  00000000`00000000 // WaitTimeout's r14
      00000001`027cf028  00000000`3b803fa0 // WaitTimeout's rdi
      00000001`027cf030  00000000`00006ac8 // WaitTimeout's rsi
      00000001`027cf038  00007ff7`eeff3c43 FABRIKAM!AccessRequest::WaitTimeout+0x87
      00000001`027cf040  00000000`ffffffff // WaitTimeout's rbx
      00000001`027cf048  10f513ec`6a161fb8
      00000001`027cf050  00000000`00000000 // WaitTimeout's r8
      00000001`027cf058  00000000`00006ac8
    

    Ooh, another red herring function pointer got caught in the local variables.

    Putting everything together results in the following annotated stack, with red herrings removed.

    00000001`027cef88  00007ffe`f1bbac9a NTDLL!NtWaitForSingleObject+0xa
    00000001`027cef90  00000000`00006ac8
    00000001`027cef98  00007ffe`ef1bd085 KERNELBASE!WaitForSingleObjectEx+0xa5
    00000001`027cefa0  00000000`00006ac8
    00000001`027cefa8  00000000`00000000
    00000001`027cefb0  00000000`026d0000
    00000001`027cefb8  00000000`00000000
    00000001`027cefc0  00000001`01739ee0
    00000001`027cefc8  00000000`00000001
    00000001`027cefd0  00000000`00000048
    00000001`027cefd8  00000001`00000001
    00000001`027cefe0  00000000`00000000
    00000001`027cefe8  00000000`00000000
    00000001`027ceff0  00000000`00000000
    00000001`027ceff8  00000000`00000000
    00000001`027cf000  00000000`00000000
    00000001`027cf008  00000000`00000000
    00000001`027cf010  00000000`00000000
    00000001`027cf018  00007ff7`eefe6540
    00000001`027cf020  00000000`00000000 // WaitTimeout's r14
    00000001`027cf028  00000000`3b803fa0 // WaitTimeout's rdi
    00000001`027cf030  00000000`00006ac8 // WaitTimeout's rsi
    00000001`027cf038  00007ff7`eeff3c43 FABRIKAM!AccessRequest::WaitTimeout+0x87
    00000001`027cf040  00000000`ffffffff // WaitTimeout's rbx
    00000001`027cf048  10f513ec`6a161fb8
    00000001`027cf050  00000000`00000000 // WaitTimeout's r8
    00000001`027cf058  00000000`00006ac8
    00000001`027cf060  00000000`3b803fb8 // Wait's rsi
    00000001`027cf068  00007ff7`eeff3cc0 FABRIKAM!AccessRequest::Wait+0x18
    00000001`027cf070  00000000`8007000e // Wait's rbx
    00000001`027cf078  00000000`fd416ff0
    00000001`027cf080  4fe80c4a`51236583
    00000001`027cf088  00000000`3b803fb8
    00000001`027cf090  00000000`ffffffff // INFINITE parameter
    00000001`027cf098  00000000`00000000
    00000001`027cf0a0  00000000`3b803fa0
    00000001`027cf0a8  00007ff7`da8375b7 FABRIKAM!DataAccess::RequestAccess+0x93
    00000001`027cf0b0  00000000`3b803fa0
    00000001`027cf0b8  00000000`3b803fb8
    00000001`027cf0c0  4fe80c4a`51236583
    00000001`027cf0c8  10f513ec`6a161fb8
    00000001`027cf0d0  00000000`00000076
    00000001`027cf0d8  00007ff7`f07a8619
    00000001`027cf0e0  000095dc`90897985
    00000001`027cf0e8  00000000`00000110
    00000001`027cf0f0  00000000`00000000
    00000001`027cf0f8  00000000`00000000
    00000001`027cf100  00007ff7`4dd8000c
    00000001`027cf108  00007ff7`f07a85e5
    00000001`027cf110  00000000`ffc03f90
    00000001`027cf118  00000001`027cf1e8
    00000001`027cf120  00000000`ffc03fc8 // VerifyData's r15
    00000001`027cf128  00000000`00000000 // VerifyData's r14
    00000001`027cf130  00000000`00000000 // VerifyData's r13
    00000001`027cf138  00007ff7`4dd8000c // VerifyData's r12
    00000001`027cf140  00007ff7`da5bc420 FABRIKAM!DataAccess::`vftable'+0x18 // VerifyData's rdi
    00000001`027cf148  00000000`ffc03f90 // VerifyData's rsi
    00000001`027cf150  00000001`027cf260 // VerifyData's rbp
    00000001`027cf158  00007ff7`f0b3459c CONTOSO!DataWrapper::VerifyData+0x428
    00000001`027cf160  00000000`3b803fa0 // VerifyData's rbx
    00000001`027cf168  00007ff7`f0ed1d30 CONTOSO!g_DataManager // VerifyData's rdx
    00000001`027cf170  00000000`fe196f10 // VerifyData's r8
    00000001`027cf188  00000000`00000001 // VerifyData's r9
    00000001`027cf190  00000000`00000000
    00000001`027cf198  00000001`027cf270
    00000001`027cf1a0  00000001`027cf480
    00000001`027cf1a8  00007ff7`00000001
    00000001`027cf1b0  00000001`027cf220 
    00000001`027cf1b8  00000000`00000000 
    00000001`027cf1c0  00000000`00000000 
    

    From this, we can also suck out the this pointer passed to Data­Access::Request­Access. We saw that it was stashed in rdi. The Wait function doesn't use rdi (because if it did, it would have saved the old value), so its rdi is the same as Request­Access's rdi. Similarly, the Wait­Timeout function does not use rdi. Therefore, when Wait­For­Single­Object saves the rdi register, it is saving the value from Data­Access::Request­Access.

    00000001`027cf028  00000000`3b803fa0 // WaitTimeout DataAccess's rdi
    

    And that is the this pointer that lets us study the Data­Access object to figure out why its access request is not completing.

  • The Old New Thing

    When does GetTickCount consider the system to have started?

    • 45 Comments

    The Get­Tick­Count and Get­Tick­Count­64 functions return "the number of milliseconds that have elapsed since the system was started." (The 32-bit version wraps around after around 50 days.) But when exactly is the system considered to have started? Is it when power is applied to the computer? When the BIOS completes POST? When the user picks the operating system from the boot menu? When the kernel switches to protected mode?

    It isn't defined exactly when the timer starts. Because that's not its purpose.

    The purpose of Get­Tick­Count is to let you measure intervals of time. It provides a common clock source so that multiple components can coordinate their actions. It also allows you to retrieve the tick count at one point, then retrieve the tick count at another point, subtract them, and conclude how much time has elapsed between those two points. The absolute value of the tick count is not meaningful. The only way to extract meaning from it is to subtract it from another tick count to get the delta.

    In fact, on debugging builds of Windows, the kernel artificially sets the Get­Tick­Count counter to "one hour before 32-bit timer tick rollover"; it effectively backdates the boot time by around 50 days. This is done to help identify bugs related to timer tick rollover.

    If your goal is to measure operating system boot time from the application of power to the computer, then Get­Tick­Count is not going to be useful. After all, Windows isn't even running at the moment you apply power to the computer. The BIOS does its work without any operating system all, so Windows has no idea how long the BIOS took to POST. The text in MSDN could be a bit more explicit and say "elapsed since Windows started", or it could be pointlessly nitpicky and say "elapsed since the Windows HAL initialized the programmable interval timer."

    Better would be if it simply described how the timer is intended to be used. "Get­Tick­Count returns a value which increases at a rate of 1000 per second." Perhaps with some clarifying text: "By convention, the zero point of the Get­Tick­Count counter is the approximate time the system booted. Note, however, that this convention is violated on occasion (such as on a checked build of Windows), so applications should not ascribe any meaning to the zero point of the tick counter."

    If you want to know how much time elapsed since the application of power, you need to use a stopwatch.

  • The Old New Thing

    How can I wait until all startup applications have finished launching?

    • 27 Comments

    A customer wanted to know how to detect that all startup applications have finished launching. They wanted to wait until everything settled down before proceeding with some task.

    What if two programs did this?

    Suppose two programs both wanted to detect that all startup applications have finished launching. Each one would sit there waiting for the other, because this is one of those I want to be the last to do something, even laster than the other guy who wants to do something last things.

    If you want to wait until the system is idle to perform some task, you can use the Task Scheduler to create an idle-time task. This has the benefit of coordinating multiple idle-time tasks so that you don't have the phenomenon of two applications both trying to do things when they think nobody else is doing anything, resulting in an Alphonse-and-Gaston state: The system goes idle, and both Alphonse and Gaston start doing their idle-time tasks. But Alphonse notices that Gaston is busy and backs off; at the same time, Gaston notices that Alphonse is busy and backs off. Now the system is idle again, and both Alphonse and Gaston start their idle-time tasks, and the cycle continues. The computer gets into this weird state where it keeps waking up, going to sleep, waking up, going to sleep, and never actually making progress.

  • The Old New Thing

    A little cheat in my Tiger Beat photo homage

    • 9 Comments

    One thing nobody has called out in my tribute to the Bill Gates Tiger Beat photo, either because it was too subtle or too obvious, is that the photo is actually a mirror image.

    The arrangement of furniture in the room was not correct: The big table was on the wrong side of the room. It was also too heavy to move around, so we cheated. We staged the entire picture as a mirror image, flipping the Windows screen shot. And then back in the virtual darkroom, Ariel flipped the photo to put the furniture on the correct side of the photo.

    Here are the clues in the photo:

    • The SONY logo on the monitor.
    • The Multiscan G500 and Trinitron branding on the monitor.
    • The Microsoft logos on the binder on the table.
    • The arrows on the recycle bin propping open the door.

    Chatter: The day after I put this article into the queue (which makes it visible to Microsoft employees), somebody posted a comment pointing it out. Coincidence? You decide.

  • The Old New Thing

    Kicking around a function that formats stuff

    • 17 Comments

    Today's "Little Program" is really a "Little Puzzle" that got out of hand.

    This started out as a practical question: This code fragment screams out for some sort of simplification. (I've changed the names of the classes.)

    class FrogProperty
    {
     public string Name { get; private set; }
     public string Value { get; private set; }
     ...
    }
    
    class ToadProperty
    {
     public string Name { get; private set; }
     public string Value { get; private set; }
     ...
    }
    
    var frogStuff = new List<string>();
    foreach (var frogProp in FrogProperties) {
      frogStuff.Add(string.Format("{0}: {1}", frogProp.Name, frogProp.Value));
    }
    frogStuff.Sort();
    Munge(frogStuff);
    
    var toadStuff = new List<string>();
    foreach (var toadProp in ToadProperties) {
      toadStuff.Add(string.Format("{0} = {1}", toadProp.Name, toadProp.Value));
    }
    toadStuff.Sort();
    Munge(toadStuff);
    
    var catStuff = new List<string>();
    foreach (var cat in Cats) {
      catStuff.Add(string.Format("{0}", cat.Name));
    }
    catStuff.Sort();
    Munge(catStuff);
    
    var dogStuff = new List<string>();
    foreach (var dogProp in DogProperties) {
      dogStuff.Add(string.Format("{0} {1}", dogProp.Name, dogProp.Value));
    }
    dogStuff.Sort();
    Munge(dogStuff);
    
    ...
    

    Clearly, the pattern is

    var stuff = new List<string>();
    foreach (var thing in thingCollection) {
     stuff.Add(string.Format(formatstring, thing.Name, [optional: thing.Value]));
    }
    stuff.Sort();
    Munge(stuff);
    

    Everything here is pretty straightforward, except for the string.Format part. Can we write a function that takes a thing and formats it in a somewhat flexible manner?

    Let's start with the Name-and-Value cases. We might try something like this:

    public static string FormatNameValue<T>(this T t, string format)
    {
     return string.Format(format, t.Name, t.Value);
    }
    

    But then we'd run into trouble, because there is no constraint on T, so the compiler will complain, "I don't know how to get a Name or a Value from an object."

    And since Frog­Property and Toad­Property do not have a common base class, you're kind of stuck.

    One way out would be to use the new dynamic type:

    public static string FormatNameValue<T>(this T t, string format)
    {
     dynamic d = t;
     return string.Format(format, d.Name, d.Value);
    }
    

    But that won't work in the Name-only case:

    cat.FormatNameValue("{0}");
    

    The cat object has a Name but no Value. The attempt to read the Value will raise an exception (even though it is never consumed by the format).

    Maybe we can turn to reflection.

    public static string FormatNameValue<T>(this T t, string format)
    {
     return string.Format(format,
                          typeof(T).GetProperty("Name").GetValue(t, null),
                          typeof(T).GetProperty("Value").GetValue(t, null));
    }
    

    This still raises an exception if there is no Value, but we can detect the missing Value before we run into trouble with it.

    static object GetPropertyOrNull<T>(this T t, string prop)
    {
     var propInfo = typeof(T).GetProperty(prop);
     return propInfo == null ? null : propInfo.GetValue(t, null);
    }
    
    public static string FormatNameValue<T>(this T t, string format)
    {
     return string.Format(format,
                          t.GetPropertyOrNull("Name"),
                          t.GetPropertyOrNull("Value"));
    }
    

    Okay, now we're getting somewhere.

    But before getting to deep into this exercise, I should point out that another way to solve this problem is to turn it inside-out. Instead of making the munger understand all of the different objects, why not make each object understand munging?

    class FrogProperty : IFormattable
    {
     public string Name { get; private set; }
     public string Value { get; private set; }
     public override ToString(string format, IFormatProvider formatProvider)
     {
      switch (format) {
      case "Munge":
       return string.Format(formatProvider,"{0}: {1}", Name, Value);
      default:
       return ToString(); // use object.ToString();
      }
     }
    }
    
    class Cat : IFormattable
    {
     public string Name { get; private set; }
     public override ToString(string format, IFormatProvider formatProvider)
     {
      switch (format) {
      case "Munge":
       return string.Format(formatProvider,"{0}", Name);
      default:
       return ToString(); // use object.ToString();
      }
     }
    }
    

    The generic helper function would then be

    var stuff = new List<string>();
    foreach (var thing in thingCollection) {
     stuff.Add(string.Format("{0:Munge}", thing);
    }
    stuff.Sort();
    Munge(stuff);
    

    Okay, fine, rain on my little puzzle parade.

    Let's ignore this very useful advice and proceed ahead with our puzzle, because we're determined to see how far we can go, even if it's in the wrong direction.

    Now that we have Format­Name­Value, we might say, "What about generalizing to cases where we want properties other than Name and Value?" One design would be to pass in a format string and list of properties you want to fill in:

    thing.FormatProperties("{0}: {1} (modified by {2})",
                           "Name", "Value", "ModifiedBy");
    

    Our Format­Name­Value function would go something like this:

    public static string FormatProperties<T>(
        this T t, string format, params string[] props)
    {
     object[] values = new object[props.Length];
     for (var i = 0; i < props.Length; i++) {
      values[i] = typeof(T).GetProperty(props[i]).GetValue(t, null);
     }
     return string.Format(format, values);
    }
    

    This suffers from a problem common to most formatters: Once you get more than a few insertions, it becomes hard to figure out which one matches up to what. So I'm going to try something radical:

    static Regex identifier = new Regex(@"(?<={)(.*?)(?=[:}])");
    
    // pedants would use
    //identifier = new RegEx(@"[_\p{Lu}\p{Ll}\p{Lt}\p{Lm}\p{Lo}\p{Nl}]" +
    //       @"[_\p{Lu}\p{Ll}\p{Lt}\p{Lm}\p{Lo}\p{Nl}\d\p{Pc}\p{Mn}\p{Mc}]");
    
    public static string FormatProperties<T>(this T t, string format)
    {
      var values = new ArrayList();
      int count = 0;
      format = identifier.Replace(format, (m) => {
        values.Add(typeof(T).GetProperty(m.Value).GetValue(t, null));
        return (count++).ToString();
      });
      return string.Format(format, values.ToArray());
    }
    

    Instead of separating the properties from the format, I embed them in the format.

    thing.FormatProperties("{Name}: {Value} (modified by {ModifiedBy})");
    

    Note that I explicitly exclude colons from identifiers. That lets me do things like this:

    var result =
      (new System.IO.FileInfo(@"C:\Windows\Explorer.exe"))
        .FormatProperties("Created on {CreationTime:F} " +
                          "{Length} bytes in size");
    

    The property names are extracted and replaced with corresponding numbers, but the format string remains, allowing it to be used to alter the final formatting of the property.

    Okay, at this point I figured I had gone far enough. The fun had run out, so I decided to stop.

Page 1 of 436 (4,356 items) 12345»