Can’t use DirectX 11.2 with Visual Studio 2012 Graphics Diagnostics

If you’re one of the people using Graphics Diagnostics in Visual Studio 2012, please be aware that the DirectX 11.2 APIs are not supported.

This includes:

• ID3D11Device2
• ID3D11DeviceContext2
• HLSL Shader Linking
• Tiled resources

Basically anything listed here:

https://msdn.microsoft.com/en-us/library/windows/desktop/dn312084(v=vs.85).aspx

Now that doesn’t mean 11.1 and below doesn’t work. In fact we fixed a number of bugs in our second update to VS 2012, and as long as you use 11.1 interfaces (or below) everything else should work on Windows 8.1

Why doesn’t it work?

If you really care to know, I’m going to go into some detail on how we actually do graphics diagnostics. This will explain why 11.2 doesn’t work in VS 2012 and what we plan to do to address this in the future.

Capturing

When you hit print screen during graphics diagnostics, we “capture” a frame of your app running. You can see all the stuff your frame did in the event list here:

So how do we “capture” this list of events?

Detours

For VS 2012 (and previous incarnations of WinPIX), we use something called detours. Detours essentially rewrites the beginning of a function so that it jmps to somewhere else. Here’s the disassembly for D3D11CreateDevice:

This is what D3D11CreateDevice looks like before it is detoured.

After detouring, this assembly changes to this (notice the new jmp):

This means calls to D3D11CreateDevice go to another create device function.

However, this other function still needs to call the original, so detours also creates a “trampoline” function to allow us to call the original:

Here’s an example of a new detoured CreateDevice:

 HRESULT CHookedD3D11Top::D3D11CreateDevice( D3D11CREATEDEVICE_TYPED_PARAMS )
 {
     if ( D3D11CreateDevice_Trampoline == NULL )
         return E_NOINTERFACE;
  
     m_pHookMgr->LogInputParams( D3D11CREATEDEVICE_PARAMS );
      
     HRESULT hr = D3D11CreateDevice_Trampoline( D3D11CREATEDEVICE_PARAMS );
      
     m_pHookMgr->LogOutputParams( D3D11CREATEDEVICE_PARAMS, hr );
      
     return hr;
 }

Essentially we:

• Intercept the call
• Log the input parameters
• Make the original call
• Log the output parameters

This allows us to log/(and more importantly) recreate all the things your app did while running.

Sounds simple, so why not just add logging for the new 11.2 apis?

Hooked objects

It’s actually a bit more complicated than that. In the example above, the ID3D11Device returned from the function is likely going to be used later, to say create textures.

The workflow here is:

1. Game calls D3D11CreateDevice
2. We intercept in our detoured HookedD3D11CreateDevice
3. We call into the RealD3D11CreateDevice
4. It returns a real device object
5. Game calls the ID3D11Device::CreateTexture
6. We don’t capture this. Well because CreateTexture is called on something we aren’t detouring.

You can’t detour a virtual function (not with the detours api anyway, more on this later) so we need to do more than just detour the straight create functions. We also need to wrap objects as they are returned.

This gives us something like so:

Now we can log our HookedDevice::CreateTexture. It’s code would look something like so:

 HRESULT CHookedD3D11Device::CreateTexture2D( D3D11CREATEDTEXTURE2D_TYPED_PARAMS )
  {
      m_pHookMgr->LogInputParams( D3D11CREATEDTEXTURE2D_PARAMS );
      
      HRESULT hr = m_pRealDevice( D3D11CREATEDTEXTURE2D_PARAMS );
      
      m_pHookMgr->LogOutputParams( D3D11CREATEDTEXTURE2D_PARAMS, hr );
  
      if (hr == S_OK)
          *ppTexture = m_pHookMgr->WrapObject(*ppTexture);
      
      return hr;
  }

This is the general pattern we follow throughout our hooking.

Things get messy in some scenarios however.

QueryInterface

QueryInterface is a special function. This is the way a COM object is queried for new types.

HRESULT QueryInterface(

  [in]   REFIID riid,

  [out]  void **ppvObject

);

In our example, our HookedDevice has its own implementation of QueryInterface. What happens when we QI for some interface we don’t know about (like say ID3D11Device2)?

There are really two things we can do (well that I’ve thought of)

Off the farm

We could just return the original m_pRealDevice object. This means if you QI for ID3D11Device2, and then start calling functions on it, we won’t hook them and capture won’t succeed. This means your app will probably run under the graphics debugger, but we’ll just miss calls.

Imagine this workflow:

1. Create device
2. QI for ID3D11Device2
3. GetImmediateContext2
4. Call draw calls on ID3D11DeviceContext2

In step 2, the QI for ID3D11Device2 is not understood by VS 2012 (well because ID3D11Device2 didn't exist yet). So it returns the original object. In step 3, the immediate context returned is not our hooked immediate context, and in step 4, the draw calls are not hooked. This means the QI for ID3D11Device2 goes “off the farm” and we can’t see our sheep anymore.

This scenario actually came up during development of VS 2012. It seems D2D actually queries a device for internal only interfaces that it knows about. This means if you use D2D in your app, you wouldn’t have been able to capture with VS 2012. Pretty crummy.

VTable patching

This prompted us to actually not return hooked objects, but instead to return the original m_pRealDevice (from D3D11CreateDevice) and patch its vtable to point to our Hooked API calls. This is another form of detouring that allows the QI to go “off the farm” but come back onto the farm whenever a call is made to a known API.

It works something like so:

The real device starts out something like so:

Then we create a hooked device that looks like this:

Then we rewire the Original Device to point to the VTable of the HookedDevice and the hooked entries to call into the original as necessary:

This allows us to always return the “original” object, but have its functions still detoured, and when subsequent QI’s happen for unknown interfaces, we can patch just those vtables for the IUnknown vtable.

Additional difficulties

This seems to fix the capture with 11.2 problem. As long as the original vtable is patched, any new interfaces would at least work (if not capture). However there’s a problem here.

DirectX itself is patching vtables. The device context is patched at the vtable by DirectX to swap out feature levels. This breaks our vtable patching. So we can only returned hooked objects for device contexts.

Supporting 11.2 = VS 2013

Given all of this background, you may be able to see how much work it is to actually hook DirectX. Months of patching vtables here, making sure parameters are copied correctly, etc. Not to mention responding to changes as they come up. DirectX changed some of the 11.1 APIs 5 times during the release of VS 2012.

The ideal alternative would be for DirectX itself to support hooking DirectX APIs. This way DirectX could keep the hooks up to date as they made changes to their APIs and handle any private interfaces that might come up.

For VS 2013, this is what we are doing. From now on, DirectX detouring will be handled internally by DirectX itself. This eliminates the need for us to keep updating our hooking every time DirectX adds new APIs.

So given all of that, we’re not porting this change to VS 2012 for a couple of reasons:

1. It would take up months of development time we could be spending on new features.
2. The new version will be free (for store apps, and somewhat useable for desktop apps)

I’ll leave the details of the new DirectX hooking as a topic for a future post