The video renderer is the last filter in the video pipe, and it is responsible for displaying the output of upstream filters. The video renderer is just a controller for the underlying display driver, and does not do any processing on the image samples themselves.
The video renderer operates in two distinct modes:
When the graph is first connected, the video renderer always tries to connect using GDI, and for that, it will need a connection with an RGB media type that matches the display format of the primary monitor. Just when it goes into Paused mode, the video renderer will try to allocate surfaces using DirectDraw. This dual mode of operation was envisioned so as to always have a fall back plan in case DirectDraw surfaces were not available in some circumstances.
Choosing an Accelerated Media Type
When the video renderer goes into Paused mode, it is time to allocate the DirectDraw surfaces. The video renderer will do so by enumerating all media types of the upstream filter, and then trying to allocate a surface matching that media type. For instance, let's assume the upstream filter is the WMV DMO. It currently supports the following output media types (the preferred media type is the first one):
The video renderer will try to allocate flipping overlay surfaces first, then non-flipping surfaces:
In this way, if the upstream filter has optimized for certain YUV formats, it can control the choice of media type. In case the display driver can also provide a surface of that type, the accelerated media type is chosen. The whole process is driven by the upstream filter, with the display driver in a passive role.
Dynamic Format Changes from the Video Renderer
Of course, for an optimal pipe, we would like to always have overlay flipping surfaces available. Nevertheless, that may not be the case in some situations. For instance, depending on the display driver capabilities, the flipping overlay may just be available when the user is watching the video at its original size. When the user is stretching or shrinking, overlays might not be available. This is controlled by the DirectDraw hardware capabilities dwMinOverlayStretch and dwMaxOverlayStretch (see http://msdn2.microsoft.com/en-us/library/aa915204.aspx). So, if the display driver doesn't support overlay stretching, and the video renderer is currently using overlays, it will need to swap to GDI (and thus to RGB format), so that GDI will do the necessary scaling.
Note that every time the upstream filter requests a new buffer from the video renderer, the video renderer will try to return a DirectDraw buffer. If all the conditions to use the DirectDraw buffer are OK (clipping, stretching, video memory, etc.), then it will use it. Just in case one of the conditions fail it will resort to using GDI.
Debugging Video Renderer Connection Problems
We have seen some common connection problems when initially bringing up new decoder filters and/or capture drivers:
Analyzing the DirectShow logs
The first step in this case is to turn on the debug zones for the DirectShow DLL, quartz.dll, and observe the connection and video renderer messages.
Run your test scenario, and save the debug output. Look for the section that says "Filter Graph Dump", and verify which filters got inserted in the graph. Here's an example of a filter graph dump:
Filter graph dumpFilter 1a199a30 'Video Renderer' Iunknown 1a199a20 Pin 1a199f10 Input (Input) connected to 1a0e1880Filter 1a0e1200 'WMVideo & MPEG4 Decoder DMO' Iunknown 1a0e11f0 Pin 1a0e16e0 in0 (Input) connected to 1a0e0600 Pin 1a0e1880 out0 (PINDIR_OUTPUT) connected to 1a199f10 Pin 1a0e1a00 ~out1 (PINDIR_OUTPUT) connected to 0Filter 1a0ecc60 'ASF ICM Handler' Iunknown 1a0ecc50 Pin 1a0ecd70 In (Input) connected to 1a0aa3a0 Pin 1a0e0600 Out (PINDIR_OUTPUT) connected to 1a0e16e0Filter 1a0ec240 'Audio Renderer' Iunknown 1a0ec230wo: GetPin, 0 Pin 1a0ec4e0 Audio Input pin (rendered) (Input) connected to 1a0eb880Filter 1a0eb220 'WMAudio Decoder DMO' Iunknown 1a0eb210 Pin 1a0eb660 in0 (Input) connected to 1a0ea800 Pin 1a0eb880 out0 (PINDIR_OUTPUT) connected to 1a0ec4e0Filter 1a0e9380 'ASF ACM Handler' Iunknown 1a0e9370 Pin 1a0e9490 In (Input) connected to 1a0aa000 Pin 1a0ea800 Out (PINDIR_OUTPUT) connected to 1a0eb660Filter 1a0a2ae0 '\Hard Disk2\clips\wmv\0-1.asf' Iunknown 1a0a2ad0 Pin 1a0aa000 Stream 1 (PINDIR_OUTPUT) connected to 1a0e9490 Pin 1a0aa3a0 Stream 2 (PINDIR_OUTPUT) connected to 1a0ecd70End of filter graph dump
After that, verify which media type the video renderer is using when trying accelerated mode (and if it succeeded). Search for "Allocating video resources":
Allocating video resourcesInitialising DCI/DirectDrawSearching for direct formatEntering ReleaseSurfacesEntering HideOverlaySurfaceEnumerated 32315659Entering FindSurfaceEntering GetMediaTypeNot a RGB formatEntering CreateYUVFlippingEntering CheckCreateOverlayGWES Hook fails surface creation. IDirectDraw::CreateSurface fails.No surfaceEntering ReleaseSurfacesEntering HideOverlaySurfaceEnumerated 3231564eEntering FindSurfaceEntering GetMediaTypeNot a RGB formatEntering CreateYUVFlippingEntering CheckCreateOverlayGWES Hook fails surface creation. IDirectDraw::CreateSurface fails.No surfaceEntering ReleaseSurfacesEntering HideOverlaySurfaceEnumerated 32595559Entering FindSurfaceEntering GetMediaTypeNot a RGB formatEntering CreateYUVFlippingEntering CheckCreateOverlayEntering InitOverlaySurfaceEntering InitDrawFormatEntering InitDrawFormatEntering GetDefaultColorKeyReturning default colour keyEntering InitDefaultColourKeyEntering SetSurfaceSizePreparing source and destination rectanglesEntering ClipPrepareEntering InitialiseClipperEntering InitialiseColourKeyoverlay color key onColour keyNo paletteFound AMDDS_YUVFLP surfaceProposing output type M type MEDIATYPE_Video S type MEDIASUBTYPE_YUY2
Note in the above log that the video renderer tried to create surfaces in the order specified by the WMV DMO. For the display driver in use for the above log, it managed to create a YUY2 surface, the third option for the WMV decoder. The last section in this blog entry has more information about FourCC codes.
Here are some solutions for common connection problems we have faced in the past.
Color Space Converter is inserted in the graph
The number one problem is that the upstream filter doesn't report any RGB format, just YUV formats. If that's the case, the video renderer can't connect directly to the filter since it requires a matching RGB format. Usually, the color space converter will be inserted in the graph in these cases. We don't want this to happen, as it will imply a memory copy of each frame buffer, so we want to make sure the upstream filter does provide RGB formats.
Sometimes the color converted gets inserted in the graph even though the upstream filter does support the needed RGB format. This can happen because the upstream filter is requiring an alignment different than 1 when the allocator is being decided. Currently, the video renderer will just accept 1-byte alignments.
Another common reason for the color converter to be inserted in the graph is when the BITMAPINFOHEADER supplied by the upstream filter doesn't contain the bitmasks correctly at the end of BITMAPINFOHEADER that is passed when getting the output media types. Please make sure that the bitmasks are inserted correctly. For instance, for RGB565, we should have:
*pdwBitfield++ = 0xF800; // Red – 5
*pdwBitfield++ = 0x07E0; // Green - 6
*pdwBitfield = 0x001F; // Blue - 5
Graph doesn't connect at all
If the upstream filter just supports a subset of YUV formats, and none of these are recognized by the color space converter, then it won't be possible at all to connect the video renderer. Again, in this case the solution is for the upstream filter to provide RGB formats.
YUV Surfaces are not used, just GDI
Another common occurrence is for the upstream filters to provide allocators. If this is the case, the video renderer will be tied to not using DirectDraw (as it can't pass upstream memory buffers to DirectDraw). If we want the optimal overlay flipping path, the video renderer *needs* to be the allocator, so that it is possible for it to provide DirectDraw surfaces upstream.
Surface Types: Controlling Which Surfaces the Video Renderer Creates
There are ways to control which accelerated surfaces the video renderer is allowed or not to create that are useful when debugging the connection process, specially to reduce the number of options and the number of tries in the display driver. This is controlled via a registry key (see http://msdn2.microsoft.com/en-us/library/aa930626.aspx):
HKEY_LOCAL_MACHINE\Software\Microsoft\DirectX\DirectShow\Video Renderer\SurfaceTypes
The following table shows the AMDDS values for use with the SurfaceTypes named value.
AMDDS_NONE
0x00
No support for Device Control Interface (DCI) or DirectDraw.
AMDDS_DCIPS
0x01
Use DCI primary surface.
AMDDS_PS
0x02
Use DirectDraw primary surface.
AMDDS_RGBOVR
0x04
RGB overlay surfaces.
AMDDS_YUVOVR
0x08
YUV overlay surfaces.
AMDDS_RGBOFF
0x10
RGB off-screen surfaces.
AMDDS_YUVOFF
0x20
YUV off-screen surfaces.
AMDDS_RGBFLP
0x40
RGB flipping surfaces.
AMDDS_YUVFLP
0x80
YUV flipping surfaces.
AMDDS_ALL
0xFF
Use all available surfaces.
AMDDS_DEFAULT
AMDDS_YUV
0xA8
(AMDDS_YUVOFF | AMDDS_YUVOVR | AMDDS_YUVFLP)
AMDDS_RGB
0x58
(AMDDS_RGBOFF | AMDDS_RGBOVR | AMDDS_RGBFLP)
AMDDS_PRIMARY
0x03
(AMDDS_DCIPS | AMDDS_PS)
If you just want to enable YUV overlay flipping surfaces for debugging purposes, you should set the SurfaceTypes registry key to AMDDS_YUVFLP. Remember to turn on all surfaces back on after you finished debugging your problem...
About FourCC codes:
Note that in the example log we list the FOURCC codes that are being used in the line "Enumerated 32315659". Here's how to map this hex number into a character sequence that will help identify the code:
Enumerated 32315659
0x32 = '2', 0x31 = '1', 0x56 = 'V', 0x59 = 'Y' ===> 0x32315659 = YV12
Enumerated 3231564e
0x32 = '2', 0x31 = '1', 0x56 = 'V', 0x4e = 'N' ===> 0x3231564e = NV12
Enumerated 32595559
0x32 = '2', 0x59 = 'Y', 0x55 = 'U', 0x59 = 'Y' ===> 0x32595559 = YUY2
Enumerated 56555949
0x56 = 'V', 0x55 = 'U', 0x59 = 'Y', 0x49 = 'I' ===> 0x56555949 = IYUV
Enumerated 59565955
0x59 = 'Y', 0x56 = 'V', 0x59 = 'Y', 0x55 = 'U' ===> 0x59565955 = UYVY
Enumerated 55595659
0x55 = 'U', 0x59 = 'Y', 0x56 = 'V', 0x59 = 'Y' ===> 0x55595659 = YVYU
Also, the file in public\directx\sdk\inc\uuids.h contains several FOURCC media subtypes definitions:
Please do leave feedback, and let us know if this has been useful. Thanks,
Lucia