A while back I posted a blog entry detailing some of the steps that were glossed over in the MSDN documentation on how to install the media encoder for video recording.
The version of encoder that should be used changed in August when Expression 4 was released. The Windows Media Encoder is now obsolete for Test Manager video recordings. Expression Encoder 4 should be used.
Download details and installation instructions can be found at http://msdn.microsoft.com/en-us/library/dd997558.aspx and Mike Snell updated his blog post with some new screen shots: http://visualstudiounleashed.com/mikesnell/post/2010/04/22/Recording-Videos-with-Microsoft-Test-Professional-2010.aspx
Giles Davies has a nice post that highlights the additional configuration settings you can use with Expression Encoder 4.