Stealing a page from Mark Russinovich's blog here.
I've been going batty trying to play ripped DVDs residing on my WHS box through my XBox 360. I'd done all the due research, and had followed all the various directions around making symbolic links on my MCE box, installing transcoder.dll, etc... I got the DVD Library to show up on the 360, but whenever I tried to play the file, I received the dreaded "video decoder error".
Having some free time on a Saturday, I decided to not come for air till I had it working.
My first step was to run Process Monitor on my local (Media Center) machine, and collect file activity during a request to play the DVD. I could see in the EHShell process where it successfully opened the .MPEG file on the local machine. However, the .MPEG file is just a link to the actual .VOB file on the WHS share. EHShell was failing to open the .VOB file on the server, even though I could see/copy the exact same file from a command prompt. The error for the CreateFile call was LOGON_FAILURE.
At that point, I realized that EHShell was running as the MCX2 account, rather than as me. OK then! that could explain why the file couldn't be accessed.
My first attempt to fix this was to go to the WHS console, and add "Read" access for the "Guest" account. Sadly, no joy. Restarting everything still resulted in the same LOGON_FAILURE.
Eventually, I poked around more in the WHS console, and noticed that the Guest account was disabled. That is, the "Shared Folders" view happily let me enable Guest access to folders, even though the Guest account wasn't active. Running through the "Enable Guest account" wizard got everything working, and I startled my wife with the whoop of joy I let out.
Maybe this should have been obvious to me sooner, but I can't recall seeing anything about this while hunting around, trying to figure out why it worked for everybody else, but not for me.
So, to summarize, if you're trying to play ripped DVDs from a WHS share using your XBox360 as a Media Center Extender, try making sure that the Guest account is enabled on your WHS setup.
Disclaimer: I'm not a Media Center Extender expert. I posted this info here to hopefully spare at least one person the pain I went through. :-)