So... I've been looking more closely at Wix and I think I'm going to build some custom actions to do a really good job installing shell extensions. Which ones do you find most problematic? I realize this is a bit of a change of pace for the blog, so I'll probably hit a few other forums, but please feel free to add comments/complaints about which extensions are problematic.
Maybe you've already done the work to install these properly, but in some cases that can be quite complicated and I could see a lot of folks just accepting some minor bugs and moving on.
Protocol Handlers (http://msdn2.microsoft.com/en-us/library/bb266527.aspx) and Schema Extensions (http://msdn2.microsoft.com/en-us/library/ms647576.aspx). Both of these require custom actions to be installed correctly - and there's no standard for it so everyone is probably doing it themselves which is going to lead to some really inconsistent behavior. None of that is going to help system stability at all.
Another one I came across recently was installing a shell folder that should appear on the desktop. I added it to the desktop namespace but until the user refreshes the desktop it isn't going to appear. You need to send a SHChangeNotify to get it to show up without user intervention. Anyway, there's no standard action for that. So unless you can accept a shortcut to the shell folder (which sends a notification), you'll want a CA there as well. And there again, you want to send as minimal a notification as possible to avoid having the system reload everything when all that you needed is a new icon to show up.
File Associations:This one is going to be a bit of a laundry list of issues, but I'll start off the first couple issues.
So there's a start - let me know as you come across other issues.