We have been using Silverlight heavily in my current SP2010 project and here are the things you should know when using silverlight in SP2010 sites

• SL Webpart does not load in IE 64bit, be sure to browse the site in IE 32 bit mode
• SL Webpart did not load in Safari on Mac, Safari on Mac is supported but Safari on PC is not (unfortunately Safari on PC works fine)
• Don’t hard code the xap path in SL webpart. I learned this the hard way, always use relative path because if you extend the web application to a different zone and you are accessing the site through a different zone url. SL webpart will fail to load if you hard code full path.
• Deploying xap files, you have a choice to create a document library and upload xap. We chose to deploy to “ClientBin” directory under layouts as it was easy to do with VS 2010 and rich sharepoint support available in VS 2010
• WCF proxy generation gave major head aches in VS 2010 for Silverlight projects, after you update the WCF service and try to update the service reference in SL project, proxy generation would fail and no proxy classes gets generated. They way I got this working is by reflecting existing WCF services deployed to SharePoint and looking at web.config settings