Why doesn't my XAP load on a cross-domain page?  Probably because you're not setting the mime-type of the XAP to "application/x-silverlight-app".

What's the point of this restriction?  Well, we found during the development of Silverlight 2 that threats involving a Bad Guy loading a good XAP on his own domain were a bit subtle and non-obvious.  It's common to use the HTML bridge (HtmlPage class) to chat back and forth with JavaScript.  As a developer does this, she's rarely thinking about what might happen if the page is malicious.  Yes, this is somewhat mitigated by the ExternalCallersFromCrossDomain attribute, but this only protects against the bad JavaScript calling into the XAP directly.  There's still the problem of the XAP calling out to a JavaScript function with potentially sensitive information.  So, we demand to see a special mime-type on the XAP to ensure that's what the author intended.

We were also worried that sites allowing users to upload arbitrary content would now need to check for files that look like XAP.  We really wanted to make sure that web developers and webmasters don't need to read all about Silverlight just to make sure it can't attack their site, even if they never plan to use it.  By demanding to see the mime-type, Silverlight knows that the web server knows what a Silverlight application is.

There's more info here.