Everything was working, I did some refactoring and starting getting this error:

Error: Unhandled Error in Silverlight Application Code: 2103 Category: InitializeError Message: Invalid or malformed application: Check manifest

In silverlight 3, it turns out that the refactoring of namespaces causes this. Very unintuitive error. I had added default App class under a namespace, but missed to change the startup object.

To fix: Right click on project, go to properties, and select the refactored name under "Startup object"