Office will only load managed code assemblies which are fully trusted by the .Net CAS policy. When developing a VSTO solution, Visual Studio tweaks the CAS policy to allow Office to load the VSTO assemblies from a default <projectname>_bin folder under the solution folder. Attempting to run the VSTO from any other location (local or remote) will fail due to lack of permissions.
If you need to enable the VSTO assembly to run from a different location such as a remote or virtual directory, the easiest solution is to modify the policy to trust the remote URL:

C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\caspol -q -m -ag 1 -url VSTO_ASSEMBLY_URL/* FullTrust -n VSTO_ASSEMBLY_NAME

The above command assigns *full trust* to any assembly in the specified directory. While this configuration provides the greatest ease-of-use while developing, it is *not* a secure solution. Never do this anywhere other than your development environment!   For deployment, the policy modifications should be much narrower, for example assigning permissions based on strong name or publisher.