When it's in another zone, of course.
One of the recommendations we give for setting policy on VSTO projects is to sign your assembly (either with an Authenticode certificate or a strongname) and then trust that key only for a specific computer (eg, your deployment server) or for a specific Zone (typically My Computer or Local Intranet). So people go ahead and they do this, and sometimes it works and sometimes it doesn't. They scratch their head for a while, and then they e-mail me and ask what's up :-)
Almost always, the problem is that although the person thinks the computer is in the Local Intranet zone, IE has a different opinion on the matter for one of two reasons:
1) You have previously added the web site to the "Trusted Zone" for some particular reason
2) You are using a dotted IP address (\\192.168.1.1) or a FQDN (\\computername.company.com) name instead of a NETBIOS name (\\computername)
The first one fails for obvious reasons -- you moved the site from the Local Intranet Zone to the Trusted Sites Zone, and the CLR is simply obeying your command. Fixing it is as easy as moving the site back to Local Intranet (if that makes sense for you) or changing your policy to trust the key inside the Trusted Sites Zone instead.
The second one is a bit less obvious. Basically (and I'm by no means a network protocol expert, so feel free correct me if I get the details wrong) IE knows that NETBIOS names (names without dots in them) can't be routed across the internet, so if it sees a name without a dot it knows it is a local name. If IE sees a name with dots in it (whether numeric or DNS) then it assumes it is coming from across the internet, and so it falls into the Internet Zone. One tricky thing here is that since an IP address is really just four bytes, you can represent an IP address as a single 32-bit number such as \\12345678, and this could indeed come from across the internet. So there is a special check to see if it is this kind of address in addition to the simple "does it have dots in it" check.
Fixing this is also quite easy; just add the computer's address to your Local Intranet Zone by going to Tools -> Internet Options -> Security -> Local Intranet Zone -> Sites -> Advanced and typing it in.
Note that the cause of this problem (but not the solution) is easily deducible by following my previous debugging tip to find out how the CLR is resolving your policy rules.