There are a couple of reasons why appdomain unloads can fail. One common failure is when all threads inside the appdomain cannot be unwound. This mostly happens when there are threads running unmanaged code. In a remoting case its possible a thread that could be performing active IO at the winsock layer when the unloading is triggered. In this scenario the unload would most certainly fail. The usual workaround is to catch the unload failure and retry till the appdomain sucessfully unloads. Another workaround is to subscribe to the AppDomain.DomainUnload event and try to make sure all threads in the current appdomain are in abortable state.
In v2.0 of the framework the time the runtime will wait for all threads to be unwound is configurable through the CLR hosting APIs. An increased timeout can give a relatively better guarantee of a successful unload. Chris Brumme has some good info on appdomains in general.