There's no way to unload an individual assembly without unloading all of the appdomains containing it. (See here for why not.) This can by done by calling AppDomain.Unload() for each AppDomain that has it loaded. (You could also use UnloadDomain() on the v1 unmanaged hosting API, but in general, I recommend using managed code whenever possible.)

I should point out that this means that even if your managed Assembly/Module/etc. instances go out of scope, GC may clean those instances up. The actual file will remain loaded until you unload all of the appdomain(s) containing it, however. (In general, that's better, anyway. Loading a file, especially by http, is too expensive to be done over and over when it can just be cached.)

If you want to unload some assemblies but not others, consider creating a new AppDomain (requires ControlAppDomain permission), execute the code using the temporary ones from within that AppDomain, and then unload that new AppDomain when you're done. I recommend minimizing the number of AppDomains, though, since there is overhead involved in creating them, transitioning between them, and unloading them. So, it's best if you do all of that work at once, in one AppDomain, minimizing the cross-domain communication, and just unload that when possible (as opposed to creating a new AppDomain for each assembly to unload, etc.).

Gotchas
Watch out for references to that assembly leaking to other appdomains. If that happens, that assembly will remain loaded if you just unload one AppDomain. For example, passing a Type object to another appdomain will cause its assembly to be loaded there.

Calling Unwrap()/AppDomain.CreateInstanceAndUnwrap()/etc. may also cause the runtime to try to load the assembly in the calling appdomain. In that case, call CreateInstance*() on a MarshalByRefObject type from a different assembly. For example, using AppDomain.CreateInstanceFrom() on the calling assembly (the one currently calling Unwrap()) may be the most convenient way. Then, call a method on it which will do all of your assembly loading. That way, those assemblies can be loaded in just the target appdomain.

Also, note that assemblies loaded as domain neutral will not be unloaded until the process exits, since other appdomains are technically still using them.

Alternatives
But, maybe you don't need to load that Assembly, anyway. If you just need info about it, you could call AssemblyName.GetAssemblyName() with the path to the file. That will temporarily load the file to get the info and then immediately unload it. That call will not cause it to show up as an Assembly in the AppDomain.

Or, if you're only worried about keeping the file locked, you could use shadow copying. That will make a copy of the file on disk and load it from the new location. The original file will not be locked by that load.  To do that, set AppDomainSetup.ShadowCopyFiles to "true" when creating the AppDomain or set AppDomain.ShadowCopyFiles to true after it's already been created.