CLR does not do anything special regarding authorization during assembly loading. It will try to access the file under the current user. However, if an assembly is already loaded, next time when the assembly is requested, CLR will return the assembly from the cache without hitting the file.
When the application impersonates a user, things get a little bit interesting. Sometimes the impersonated user does not have access to the assembly. Sometimes the impersonated user has read access to the assembly, but the appdomain has shadow copy enabled, and the impersonated user does not have write access to the shadow copy location. All those cases will result in access denied when the application tries to load the assembly.
In most cases, this access deny is perfectly acceptable. However, in some cases, you may want the user to be able to use the assembly. For example, you developed a web application. You would like to run the web application under the user accessing the web application, but the user does not necessarily have the right to write to "Temporary ASP.NET files" location where ASP.NET stores the shadow copied assembly.
You can workaround this problem by pre-loading the assemblies using a high privileged user. Since CLR does not touch the file after the assembly is loaded, you can sidestep the access deny problem.
(This technique does not work for Assembly.LoadFrom/LoadFile, for obvious reason.)
However, there is apparent security implications, and you have to evaluate it carefully.