I typically recommend that you build and test your assemblies against the same version of .NET that you will be running them against. That way, you'll have correct references and avoid surprises from behavior differences between builds.
Older assembly, newer .NET
But, sometimes you don't run against the same version that you built against. For example, the latest available CLR is the default when interop causes it to be loaded. Or, if the version of the CLR the assembly was built against is not available, the latest version can also be used instead. If there is a compatibility problem with that for your application, you can force the use of a preferred CLR version by using an app.config.
Yes, there are going to be behavior differences between versions - no way around it. It is the .NET team's goal to be as backwards compatible as realistically possible between versions. But, some changes are required - new features are added which sometimes require (hopefully minimal) behavior changes in other features. If we were required to never break any app, ever, we wouldn't be able to fix any bugs or add any features for fear that someone, somewhere relied on the broken behavior in some strange way. As Dll Hell has shown in the past, new versions of components aren't really compatible. The only guaranteed 100% compatible version is the same one you tested against.
V2+ assembly, older .NET
Say you see a BadImageFormatException for hresult 0x80131107 (CLDB_E_FILE_OLDVER) with message "Version [...] is not a compatible version." That means that you've tried to load an assembly built against a newer version of the CLR than was running.
This fails because new files can't be opened on an older CLR that does not understand them. You will need to either use the v2 CLR to load the v2/v1 assemblies, or load only v1 assemblies on the v1 CLR. That's just good practice, anyway. If you try to use an older .NET than you built against, a method you built against may not exist, etc.