A first hand look from the .NET engineering teams
We’ve received several reports that our NuGet packages broke the NuGet package restore feature. In this post, I’ll explain what the issue is, how you can work around it, and finally how we plan on fixing this issue in the long term.
Microsoft.Bcl.Build and Microsoft.Bcl.Compression require custom target files, which do not work well with NuGet’s package restore feature. The easiest way to fix the package restore issues is by checking in any .targets files that are stored under the packages directory.
When you add a NuGet package to your project, NuGet essentially does two things:
In order to build your project on a build server you have to check in all sources as well as all 3rd party libraries. Many developers cringe when binaries need to be checked in to version control as they typically aren’t stored very efficiently and cause bloat over time. This is especially problematic for distributed version control systems (DVCS) like git or Mercurial where developers have to download the repository with the entire history (typically referred to as “cloning”).
For that reason NuGet has a feature called package restore. You need to enable that feature explicitly by right-clicking your solution and invoking the Enable NuGet Package Restore menu item:
After package restore is enabled you can delete the “packages” directory from your solution. Rebuilding the project will automatically re-create this directory and retrieve all missing packages; whether they are downloaded from the internet or a local file share. This allows excluding the packages directory from version control as the build machine can retrieve the packages and thus doesn’t need a checked-in version of the NuGet packages.
Two of our packages provide a custom targets file:
Targets files are MSBuild files that provide additional functionality that extend the build process. We use it for several features for which NuGet doesn’t provide declarative features today, for example, binding redirects and choosing the correct binary for the selected architecture. We also use it for additional diagnostics, as explained in this blog post (section “The CPU architecture matters”).
Installing Microsoft.Bcl.Build will change your project by adding an <import> entry for the target file to your project. The target file is located in the packages folder
<Import Project="..\packages\Microsoft.Bcl.Build.1.0.7\tools\Microsoft.Bcl.Build.targets" />
If package restore is enabled, it’s likely the packages folder is excluded so the .targets file isn’t available on the build machine. You may now think “hold on – isn’t package restore supposed to take care of this?” Unfortunately, not in this case. In order to restore packages you need to build the project, since package restore is plugged into the build process. Before building a project MSBuild loads the project, which in turn requires loading all necessary .targets files. Since the .targets files can’t be located at load time the build fails well before package restore has a chance of executing. You will see an error message like this:
YourProject.csproj(44,3): error MSB4019: The imported project "D:\YourProject.csproj\packages\Microsoft.Bcl.Build.1.0.7\tools\Microsoft.Bcl.Build.targets" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
You may be aware that NuGet 2.5 added first class support for installing custom target files. The end result is very similar, except that the import looks like this:
The import is conditioned on the existence of the file which allows loading the project file even if the .targets file doesn’t exist. Unfortunately this doesn’t solve the problem either.
On the build machine, the build will load the project without our .targets file, run package restore and then build the project. Since MSBuild evaluates all <import> entries before building, the .targets files don’t participate in the build because they weren’t present when the project was loaded. Depending on what a custom .targets file does this can have virtually any impact on the build, for example the build may simply still fail or – worse – succeed with incorrect outputs. For that reason we decided to not use conditional imports because the build at least fails in a deterministic and predictable way.
On the developer machine you have a very similar problem. When a project has missing .targets files it will not successfully open in Visual Studio. You might think that conditional imports would be a good solution there but you’ve fundamentally the same problem as on the build machine: the first build will restore the packages but the build outputs might be bogus without yielding an error. This behavior would remain until you re-open the solution. This issue would be quite hard to diagnose.
In order to make package restore work for packages that have custom target files you have three options:
The first option isn’t a really an option and we certainly wouldn’t recommend it either.
The second option requires changing the way your build is hooked up to your build environment. You need to run nuget.exe to restore packages before you actually run MSBuild on your solution:
nuget.exe install .\PortableClassLibrary3\packages.config
This ensures that by the time MSBuild loads the project all .targets are already present.
The third option is the easiest solution as it doesn’t require any changes to your environment. You only need to check-in the .targets files – you don’t need to check in any other files from the packages directory. Since target files are simply text files, they aren’t causing the repository to bloat over time. This also solves the issue on the developer’s machine whereby the second option would still require each team member to run package restore before the affected projects can be opened in Visual Studio.
We are working with the NuGet team to make sure we don’t have to use .target files moving forward. The idea is to add some features that allow packages to be declarative as opposed to adding code that runs during the build.
We understand that this isn’t good long term solution and are working with NuGet to greatly reduce the need for custom .targets files.
Please let us know if you have any questions or additional concerns.
Interesting Finds: June 13, 2013
Please could you release the libraries as normal downloads? Not everyone can use Nuget. Not everyone wants to use Nuget.
There is an even bigger problem. You can't open any visual studio projects if these target files don't exist... For example If I pull down a project from source control and I have no packages restored. I get an error saying unable to open project..
You know, back in the "old days" you'd just post a file and I'd download it and have it forever. Sometimes too much automation is a bad thing.
Here is another work around that I believe works.
Add Microsoft.Bcl.Build as a solution target.
These are downloaded before projects are built, AFAIK, so this should work.
Create a packages.config file in \.nuget with the following and check it in.
<?xml version="1.0" encoding="utf-8"?>
<package id="Microsoft.Bcl.Build" version="1.0.6" targetFramework="net45" />
1. Add dummy project (NugetHelper for example), add package.config with
<package id="Microsoft.Bcl.Build" version="1.0.6" targetFramework="net45" />
2. Open Menu -> Project -> ProjectDependencies and make NugetHelper to build before other projects in solution
<Import Project="..\packages\Microsoft.Bcl.Build.1.0.6\tools\Microsoft.Bcl.Build.targets" />
<Import Project="..\packages\Microsoft.Bcl.Build.1.0.6\tools\Microsoft.Bcl.Build.targets" Condition="Exists('..\packages\Microsoft.Bcl.Build.1.0.6\tools\Microsoft.Bcl.Build.targets')" />
this will restore Microsoft.Bcl.Build.targets before actually loading it in your main project
@Matt, @Neil: We don't have any plans to release standalone binaries. Most of our packages are multi-platform and despite today's shortcomings we believe NuGet packages provide the best experience. We are very much committed to work with NuGet on addressing all shortcomings. Can you explain why you don't want to use NuGet? Feel free to shoot me an email at immol at Microsoft dot com. Thanks!
@Blake: That's why we recommend checking in the .targets files.
@Henri, @Andriy: Thanks, these are interesting workarounds. They pretty much have the same characteristics as option 2 in the sense that it only helps with the build machine. As Blake pointed out, it still doesn't help for developer machines because you need to restore the packages before you can load the package/get a reliable ouput.
What I did to fix this issue was loaded up the Package Manager Console (you could also use the regular command line) and fired up this following nuget command.
nuget install .\GlobalInit_MVC\packages.config -OutputDirect packages
This worked great for us.
I am using AnkhSVN. The NuGet.targets file is versioned. This does not, apparently, help. What now?
@Jeremy Holovacs: Sorry for the confusion. The NuGet.targets file needs to be checked in order to make package restore work in general. However, if you are using a package that depends on Microsoft.Bcl.Build, you ALSO need to check in the packages\Microsoft.Bcl.Build<version>\tools\Microsoft.Bcl.Build.targets file as well.
I created a gist for everyone using git for source control, which solved the problem for us
Thanks a ton, it really helped me, i am surprise how deep knowledge you have about it. Really great post. i have rated it 5 stars.
I'm having this problem, yet I don't have NuGet Package restore turned on and I'm checking in all of our DLLs into TFS. Yet, TFS online is still giving me this build error.
To be more clear, I'm checking in the entire packages dir, with everything underneath. Is TFS Online ignoring that, dumping the packages dir and then doing restore? Either way, I can't seem to get Workaround #3 to work at all.
Can you paste the exactly build error that you are getting?