Improved Package Restore

Improved Package Restore

Rate This
  • Comments 38

As we’ve previously explained we receive various reports on how our NuGet packages don’t play nicely with NuGet’s package restore feature. In this blog post I’ll talk about an update we shipped to our infrastructure package Microsoft.Bcl.Build that reduces the impact.

The Issue

To recap, some of our packages need to participate in the build in order to work properly. Examples of tasks that require build participation include generating binding redirects or selecting the appropriate native dependency based on the CPU architecture the build is producing.

Build participation is implemented by adding an import to an MSBuild .targets file. Generally, build participation isn’t optional which is why we decided not to use NuGet’s feature to add a target file import. NuGet will add the import with a condition that checks whether the .targets file already exists, i.e. whether the NuGet package was already restored.

Both solutions aren’t ideal:

  • A non-optional import ensures that building will fail when packages are missing because the imported target file can’t be found. However, this also means that you can’t open the solution in Visual Studio (see picture below). Also, the error message isn’t really actionable as it doesn’t include any instructions on how to fix the issue.

  • An optional import allows the project to load and build. However, even if you have package restore enabled, the first build may not work because by the time the project was build, the targets file was still missing. This can result in build breaks or – even worse – the build may succeed but with incorrect outputs.

The imported project "<solution-path>\packages\Microsoft.Bcl.Build.1.0.8\tools\Microsoft.Bcl.Build.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.

The Improvement

We’ve updated Microsoft.Bcl.Build to use a different approach. The new version will use a conditional import similar to what NuGet’s automatic import feature does. This will always allow the project to load in Visual Studio.

However, Microsoft.Bcl.Build also adds a target to your project that will run after the build is finished. This target checks whether the current build restored packages and if so fail the build with an actionable error message:

The build restored NuGet packages. Build the project again to include these packages in the build. For more information, see http://go.microsoft.com/fwlink/?LinkID=317568.

Building a second time will fix this error. Please note that this error will only appear if packages were missing so it’s not like you always have to build twice.

This solution doesn’t address build server / continuous integration (CI) scenarios. In order to successfully use package restore on the build server, you have two options:

  1. Check-in the .targets file.
  2. Explicitly run NuGet package restore prior to building your project/solution.

NuGet 2.7

Prior to version 2.7, NuGet makes package restore a bit more complicated than most people would like it to be:

  1. First of all you need to manually enable it for the solution. This will add a NuGet.exe, NuGet.config and NuGet.targets file to your solution and you are expected to check those files in.
  2. Explicitly running package restore for the solution isn’t a single step. You need to run a command for each project, such as nuget.exe install .\Project1\packages.config.

NuGet 2.7 makes this a lot easier:

  1. You no longer need to enable package restore explicitly – when building in VS all packages are restored automatically.
  2. Running package restore on a build machine is now super easy. You only need to check-in NuGet.exe (nothing else) and you can put it wherever you want. It can even be in a well-known location on your build server and excluded from every solution if desired. Prior to the build you simply run nuget.exe restore path\to\my\solution.sln.

Also, the NuGet team is talking to all major providers of build/CI servers (incl. TFS) so that at some point the second step can be handled automatically by the build servers. For more details, have a look at the NuGet 2.7 release notes and the new Package Restore documentation.

Summary

The new version of Microsoft.Bcl.Build will ensure that solutions containing our packages will load successfully even if packages aren’t restored yet. This affects all .NET NuGet packages that depend on it, such as Microsoft.Net.Http, Microsoft.Bcl, and Microsoft.Bcl.Async.

Microsoft.Bcl.Build will give an actionable error message in cases the package was missing by asking you to build again.

When coupled with NuGet 2.7 where package restore is automatic in Visual Studio and isn’t implemented through MSBuild, the experience is transparent and smooth. However, this doesn’t address build server scenarios yet so you still need run nuget.exe restore solution.sln prior to build, or check-in the .targets file if preferred.

Leave a Comment
  • Please add 5 and 5 and type the answer here:
  • Post
  • > Also, the NuGet team is talking to all major providers of build/CI servers (incl. TFS) so that at some point the second step can be handled automatically by the build servers

    Is it fair to say that this includes Azure Web Site deployments?

  • The previous versions always redownloads all the packages. Is that fixed now?

  • Does this mean i can delete now the ".nuget" folder and its content for all of my projects.

    The targets file in there is not more necessary?

  • @ta.speot.is: Good question, I'll ping the NuGet guys and come back to you.

    @Works on mono: not sure I fully understand. Are you saying that NuGet on Mono used to always download packages regardless whether they were already downloaded? I'll have to ask the NuGet folks for this one.

    @Mike: That's correct. On the build machine you just need to run nuget.exe restore mysolution.sln but it's up to you where you put nuget.exe.

  • @Immo: that is correct. A restore on a packages.config with an OutputDirectory target does re-download all packages if running on mono/linux/osx as opposed to a restore on a packages.config on a Windows system.

  • @Works on mono: I've asked the NuGet team and Fei Ling looked into it. This is what he had to say:

    I then tried all cases of restore, and in all cases, nuget restore works as expected, i.e. only missing packages are downloaded. The cases that I tried are:

    • Nuget.exe restore
    • Nuget.exe restore –OutputDirectory dir
    • Nuget.exe restore packages.config –OutputDirectory dir
    • Nuget.exe install packages.config –OutputDirectory dir

    So, whatever that bug is, it was already fixed now. Customer can try the latest 2.7 build (build.nuget.org/viewLog.html).

    Does this help?

  • I got the error described above on my Team Foundation Service/git CI build:

     "The build restored NuGet packages. Build the project again to include these packages in the build."

    I was able to get around it by adding the .targets file like you said:

     git add -f Microsoft.Bcl.Build.targets

    Visual Studio's git tools wanted to ignore this change since it was overriding .gitignore, so I had to do a git commit from the command line instead.

    Is Visual Studio going to know to push this file if there are ever changes?  Since the file is under a version-specific directory (Microsoft.Bcl.Build.1.0.10), will I need to drop to the command prompt and add it every time there is a new version?

    Maybe I would be better off using option 2 (explicitly running NuGet package restore), but I'm not sure I can do that on a Team Foundation Service build.

  • @Thomas M. Mueller:  You can fix your .gitignore file like this:

    packages

    !packages/**/*.targets

    This will exclude all packages folders but will re-include all contained .targets files.

    BTW: I'll publish a blog post with a walkthrough that will address your specific scenario (git + TFS + package restore). In this post, I'll show how you can avoid check-in any .targets files.

  • Hi,

    I am trying to use Microsoft.Bcl.Build in an ASP.NET MVC application. My web.config root element uses a namespace xmlns="schemas.microsoft.com/.../v2.0". My issues is that I cannot build the project due to an exception in the Roxel.BuildTasks.EnsureBindingRedirects build task. When the task tries to load the application's web.config file, it checks the root element like this:

    (xDocument.Root == null || xDocument.Root.Name != "configuration")

    In my case this check can never pass as the root element name contains a namespace - "{schemas.microsoft.com/.../v2.0}configuration". Is this a bug, and you are better off checking Name.LocalName instead of Name?

  • @Roy Platter: Thanks for reporting this issue. I've filed a bug on our side to fix this. Meanwhile, you should be able to unblock yourself by removing the namespace -- AFAIU this namespace is used to provide IntelliSense, so it shouldn't be required.

  • I've been struggling with an error in my build for a few days:  C:\a\src\AcsWebApplication-Staging\AcsWebApplication\AcsWebApplication.csproj (994): The build restored NuGet packages. Build the project again to include these packages in the build. For more information, see go.microsoft.com/fwlink.

    I've tried every solution and nothing seems to work.  I'm running NuGet 2.7 (confirmed by using the update command).  Today I clicked on the link in that the build log displays for this error and I came across the error in the proj file.  The way I'm reading this is that I can't win for losing.  There are two errors, one for the target file for not existing and one for having the target exist.  Should this target file be gone at some point?  Because after a build the .target file still exist and I run up against this error.

    <Import Project="..\packages\Microsoft.Bcl.Build.1.0.10\tools\Microsoft.Bcl.Build.targets" Condition="Exists('..\packages\Microsoft.Bcl.Build.1.0.10\tools\Microsoft.Bcl.Build.targets')" />

     <Target Name="EnsureBclBuildImported" BeforeTargets="BeforeBuild" Condition="'$(BclBuildImported)' == ''">

       <Error Condition="!Exists('..\packages\Microsoft.Bcl.Build.1.0.10\tools\Microsoft.Bcl.Build.targets')" Text="This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them.  For more information, see go.microsoft.com/fwlink HelpKeyword="BCLBUILD2001" />

       <Error Condition="Exists('..\packages\Microsoft.Bcl.Build.1.0.10\tools\Microsoft.Bcl.Build.targets')" Text="The build restored NuGet packages. Build the project again to include these packages in the build. For more information, see go.microsoft.com/fwlink HelpKeyword="BCLBUILD2002" />

     </Target>

  • Can you guys please explain me a thing here.

    When I run my MVC4 project it's run fine in VWD 12 Express. Now I turn to 2013 Preview now run my project. My visual studio is crashed.

    My CPU is still using 3%  Don't know What VWD do. Now I stopped it run VWD again. This time take a lot of time to build.

    Now Everytime I build he show me a dialog that he restore something. Why I waste some second every time I build something or it will be better to don't confuse me with that dialog. If this is something additional then feature exist in 2012 then how I can disable it.

  • There is an issue with the Import of the Microsoft.Bcl.Build target causing MS Build Warning 3247 for Web Application Projects. Refer this forum post: social.msdn.microsoft.com/.../ms-build-gives-warning-msb3247-found-conflicts-between-different-versions-of-same-dependent

  • So now if I have a solution with 10 projects, and in 1 project I want to use Microsoft.Net.Http, I have to install it in all 10 projects because VS gives warnings like 'All projects referencing xxx.csproj must install nuget package Microsoft.Bcl.Build' and after fixing that new warnings asking for Microsoft.Net.Http in every project. Is that really how this should work?

  • I'm running Visual Studio 2012 (Version 11.0.60610.01 Update 3).  I've installed NuGet 2.7.  I've created an SSIS project, and inserted a C# Script Task.  I add references with NuGet and they install fine with no errors.  I run the "Code Analysis" and it returns "Rebuild All succeeded".  But when I try to run "Build" it returns the error "This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them..." and refers me to this article.  Under "Package Manager Settings", both "Allow NuGet to download missing packages" and "Automatically check for missing packages during build in Visual Studio" are CHECKED.  How do I resolve this error, I've spent hours and hours trying to troublesheet it with no resolution?

Page 1 of 3 (38 items) 123