Improved Package Restore

Improved Package Restore

Rate This
  • Comments 38

As we’ve previously explained we receive various reports on how our NuGet packages don’t play nicely with NuGet’s package restore feature. In this blog post I’ll talk about an update we shipped to our infrastructure package Microsoft.Bcl.Build that reduces the impact.

The Issue

To recap, some of our packages need to participate in the build in order to work properly. Examples of tasks that require build participation include generating binding redirects or selecting the appropriate native dependency based on the CPU architecture the build is producing.

Build participation is implemented by adding an import to an MSBuild .targets file. Generally, build participation isn’t optional which is why we decided not to use NuGet’s feature to add a target file import. NuGet will add the import with a condition that checks whether the .targets file already exists, i.e. whether the NuGet package was already restored.

Both solutions aren’t ideal:

  • A non-optional import ensures that building will fail when packages are missing because the imported target file can’t be found. However, this also means that you can’t open the solution in Visual Studio (see picture below). Also, the error message isn’t really actionable as it doesn’t include any instructions on how to fix the issue.

  • An optional import allows the project to load and build. However, even if you have package restore enabled, the first build may not work because by the time the project was build, the targets file was still missing. This can result in build breaks or – even worse – the build may succeed but with incorrect outputs.

The imported project "<solution-path>\packages\Microsoft.Bcl.Build.1.0.8\tools\Microsoft.Bcl.Build.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.

The Improvement

We’ve updated Microsoft.Bcl.Build to use a different approach. The new version will use a conditional import similar to what NuGet’s automatic import feature does. This will always allow the project to load in Visual Studio.

However, Microsoft.Bcl.Build also adds a target to your project that will run after the build is finished. This target checks whether the current build restored packages and if so fail the build with an actionable error message:

The build restored NuGet packages. Build the project again to include these packages in the build. For more information, see http://go.microsoft.com/fwlink/?LinkID=317568.

Building a second time will fix this error. Please note that this error will only appear if packages were missing so it’s not like you always have to build twice.

This solution doesn’t address build server / continuous integration (CI) scenarios. In order to successfully use package restore on the build server, you have two options:

  1. Check-in the .targets file.
  2. Explicitly run NuGet package restore prior to building your project/solution.

NuGet 2.7

Prior to version 2.7, NuGet makes package restore a bit more complicated than most people would like it to be:

  1. First of all you need to manually enable it for the solution. This will add a NuGet.exe, NuGet.config and NuGet.targets file to your solution and you are expected to check those files in.
  2. Explicitly running package restore for the solution isn’t a single step. You need to run a command for each project, such as nuget.exe install .\Project1\packages.config.

NuGet 2.7 makes this a lot easier:

  1. You no longer need to enable package restore explicitly – when building in VS all packages are restored automatically.
  2. Running package restore on a build machine is now super easy. You only need to check-in NuGet.exe (nothing else) and you can put it wherever you want. It can even be in a well-known location on your build server and excluded from every solution if desired. Prior to the build you simply run nuget.exe restore path\to\my\solution.sln.

Also, the NuGet team is talking to all major providers of build/CI servers (incl. TFS) so that at some point the second step can be handled automatically by the build servers. For more details, have a look at the NuGet 2.7 release notes and the new Package Restore documentation.

Summary

The new version of Microsoft.Bcl.Build will ensure that solutions containing our packages will load successfully even if packages aren’t restored yet. This affects all .NET NuGet packages that depend on it, such as Microsoft.Net.Http, Microsoft.Bcl, and Microsoft.Bcl.Async.

Microsoft.Bcl.Build will give an actionable error message in cases the package was missing by asking you to build again.

When coupled with NuGet 2.7 where package restore is automatic in Visual Studio and isn’t implemented through MSBuild, the experience is transparent and smooth. However, this doesn’t address build server scenarios yet so you still need run nuget.exe restore solution.sln prior to build, or check-in the .targets file if preferred.

Leave a Comment
  • Please add 4 and 3 and type the answer here:
  • Post
  • @Robin Ainscough: I'm sorry to hear that your NuGet experience is so frustrating. A couple of things:

    * Beginning with NuGet 2.7 you shouldn't suffer as much because NuGet has support for automatic package restore (see docs.nuget.org/.../package-restore)

    * Microsoft.Bcl.Build should fail with an actionable error message, so your solution shouldn't be completely hosed. If that's the case we'd love to take a look because that's clearly not the correct behavior.

    If you want, we can discuss more details via email to resolve this issue to your satisfaction. You can reach me under immol at microsoft dot com.

  • Sorry guys, this is a complete and utter mess.  NuGet sucks, having builds fail like this sucks and this is not making anyones life easier, the combination of NuGet and Microsoft.Bcl.Build has cost me 1000 times more time and effort ficing the mess it makes that hand managing my dependencies ever did, the whole thing is a failure.

  • @James Green: No need to apologize for the criticism. We're very much aware that this is a cause of major dissatisfaction. Microsoft.Bcl.Build is only a workaround. We're working with the NuGet team to avoid the need for it. For the time being, make sure to read this post on how to leverage the new package restore. While this doesn't solve everything, it should make it a bit less painful.

    blogs.msdn.com/.../nuget-package-restore-with-team-foundation-build.aspx

  • Hi Immo,

    Thanks for your article.  Can you provide a bit more info on where Visual Studio Online drops the packages?

    I am getting positive info at the top of my build log that my packages have been "Installing ..." and "Successfully installed ...", however further down when it comes to the individual projects, the build process still fails to "find" those packages and goes those a whole series of hints to locate them, eventually giving up.  Even when generating a "Detailed" build log, the path to the packages are not reported in the log.

    I added my "build.proj" two levels above (towards root) the solution file, so could this be the cause?

    Thanks.

    Project "C:\a\src\TFS\RSPlatform\build.proj" on node 1 (default targets).

    Target "RestorePackages" in project "C:\a\src\TFS\RSPlatform\build.proj" (target "Build" depends on it):

    Using "Exec" task from assembly "Microsoft.Build.Tasks.v4.0, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a".

    Task "Exec"

     "C:\a\src\TFS\RSPlatform\Main\Tools\NuGet\NuGet.exe" restore "C:\a\src\TFS\RSPlatform\Main\Source\RSPlatform Team Project.sln"

     Installing 'Bytescout.Spreadsheet 2.4.0.1346'.

     (etc)

     Successfully installed 'Bytescout.Spreadsheet 2.4.0.1346'.

      (etc)

    Done building target "RestorePackages" in project "build.proj".

    .....

     Assemblies:

         Bytescout.Spreadsheet, Version=2.4.0.1346, Culture=neutral, PublicKeyToken=f7dd1bd9d40a50eb, processorArchitecture=MSIL

    .....

     Primary reference "Bytescout.Spreadsheet".

    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Microsoft.Common.targets(1605,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "Bytescout.Spreadsheet". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [C:\a\src\TFS\RSPlatform\Main\Source\Support\A2Refresh\A2Refresh.csproj]

    etc.

  • Hi Immo

    Recently adding BCL.Build causes our WiX project to fail. The MSBuild warning says the WiX project should also reference the BCL.Build package because the WiX project has a project reference to another project that uses BCL.Build.

    This MSBuild error is wrong - the WiX project cannot have the BCL.Build NuGet package installed - and the project references the WiX project have is for harvesting outputs (heat.exe in WiX parlance).

    How to I get MSBuild to not give this error and fail the builds?

    10>E:\Development\Projects\XXXX\packages\Microsoft.Bcl.Build.1.0.14\tools\Microsoft.Bcl.Build.targets(225,5): error : All projects referencing Service.Windows.ApplicationServer.csproj must install nuget package Microsoft.Bcl.Build. For more information, see go.microsoft.com/fwlink.

    10>E:\Development\Projects\XXXX\packages\Microsoft.Bcl.Build.1.0.14\tools\Microsoft.Bcl.Build.targets(225,5): error : All projects referencing Service.Windows.Automation.csproj must install nuget package Microsoft.Bcl.Build. For more information, see go.microsoft.com/fwlink.

    10>E:\Development\Projects\XXXX\packages\Microsoft.Bcl.Build.1.0.14\tools\Microsoft.Bcl.Build.targets(225,5): error : All projects referencing UI.Web.csproj must install nuget package Microsoft.Bcl.Build. For more information, see go.microsoft.com/fwlink.

    10>E:\Development\Projects\XXXX\packages\Microsoft.Bcl.Build.1.0.14\tools\Microsoft.Bcl.Build.targets(225,5): error : All projects referencing UI.Windows.csproj must install nuget package Microsoft.Bcl.Build. For more information, see go.microsoft.com/fwlink.

    1

  • Hi! Is there any ticket we can track to see how this issue is proceeding? From the comments, I guess some changes need to be performed on nuget for this to work correctly?

    I don't want to update my current references to the async package to use nuget until we can avoid the requirement to add the Microsoft.BCL dependency to every project referencing it. We have projects that might be referenced by multiple solutions and I don't want to have to track which projects I'm breaking by changing a referenced project dependency to use nuget...

  • I have a continuous integration build set up at VS Online.  Locally the solution builds perfectly fine.  But fails at the build server.  Getting the same error.

    The build restored NuGet packages. Build the project again to include these packages in the build. For more information, see go.microsoft.com/fwlink.

    I built it more than one time, but every time i get the same error.

  • Santiago Aguiar: There is no ticket to track.

    Sabarinath: Which TFS build template are you using? You need to use one of the newer ones that also include automatic package restore. It seems you're using the old MSBuild-Integrated package restored restore which was deprecated. See this topic for details: docs.nuget.org/.../package-restore.

Page 3 of 3 (38 items) 123