Improved Package Restore

Improved Package Restore

Rate This
  • Comments 38

As we’ve previously explained we receive various reports on how our NuGet packages don’t play nicely with NuGet’s package restore feature. In this blog post I’ll talk about an update we shipped to our infrastructure package Microsoft.Bcl.Build that reduces the impact.

The Issue

To recap, some of our packages need to participate in the build in order to work properly. Examples of tasks that require build participation include generating binding redirects or selecting the appropriate native dependency based on the CPU architecture the build is producing.

Build participation is implemented by adding an import to an MSBuild .targets file. Generally, build participation isn’t optional which is why we decided not to use NuGet’s feature to add a target file import. NuGet will add the import with a condition that checks whether the .targets file already exists, i.e. whether the NuGet package was already restored.

Both solutions aren’t ideal:

  • A non-optional import ensures that building will fail when packages are missing because the imported target file can’t be found. However, this also means that you can’t open the solution in Visual Studio (see picture below). Also, the error message isn’t really actionable as it doesn’t include any instructions on how to fix the issue.

  • An optional import allows the project to load and build. However, even if you have package restore enabled, the first build may not work because by the time the project was build, the targets file was still missing. This can result in build breaks or – even worse – the build may succeed but with incorrect outputs.

The imported project "<solution-path>\packages\Microsoft.Bcl.Build.1.0.8\tools\Microsoft.Bcl.Build.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.

The Improvement

We’ve updated Microsoft.Bcl.Build to use a different approach. The new version will use a conditional import similar to what NuGet’s automatic import feature does. This will always allow the project to load in Visual Studio.

However, Microsoft.Bcl.Build also adds a target to your project that will run after the build is finished. This target checks whether the current build restored packages and if so fail the build with an actionable error message:

The build restored NuGet packages. Build the project again to include these packages in the build. For more information, see http://go.microsoft.com/fwlink/?LinkID=317568.

Building a second time will fix this error. Please note that this error will only appear if packages were missing so it’s not like you always have to build twice.

This solution doesn’t address build server / continuous integration (CI) scenarios. In order to successfully use package restore on the build server, you have two options:

  1. Check-in the .targets file.
  2. Explicitly run NuGet package restore prior to building your project/solution.

NuGet 2.7

Prior to version 2.7, NuGet makes package restore a bit more complicated than most people would like it to be:

  1. First of all you need to manually enable it for the solution. This will add a NuGet.exe, NuGet.config and NuGet.targets file to your solution and you are expected to check those files in.
  2. Explicitly running package restore for the solution isn’t a single step. You need to run a command for each project, such as nuget.exe install .\Project1\packages.config.

NuGet 2.7 makes this a lot easier:

  1. You no longer need to enable package restore explicitly – when building in VS all packages are restored automatically.
  2. Running package restore on a build machine is now super easy. You only need to check-in NuGet.exe (nothing else) and you can put it wherever you want. It can even be in a well-known location on your build server and excluded from every solution if desired. Prior to the build you simply run nuget.exe restore path\to\my\solution.sln.

Also, the NuGet team is talking to all major providers of build/CI servers (incl. TFS) so that at some point the second step can be handled automatically by the build servers. For more details, have a look at the NuGet 2.7 release notes and the new Package Restore documentation.

Summary

The new version of Microsoft.Bcl.Build will ensure that solutions containing our packages will load successfully even if packages aren’t restored yet. This affects all .NET NuGet packages that depend on it, such as Microsoft.Net.Http, Microsoft.Bcl, and Microsoft.Bcl.Async.

Microsoft.Bcl.Build will give an actionable error message in cases the package was missing by asking you to build again.

When coupled with NuGet 2.7 where package restore is automatic in Visual Studio and isn’t implemented through MSBuild, the experience is transparent and smooth. However, this doesn’t address build server scenarios yet so you still need run nuget.exe restore solution.sln prior to build, or check-in the .targets file if preferred.

Leave a Comment
  • Please add 8 and 7 and type the answer here:
  • Post
  • "So now if I have a solution with 10 projects, and in 1 project I want to use Microsoft.Net.Http, I have to install it in all 10 projects because VS gives warnings like 'All projects referencing xxx.csproj must install nuget package Microsoft.Bcl.Build' and after fixing that new warnings asking for Microsoft.Net.Http in every project. Is that really how this should work?"

    Thanks Marcel. No Joke. My SLN has currently 10 projects too. Seven references the one using Net.Http and all want BCL and Net.Http!

    So please.. Can someone answer this quersion?!

    Thanks

    Holger

  • @L Parrish: If you get the error, your build restored packages as part of the same build. You have to build twice in order to get rid of the error. If you are building on a build machine where building twice isn't an option you either have to check-in the targets file or run package restore prior to build, e.g. "nuget.exe install .\Project1\packages.config".

    @Jaans: Removing the import to the targets file isn't the correct fix. You either check-in the target file or restore packages prior to build.

    @Marcel, Holger: All upstream projects need this reference, yes. However, you can simplify this tremendously by right clicking the solution and invoke "Manage NuGet Packages". In the dialog you can install a NuGet package to multiple projects at once.

    @michael.o.hanna@multco.us: Sorry to hear that you lost that much time trying to identify the issue. When you build in the IDE, building again should fix this error. If it doesn't please send us an email to netfxnuget at microsoft dot com so that we can trouble shoot. Thanks!

  • Using nuget 2.7.3 and the conditional import - it works initially - so a lot better than it was.

    However get problems after a nuget safe update

    This updates the version in packages.config

    but not the version in the conditional import in the csproj

    Once these are out of sync project build fails and have to hand edit the csproj.

    Using nuget safe updates as a matter of course in automated builds, so hit this whenever a new version comes out.

    thanks

  • This solution is garbage.

    Someone has taken a good thing and made it worse over time. Who is responsible and how do we get rid of them? So many labor hours are being lost over this targets file, over and over. Every time something goes wrong it's attributed to an improvement, or a fix.. get real, you're not improving anything, you're just creating more work. Is it not clear? How can you not see what you're doing, are you in denial?

    You made a bad decision with these build tasks and now you're working feverishly to make it work, again and again.

    Please, stop screwing us already. This load of garbage should be OPTIONAL, instead it's being referenced from other Microsoft packages. What a load of crap, I don't know what has happened at Microsoft in the past 3-5 years but I'm slowly losing my patience, and I've been a loyal Microsoft developer for 20 years. You're making it increasingly hard to continue being an advocate, and without people like me everyone would be using non-Microsoft products by now.

    Get rid of this *** already.

  • How do I opt out of this package? Don't want it, don't need it, only causes problems.

    Thanks.

  • Shaun, agreed.

  • In a continuous integration build on the TFS server, I'm getting the error:

    The "EnsureBindingRedirects" task could not be loaded from the assembly C:\a\src\packages\Microsoft.Bcl.Build.1.0.13\tools\Microsoft.Bcl.Build.Tasks.dll.

    I checked in the targets file, as suggested. That gave me an error that NuGet.exe could not be found, so I checked that in too. Then it went back to this error again. What? Why?

    This is insane. I neither know nor care about any of this crap. I just want my code my build. It builds on my PC. It refuses to build on the TFS server. Why? No-one seems to be able to tell me. I've been trying to solve this issue for three hours now and the best I can get out of it is people at MS saying "yeah it's a bit crap... screw you anyway". Oh, maybe I can run nuget.exe on the build sever... oh wait, that doesn't seem to be even possible (I certainly can't find any instructions as to HOW to do it), as "the build server" is a VM somewhere in Microsoft's cloud. Thanks though.

  • @Phil: Thanks for reporting this. That looks like a bug to me. We're looking into it.

    @Shaun Wilson: I'm sorry that you feel like we're screwing you over. We aren't exactly crazy about injecting .target files either.

    However, using target files over not using them isn't exactly a matter of taste. We're leveraging target file because there is no other way to make our component work. Simply removing it works for some folks but it depends on the exact project configuration this package is being used in.

    We fully understand that target files have issues for restoring packages during the build. Thus, we recommend ensuring packages are restored before your project is being built.

    Eventually we want to get rid of these target files. We're only using them in order to work around limitations that exist in NuGet today. We're working with the NuGet team to address those limitations so that we don't have to rely on target files moving forward. Unfortunately, this is a journey and we aren't there yet. Sorry to disappoint.

  • The improved package restore experience works in my C# projects.   Build once and the Microsoft.Bcl.Build target error is returned.  Build again and the error is fixed.   This does not work with my VB.NET projects.   Tested Silverlight 5 and Windows - WPF (.NET 4.0), in VS2012 and VS2013 (both Ultimate editions).   With VB.NET projects, when the nuget package is first restored, all builds return the Microsoft.Bcl.Build target error.   I must close and reopen Visual Studio to build the project successfully, which makes for a clunky process.   Is this a bug?  If so, is it possible to resolve this?  

  • @Rex: You're right, VB apparently doesn't re-evaluate the missing .targets after the first. As a workaround, build once, close the solution and reopen. Now you shouldn't see any errors.

    Alternatively, you can remove the automatic package restore during build (i.e. remove the nuget.exe, nuget.config and related imports) and use the approach outlined here:

    blogs.msdn.com/.../nuget-package-restore-with-team-foundation-build.aspx

  • @Immo  Thanks for the link.  I included a nuget.exe, added the prebuild step, and it worked for both C# and VB projects.  

    On the original issue I reported with VB projects and the solution advertised in this blog post... is it fixable in the Microsoft.Bcl.Build package?  

  • @Rex: I don't think it's fixable in Microsoft.Bcl.Build (MBB). MBB already participates in the build to fix certain issues but the issue that you're hitting is that the VB apparently doesn't re-evaluate the project after the first build, thus resulting in the same error condition. However, we can take a deeper look next year.

  • I tend to agree with Sean, this solution has caused numerous problems for us.  This seems like an ugly hack that needs a better fix.

    It's worked well for me to just ignore the Microsoft.Bcl.Build import in CI builds, but leave it in for dev environments.  You can do that by passing /p:BclBuildImported=ignore  on the command-line, or including a *.targets file in your build environment that contains this property:

    <PropertyGroup>

     <BclBuildImported>Ignore</BclBuildImported>

    </PropertyGroup>

    Where "Ignore" could be any text.

    But, since there's no documentation anywhere on exactly what Microsoft.Bcl.Build does, I can't be 100% sure whether this is safe or not.  But so far it has worked fine.

    I created this SO post, but no one has provided a good answer on what Microsoft.Bcl.Build does:

    stackoverflow.com/.../what-does-the-microsoft-bcl-build-nuget-package-do

  • Thanks restore Nuget task helped to resolve CI error

  • I tend to agree, my solution is now completely trashed with NuGet errors and I have no idea how to resolve going on day 2.  It boggles my mind that professional developers produces such a poorly tested product/process ... was there a "design" team even involved or was it code and crash type of process?

    It's bad enough living with VS 2012 bugs, and also make our own applications with limited bugs ... this is entire process of developing is for fools ... it didn't used to be like this and ROI on application development was considerably better with applications that are equally powerful (feature rich) and maintainable.  This NuGet process is a nightmare ... on par with the design team that thought up Windows 8 Metro.

    The "NuGet.exe" is only located in two folders on my PC, and they are in Telerik Examples ... not the place I expected it to be ... so can someone provide a little more details on how to resolve missing package references?

    You should consider renaming NuGet to NutGet ... only "nuts" get it ... the rest of the human race are left wonder what planet did this come from?

    Rob

Page 2 of 3 (38 items) 123