Transferring (Large) Files Using BITs

Transferring (Large) Files Using BITs

  • Comments 19

Have you had a file copy fail?  Does that drive you crazy or what?  How about when it is a REALLY big file and it takes a couple of hours and JUUUUUST before it finishes something happens to interrupt the transfer and you have to start all over again.  Does that make you want to shove a sharpened #2 pencil up your nose or what?

I got to thinking about these things when I saw that the W7 and WS08/R2 beta bits were available and that the demand was so high that the servers were having a hard time keeping up.  Can you image how you would react if you downloaded a GB and then the transfer failed?  Can’t someone HELP?  Can’t someone give us the get semantics of Windows Update but for files that **I** care about?

Funny you should mention that.

Have I mentioned that PowerShell V2 rocks?  In CTP3, we are exposing a new set of Cmdlets that give you access to BITS.   BITS stands for Background Intelligent Transfer Service.  BITS is an awesome service.  It is the special sauce behind Windows Update.  Have you ever gone to WU and seen a couple hundred MB service pack and wondered, “how is every PC on the planet going to get a couple hundred MB without causing the Internet to melt?”  Well BITS is the answer to that question.  We think everyone should be using BITS.  That means both ISV products and end users.  Whenever you use BITS you get a better experience.  You get reliable transfers, adaptive network bandwidth usage to minimize the impact on your system and you get peer caching which minimizes traffic.  You can learn more about BITS by downloading a document HERE.

BTW - It turns out that we have been shipping the ability to do this for quite a while through a WIN32 command called BITSADMIN but few  know about it and it is difficult to use.  We think we still have usability issues with these Cmdlets so we might change them before we ship.

[0]PS> # You need to import the FileTransfer Module to get BITS cmdlets
[1]PS> # Notice that the NOUN is FileTransfer
[2]PS> Import-Module FileTransfer -Verbose
VERBOSE: Importing cmdlet 'Add-FileTransfer'.
VERBOSE: Importing cmdlet 'Clear-FileTransfer'.
VERBOSE: Importing cmdlet 'Complete-FileTransfer'.
VERBOSE: Importing cmdlet 'Get-FileTransfer'.
VERBOSE: Importing cmdlet 'New-FileTransfer'.
VERBOSE: Importing cmdlet 'Resume-FileTransfer'.
VERBOSE: Importing cmdlet 'Set-FileTransfer'.
VERBOSE: Importing cmdlet
'Suspend-FileTransfer'.

[3]PS> # This is the HTTP file that I want to download.  I could have
[4]PS> # also used an UNC Path
[5]PS> $file = "
http://download.microsoft.com/download/D/0/E/D0E6D2C1-2593-401
7-B26D-7375BC9263D5/PowerShell_Setup_amd64.msi"

[6]PS> # This creates a new BITS job running in the background.  After
[7]PS> # this command the file will get transferred even if
[8]PS> #    this process terminates
[9]PS> #    this machine (or the server) reboots.
[10]PS> #   the network goes down and comes back up.
[11]PS> New-FileTransfer -ServerFileName $file -ClientFileNamePrefix c:\Kits -
DisplayName PSAMD  -Asynchronous

JobId           DisplayName     TransferType    JobState       OwnerAccount
-----           -----------     ------------    --------       ------------
5652f872-c8d... PSAMD           Download        Connecting     RugratsVist...

[12]PS> $job = Get-FileTransfer PSAMD

[13]PS> # Let's take a look at what we have
[14]PS> $job |format-list *

JobId                   : 5652f872-c8db-4423-a28f-70c10e1600f4
DisplayName             : PSAMD
Description             : This is a transfer using Background Intelligent Tra
                          nsfer Service.
TransferType            : Download
JobState                : Transferring
OwnerAccount            : RugratsVista\jsnover
Priority                : Normal
MinimumRetryDelay       : 600
NoProgressTimeout       : 1209600
TransientErrorCount     : 0
ProxyUsage              : SystemDefault
ErrorContext            : None
ErrorCondition          : NoError
InternalErrorCode       : 0
ErrorDescription        :
ErrorContextDescription :
BytesTotal              : 10313216
BytesTransferred        : 1063897
FilesTotal              : 1
FilesTransferred        : 0
CreationTime            : 1/11/2009 9:03:06 AM
ModificationTime        : 1/11/2009 9:03:11 AM
TransferCompletionTime  : 1/1/0001 12:00:00 AM
FileList                : {
http://download.microsoft.com/download/D/0/E/D0E6D
                          2C1-2593-4017-B26D-7375BC9263D5/PowerShell_Setup_am
                          d64.msi}
ProxyList               :
ProxyBypassList         :

[15]PS> # Notice that you don't see anything until the job is complete
[16]PS> # Actually, in -Asynchronous mode, you want see anything
[17]PS> # until *YOU* complete the FileTransfer (with a Complete-FileTransfer)

[18]PS> dir c:\Kits\power*

[19]PS> # The transfer is smart about bandwidth utilization and will
[20]PS> # scale back when it detects interactive use or competition for
[21]PS> # the network.  You can control this with the -Priority flag
[22]PS> # which takes values:
[23]PS> #    Forground, High, Normal, Low
[24]PS> # You can also manually suspend the job for any reason
[25]PS> Suspend-FileTransfer $job

JobId           DisplayName     TransferType    JobState       OwnerAccount
-----           -----------     ------------    --------       ------------
5652f872-c8d... PSAMD           Download        Suspended      RugratsVist...

[26]PS> $job

JobId           DisplayName     TransferType    JobState       OwnerAccount
-----           -----------     ------------    --------       ------------
5652f872-c8d... PSAMD           Download        Suspended      RugratsVist...

[27]PS> # This is how you resume the FileTransfer.
[28]PS> Resume-FileTransfer $job -Asynchronous

JobId           DisplayName     TransferType    JobState       OwnerAccount
-----           -----------     ------------    --------       ------------
5652f872-c8d... PSAMD           Download        Connecting     RugratsVist...

[29]PS> $job

JobId           DisplayName     TransferType    JobState       OwnerAccount
-----           -----------     ------------    --------       ------------
5652f872-c8d... PSAMD           Download        Transferred    RugratsVist...

[30]PS> # When the job is complete, you have to use this command to
[31]PS> # make the files show up and to clean up the job
[32]PS> Complete-FileTransfer $job

[33]PS> dir c:\Kits\power*

    Directory: C:\Kits

Mode                LastWriteTime     Length Name
----                -------------     ------ ----
-a---        12/15/2008   5:08 AM   10313216 PowerShell_Setup_amd64.msi

Everyone should use BITS.

Experiment, Enjoy, Engage.

Jeffrey Snover [MSFT]
Windows Management Partner Architect
Visit the Windows PowerShell Team blog at:    http://blogs.msdn.com/PowerShell
Visit the Windows PowerShell ScriptCenter at:  http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx

Leave a Comment
  • Please add 7 and 1 and type the answer here:
  • Post
  • I just posted a blog about our new File Transfer Cmdlets in which I artfully dodged a difficult question:

  • "the servers were having a hard time keeping up."

    Imagine if I could download some of the data from other regular folks who already have it, while they download some of the data from me?  Think how much less load there could be on the server!

    I think I'll make this, and call it... BITS-torrent.

  • From a usability standpoint, having to run Complete-FileTranfer in -Asynchronous mode kinda defeats the purpose of it being Asynchronous. I would be much happier if I didn't have to do that. Otherwise, this functionality rocks.

  • I am just getting started with PowerShell so excuse what might be a simple question. In your blog you say:

    "[3]PS> # This is the HTTP file that I want to download.  I could have also used an UNC Path "

    What is the required setup on the source and destination devices for this to work with a UNC.  The BITS doc I have found so far indicates that IIS is required on at least one end of the copy (assuming the need for HTTP service) and does not mention using UNC.  I have some exceedingly large files to copy and this may be the ticket for guarenteed delivery without having the bandwidth police looking for my head on a platter.

    thx

    hrb

  • hrb - in response to your concern about consuming too much bandwidth, I'm not sure how to do it with PSv2 and BITS, but you can do it today with robocopy.exe in the resource kit.

    Robocopy has a parameter specifically for slowing down the transfer and it can restart a large transfer if it fails.

    /IPG:n :: Inter-Packet Gap (ms), to free bandwidth on slow lines.

    Jeffrey,

    It would sure be nice to have a property on the file transfer Job called "PercentComplete". I know it can be computed with simple division (BytesTransferred/BytesTotal), but since it's likely to be one of the most common things folks look for storing it as a property would be very nice.

    Can't wait to try it out!

    -Chris

  • I was in a meeting yesterday talking about BITS and what an awesome service it was.  I described

  • As someone who does a far amount of downloading of stuff – especially technical documents etc I was interested

  • Je viens de le voir dans un post sur le blog du team PowerShell , et un autre sur le blog WMI : PowerShell

  • It seems you cannot download files using BITS from an IIS website which requires client certificates? Downloading over SSL with the server ignoring client certificates works fine but not when client certificates are required. Is this true?

  • Query: I am using the windows 7 (build 7100 .. RC). I cannot import the module there. I thought it was coming with the latest Powershell ... will Win7 include this module?

    When I download the CTP 3 and install in a Vista (because I cannot uninstall in Win 7) it works.

    Tks in advance

    Best

  • The module has been renamed, FileTransfer is now BitsTransfer

  • Yeah .. I saw this information somewhere :) Thank you.

    Taking advantage of the message.

    My application use BITS ... and I use the SetNotifyCmdLine when it runs ok http://msdn.microsoft.com/en-us/library/aa362988(VS.85).aspx.

    Is there any way to do this in commandlets? If I understood right the start-bitstransfer accepts just files to be downloaded or uploaded. Is that right? How can I indicate to run an EXE if the transmission was ok?

    Regards

    Steven Alexander

  • This is great! However, it would be nice if I could do it remotely. I run a lab with 30 computers and we regularly push large files out from a server to all the machines. Unfortunately, BITS for some reason (probably security) does not work when remotely invoking commands.

  • I see it doesn't currently support downloading from FTP sites. is there any plans to include FTP transfers too?

  • Wish that bits had an option for unbuffered copy operations. As it is this is virtually useless for moving truly large (> 4 or 5 GB) files.

    Unfortunately the classic buffered/unbuffered I/O problem strikes again when using this. Its a shame because I like the resumability, hash check at end, etc that bits provides. But I'm writing a utility to move large backup files around (4 up to 100+ GB) on a backplane and this simply won't cut it. For now I'm going to have to go back to Eseutil or maybe Msoft's RichCopy and call those from power shell. :(

Page 1 of 2 (19 items) 12