Automating the world one-liner at a time…
Have you had a file copy fail? Does that drive you crazy or what? How about when it is a REALLY big file and it takes a couple of hours and JUUUUUST before it finishes something happens to interrupt the transfer and you have to start all over again. Does that make you want to shove a sharpened #2 pencil up your nose or what?
I got to thinking about these things when I saw that the W7 and WS08/R2 beta bits were available and that the demand was so high that the servers were having a hard time keeping up. Can you image how you would react if you downloaded a GB and then the transfer failed? Can’t someone HELP? Can’t someone give us the get semantics of Windows Update but for files that **I** care about?
Funny you should mention that.
Have I mentioned that PowerShell V2 rocks? In CTP3, we are exposing a new set of Cmdlets that give you access to BITS. BITS stands for Background Intelligent Transfer Service. BITS is an awesome service. It is the special sauce behind Windows Update. Have you ever gone to WU and seen a couple hundred MB service pack and wondered, “how is every PC on the planet going to get a couple hundred MB without causing the Internet to melt?” Well BITS is the answer to that question. We think everyone should be using BITS. That means both ISV products and end users. Whenever you use BITS you get a better experience. You get reliable transfers, adaptive network bandwidth usage to minimize the impact on your system and you get peer caching which minimizes traffic. You can learn more about BITS by downloading a document HERE.
BTW - It turns out that we have been shipping the ability to do this for quite a while through a WIN32 command called BITSADMIN but few know about it and it is difficult to use. We think we still have usability issues with these Cmdlets so we might change them before we ship.
[0]PS> # You need to import the FileTransfer Module to get BITS cmdlets [1]PS> # Notice that the NOUN is FileTransfer [2]PS> Import-Module FileTransfer -Verbose VERBOSE: Importing cmdlet 'Add-FileTransfer'. VERBOSE: Importing cmdlet 'Clear-FileTransfer'. VERBOSE: Importing cmdlet 'Complete-FileTransfer'. VERBOSE: Importing cmdlet 'Get-FileTransfer'. VERBOSE: Importing cmdlet 'New-FileTransfer'. VERBOSE: Importing cmdlet 'Resume-FileTransfer'. VERBOSE: Importing cmdlet 'Set-FileTransfer'. VERBOSE: Importing cmdlet 'Suspend-FileTransfer'.
[3]PS> # This is the HTTP file that I want to download. I could have [4]PS> # also used an UNC Path [5]PS> $file = "http://download.microsoft.com/download/D/0/E/D0E6D2C1-2593-401 7-B26D-7375BC9263D5/PowerShell_Setup_amd64.msi"
[6]PS> # This creates a new BITS job running in the background. After [7]PS> # this command the file will get transferred even if [8]PS> # this process terminates [9]PS> # this machine (or the server) reboots. [10]PS> # the network goes down and comes back up. [11]PS> New-FileTransfer -ServerFileName $file -ClientFileNamePrefix c:\Kits - DisplayName PSAMD -Asynchronous
JobId DisplayName TransferType JobState OwnerAccount ----- ----------- ------------ -------- ------------ 5652f872-c8d... PSAMD Download Connecting RugratsVist...
[12]PS> $job = Get-FileTransfer PSAMD
[13]PS> # Let's take a look at what we have [14]PS> $job |format-list *
JobId : 5652f872-c8db-4423-a28f-70c10e1600f4 DisplayName : PSAMD Description : This is a transfer using Background Intelligent Tra nsfer Service. TransferType : Download JobState : Transferring OwnerAccount : RugratsVista\jsnover Priority : Normal MinimumRetryDelay : 600 NoProgressTimeout : 1209600 TransientErrorCount : 0 ProxyUsage : SystemDefault ErrorContext : None ErrorCondition : NoError InternalErrorCode : 0 ErrorDescription : ErrorContextDescription : BytesTotal : 10313216 BytesTransferred : 1063897 FilesTotal : 1 FilesTransferred : 0 CreationTime : 1/11/2009 9:03:06 AM ModificationTime : 1/11/2009 9:03:11 AM TransferCompletionTime : 1/1/0001 12:00:00 AM FileList : {http://download.microsoft.com/download/D/0/E/D0E6D 2C1-2593-4017-B26D-7375BC9263D5/PowerShell_Setup_am d64.msi} ProxyList : ProxyBypassList :
[15]PS> # Notice that you don't see anything until the job is complete [16]PS> # Actually, in -Asynchronous mode, you want see anything [17]PS> # until *YOU* complete the FileTransfer (with a Complete-FileTransfer)
[18]PS> dir c:\Kits\power*
[19]PS> # The transfer is smart about bandwidth utilization and will [20]PS> # scale back when it detects interactive use or competition for [21]PS> # the network. You can control this with the -Priority flag [22]PS> # which takes values: [23]PS> # Forground, High, Normal, Low [24]PS> # You can also manually suspend the job for any reason [25]PS> Suspend-FileTransfer $job
JobId DisplayName TransferType JobState OwnerAccount ----- ----------- ------------ -------- ------------ 5652f872-c8d... PSAMD Download Suspended RugratsVist...
[26]PS> $job
[27]PS> # This is how you resume the FileTransfer. [28]PS> Resume-FileTransfer $job -Asynchronous
[29]PS> $job
JobId DisplayName TransferType JobState OwnerAccount ----- ----------- ------------ -------- ------------ 5652f872-c8d... PSAMD Download Transferred RugratsVist...
[30]PS> # When the job is complete, you have to use this command to [31]PS> # make the files show up and to clean up the job [32]PS> Complete-FileTransfer $job
[33]PS> dir c:\Kits\power*
Directory: C:\Kits
Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 12/15/2008 5:08 AM 10313216 PowerShell_Setup_amd64.msi
Everyone should use BITS.
Experiment, Enjoy, Engage.
Jeffrey Snover [MSFT] Windows Management Partner Architect Visit the Windows PowerShell Team blog at: http://blogs.msdn.com/PowerShell Visit the Windows PowerShell ScriptCenter at: http://www.microsoft.com/technet/scriptcenter/hubs/msh.mspx
I just posted a blog about our new File Transfer Cmdlets in which I artfully dodged a difficult question:
"the servers were having a hard time keeping up."
Imagine if I could download some of the data from other regular folks who already have it, while they download some of the data from me? Think how much less load there could be on the server!
I think I'll make this, and call it... BITS-torrent.
From a usability standpoint, having to run Complete-FileTranfer in -Asynchronous mode kinda defeats the purpose of it being Asynchronous. I would be much happier if I didn't have to do that. Otherwise, this functionality rocks.
I am just getting started with PowerShell so excuse what might be a simple question. In your blog you say:
"[3]PS> # This is the HTTP file that I want to download. I could have also used an UNC Path "
What is the required setup on the source and destination devices for this to work with a UNC. The BITS doc I have found so far indicates that IIS is required on at least one end of the copy (assuming the need for HTTP service) and does not mention using UNC. I have some exceedingly large files to copy and this may be the ticket for guarenteed delivery without having the bandwidth police looking for my head on a platter.
thx
hrb
hrb - in response to your concern about consuming too much bandwidth, I'm not sure how to do it with PSv2 and BITS, but you can do it today with robocopy.exe in the resource kit.
Robocopy has a parameter specifically for slowing down the transfer and it can restart a large transfer if it fails.
/IPG:n :: Inter-Packet Gap (ms), to free bandwidth on slow lines.
Jeffrey,
It would sure be nice to have a property on the file transfer Job called "PercentComplete". I know it can be computed with simple division (BytesTransferred/BytesTotal), but since it's likely to be one of the most common things folks look for storing it as a property would be very nice.
Can't wait to try it out!
-Chris
I was in a meeting yesterday talking about BITS and what an awesome service it was.  I described
As someone who does a far amount of downloading of stuff – especially technical documents etc I was interested
Je viens de le voir dans un post sur le blog du team PowerShell , et un autre sur le blog WMI : PowerShell
It seems you cannot download files using BITS from an IIS website which requires client certificates? Downloading over SSL with the server ignoring client certificates works fine but not when client certificates are required. Is this true?
Query: I am using the windows 7 (build 7100 .. RC). I cannot import the module there. I thought it was coming with the latest Powershell ... will Win7 include this module?
When I download the CTP 3 and install in a Vista (because I cannot uninstall in Win 7) it works.
Tks in advance
Best
The module has been renamed, FileTransfer is now BitsTransfer
Yeah .. I saw this information somewhere :) Thank you.
Taking advantage of the message.
My application use BITS ... and I use the SetNotifyCmdLine when it runs ok http://msdn.microsoft.com/en-us/library/aa362988(VS.85).aspx.
Is there any way to do this in commandlets? If I understood right the start-bitstransfer accepts just files to be downloaded or uploaded. Is that right? How can I indicate to run an EXE if the transmission was ok?
Regards
Steven Alexander
This is great! However, it would be nice if I could do it remotely. I run a lab with 30 computers and we regularly push large files out from a server to all the machines. Unfortunately, BITS for some reason (probably security) does not work when remotely invoking commands.
I see it doesn't currently support downloading from FTP sites. is there any plans to include FTP transfers too?
Wish that bits had an option for unbuffered copy operations. As it is this is virtually useless for moving truly large (> 4 or 5 GB) files.
Unfortunately the classic buffered/unbuffered I/O problem strikes again when using this. Its a shame because I like the resumability, hash check at end, etc that bits provides. But I'm writing a utility to move large backup files around (4 up to 100+ GB) on a backplane and this simply won't cut it. For now I'm going to have to go back to Eseutil or maybe Msoft's RichCopy and call those from power shell. :(