AzCopy – Using Cross Account Copy Blob

AzCopy – Using Cross Account Copy Blob

Rate This
  • Comments 13

You can always download latest version from aka.ms/AzCopy.

New features have been added in this release

  • Support of Cross-Account Copy Blob: AzCopy allows you to copy blobs within same storage accounts or between different storage accounts (visit this blog post for more details on cross account blob copy). This enables you move blobs from one account to another efficiently with respect to cost and time. The data transfer is done by the storage service, thus eliminating the need for you to download each blob from the source and then upload to the destination. You can also use /Z to execute the blob copy in re-startable mode.
  • Added /MOV: This option allows you move files and delete them from source after copying. Assuming you have delete permissions on the source, this option applies regardless of whether the source is Windows Azure Storage or the local file system.
  • Added /NC: This option allows you to specify the concurrent network calls. By default, when you upload files from local computer to Windows Azure Storage, AzCopy will initiate network calls up to eight times the number of cores this local computer had to execute concurrent tasks. For example, if your local computer has four cores, then AzCopy will initiate up to 32 (eight times of 4) network calls at one time. However, if you want to limit the concurrency to throttle local CPU and bandwidth usage, you can specify the maximum concurrent network calls by using /NC. The value specified here is the absolute count and will not be multiplied by the core count. So in the above example, to reduce the concurrent network calls by half, you would specify /NC:16
  • Added /SNAPSHOT: This option allows you transfer blob with snapshots. This is a semantic change, as in AzCopy CTP 1 (Released in October, 2012), it would transfer a blob’s snapshots by default.  However starting from this version, by default AzCopy won’t transfer any snapshots while copying a blob. Only with /SNAPSHOT specified, AzCopy will actually transfer all the snapshots of a blob to destination, however these snapshots will became separate blobs instead of snapshots of original base blob in destination, so each of the blobs will be full charged for (not block reuse between them). The transferred blob snapshots will be renamed in this format: [blob-name] (snapshot-time)[extension].

    For example if readme.txt is a source blob and there are 3 snapshots created for it in source container, then after using /SNAPSHOT there will be 3 more separate blobs created in destination container where their names will looks like

    readme (2013-02-25 080757).txt
    readme (2012-12-23 120657).txt
    readme (2012-09-12 090521).txt

    For the billing impact compare blob snapshots to separate blob, please refer to this blog post Understanding Windows Azure Storage Billing
  • Added /@:response-file: This allows you store parameters in a file and they will be processed by AzCopy just as if they had been specified on the command line. Parameters in the response file can be divided as several lines but each single parameter must be in one line (breaking 1 parameter into 2 lines is not supported). AzCopy parses each line as if it is a single command line into a list of parameters and concatenates all the parameter lists into one list, which is treated by AzCopy as from a single command line. Multiple response files can be specified, but nested response file is not supported, instead it will be parsed as a location parameter or file pattern. Escape characters are not supported, except that “” in a quoted string will be parsed as a single quotation mark. Note that /@: after /- does not means a response file, it is treated as a location parameter or file pattern like other parameters

Examples

Here are some examples that illustrate the new features in this release.

Copy all blobs from one container to another container under different storage account

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /S

The above command will copy all blobs from the container named “sourcecontainer” in storage account “sourceaccount” to another container named “destcontainer” in storage account “destaccount”

If you have base blob with snapshots, please add/Snapshot to move all snapshots with base blob to destination, please be noted the blob snapshot will be renamed to this format in destination: [blob-name] (snapshot-time)[extension]

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /S /SNAPSHOT


For example if you have readme.txt with 3 snapshots in source container, then it will be as below in destination container
readme.txt

readme (2013-02-25 080757).txt
readme (2012-12-23 120657).txt
readme (2012-09-12 090521).txt

If you’d like to delete those blobs from the source container when the copy is complete, then just added /MOV as below

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /MOV /S

You can also create a response file to make it easier of running same command again and again. Create a txt file called “myAzCopy.txt” with content below

#URI of Source Container
https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/
#URI of Destination Container
https://<destaccount>.blob.core.windows.net/<destcontainer>/

Then you can run the command below to transfer files from source container to destination container

AzCopy /@:C:\myAzCopy.txt /sourcekey:<key> /destkey:<key> /MOV /S

Leave a Comment
  • Please add 3 and 3 and type the answer here:
  • Post
  • Hello,

    Thanks a lot for this amazing tool. Is there any plan to add a timeout switch to enable large file copy (by increasing it). Or, will you release the source code so we can submit contribution to this tool.

    Best regards,

    Cyprien

  • Hello, Does this copy also the Blob metadata? Thanks.

  • @Ronen - Yes it copies the metadata. It internally uses (async) CopyBlob across storage accounts see msdn.microsoft.com/.../dd894037.aspx for details on other properties that the API sets.

    @Sandorfr - Timeout switch is a good request. We will add it to the list of feature requests.

  • Is there a companion command to get the names of the files that are in the source container?

  • @Jeff - The is no separate companion command, however you can use the /L and /V parameters together get AzCopy to display the files that would be copied.

    AzCopy localFolder account.blob.core.windows.net/somecontainer * /DestKey:xxxxxxxxxx /L /V

  • I am not sure what I am doing wrong but nothing I try seems to work. It doesn't throw any errors but no matter what I try it always prints the following:

    Transfer summary:

    -----------------

    Total files transferred: 0

    Transfer successfully:   0

    Transfer failed:         0

    I tried copying blobs between storage accounts in the same subscription. I tried copying blobs between storage accounts in different subscriptions. I tried making the blob containers public. I tried blob URLs with and without SSL. I tried specifying a single blob in the source and a container in the destination. I tried specifying containers in the source and destination. I tried with and without trailing slash in the container path. All to no avail. It resolutely produces the above given output.

    I tried running it with Fiddler on and all I see is 2 blob listing calls for the source and destination accounts with nothing after that. At this point I am ready to give up and write my own thing. Is the source for this tool up on github? I tried searching in the Windows Azure github account and I couldn't find it there!

  • @Raj - Source of this tool is not yet opened to public. Could you please try to add /S to see whether it resolves your issue?

    AzCopy sourceaccount.blob.core.windows.net/sourcecontainer destaccount.blob.core.windows.net/destcontainer /SourceKey:****** /DestKey:****** /S

  • What is the Microsoft.WindowsAzure.Storage.DataMovement assembly that ships with AzCopy?  There doesn't seem to be any information out there on it, other than it exists.

  • @Reggie

    This Data Movement assembly is an internal library used by AzCopy today.

    We are considering to expose this library so that customer can introduce more data movement capabilities in their own applications,

    however we don’t have a date to share yet.

  • Any word yet on sharing the code or assembly for this?

    Honestly, you are getting throughputs that are 3X-5X faster than we get with our code.

    Thanks,

    Mark

  • Hi Mark,

    We will consider to open source AzCopy in the future, but do not have a timeline yet.

    Regards

    Zhiming

  • Is there a way to know how many threads AzCopy is executing.  I started to copy 2 small files (3GB each) from a directory as a test and I'm trying to get some statistics.  I'm using /V switch but I don't believe it will provide the statistics I need.

    - How many threads is AzCopy executing

    - Time taken to copy the files

  • Hi Gus,

    Thanks for your feedback.

    AzCopy will by default start CORE*8 concurrent operations to copy the blobs, you can use option /nc to change the number.

    And AzCopy does not show ‘time taken to copy the files’, we will add it to the list of feature request.

    Zhiming

Page 1 of 1 (13 items)