How to: Copy very large files across a slow or unreliable network

Grant Holliday’s blog

Senior Service Engineer, Microsoft Visual Studio Team Foundation Service

How to: Copy very large files across a slow or unreliable network

  • Comments 12

To prepare for the DevDiv TFS2010 upgrade we had to copy 8TB of SQL backups about 100 miles across a WAN link so that we could restore it on our test system.  The link speed was reasonably good and the latency fairly low (5ms), but when you’re dealing with files this big then the odds are against you and using sneakernet can be a good option. In our case it wasn’t an option and we had to find the next best solution.  In the end we were able to copy all 8TB over 7 days without having to resume or restart once.

The 8TB backups were spanned across 32 files of 250GB each which makes them a little easier to deal with.  The first problem that you’ll encounter when using a normal Windows file copy, XCopy, RoboCopy or TeraCopy to copy these large files is that your available memory on the source server will start to drop and eventually run out. The next problem you’ll encounter is the connection will break for some reason and you’ll have to restart or resume the transfer.

Fortunately the EPS Windows Server Performance Team have a blog post on the issue and a great recommendation: Ask the Performance Team : Slow Large File Copy Issues

The problem lies in the way in which the copy is performed - specifically Buffered vs. Unbuffered Input/Output (I/O).

Buffered I/O describes the process by which the file system will buffer reads and writes to and from the disk in the file system cache.  Buffered I/O is intended to speed up future reads and writes to the same file but it has an associated overhead cost.  It is effective for speeding up access to files that may change periodically or get accessed frequently.  There are two buffered I/O functions commonly used in Windows Applications such as Explorer, Copy, Robocopy or XCopy:

  • CopyFile() - Copies an existing file to a new file
  • CopyFileEx() - This also copies an existing file to a new file, but it can also call a specified callback function each time a portion of the copy operation is completed, thus notifying the application of its progress via the callback function.  Additionally, CopyFileEx can be canceled during the copy operation.

So looking at the definition of buffered I/O above, we can see where the perceived performance problems lie - in the file system cache overhead.  Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete.  This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data.  Many applications accomplish this by calling CreateFile() to create an empty destination file, then using the ReadFile() and WriteFile() functions to transfer the data.

  • CreateFile() - The CreateFile function creates or opens a file, file stream, directory, physical disk, volume, console buffer, tape drive, communications resource, mailslot, or named pipe. The function returns a handle that can be used to access an object.
  • ReadFile() - The ReadFile function reads data from a file, and starts at the position that the file pointer indicates. You can use this function for both synchronous and asynchronous operations.
  • WriteFile() - The WriteFile function writes data to a file at the position specified by the file pointer. This function is designed for both synchronous and asynchronous operation.

Which Tool? ESEUTIL

Yes, the tool has some limitations – but in my experience it’s well worth the time investment to get running. See How to Run Eseutil /Y (Copy File)

To get the utility, you need access to an Exchange server or to install Exchange in Administrator-only mode. When you install Exchange in Administrator-only mode, the appropriate binaries are copied to your computer and you can then copy these three files off and use them on another computer:

ese.dll
eseutil.exe
exchmem.dll

It does not accept wildcard characters (such as *.* to copy all files), so you must have to specify a file name and copy one file at a time. Or use a command like: FOR %f IN (d:\backups\*.BAK) DO ESEUTIL /Y "%f"

COPY FILE:
    DESCRIPTION:  Copies a database or log file.
         SYNTAX:  D:\BIN\ESEUTIL /y <source file> [options]
     PARAMETERS:  <source file> - name of file to copy
        OPTIONS:  zero or more of the following switches, separated by a space:
                  /d<file> - destination file (default: copy source file to
                             current directory)
                  /o       - suppress logo
          NOTES:  1) If performed on arbitrary files, this operation may fail
                     at the end of the file if its size is not sector-aligned.

Example

D:\>d:\bin\eseutil /y c:\Backups\Backup1.bak /d \\destination\c$\Backups\Backup1.bak

Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 08.01
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating COPY FILE mode...
     Source File: c:\Backups\Backup1.bak
Destination File: \\destination\c$\Backups\Backup1.bak

                      Copy Progress (% complete)

          0    10   20   30   40   50   60   70   80   90  100
          |----|----|----|----|----|----|----|----|----|----|
          ...................................................

Operation completed successfully in 7.67 seconds.

Other Tools

If you read the comments of the performance team’s blog post, you’ll see that XCopy has a /J option in Windows 7 and Windows 2008 R2 that does unbuffered I/O. However that’s not an option when you haven’t upgraded to R2 yet.

/J           Copies using unbuffered I/O. Recommended for very large files.

Which Direction?

Through trial and error, we determined that it was much more reliable to run eseutil.exe on the SOURCE server and push the files to the remote share.  This seemed to absorb any network blips and required no manual interruption over the 7 days it took us to copy the files.

Verifying hashes

The third problem you want to avoid is getting the files copied and then finding out that they match in size but the contents are corrupt.  You can check for this by generating hashes on both the source and target systems and comparing them after the copy.

You can download the Microsoft File Checksum Integrity Verifier fciv.exe tool from Microsoft Downloads.

Then run it like this on each system:

fciv.exe C:\Backups -type *.bak -r -wp -xml hashes.xml

  • Hi Grant,

    Thanks for the useful pointer.

    Is there any way you (or someone there) could sponsor getting the relevant files from the Exchange utility added to downloads.microsoft.com ?

  • Given that unbuffered copies are built-in to Win 2008 R2 and the Win 7 version of XCopy, there's not a great need for this.

    That said, I'll try and find an owner and see if it's possible.

  • Ehm,

    wouldn't it have been easier to just copy the 8 TB to other disks, put those disks in a box, put the box in a car, drive to other location, copy 8 TB on the host system?

    You could do the copy in parallel. Way faster and with the same verification just as secure.

  • That would be "sneakernet" (i.e., you transfer it using your feet), which he mentioned at the beginning wasn't an option.  I'm also curious why *not* exactly; it seem to me that sending it 100 miles with a trusted courier would take far less than a week (if you drove it yourself, you'd just need to take an afternoon off and you'd still get home in time for dinner...), plus it wouldn't saturate your connection during the 7 days it took to deliver digitally.

    Side question: was saturation a problem?  I.e, sending this unusually large amount of data flooding across the WAN during those 7 days?

  • It wasn't a saturation problem or a transportation problem - it was a security problem.  Only a handful of people are allowed physical access to the facility, so it wasn't an option to go and plug in a drives to the server or get on the LAN.

  • I use MiniCopier as a copy manager to copy, move files and folders to other locations. It lets me control and view  the copy process. The application supports the resuming of failed copy attempts.

    bstdownload.com/.../minicopier

  • How to determine when to use eseutil /y?  Should it only be used if the size of the file being copied is larger than the file system cache? Is file system cache same as virtual memory?  If not, how can I find out the size of the file system cache?

    When I try copying a 3GB  file using xcopy, xcopy /j, and eseutil /y in a Windows 7 64bit machine with 8G RAM and 11G Virtual Memory, xcopy is little bit faster than eseutil /y and xcopy /j is much SLOWER than both xcopy and eseutil /y.  

  • Additional info: The eseutil.exe I used is from exchange server 2003 so it is 32 bit version.

  • You'll know when you need eseutil: when all the others fail.

  • You may also wish to look at msoft's richcopy which supports unbuffered copy, resume, verify hash, fine tuning concurrent copying, etc.  

    blogs.technet.com/.../richcopy-bulk-file-copy-tool-released-get-it-here.aspx

    A fairly big (in my needs) downside is richcopy does not provide a good cli, i.e. its nearly impossible in my experience to integrate it into powershell etc. But its what Msoft uses internally in general for big copying.  

    Also, note that in at least pre win2k8 operating systems network copies are always buffered. I.e. you MUST run richcopy/eseutil etc from the source server to avoid the file system cache problem. Not sure if they've addressed the problem in network "pull" in win2k8 R2 or not (anyone know?)

  • oops, typo - "Also, note that in at least pre win2k8 operating systems network copies are always buffered."

    Should have said that network pulls (i.e. where destination server requests the file from source server via unc) will always be buffered in pre-win2k8

  • XCOPY /Z - restartable option in XP

Page 1 of 1 (12 items)