WHat's wrong with this code, part 9 (File Copy) - the answers

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Joku — Thu, 10 Feb 2005 19:41:00 GMT

I do not know if you are coding a new algorithm for Long.... but here's my take and then a question.

If the case is that the source file size is, say, less than half of the available physical memory, then use apply logic:

IF source & dest can be determined to be on the same physical drive, see if the file is fragmented, if it is, then copy as normally and let NCQ handle the head position ordering. If file is in say less than 10 fragments, have the file read into the file cache to the point that half of the avail physical memory is used. Now start writing the file, stopping reading until writing is done. Handle as transaction.

The question I have is, I have not yet found a copy program that uses memory to speed up the copy, instead the programs seem to prefer reading small blocks and have the head seek back and forth the drive. Even if there is gigabyte of physical memory free. Now is this just bad design or can you give some good insight to why the available memory is not used to full/even half extent?

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Larry Osterman — Thu, 10 Feb 2005 19:47:00 GMT

I'm certainly not messing with CopyFile, that's the base team's job, I'm in multimedia :)

Those might be reasonable assumptions, I'd have to think about it a bit...

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Capt. Jean-Luc Pikachu — Thu, 10 Feb 2005 19:58:00 GMT

Arg, I misread <a href="http://www.parashift.com/c++-faq-lite/freestore-mgmt.html#faq-16.5">the FAQ</a>. Also, I just recently started reading, so I wasn't aware of your view on exceptions. ^^;;

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Stuart Dootson — Thu, 10 Feb 2005 19:59:00 GMT

One small point about what you said about 'operator new' - it's not just #include <new> that causes use of a throwing new - any C++ library header will causes this (so long as it brings int eh C++ runtime libs). It's a small point (I thought you were more wrong, but I read the VS.NET 2003 docs and learnt what the actual situation is - which is nice, learning something new everyday is always good), but I'd guess you'll find there are a lot of people who include some part of the C++ runtime and might still be checking that new functions correctly.

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Anon — Thu, 10 Feb 2005 21:00:00 GMT

I'd really like to know more about memory allocation vs. scalability, if you ever run out of topics :-) My first encounter with the problem was a while ago: I noticed that the behaviour of the application I was working on degraded significantly during subsequent repitions, and identified the cause as being due to excessive memory allocation and fragmentation.

I managed to tweak things by simply tidying up the number of times data was copied and being more careful about lifetime, but I didn't really get any satisfying advice from the internet: there are a lot of products being sold, and a lot of college-level articles about memory allocators, but nothing that gave sound advice for scalable memory management (except, of course, articles describing how GC mitigates these problems)

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Bob Frankston — Thu, 10 Feb 2005 21:21:00 GMT

I accidentally glanced at the comments and noticed the comment about 64 bit operations not being atomic. I guess I've been around too long and don't see any reason to assume any such operation is atomic. If the CLI doesn't make this promise then you can't assume it for any operation.

With HT processors we're going to see a lot of subtle bugs appear and they'll all be blamed on mysterious viruses or maybe ghosts left in the CPU from processes that died horrible deaths.

Seriously the current processors are the result of years of single threaded programming. Maybe we should make atomicity a well-defined part of the architecture instead of just assuming it because the program seems to work.

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Larry Osterman — Thu, 10 Feb 2005 22:13:00 GMT

Wow, Bob - long time no hear.

Your point is actually a huge issue. On our currently supported 32bit processors, 32bit aligned memory operations are guaranteed (by the processor) to be atomic. However, in general, you're totally right - on 64bit processors this is explictly NOT true - on those machines the rules on write ordering have been significantly relaxed, which can cause all sorts of issues.

I used a great deal of care writing the example to never have shared access to any variables - there is only one thread operating on any data structure (except for the one case I left in as a bug) at any time.

Your comment about the ordering issue is why I suggested using the InterlockedExchangeAdd API - it asserts a memory barrier after the add to ensure that all (logical and physical) processors have a consistant view of the data.

Btw, the biggest likely item to fail in the presence of misordered writes is the Double Checked Lock Pattern - here's a great discussion of why it fails on modern processors: http://www.cs.umd.edu/~pugh/java/memoryModel/DoubleCheckedLocking.html

The problem is that there is a lot of existant code that uses it that will start breaking in really subtle ways.

re: WHat's wrong with this code, part 9 (File Copy) - the answers

Raymond Chen — Fri, 11 Feb 2005 07:27:00 GMT

"The problem with this is that 64bit math isn't atomic on 32bit processors."

Not even 32-bit math is atomic on 32-bit processors. Given

int i = 0;

// Thread A
i += 2;

// Thread B
i += 3;

the final result might be i=2, or it could be i=3 or it could be i=5. if you want atomic arithmetic you must use an Interlocked function.

Larry gets taken to task on concurrency

Larry Osterman's WebLog — Fri, 11 Feb 2005 21:27:00 GMT

re: WHat's wrong with this code, part 9 (File Copy) - the answers

RJ — Tue, 15 Feb 2005 03:38:00 GMT

I especially like your mention of using unbuffered I/O. It can help on low memory computers (256 is low memory? <grin>).

When using WinRAR on a 4GB archive, my computer becomes unusable for the duration. No UI application is active enough to stay in memory. But altering the source, which they provide, to use sequential__scan on the input and unbuffered I/O on the output file prevents the problem.

Concurrency, part 11 - Hidden scalability issues

Larry Osterman's WebLog — Fri, 04 Mar 2005 20:48:00 GMT