Concurrency, Part 8 - Concurrency for scalability

re: Concurrency, Part 8 - COncurrency for scalability

Ricky Datta — Mon, 28 Feb 2005 21:31:00 GMT

If you give us an example of IO competion usage (File, sockets) using C# managed code - that will be great.

Ricky

re: Concurrency, Part 8 - Concurrency for scalability

Larry Osterman — Mon, 28 Feb 2005 21:32:00 GMT

Ricky, are the MSDN overlapped I/O and I/O completion port examples not sufficient? I could write something up but I'd think that there was enough sample code for that already on the web.

Or do you have something more specific in mind?

re: Concurrency, Part 8 - Concurrency for scalability

Ricky Datta — Mon, 28 Feb 2005 21:48:00 GMT

IO Completion ports are kind mystery..exotic to ordinary Win32 programmer. On the other hand, developer education of completion ports are essential for scalable windows server development.

CLR FX 2.0 implements IO completion ports, which is a high productivity devlopment environment. There is very little doc/samples/established examples of how to use the following :

AcceptEx (CLR version)
ConnectEx (CLR version)
TransmitFile (CLR version)
TransmitPackets (CLR version)
Asynchronus File/Socket operations

In my opinion asynchronous server techniques using IO completion are usually found in MS products, never in 3rd party apps.

If you could write about these, that would be great.

Ricky

re: Concurrency, Part 8 - Concurrency for scalability

Jerry Pisk — Mon, 28 Feb 2005 22:27:00 GMT

You don't need to worry about synchronization if the data you share is read only. But it has to be read only for the lifetime of your threads. An example would be a B-tree index of your static data (such as resource strings) that is only built once, at your application startup, before you fire off your worker threads.

re: Concurrency, Part 8 - Concurrency for scalability

Larry Osterman — Mon, 28 Feb 2005 22:30:00 GMT

Jerry, that's true. But that relies on the fact that the data is static. While this is likely to be true for some data sets (census data, for example) for others (Amazon.Com's book list) it's not.

re: Concurrency, Part 8 - Concurrency for scalability

Mike Dimmick — Mon, 28 Feb 2005 23:10:00 GMT

Ricky, I seem to recall there's a pretty good treatment of I/O Completion Ports - and sample code - in "Programming Server-Side Applications for Microsoft Windows" by Jeffrey Richter and Jason Clark. Unfortunately it's out of print - why, MS Press? This is an essential book! I note that Jeff's other Win32 book "Programming Applications for Microsoft Windows" is also out of print.

As for the .NET Framework 2.0, it's still in beta and a lot of the documentation is unfinished. Hopefully Beta 2 will be a lot better in that regard.

re: Concurrency, Part 8 - Concurrency for scalability

Eric Lippert — Mon, 28 Feb 2005 23:52:00 GMT

Good series Larry!

> But if your application is CPU bound then the obvious solution is to throw more CPUs at the problem.

That's one obvious "solution" that turns out to not always work. Another obvious "solution" is to performance tune the application so that each operation is faster, AND throw more CPUs at the problem.

Counter-intuitively, making the processor do less work on a given thread can actually DECREASE your performance, or at least wreck scalability. Scalability is really, really hard to wrap your mind around.

I wrote up an analogy that shows how speeding can slow you down here:

http://weblogs.asp.net/ericlippert/archive/2003/12/01/53411.aspx

re: Concurrency, Part 8 - Concurrency for scalability

Drew — Tue, 01 Mar 2005 00:08:00 GMT

<pedantic>
The singular is quantum. Quanta is plural. Because a slice of CPU time is neuter although I have no idea why.
</pedantic>

I'm also really enjoying this series. And not just because of the irony inherent in a series on parallelism. I'm eagerly awaiting tomorrow's installment.

re: Concurrency, Part 8 - Concurrency for scalability

SeanH — Tue, 01 Mar 2005 00:15:00 GMT

Jerry, Larry mentioned writing scalable concurrent applications need a paradigm-shift in terms of how software developers think about writing applications, and I think one of these shifts is the concept of shared data and how we access it.

I think writing scalable concurrent applications will mean having minimal access to the shared data. My guess is, you will have one huge container with a well-staged way of modify contents of that container through some sort of message passing mechanism where data to be modified is deep copied and wrapped around a message that is sent to a worker thread and the modifications sent back through another message. The container will then be in charge of determining which modifications will be committed.

re: Concurrency, Part 8 - Concurrency for scalability

Larry Osterman — Tue, 01 Mar 2005 00:33:00 GMT

Drew, you're 100% right. But every discussion I've ever had with the kernel group refers to the unit of scheduling as a quanta, not quantum. I don't get it, but...

re: Concurrency, Part 8 - Concurrency for scalability

Mike Flasko — Tue, 01 Mar 2005 08:22:00 GMT

Larry, I agree with Ricky, a series on IO Completion ports would be great.

As for multithreading I typically use the following train of thought when developing:

1. identify tasks suitable to a secondary thread
2. identify shared data (among threads)
3. of that shared data which of the data is writable (aka. find the race conditions)
4. provide mutual exclusion for the variables in step 3.

** 5. If necessary, ensure my critical sections do not lead to thread starvation.

re: Concurrency, Part 8 - Concurrency for scalability

MartinK — Tue, 01 Mar 2005 10:45:00 GMT

This series gets better and better. Two things that I'd welcome a bit of expansion of:

1. "Avoid synchronous I/O because it blocks". Out of interest, is this often true of WriteFile? Reading Inside Windows 2000, it looks as if WriteFile practically never writes a file, just schedules a lazy write and returns at once... in which case safe to use, surely?
2. Does the deep scheduling stuff inside Windows work? I'm not being offensive, it's just that we all know that concurrent code is virtually impossible to debug, and I don't want to spend 24 hours chasing an evanescent bug in my own code and then discover that it is one of those "edge cases in ntdll.dll that were never thought through properly" [Rob Earhart]. This may be why it's mostly MS that use their own completion ports and so on... the internal company "lore" will tell them which documented features have been well used and which ones haven't been used enough to be safe to rely on.
3. [ok, so I can't count!] A server app can balance the work it does for its clients in all sorts of ways, intelligently allocating CPU, memory, disk I/O... but the one thing it seems impossible to balance is bandwidth back to the clients. Example: my protocol splits big messages into a stream of smaller packets, the idea being that a small message can insert itself into the stream between those packets - helps responsiveness at the client end. Also there is then some hope of bandwidth allocation so that a client who's just requested 10MB won't lock out lots of other clients who only want 1KB each to get on with their work.
But - Windows provides massive buffers for socket output and happily hoovers up huge quantities of data that it then queues. Good for raw performance, but it makes for hellish latency [the insert-a-quick-little-packet solution won't work when there are already megabytes in the queue "owned" by Windows] and for no hope of balancing the bandwidth requirements of multiple clients.
I'm probably missing something here - one little API or callback or port or something - so any pointers would be welcome.

re: Concurrency, Part 8 - Concurrency for scalability

Larry Osterman — Tue, 01 Mar 2005 14:47:00 GMT

Martin, for #1: If you're doing a synchronous write, the I/O subsystem takes a lock around each file object - which might block your thread if someone else is accessing the file object. The fast I/O path won't typically take many locks, you're right. But you're not guaranteed you'll take the fast I/O path.

#2: The NT scheduler's not bad, w.r.t. scalability, and it's getting better all the time. Earhart reads this blog (at least he comments on here :)), he may be able to answer better than I.

#3: Let me look into that one, I'm not sure that there are solutions for rate metered delivery on the server side (other than setting the socket buffer size to something small)

re: Concurrency, Part 8 - Concurrency for scalability

Skywing — Tue, 01 Mar 2005 16:03:00 GMT

Synchronous WriteFile will block after you've written more data than the cache manager will allocate for a particular file. You should not rely on it completing instantly, particularly when the system is under load.

I use I/O completion ports in my network server apps and I've not yet run into any problems with them.

re: Concurrency, Part 8 - Concurrency for scalability

Andreas Häber — Tue, 01 Mar 2005 19:51:00 GMT

MartinK, think I can help you with #3:
Winsock uses an algorithm named Nagel which causes this buffering. If you enable TCP_NODELAY (through setsockopt or any equivalent) then Winsock will send packages straight away.

Also you can change the send and receive buffers through setsockopt.

For more/better information see http://msdn.microsoft.com/library/default.asp?url=/library/en-us/winsock/winsock/setsockopt_2.asp

re: Concurrency, Part 8 - Concurrency for scalability

MartinK — Wed, 02 Mar 2005 11:02:00 GMT

Andreas, thanks. I'm not worried about small-packet latency. It's the opposite problem: if Windows lets me output 1MB of data to a socket that's connected to a dial-up client, then anything I send immediately afterwards will take at least 150 seconds to arrive. Having smaller buffers would indeed be the answer, so that my application would be able to make intelligent decisions about which packets of data to send when.
Unfortunately there's more buffering inside Windows than just the socket buffers. I've run a test: getsockopt() reports a send buffer size of 8192 bytes, but a call to send() that asks to send 1000000 bytes returns immediately, reporting that it's sent 1000000 bytes, not 8192.

Concurrency, Part 9 - APIs that enable scalable programming

Larry Osterman's WebLog — Wed, 02 Mar 2005 20:37:00 GMT

Concurrency, Part 9 - APIs that enable scalable programming

Larry Osterman's WebLog — Wed, 02 Mar 2005 20:45:00 GMT