Processing Streams with PLINQ

re: Processing Streams with PLINQ

leniel — Mon, 06 Jul 2009 07:56:22 GMT

Hi igoro,

It's great to know this is possible with PLINQ! :)

Thanks for sharing this advice.

Leniel Macaferi

http://leniel.net

re: Processing Streams with PLINQ

Aaron — Mon, 06 Jul 2009 11:14:43 GMT

"Thus, a query that contains an OrderBy operator does not (and cannot) run in a streaming fashion"

Um, what? You might not _want_ to run it in a stream fashion, but maybe _I_ do. For example, expr.OrderBy(x => x).Take(3) need only partially sort the sequence. Yes, OrderBy reads the whole input stream. But streaming as lazy processing need not sort the whole input before providing the first result.

"the algorithms we use are optimized for throughput rather than latency"

In this context, OrderBy's eager behavior makes sense. Whole-sequence sort is always going to be faster than a complete set of partial sorts, by some constant factor.

re: Processing Streams with PLINQ

Robert Westerlund — Mon, 06 Jul 2009 17:42:12 GMT

Aaron:

I might be missunderstanding something, and if I do please let me know, but as far as my understanding goes, it is correct to say that you cannot run a sorting algorithm without having all values.

Given your example, expr.OrderBy(x => x).Take(3) in a situation where expr would stream its return value in two chunks. Suppose that the OrderBy would:

1) Sort the initial chunk before the second chunk arrives

2) Return the sorted items as soon as it has sorted them (in this case, before receiving the second chunk)

Given that we would get the following two chunks (3,6,5,7) and (8,1,9) the OrderBy-method would sort the first chunk (resulting in 3,5,6,7) and the Take(3) would return 3,5,6. As can be seen, this behaviour would not be correct, since a correct result in the end would have been 1,3,5. Thus, if we would have been handed the 3 as first result, it would have given us the wrong result and thus not successfully sorted the items.

Therefore, OrderBy needs access to all items in order to be able sort correctly.