Query Data with Parallel LINQ

DotNetKicks.com — Tue, 16 Dec 2008 09:18:58 GMT

You've been kicked (a good thing) - Trackback from DotNetKicks.com

re: Query Data with Parallel LINQ

Adam — Tue, 16 Dec 2008 15:56:47 GMT

I hope that the C# team will consider building this in behind the scenes. It seems to me that adding the parallel functionality to IEnumerable, ToList etc would not only make upgrades easier but further simplify the transition for developers.

re: Query Data with Parallel LINQ

James Wilkinson — Tue, 16 Dec 2008 16:15:51 GMT

I second Adam's comment - why not make it the default?

If your machine only has one processor, it'll be a linear operation, but on multi-core or multi-cpu, you'll get the benefits without having to write code to specifically target it.

Are there any gotchas that means it couldn't be the default? (other than the ParallelQuery<> return type..)

Always measure

Anders Borum — Tue, 16 Dec 2008 16:28:37 GMT

Hi Charlie,

an interesting article (as always). At the PDC 2008, I had a nice talk with the PM of the PLINQ team (who gave a really great presentation on the technology - of which most had already been made presented before, but still, it was nice to get the full overview and thoughts).

When we look at the typical examples today for parallel LINQ, most use fairly long working sets (including those presented in this post). The common scenarios I face in the ECM industry(enterprise content management), the working sets are comprised of fewer items - however, having each item require more complex processing (thus, still a canditate for parallelization).

I think it's important to communicate that parallelization may be of great use, even though you're facing smaller working sets of, say, 16 or 32 items. I overhead a few conversations at the PDC where other developers were unsure whether PLINQ would make a difference.

So, I guess the best answer is: "measure measure measure" (stopwatch approach or profiling - of which the latter usually yields interesting results because the hotspots are not always where you'd think).

With all the cores available, it would be really nice to see the PLINQ library have a lightning fast start-up, so that smaller and smaller working sets (with low processing requirements) can be run in parallel.

Perhaps a future CLR (version 5.0) would be able to make assumptions like this and parallelize when it makes sense. This would remove the requirement of explicit using the .AsParallel() etc. operators (although I personally find it important that this part of the code is explicit).

Query Data with Parallel LINQ

Public Sector Developer Weblog — Tue, 16 Dec 2008 16:59:29 GMT

Charlie Calvert published a blog entry on the subject. It is a nice, quick read on PLinQ. Check it out:

re: Query Data with Parallel LINQ

Juliano — Tue, 16 Dec 2008 17:22:12 GMT

I hope this in behind

I can't believe that's don't work by default, have no sense.

re: Query Data with Parallel LINQ

Marcel Popescu — Tue, 16 Dec 2008 17:38:28 GMT

Any plans of adding stuff from the CCR (http://msdn.microsoft.com/en-us/library/bb905450.aspx) in this library?

re: Query Data with Parallel LINQ

Nicholas Paldino [.NET/C# MVP] — Tue, 16 Dec 2008 20:13:15 GMT

This statement is incorrect:

Rather than returning IEnumerable<T>, this version of PLINQ returns IParallelEnumerable<int>:

IParallelEnumerable<T> ^is^ an IEnumerable<T> as it derives from IEnumerable<T>.

re: Query Data with Parallel LINQ

James Miles — Wed, 17 Dec 2008 04:50:02 GMT

Hi Guys,

Regarding;

>“I hope that the C# team will consider building this in behind the scenes. It seems to me that adding the parallel functionality to IEnumerable, ToList etc would not only make upgrades easier but further simplify the transition for developers.”

>“I second Adam's comment - why not make it the default?”

There are so many exciting technologies targeting the parallel computing problem space it is not yet clear what the best approach or solution is going to be. Anders has said as much in a number of interviews/posts.

PLINQ may turn out to be the solutions to all parallel programming problems, or they might just replace the CLR with a PCLR! ;)

Cheers,

James

re: Query Data with Parallel LINQ

DanielMoth — Wed, 17 Dec 2008 08:55:34 GMT

RE: "turn on by default" comment:

If your query mutates shared state (which is easily done) then PLINQ by default would yield incorrect results. Same goes for ordering expectations (if you have any). So, right now you need to opt-in for good reasons – only you know if the semantics will be preserved while gaining the speedup...

RE: CCR

We are looking at bringing a similar programming model into a future version of .NET, but that will not happen in v4.

Cheers

Daniel

re: Query Data with Parallel LINQ

Charlie Calvert — Thu, 18 Dec 2008 05:21:35 GMT

Thanks everyone for the great comments here.

James is correct. Anders H. has indeed said that the team is looking at all of these technologies, and will probably incorporate some of them into C# proper, once there has been a chance to see what solutions work best in which circumstances. That might take time.

Daniel Moth also raises good points about the issues surrounding mutable and immutable code and ordering expectations. It is not always possible to reliably detect these issues at compile time within a reasonable period of time, and that sets some restraints on our (and your) ability to detect which code is a good candidate for parallelism.

The comments by Anders Borum about the importance of measuring your code to see if it is yielding the results you expect are very relevant. The startup time issue, as is often the case with .NET code, is one that needs to be watched. The team, of course, is looking at this issue carefully.

Finally, remember that PLINQ sets the entry bar to writing concurrent code very low. This is just an introductory article to help you get started. As the comments shown here demonstrate, however, this is complex subject that deserves careful study. How does Andre Gide's old saying go: "Please don't understand me too quickly...." Use this article to help you get started, but continue to explore this subject as it develops over the coming months.

re: Query Data with Parallel LINQ

Macke — Fri, 19 Dec 2008 10:42:56 GMT

Its pretty obvious why this isnt made default, really. For example.

int i = 0;

foreach(int x in list)

x = i++;

this simple code would screw up pretty much since the elements doesnt come in a given order.

re: Query Data with Parallel LINQ

Gregory Kornblum — Sat, 20 Dec 2008 04:26:52 GMT

I am highly anticipating this. I can see this really improving our ability to handling multi-threaded collections. It is not like it is such a hard task without it but having much leaner code is always a good thing.

#.think.in infoDose #12 (15th Dec - 19th Dec)

#.think.in — Mon, 22 Dec 2008 01:47:56 GMT

#.think.in infoDose #12 (15th Dec - 19th Dec)

re: Query Data with Parallel LINQ

Erik Cox — Mon, 22 Dec 2008 17:07:13 GMT

It's nice to see some examples of LINQ and PLINQ queries....I mean this was a very interesting reading for me...I mean I knew very little about LINQ..but after reading your post I got a lot clearer picture about what LINQ really is..

Erik Cox

http://www.notionsolutions.com

re: Query Data with Parallel LINQ

Alexander — Fri, 26 Dec 2008 13:32:28 GMT

This looks the same as I thought of and build somewhat more than 4 years ago,.. off course not with LINQ. Nice to know that my ideas are shared by Microsoft :-)

re: Query Data with Parallel LINQ

RussellH — Thu, 08 Jan 2009 20:37:56 GMT

I have to confess that I don't get how PLINQ could lead to much anything but chaos if not used very carefully.

The trouble, I as see it, is that PLINQ appears to assume that the process running PLINQ is the only one on the box. It tries to monopolize all the processors on the box. Is that true? That would be bad enough on a client application, but it would be disastrous on a sever application such as an ASP.NET or SQL Server application.

My experience with modern desktop systems is that a user may have two or or even three monitors hooked up and may frequently have more than one application doing work at once. At any given time there may be two or three runnable threads that can be scheduled on a processor, so it scales fairly well. With PLINQ you have the possibility that each application will try to start as many threads as there are processors, and cause all kinds of overhead with task switching and scheduling. Performance is likely to be worse if the technology is not applied carefully.

In the case of sever code, I just can't see how PLINQ makes any sense at all unless the number of users of the application or web service is very small (like one user at a time). IIS and ASP.NET are already written to schedule multiple threads (in thread pools) to do processing in a way that uses available resources as fully as possible, while still attempting to be "fair" to any given client. The same goes for SQL Server or any multiple user database system. As someone who has written a fair amount of server code, I can say that doing this kind of thing right is hard. If you throw in a process that wants to start creating its own threads (PLINQ), you are likely to kill performance and scalability.

Maybe I don't understand enough about PLINQ yet, but it seems an absolute requirement that PLINQ be very aware of the environment the process is running it. It will need to know how many other processes are currently running on a system. It will need to know if it running on a server or client process.

The technology looks interesting, but I think Microsoft needs to be very careful about putting the barrier of entry too low.

re: Query Data with Parallel LINQ

Jason — Mon, 12 Jan 2009 21:24:55 GMT

My thoughts:

Due to the aforementioned issues (i.e. parallelism changing order of output), it should be necessary to add explicitly.

However, it certainly doesn't preclude a CodeRush or similar IDE enhancement from automatically listing use of LINQ (with reminders if order is missing) and making a conversion to .NET40 with parallelism much easier (i.e. similar to refactoring).

[re: RusselH]

I think you are overconcerned about the addition of an a low barrier to add parallelism to applications. Many applications have more than one thread and any available processor can handle a thread (i.e. the worries you have presented already exist). All PLINQ does is allow .NET to do opportunistic multi-threading (albeit explicitly I assume). The workload on a system doesn't change because you add the possibility of doing the work more efficiently. Sure the workload will be spread over more processors at once, but other processes are already doing this by their existence (i.e. having more processes than processors already creates the scenario you are worried about, notwithstanding many of them being multithreaded)

As far as having a lot of threads affecting user experience on the client, this issue already exists, and is already handled by:

1. Tuning already in place for client and server machines; i.e. dynamic thread priority giving foreground windows applications higher priority on client machines.

2. Thread priority in general.

That said, the multi-processor environment along with many applications using more parallelism could increase the need for adding better UI for tuning in the highly parallelized computing world (i.e. slider controls on Task Manager for adjusting priority; or an admustment on the window ctrl-menu to adjust it's priority; or new system wide options for allowing QoS metrics for processors).

I think PLINQ is the wrong place to address concerns which are more global to multi-processor / multi-threaded in general. However, should advanced tuning metrics make their way into the OS in general, then PLINQ should gain/provide exposure to them as well.

My 2 cents.

Jason L

re: Query Data with Parallel LINQ

Charlie Calvert — Wed, 14 Jan 2009 16:47:00 GMT

A couple quick clarifications:

* When you call AsParallel(), you can pass in the number of processers that you want to use. That is, you need not use all the processors on a particular system.

* Secondly, this is but one of several parallel programming initiatives that Microsoft is working on at this time. I think PLINQ is great because it is so easy to use, but it can be thought of as but one in a series of tools that is Microsoft developing in the parallel programming arena. I say this not because I'm backing off of my support for PLINQ, which I think is great, but just to be sure that everyone understands that this is not the companies only effort in this area. To learn more, visit the concurrency site, found as the first link near the top of this post.

re: Query Data with Parallel LINQ

Charlie Calvert — Wed, 14 Jan 2009 16:49:04 GMT

A couple quick clarifications:

* When you call AsParallel(), you can pass in the number of processers (threads) that you want to use. That is, you need not use all the processors on a particular system.

* Secondly, this is but one of several parallel programming initiatives that Microsoft is working on at this time. I think PLINQ is great because it is so easy to use, but it can be thought of as but one in a series of tools that is Microsoft developing in the parallel programming arena. I say this not because I'm backing off of my support for PLINQ, which I think is great, but just to be sure that everyone understands that this is not the company's only effort in this area. To learn more, visit the concurrency site, found as the first link near the top of this post

IEnumerable vs. IParallelEnumerable

Jason — Mon, 19 Jan 2009 20:14:32 GMT

@Nicholas Paldino:

I don't see how the statement mentioning PLINQ returns back a different interface is false. Look at the type of the implicit var: what is it? Answer: IParallelEnumerable<T>. The statement was not about polymorphism or contravariance in .Net. The statement didn't say you couldn't pass IParallelEnumerable<T> to other APIs expecting IEnumerable<T>.

On the contrary, it's very important that you know the type is now IParallelEnumerable<T> because if you do pass that type to other APIs or you reuse that type in another query, do know that PLINQ WILL BE USED IN THOSE QUERIES.

C# MVP? really?

re: Query Data with Parallel LINQ

Mikael — Tue, 10 Feb 2009 17:59:45 GMT

@Nicholas Paldino

Incorrect? In no way. Even though IParallelEnumerable<T> inherits from IEnumerable<T> (and, in essense, is IEnumerable<T>) there is nothing incorrect in pointing out that it actually returns IParallelEnumerable<T> and not _just_ IEnumerable<T>.

What is interesting here is the most derived interface, because the extension methods used by the compiler are hooked into that.

I would hope that someone who puts ".NET/C# MVP" after his name would understand this. As a simpler example; someone ordering a Coors Light would not be served in exactly the same way as someone ordering just a beer, even though a Coors Light is a beer (though I can see an argument being made against that).

re: Query Data with Parallel LINQ

Grant Rettke — Fri, 13 Feb 2009 18:40:27 GMT

Great post!

Thanks Charlie.