The Wayward WebLog : PDC

LINQ to SQL: The Mapping Engine

mattwar — Thu, 07 Jun 2007 03:11:00 GMT

The primary purpose of any ORM system is to map relational data onto objects in your programming environment. Mapping here refers to the meaning in the mathematical sense that there is a correspondence from one item to the other. For example, a database row might map to an object, or a field in a database might map to a property. Some mappings are simple, like the ones I’ve already mentioned; others are more complex such as parts of multiple rows combining to form a single object. ...(read more)

IQueryable’s Deep Dark Secret

mattwar — Fri, 01 Jun 2007 23:05:00 GMT

I love the IQueryable interface, but it’s got a dark checkered past that most of you might not know about. IQueryable is a great way to expose your API or domain model for querying or provide a specialized query processor that can be used directly by LINQ. It defines the pattern for you to gather-up a user’s query and present it to your processing engine as a single expression tree that you can either transform or interpret. It’s the way LINQ becomes ‘integrated’ for many LINQ to XXX products. Yet it was not supposed to be that way; with all that ease of use, plugging automatically into LINQ with an abundance of pre-written query operators at your disposal. You were not supposed to use it for your own ends. It was not meant for you at all. ...(read more)

The Origin of LINQ to SQL

mattwar — Thu, 31 May 2007 21:52:00 GMT

LINQ to SQL, possibly Microsoft’s first OR/M to actually ship in ten years of trying, was never even supposed to exist. ...(read more)

Oops, we did it again

mattwar — Thu, 11 May 2006 01:49:00 GMT

A new update to LINQ preview is available at MSDN site.

What's the coolest new feature? IMHO, its IQueryable<T>.

DLINQ's query mechanism has been generalized and available for all to use as part of System.Query. It implements the Standard Query Operators for you using expression nodes to represent the query. Your queries can now be truly polymorphic, written over a common abstraction and translated into the target environment only when you need it to.

public int CustomersInLondon(IQueryable<Customer> customers) {

int count = (from c in customers

where c.City == "London"

select c).Count();

return count;

}

Now you can define a function like this and it can operate on either an in memory collection or a remote DLINQ collection (or you own IQueryable for that matter.) The query is then either run entirely locally or remotely depending on the target.

If its a DLINQ query a count query is sent to the database.

SELECT COUNT(*) AS [value]

FROM [Customers] AS [t0]

WHERE [t0].[City] = @p0

If its a normal CLR collection, the query is executed locally, using the System.Query.Sequence classes definitions of the standard query operators. All you need to do is turn your IEnumerable<Customer> into IQueryable<Customer>. This is accomplished easily with a built-in ToQueryable() method.

List<Customer> customers = ...;

CustomersInLondon(customers.ToQueryable());

Wow! That was easy. But, how is this done? How can you possible turn my List<T> into some queryable thingamabob?

Good question. Glad you asked.

Check out this little gem:

Expression<Func<Customer,bool>> predicate = c => c.City == "London";

Func<Customer,bool> d = predicate.Compile();

Now you can compile lambda expressions directly into IL at runtime!

ToQueryable() wraps your IEnumerable<T> in IQueryable<T> clothing, uses the Queryable infrastructure to let you build up your own expression tree queries, and then when you enumerate it, the expression is rebound to refer to your IEnumerable<T> directly, the operators rebound to refer to System.Query.Sequence, and the resulting code is compiled using the built-in expression compiler. That code is then invoked producing your results.

Amazing, but true.

But wait, there is more! Yes, it is true, not only can you slice, but you can also dice!

That's not the end of IQueryable<T>'s little bag of tricks. What else could there be? How about dynamic queries, ones you can build up at runtime? Yes? No? Say it isn't so!

It's true. IQueryable<T> is fully dynamic. Sort of. Each IQueryable<T> has a method CreateQuery<S>() that creates a new IQueryable<S> (of the same concrete type) with a single Expression tree argument. That is, if I have an expression tree representing the code that defines a query, I can make an IQueryable<S> out of an old IQueryable<T>.

That's great you say, but you probably don't statically know what the 'T' and 'S' are in your program if you are building dynamic queries at runtime. True, true, we've thought of that too. You see, IQueryable<T> has a smaller cousin, IQueryable, just like IEnumerable<T> has its little friend IEnumerable. IQueryable is non-generic, but its ever so much a query as IQueryable<T>. In fact, it has its own CreateQuery() method that builds a new IQueryable so you can do all this query building without all that static info getting in your way.

So how do you build the expression trees that make up a query. Check out the System.Query.QueryExpression class. It is a collection of static methods that correspond to the standard query operator pattern, but are more loosely typed, taking arguments that are Expression's instead of Expression<T>'s. These methods do all the work necessary to build up the call nodes that represent calls to the more fully typed IQueryable<T> query operator methods.

Here's how you can dynamic build up the query in the prior example.

IQueryable q = ...;

ParameterExpression p = Expression.Parameter(typeof(Customer), "c");

Expression body = Expression.EQ(

Expression.Property(p, typeof(Customer).GetProperty("City")),

Expression.Constant("London")

);

LambdaExpression predicate = QueryExpression.Lambda(body, p);

Expression where = QueryExpression.Where(q.Expression, predicate);

q = q.CreateQuery(where);

You can then call GetEnumerator() on 'q' to execute the query.

Too hard you say? Do I have to write all that code? What if I just want my users to type in a little filter expression. Do I have to turn that into all those nodes?

Ah, well, we've thought of that too. Try this out.

IQueryable q = ...;

ParameterExpression p = Expression.Parameter(typeof(Customer), "c");

LambdaExpression predicate =
QueryExpression.Lambda("c.City = 'London'", p);

Expression where = QueryExpression.Where(q.Expression, predicate);

q = q.CreateQuery(where);

Or if you don't want them to have to type silly dot's, just don't give the parameter a name.

IQueryable q = ...;

ParameterExpression p = Expression.Parameter(typeof(Customer), "");

LambdaExpression predicate =
QueryExpression.Lambda("City = 'London'", p);

Expression where = QueryExpression.Where(q.Expression, predicate);

q = q.CreateQuery(where);

Viola! Instant filter. Instant dynamic query. Run it here. Run it there. Why, you can run it everywhere.

But that's enough from me. Why are you wasting time reading this when you could be downloading the preview and trying it out for yourself?

Matt

Beyond the Query

mattwar — Wed, 19 Apr 2006 10:20:00 GMT

I have been working a long time to bring queries into a modern programming language. Seven years ago I looked beyond ORM and saw the next horizon, a new world where boundaries between data are blurred and popular paradigms from different disciplines combine. Many have tried to convince me that it is simply a mirage, and maybe in some respects it is, but as with anything else there is often a lot of utility born out of imperfection. I have been marching steadily toward that horizon ever since.

I also realize much of it has been done before. From academic languages to real world database scripting engines, the concept of the query itself is nothing new. Yet, it has never been mainstream before, baked in as a first-class feature of a general purpose programming language. It is no longer a novelty, applicable to only a single domain, each domain with its own variant, bound in shackles, caged like a dangerous animal. It is becoming something real, something you can reason about and rely on, a permanent fixture of the natural programming environment, a part of its substrate.

And I’m not just out to promote a particular language. In a few years, there won’t be a language without queries. Eventually, queries will become one of the most used tools in your toolbox. And not just for databases; all aspects of your application will be queryable to some degree, allowing you the freedom to apply set-like operations anywhere and everywhere. Queries will bring power through simplicity and abstraction. What was once proscriptive will become descriptive, pushing more and more of the gritty details down into the runtime. And still the benefit will go beyond merely your personal productivity boost. The next horizon is already in sight. Queries will eventually become runtime optimized. They will enable automatic parallelism, scaling and distributed processing.

Yet, we are not there yet. With the help of a lot of other people, I’ve only just arrived at that first horizon, a place where the mirage forms into solid shapes and the world that I only imagined becomes substantive, where I can look out across the landscape of all that was devised and see it functioning, its clockwork gears spinning, its implications twinkling like dew upon fields of freshly grown grass. And I can simply query it, all of it, the grass, the trees, anything identifiable and enumerable. And it just works.

Yet in the midst of all that, I am overcome with another thought. Another horizon has opened up in my mind and I can see a bit further, beyond the benefits and the boosts, beyond the query itself. I can see now that it was only the first stepping stone. Something much bigger is out there. A paradigm shift is coming, where all things become fluid, all boundaries dissolve and intent becomes first class.

And it has certainly all been done before.

Matt

Citizen Statistics

mattwar — Wed, 01 Feb 2006 10:23:00 GMT

I was on my way to post that my site had not actually been blocked by a foreign government, aside from all the rumors spread recently, that the truth was I had just been incredibly negligent at forming any coherent thoughts or communicating in general beyond my regular guest appearances on several late night talk shows, when to what to my wondering eyes did appear, but a survey popup for MSN.

What had I done to deserve this? Was it some crime in another life, a karmic imbalance on my metaphysical spreadsheet? The appearance had been dramatically tied to my subconscious desire to post. Was there actually a connection? Had the angst of a disillusioned technorati, spurned by my inattentiveness, an undercurrent of outrage simmering for the last few months as the euphoria of the Yoda post was left to wane, whither, and finally die, leaving only the deep unavoidable depression, a vacuous hole unable to be filled by the inadequacies of the surrogate posters, the trivial news bits, the mindless slash-dot like debates, could all of this negativity have possibly culminated into a single psychic backlash, a lightning arc of loathing striking out across the electric ether and manifesting itself upon the screen of this singular purveyor of punditry, alone, at his desk, at midnight, a gruesome request to please answer a few questions, fill in a few blanks like a good sample citizen of our target-market society?

I suppose I’ll never know. However, rest assured the message has been received, loud, clear and in stylized markup. For back to the grindstone I go, feverously cranking out follow on posts with clever and witty humor, enlisting all sorts of Star Wars euphemisms and characterizations in an attempt to bring peace to the cosmos of our disenfranchised, though somewhat wayward, community.

And fear not from where I shall draw forth such an abundance of interesting topics for my specialized editorials. With Ruby on Rails on the loose, finding something to take pot shots at has never been easier.

Still, though, with all the other options available to you the reader, and with a blogroll share dipping into the low zero-percentile, it has come to my attention that I need to offer a more targeted and customized service if I want to find a true niche and a loyal reader base. So if you could take a few moments to fill out the following questionnaire it would be much appreciated. Your answers will be tallied, tabulated, scored, scorched, indexed, cross-indexed, pivoted, hot-keyed and your phone numbers placed on my speed dial, all in conformance with our standard, substandard, privacy and security policies.

Your personal anonymous input is highly valued.

How do you feel about filling out questionnaires?

[ ] S’okay, S’alright.

[ ] Don’t bug me, I bite

[ ] I’ve got a loaded gun and know how to use it.

Which sort of sarcasm do you prefer?

[ ] Dark

[ ] Semisweet

[ ] Milk

[ ] Candy coated

How many readers made it this far?

[ ] All of them

[ ] None of them

[ ] Dang, it’s just me again, isn’t it?

Which sign off do you prefer?

[ ] But I digress

YODA the Programming Language

mattwar — Mon, 10 Oct 2005 08:34:00 GMT

Now that LINQ is out the door, so to speak, I can start focusing my efforts on the next next technology here at Microsoft. Ever since I joined up with the C# team nearly two years ago I’ve been frustrated by my inability to wax poetic about all the goodness we were working on. I was sworn to secrecy. Mum was the word. Perhaps if you were paying attention to the work in C# 2.0 and C-Omega, you may have guessed what was to come. Looking back, it’s easy enough to recognize it in the design of Generics, Iterators and Anonymous methods. The existence of Nullables in there as well should have made it obvious. We were planning ahead for the big pay off, language integrated query.

You may be amazed that so much planning goes on in the features that we roll out version to version. Sometimes big ideas and far-reaching visions take many releases to come to fruition. You cannot always do them in one release. Sometimes you have to take a risk and dole them out piece by piece. This may cause a bit of confusion at first, when no one can truly understand why a particular feature was included and not others, or why one design was chosen. Yet once all the pieces are together you can finally make sense of it all, and then as if by magic it all just seems right.

Of course, this time is no different than the last. We planned ahead and baked in features into C# 3.0 that will lay the foundation for the next big thing. LINQ is only the start of the revolution. Yet, unlike last time, the features we added now make it rather obvious what is coming next. If you connect the dots in your mind and extrapolate just a bit you’ll see what I’m getting at. You’ll begin to notice that’s its not just about query in the language, or first order functions or everything else that seems to be borrowed from languages of yore. No, these pieces are just stepping stones for what is yet to come, something truly original; a next generation programming language.

The evidence is there right now, plain to see. The query expression exhibits it quite clearly. Yet, that’s just for starters. Eventually, the same degree of innovation will influence the rest of the language, and then you’ll have something quite different, a language the likes of which you have not seen before, one that is simple and elegant, yet secretly powerful: a YODA-like programming language.

You see, putting the ‘from’ ahead of the ‘select’ was not by accident. Soon the whole language will be re-arranged in an effort to make expressing intent easier and more logical.

Instead of the cryptic c-like syntax below:

public void Main(string[] args) {

Console.WriteLine(“Hello World”);

}

We will now have eloquent YODA-like syntax:

(args of string many are they) Main is what they seek yet return they do not.

Brace you must

Written it is, the Console. “Hello World”

I know it’s difficult to believe, as strange as it seems. Yet, sometime in the future, everyone will be writing software this way. Knowing this, it makes my work so much more invigorating. I can literally feel the electricity in the air around here. It’s like some queer energetic force.

But I digress

Matt

What the World is saying about LINQ

mattwar — Thu, 22 Sep 2005 04:07:00 GMT

“LINQ is totally awesome. It’s like this thing that you use to condense your entire application into one line of demo code. Sweet!” - Stanley Morgan

“LINQ is divine but DLINQ is a delinquent. It gives programmers too much power and makes programming against data way too easy. Is nothing sacred anymore?” – Ned Flanders

“Lambdas are old school. LISP had these decades ago. If all you want to do is be a sheep then go right ahead. I prefer ACME.” – Wyle E Coyote

“I can’t remember what they are called, but those things you make to build up a result out of a bunch of braces and stuff are really cool.” - Anonymous

“LINQ will change the way we think about thinking about programs without requiring us to actually think about them.” – National Research Council

“LINQ puts the 3GL back in the 4GLs, together making a 7GL!” – Guy Steele

“LINQ is the next best thing since sliced bread. Before LINQ we had to cut those database loaves up ourselves. It was a mess. Now programming against data is as easy as simply selecting the slices we want. Anyone seen that plastic doodad that holds the bag closed?” – Heimlich

“LINQ is the best party game I’ve played in a long time.” – Jeff Bezos

LINQ Designer Receives Prestigious Award at PDC

mattwar — Tue, 20 Sep 2005 09:36:00 GMT

In the wake and fanfare of the disclosure of the LINQ technology preview at the PDC 2005 in Los Angeles this last week, a member of the LINQ design team was presented with a prestigious award for outstanding achievement above and beyond his peers. Erik Meijer, most known for his antics as "Head in the Box" for VB TV and author of many research and academic papers such as "Programming with Circles, Triangles, Rectangles and other 3rd grade geometry", was surprised to find himself the award’s recipient. “The competition was pretty tough,” he was heard to say as he was lead off stage. “I had no idea I would win this thing. I just knew I had to try.”

I interviewed Erik the next day, sometime past noon when he had recovered from the celebration enough to speak coherently.

Did you ever think that your work on Haskell, VB, C-Omega and now LINQ would culminate in this?

I’m not sure what you mean. Really, that prior work speaks for itself. There’s a lot of crazy stuff in there, even for me. If you lay them out, side by side, you can see each one progressing further and further, each language referring back and forth to each other over time. It’s so Escher.

Speaking of crazy, you’re quite well known for your antics with “Head in the Box” and your wild ideas on programming languages evidenced by your numerous publications. Are these things what drove you toward the achievement last night?

Definitely. Everyone knows I’m a risk taker. I like to do wild and crazy things. Languages are just one of them. It’s like surfing, parasailing or sky-diving, you’ve just got to hang and on a go for it.

Did you always think you had what it takes to be a winner like this?

It’s not something you think about. It’s just something you do. Like when the guys in the data team asked me to help think about querying in programming languages and I didn’t really know what to do, so I just started spouting off about triangles and whatnot, waiting for their eyes to glaze over. That gave me the time to put together a plan. It’s just a skill I have I guess, my competitive side showing through. When everything came around again with LINQ I knew the geometry play would not be enough, so I tossed ‘Monads’ at them like a stun grenade. You should have been there.

What about ‘homomorphic lifting’? You’ve been heard to utter that one on occasion.

Yes, that too. Except that one still works. It’s always good for a laugh. It puts the design team into a tizzy.

So what do you think sealed the win for you last night? Was it your zest for life? Was it your competitiveness?

Well, it definitely had something to do with the large protein shake made from fish guts, liver and insect parts.

Well, there you have it, Erik Meijer, LINQ designer and winner of Universal Studio’s Fear Factor Live. It’s the programming that drove him crazy; crazy enough to be a winner.

LINQ, BB’s and the HTTP’ness

mattwar — Wed, 14 Sep 2005 00:42:00 GMT

The Language Integrated Query (LINQ) demo during the PDC demo was exciting. There was applause all around whenever different products were shown and different speakers were introduced, yet when the LINQ demo was shown there was a lot of talk within the crowd, oohs and aahs, and a bunch of OMG-this-is-cool’s. The attendee sitting next to me pulled out his notepad and started writing down everything he saw. After two hours of keynote before it, it was the only thing worthy to write down.

Anders gave the demo along with Don Box as part of a large multi-technology demo showing off the LINQ project, Indigo, Atlas and Avalon. The LINQ part was first and formed the foundation of the rest of the demo. The demo was written using the new preview version of the C# 3.0 compiler and an object-relational API we refer to as D-LINQ, both of which I’ve been working on for the last two years. Similar language features that enable queries in C# will eventually be in VB as well, or BB as Anders was heard to call it by the automatic voice to text translator overlaid on the keynote’s projection screen. Don’t you just love technology?

Anders had Don (his code monkey) write a query against the framework to find all the processes with a working set greater than 4 MB. The code looked something like this.

var q = from p in Process.GetProcesses()

where p.WorkingSet > 1024 * 1024 * 4

select new {p.ProcessName, p.WorkingSet};

foreach(var p in q) {

Console.WriteLine("{0,-30} {1}", p.ProcessName, p.WorkingSet);

}

The new C# 3.0 language has a construct known as a query expression. It looks a little bit like SQL yet not exactly. The select clause comes at the end. It written this way so the ‘range variables’ are specified up front making it possible for intellisense to aid you as you type. The select clause specifies a single expression, however you can use a new language feature known as anonymous type initializers (show in this example) to project the results into a new shape.

After the queries were written, Anders turned the floor back over to Don who with the help of Chris Anderson went off to turn the application into a web service. At this point Don muttered something about the protocol layer, noting its HTTP’ness. That’s when I knew I had a blog post.

Anyway, I’m busy working at the PDC, no time to explain more. Check out the LINQ online website for more details. Download the preview compiler and the whitepapers. There’s also a download for the VB 9.0 preview with a bunch of XML integration. Check that out too. J

Matt

ObjectSpaces:There and Back Again

mattwar — Tue, 13 Sep 2005 01:14:00 GMT

It seems fitting to make one last tribute to ObjectSpaces on the eve of the PDC. Last time around Luca was good enough to break the news to everyone, that ObjectSpaces was undergoing realignment with WinFS. That it was being pulled out of Whidbey and would or should appear sometime later. But we all knew what that meant.

ObjectSpaces was a big part of my life, my career for a long time. I was involved in its birth, its conception. The first preview appearing years ago at the PDC; that was my prototype, my code, as awkward and ugly as it was, opath, spans and all. A talented team of developers gave you the next preview and so on. I went away to work on something else, but ObjectSpaces was still on my mind. In fact, the reason I walked away was for the chance to work on what I believed to be the next logical step forward. ORM was only half the picture. The state of the art was still API’s, with queries in strings; opath was still another language.

With the Xen project we proved that modern programming languages could be modified to include basic concepts such as query. We borrowed a lot from other languages, functional and otherwise. Streams became a natural concept as well as query expressions, (aka comprehensions to language-heads). Yet we went beyond and tied it all together, all the way down to the database, and by doing so put the ease of 4GL database programming back into the hands of developers, without co-opting the language, without tying it directly to one product, making query more general purpose, making it work for XML and any in-memory data structure.

Now we are on the verge of a new era. Language Integrated Query in C#, VB and anyone else that wants to play along.

Details tomorrow.

Matt

Language Integrated Quirks

mattwar — Mon, 29 Aug 2005 23:35:00 GMT

It’s done. The pencils are down. The bits are baked. No more time for polish and there’s no turning back. Like a snapshot of the last two years of my life, the binaries have frozen in place all that my fingers have wrought, all the lines of logic, all the code, all the defects, all on disk now, a stable moment in time, a silent pause before the wind picks up again the bits are carried away like seeds, replicated far out into the void.

It’s somewhat of a relief to be done for now, but there’s also an angst welling within me. Because I know what is still left undone. I know the flaws, the things that should be working but are not, some that were not caught until too late, other’s due to my own disastrous oversights, miscalculations, certainty gone awry.

Yet then I am reminded by others of the majority that works as advertised, all that runs miraculously right out of the box, and all the feedback and reviews of others; the big thumbs up. The buzz is electric.

I’m proud of what is in there, bugs and all. Still, I relish the opportunity to dive back in and wrench it all apart, adjusting, tweaking, transfiguring. Fortunately, it’s not the final word. It is, after all, only a preview, a slice in time of the development cycle, primed and packaged for the PDC.

Language Integrated Query, you’ll see what it’s all about in two weeks time. You’ll get the disk and install it for yourself in the comfort of your own home. You’ll run the samples and see them fly. You’ll dig deeper and find the flaws. I’m okay with that. That’s why we are handing it out now. We want to hear from you.

If you are at the PDC you can even stop by and tell me what you think. I’ll be bouncing back and forth between the language and data track lounges. Though, maybe not literally unless there’s a big trampoline.

Matt

C# 3.0 Revealed?

mattwar — Mon, 13 Jun 2005 21:49:00 GMT

Professional Developers Conference PDC September, 2005