The Wayward WebLog : Rave

Oops, we did it again

mattwar — Thu, 11 May 2006 01:49:00 GMT

A new update to LINQ preview is available at MSDN site.

What's the coolest new feature? IMHO, its IQueryable<T>.

DLINQ's query mechanism has been generalized and available for all to use as part of System.Query. It implements the Standard Query Operators for you using expression nodes to represent the query. Your queries can now be truly polymorphic, written over a common abstraction and translated into the target environment only when you need it to.

public int CustomersInLondon(IQueryable<Customer> customers) {

int count = (from c in customers

where c.City == "London"

select c).Count();

return count;

}

Now you can define a function like this and it can operate on either an in memory collection or a remote DLINQ collection (or you own IQueryable for that matter.) The query is then either run entirely locally or remotely depending on the target.

If its a DLINQ query a count query is sent to the database.

SELECT COUNT(*) AS [value]

FROM [Customers] AS [t0]

WHERE [t0].[City] = @p0

If its a normal CLR collection, the query is executed locally, using the System.Query.Sequence classes definitions of the standard query operators. All you need to do is turn your IEnumerable<Customer> into IQueryable<Customer>. This is accomplished easily with a built-in ToQueryable() method.

List<Customer> customers = ...;

CustomersInLondon(customers.ToQueryable());

Wow! That was easy. But, how is this done? How can you possible turn my List<T> into some queryable thingamabob?

Good question. Glad you asked.

Check out this little gem:

Expression<Func<Customer,bool>> predicate = c => c.City == "London";

Func<Customer,bool> d = predicate.Compile();

Now you can compile lambda expressions directly into IL at runtime!

ToQueryable() wraps your IEnumerable<T> in IQueryable<T> clothing, uses the Queryable infrastructure to let you build up your own expression tree queries, and then when you enumerate it, the expression is rebound to refer to your IEnumerable<T> directly, the operators rebound to refer to System.Query.Sequence, and the resulting code is compiled using the built-in expression compiler. That code is then invoked producing your results.

Amazing, but true.

But wait, there is more! Yes, it is true, not only can you slice, but you can also dice!

That's not the end of IQueryable<T>'s little bag of tricks. What else could there be? How about dynamic queries, ones you can build up at runtime? Yes? No? Say it isn't so!

It's true. IQueryable<T> is fully dynamic. Sort of. Each IQueryable<T> has a method CreateQuery<S>() that creates a new IQueryable<S> (of the same concrete type) with a single Expression tree argument. That is, if I have an expression tree representing the code that defines a query, I can make an IQueryable<S> out of an old IQueryable<T>.

That's great you say, but you probably don't statically know what the 'T' and 'S' are in your program if you are building dynamic queries at runtime. True, true, we've thought of that too. You see, IQueryable<T> has a smaller cousin, IQueryable, just like IEnumerable<T> has its little friend IEnumerable. IQueryable is non-generic, but its ever so much a query as IQueryable<T>. In fact, it has its own CreateQuery() method that builds a new IQueryable so you can do all this query building without all that static info getting in your way.

So how do you build the expression trees that make up a query. Check out the System.Query.QueryExpression class. It is a collection of static methods that correspond to the standard query operator pattern, but are more loosely typed, taking arguments that are Expression's instead of Expression<T>'s. These methods do all the work necessary to build up the call nodes that represent calls to the more fully typed IQueryable<T> query operator methods.

Here's how you can dynamic build up the query in the prior example.

IQueryable q = ...;

ParameterExpression p = Expression.Parameter(typeof(Customer), "c");

Expression body = Expression.EQ(

Expression.Property(p, typeof(Customer).GetProperty("City")),

Expression.Constant("London")

);

LambdaExpression predicate = QueryExpression.Lambda(body, p);

Expression where = QueryExpression.Where(q.Expression, predicate);

q = q.CreateQuery(where);

You can then call GetEnumerator() on 'q' to execute the query.

Too hard you say? Do I have to write all that code? What if I just want my users to type in a little filter expression. Do I have to turn that into all those nodes?

Ah, well, we've thought of that too. Try this out.

IQueryable q = ...;

ParameterExpression p = Expression.Parameter(typeof(Customer), "c");

LambdaExpression predicate =
QueryExpression.Lambda("c.City = 'London'", p);

Expression where = QueryExpression.Where(q.Expression, predicate);

q = q.CreateQuery(where);

Or if you don't want them to have to type silly dot's, just don't give the parameter a name.

IQueryable q = ...;

ParameterExpression p = Expression.Parameter(typeof(Customer), "");

LambdaExpression predicate =
QueryExpression.Lambda("City = 'London'", p);

Expression where = QueryExpression.Where(q.Expression, predicate);

q = q.CreateQuery(where);

Viola! Instant filter. Instant dynamic query. Run it here. Run it there. Why, you can run it everywhere.

But that's enough from me. Why are you wasting time reading this when you could be downloading the preview and trying it out for yourself?

Matt

Xen meets its Omega

mattwar — Wed, 14 Jul 2004 16:11:00 GMT

I got a surprise in my inbox this morning. An email from Erik Meijer read:

The impossible has happened: X# became Xen, Xen became Comega, and Comega has shipped.

http://www.research.microsoft.com/research/downloads/default.aspx.

Enjoy

Matt

Feeling the Love

mattwar — Fri, 11 Jun 2004 20:33:00 GMT

I think I've been spoiled these last so many years. It's gotten to the point where I cannot program without it. I've tried to do it, on my own, for different projects, on the weekend, late at night, but I find that its just not the same. I actually need it now. I hunger for it when I don't have it. It's as if my brain turns off and won't let me proceed unless it is there. I just can't live without my Intellisense.

Try, if you dare. Use Notepad or even your favorite code editor. You don't get the pop ups do you? You don't get reminded after each dot what would logically come next. You don't get the instant reminder of the contents of your code base and the volumes of APIs awaiting out there in the void. You don't get the handy squiggles underlining your errors, long before you even bother to compile. You don't get the instant rewrite, correcting your errors as you type. You don't feel the love of an editor that really, truly wants to be your friend.

I need it so much that I'm willing to do almost anything to keep it. Since I work on the compiler, in the same immense codebase as the rest of the product, I experiences all the growing pains as the editor and its ilk are torn down and refitted. I experience the bugs as they come, working around those I can, being stalled by those that I cannot. I work in the face of a appallingly slow debug build and the pestering assert dialogs that lurk behind the innocent facade. Yet, still I persist. Why? Because its my editor. Its got the intellisense. I need it to even breathe.

Matt

Silencing the Audio Angst

mattwar — Mon, 15 Mar 2004 05:36:00 GMT

After all my whining about my audio problem and how all new technology solutions are still wrong, I went ahead and made the decision to just put together the best thing I could with what was available. After all, it will be Betsy’s birthday soon and she did ask me to set this whole thing up two years ago. I guess I’m on top of it now, at least for a procrastinating, indecisive techno-goober like myself. Some things just take their own sweet time to get right, and as Dirk Gently would have stated, “I was two years late but moving fast.”

I decided to buy the shelf-top mini-stereo that I found at Fry’s last week. Maybe it was my tech addiction, yearning, calling me back to the dealer. I don’t know. The thing is, that stereo had oodles of line-inputs and outputs; just what I so desperately needed, and what is so hard to find in a shelf-system. It’s not like there weren’t many excellent systems to choose from. It’s just that I have this vision for what it will eventually become, so I did not want to get trapped with some single purpose device. I wanted something I could blend into my digital home when all the pieces started falling into place. I knew that right now it would only just be a simple stereo, not the fulcrum of my digital music utopia. But I was determined to buy it anyway, come hell or high-water.

Yet, when I got to the store, and after I stood in awe of its majesty one more time, I discovered that Fry’s did have a wireless music solution in the aisle with all the routers and access points. It was the Netgear Wireless Digital Music Player, one of the models I had been previously researching. It did not seem too costly, actually cheaper than I had priced it for online, and since I was getting the stereo anyway I decided to throw it in my cart and give it a try. I’m sure the Fry’s marketing/research group would love to know that a wireless network product was an impulse buy. “Look, we'll just line up these boxes by the check-out stand; Nail clippers, Juicy fruit, and Netgear.”

The real shocker, however, was that I was still not done. That should have been the limit of my shopping, the climax of my high-tech sojourn. Yet, somehow more pieces kept falling into that same cart. Excluding a few toys and a glowing crystal necklace for Charlie, I was about to find what I needed to put the final touches on my audio phonic masterpiece.

The wireless speakers, there they were down the aisle with audio cables and what not. I was looking for a y-adapter, and instead I got ya-ya-googly eyes. These were the newer model RCA wireless speakers. The ones I already had were the older model and they sucked. The reception was bad, lots of low-level static and popping even at close range, and the lack of bass was infuriating. In fact, the fact that the older speakers were so bad was my primary motivator to find a better solution through digital wireless. The newer models promised a greater range and auto-channel synching. That meant no more praying to the tiny god of wheels and widgets to help position the potentiometer in order to tune in the tunes. I swear, getting that bit right was like trying to tweak the shower knob to find the right balance between scalding hot and icy cold. Anyway, because the older ones were so bad, I had simply dismissed the newer ones as most certainly junk to the nth degree. But now, with the advent of my new impulsiveness, I was all the more daring. It was a new, kindler, gentler, more positive version of me that was willing to give it a try. So I hefted a few of those into my cart as well. I figured, if they sucked I’d just bring them back. If the didn’t, then I’d have my ultimate audio.

To keep a long story short, they did not suck. They sound rather good. They’ve got better bass and they don’t snap, crackle and pop. The stereo is now set up, the wireless speakers are tuned in and the Netgear music player is sucking down digital tracks off my PC jukebox. The music is playing throughout the house.

Best of all, Betsy is smiling. It was worth the wait.

Matt

ObjectSpaces: What's the big idea?

mattwar — Fri, 05 Mar 2004 21:43:00 GMT

Should your data become objects or your objects become data? That’s the debate. If you build apps focused around objects and just want those suckers persisted then you want your objects to become data. If you’ve got a database and you are building apps around that data using an object-oriented language you probably want your data to become objects. Which is more prevalent, I don’t know, probably the latter.

What I am certain of, however, is that many people out there are struggling with data some way or another. You've got your ADO.Net, or just plain ADO, or maybe you're an Einstein and do it all against OLEDB or ODBC or some other API du jour, and if you were still writing quick and dirty apps these API's would suffice. I know. I've done it too.

I've recently been working in my 'spare' time on a web app that's gotten me sucked back into the old-school ASP programming. I found myself writing code against ADO, connections, commands and recordsets. I had to pull out my own book on the subject just to remember what to do. Luckily, I also spent the last nine years working on this technology, so when it came to crafting up some SQL queries it was not too tough, except they weren't really SQL queries, they were Jet SQL queries meant for an Access database. Sure, I should have been using a version of SQL Server, but it wasn't my app, I was only volunteering my time to put in new features. So I struggled to get the queries just right, trying to figure out what to do using the Access product documentation, which is great if you want to learn how to use a menu or item in the user-interface, but when it comes to documenting the query language it is none too helpful. It was at this time that I wished the application were more of an honest-to-goodness professional style app with a good data access abstraction layer, then maybe I wouldn't find myself scratching my head at 3 am wondering how to translate a subquery.

Sure, you've all been there at some point in the past. Most you have moved on to better designs and practices. That's why you build abstraction layers. I know this is true, because as customers you tell us that you do. I've visited customer sites. I've seen some of the apps. They are marvelous and huge, testimonies to man's ingenuity, and they can't get that way without abstraction. But a data access layer can be difficult to build, difficult to get right. It can consume a project team.

That's why we've built ObjectSpaces, so you don't have to. ObjectSpaces is the ideal data abstraction layer. It lets you define your application in terms of your objects, and it gives a declarative way to describe how your objects correspond to your storage system. You do this once and you never have to look back. Okay you might have to look back and tweak it, but you don’t have to tweak your code, just the mapping.

ObjectSpaces is the ultimate abstraction layer because it does not lock you into a fixed set of library methods like GetMyObjectX and GetMyObjectY. You can still write queries. You just don’t have to query the storage model in the database. Your queries refer to the objects you use in your code. Best of all, there’s just one query language that you use, regardless of where the objects are stored. If the objects are in SQL server, then OPath translates to SQL server’s dialect. If the objects are in Jet SQL then that’s what is used to communicate to your data.

So are your objects data, or are your data objects? It does not matter really which way you come at the problem. ObjectSpaces works both ways by taking the middle of the road approach. You don’t just get a turn-key system that takes your object structures and dumps them into an uncertain stasis, inaccessible to other apps in your business. That’s never worked too well and is one of the primary reasons object databases failed. And you don’t just get a system that gives you straight objects in place of database tables. You get to design your own objects, and you get to customize how they are stored. It might sound like a little more work up front, but it is well worth it. A working abstraction layer with endless tweaking ability, that’s the big idea.

Matt

Drugged out and full of bliss

mattwar — Sat, 28 Feb 2004 07:28:00 GMT

I've got a new drug. I'm not sure how long its going to last. Thanks to all the feedback from my prior post. I got a lot of ideas from everyone who replied, and I acted on one of them. I purchase the Harmony SST-659 universal remote. It is programmed via an online wizard from a large database of known devices, so it was relatively easy to set up. The best part is, now my TV monstrosity has been turned into a one click-on device! Betsy thinks its the best improvement to the TV room since the TiVo was hooked up, and I agree. All the other remote are packed away. Now there's finally room on the ottoman for my feet!

Matt