Man vs Code : ADO.net

Stop, Drop, and Roll

aconrad — Tue, 29 Mar 2005 00:32:00 GMT

A couple of weeks ago, I made a post that I thought some sort of client side data structure, which was designed to store projections of a domain model instances (both in memory and persisted in a data store), was a necessary part of any O/R framework. My conjecture was that this was required due to the current limitations of the mainstream programming languages with respect to dynamic typing in addition to the fact that most client side caches provide a common set of services which can be applied to both tuples and object graphs.

Amazingly enough, I was flamed by both camps – the dataset crowd and the O/R guys, which is quite a feat. However, surprising to me, was how polarized this issue had become.

Obviously, either I was not correct, not clear about my points, or some in the development community didn’t want to hear what I said.

This entry is an effort to clear up my original post – if you still don’t agree, please feel free to flame away.

So – my original points were:

Projections, whether over relation tables or objects graphs are a very important cog of most enterprise applications.
None of the .net languages or Java have a good way of natively exposing or storing the results of the projection. Hence, types like DataRow exist.
Existing relation caches (for example, like the DataSet) offer a number of services which are interesting to both tuples and object graphs.

There is no doubt in my mind that #2 from above will change long term. Even some of the work coming out of Microsoft shows that better solutions are being worked on. For example.

But as of today, it seems like current O/R frameworks have an inherent weakness with projections. The two most popular existing solutions I have seen is to support dynamic instantiation or return object arrays. My belief is utilizing a client side relation cache (for example – like the DataSet) is superior to the latter in almost every case or to the latter in cases where this is going to cause type inflation (MyPartialCustomer) or when clean updateability is required.

I would also push back of the DataSet crowd with some similar arguments. First, you may not believe it, but O/R frameworks are designed to take care of a lot of code that you currently have to write yourself. Second, do you really like the DataSet (i.e. The API and architecture) or do you like the rich services it provides which in most cases are the same services required by a good O/R framework? My guess is that in most cases it is the services.

Dataworks now has a group blog

aconrad — Mon, 21 Mar 2005 22:33:00 GMT

My current group now has a team Blog:

http://blogs.msdn.com/dataaccess/

Alyssa Henry, our product unit manager provided a good description of the technologies which it will cover.

Nix the DataSet??????

aconrad — Wed, 16 Mar 2005 19:50:00 GMT

Some interesting comments from my entry on new DataSet features…

Please nix the DataSet from the framework entirely and re-focus on the domain-oriented data access work you were doing prior to Tech-Ed 04. If that's not possible, please have the Visual Studio team desist from distracting .NET culture from Domain-Driven Design with that nasty, nasty typed DataSet RADule :)

I hope this helps, Scott Bellware

http://www.geekswithblogs.com/sbellware/

I don't advocate that for multiple reasons.

Sahil Malik

http://codebetter.com/blogs/sahil.malik/

Scott is an old friend from the ObjectSpaces days, so I think he is partially kidding. And I can’t say I totally disagree with him WRT the typed DataSet because I think it has been shown that O/R solutions can do better then code generation. But that is a different topic for a different time.

What I do want to talk about is projections in terms of an O/R solution – which is an interesting topic when discussing the DataSet.

Object instances in O/R solution can basically have five states:

Persisted in the data store (shredded or not).
In memory, but not materialized.
In memory, materialized as an object instance.
Partially in memory, materialized as an object instance.
Partially in memory, not materialized.

#1 is the traditional way of storing data in relational databases. Shredded means the individual properties including references have been set as column values and that the object identity is defined by a single row of storage (in fancier mappings – this row itself might be a project of base tables and/ or views). With relational database advances, there is also the ability to store the object instance wholly (i.e. Sql Server 2005 UDTs, etc) or even partially shred it and store the rest in an untyped bag (i.e. the Xml DataType)

#2 When O/R frameworks pull data out of the data store there is a transition between the rowsets being returned from the data store and the object graph returned to the consumer. Basically, this means that the object instances live in an un-materialized state. Since materialization is not cheap, many O/R solutions – particularly with client side caches, store the data in an “un-materialized” state until the specific object instances are required by the consumer. In fact, with some O/R frameworks – users can access this data.

#3 This is the in-memory materialized domain model – basically the corner stone feature of all O/R frameworks.

#4 Same as 3, but value properties and/ or references are delayed loaded. In terms of domain model consumer and for this discussion, #3 & #4 are the same. (I know there are some issues here, see my comments in the past on this).

#5 are projections of the domain model which have not been materialized as object instances – and may never get materialized. (Actually, #2 is probably a sub case of this). In fact, depending on what the projection is – it might not even be possible or even desirable to materialize the objects.

Take for example a scenario where a domain model consumer wants to search for all the customers with a given last name. From there they want to display the results to the application user and let them select the specific customer. Now, let’s assume that the Customer type is fairly complex and has 30 or so properties of which only a handful are useful for the user to visually select the “right customer”. Performance wise, this would only mean selecting a few of the type properties (a projection of the domain model) in the initial query. This design issue is quite familiar to anyone who has ever written app that binds query results to a grid. For O/R solutions this presents an interesting issue in that the properties in the project may not allow for materialization of the object instances for the retrieved type. For example, the type has several properties which either don’t have meaningful defaults, or if not initialized to persisted values leave an object instance in an illegal state.

Further, for a common last name, performance wise it might not even be desirable to materialize all the customers as object instances even if one could. So the question is, if the objects are not materialized, how is it stored/ accessible in memory. Obviously, “rowsets” are a very good idea since that is generally how relational data is exposed through data access APIs. So, a relational, client side cache (like the DataSet) would seem to be really useful. Interestingly, for this sole purpose, the DataSet is probably overkill. Really all is needed is some sort of client side collection object, even arrays could work. However, for the scenario discussed above, one is going to need other features like binding and sort/ filter/ find capabilities – which the DataSet is very good at. Ironically, these are features that are required by most applications independent of data access model.

My belief has always been that on average roughly two-thirds of all queries executed by applications requiring database persistence are projections WRT to the domain model. Unless an O/R framework has a solution for this, it either forces the domain model designer to include awkward “partial types” (i.e. PartialCustomer), utilize some sort of weak typing hack in their domain model (which kinds of goes against O/R in the first place), or translate projection queries into queries which can always generate results that can be materialized (while accepting the performance overhead).

DataSet feature requests

aconrad — Mon, 28 Feb 2005 21:27:00 GMT

We have begun the initial planning for the next version of the DataSet (post VS 2005). If you have a feature you would like to see, let me know.

I can also pass on feature requests for other components in System.Xml and ADO.net to the proper folks.

Everybody's favorite topic - NULL

aconrad — Mon, 28 Feb 2005 21:18:00 GMT

In V1 of .net, value types could not be null. Since, this was contradictory to relational databases use of null, the DataSet needed to have a concept of nullable value types – hence DBNull was invented.

DBNull is simple type in the System namespace. It does not have a constructor, but a static method (DBNull.Value) that returns a singleton instance of the class. Even though it is designed to support relational database scenarios, it does not support 3 part logic. For example equality is supported in a CLR manner:

Console.WriteLine(DBNull.Value == DBNull.Value); // returns true

Setting null for a value type means explicitly setting the value to be DBNull.Value:

DataColumn nullableColumn = new DataColumn("myInt", typeof(int));

table.Columns.Add(nullableColumn);

DataRow row = table.Rows.Add();

row["myInt"] = DBNull.Value;

One can explicitly check if a value is DbNull by comparing to DBNull.Value or calling DataRow.IsNull:

Console.WriteLine(row["myInt"] == DBNull.Value); // True - but bad news

Console.WriteLine(row.IsNull("myInt")); // True - the recommended way

Even though both work, calling DataRow.IsNull() is the recommended method of checking for DBNull valued column values. In fact, even though currently supported, no code should be doing the former. You will see the reason why in a moment.

Setting null for reference types is similar, but a little different in that since either null or DBNull.Value can be provided:

DataColumn nullableReferenceColumn = new DataColumn("myType", typeof(myCustomerType));

table.Columns.Add(nullableReferenceColumn);

row = table.Rows.Add();

row["myType"] = null;

row["myType"] = DBNull.Value;

Note – however, internally the DataSet translates the null value to be DBNull.Value. This can be seen by retrieving the value:

Console.WriteLine(row["myType"] == DBNull.Value); // True - but also bad news

Console.WriteLine(row["myType"] == null); // False - even though reference was set to null

Console.WriteLine(row.IsNull("myType")); // True - the recommended way

From this example, one can see that DataRow.IsNull() will return true whether the value was set to DBNull.Value or null. In a way, IsNull abstracts the DataRow consumer from what specific “null” the underlying value is. This becomes more obvious with the introduction of nullable types in V2.

BIG DISCLAIMER – THE FOLLOWING BEHAVIOR IS FROM VISUAL STUDIO 2005 BETA 1 AND IS BROKEN. HENCE IT WILL EITHER BE NOT SUPPORTED OR FIXED IN RTM OF VS 2005.

Now in V2, we have nullable value types through the use of Nullable<T>.

As of Beta 1 of Whidbey, the DataSet allows nullable types to be used as DataSet columns – but with some interesting problems.

If a column is a nullable value type, one can explicit set it to null like reference types in V1:

DataColumn nullableIntColumn = new DataColumn("myNullableInt", typeof(int?));

table.Columns.Add(nullableIntColumn);

DataRow row = table.Rows.Add();

row["myNullableInt"] = (int?)null;

//Note – this is different from setting the value to DBNull:

row["myNullableInt"] = DBNull.Value

In addition, one must set the null value of int? and not reference null. For those of you not completely familiar with nullable type support in C#, the following won’t even compile – which may surprise some:

int? c = (object)null; // compile error Cannot implicitly convert type 'object' to 'int?'.

// An explicit conversion exists (are you missing a cast?)

And casting the null value of int? to be an reference type boxes the value:

int? c = null;

object h = (object)c;

Console.WriteLine(h == null); // false!

Console.WriteLine(((int?)h) == null); // unboxed ... now true.

Remember, when accessing a column value via DataRow[], an object is returned. Henced, boxing happens automatically for nullable types, and the reason behind the following somewhat bizzare behavior with retrieving the nullable value type:

// warning - the following code is broken VS 2005 Beta1 and will change before RTM

Console.WriteLine((int?)row["myNullableInt"] == null); // true – unboxing value

Console.WriteLine(row["myNullableInt"] == null); // false

Console.WriteLine(row.IsNull("myNullableInt")); // false! Broken!

One of the interesting design questions is what to do whether DbNull.Value should be returned for DataRow[] for nullable value types. Current behavior is it is not:

Console.WriteLine(row["myNullableInt"] == DBNull.Value); // false, boxed value is null, but reference is not.

At this point, it may become obvious that unless one is ready to put the following all over their code in cases where the code needs to work independent of column type(or disallow nullable value typed columns), DataRow.IsNull() is by far the better technique:

if (row["myNullableInt"] == null || row["myNullableInt"] == DBNull.Value ||

!((System.INullableValue)row["myNullableInt"]).HasValue)

Console.WriteLine("null"); // value is null

One other interesing problem with the current support in Beta1 is that the AllowDBNull constraint is not enforced:

nullableIntColumn.AllowDBNull = false; //only enforces DBNull, not (int?) null

nullableIntColumn.DefaultValue = (int?)5;

DataRow row2 = table.Rows.Add();

row2["myNullableInt"] = (int?)null; // obviously

As noted before, the DataSet behavior WRT to nullable types will either be changed or not supported for RTM of VS 2005. However, it is very probable that it will be supported some time in the future.

Dataset performance improvements in frameworks V2

aconrad — Wed, 16 Feb 2005 21:30:00 GMT

System.Data.DataSet contains some very significant performance improvements over prior versions – particularly for large result set scenarios.

In particular,

Highly efficient Binary Remoting -- For large Dataset, orders of magnitude better than v1.x
Row Insert, Delete and Modify operations – For large Dataset, orders of magnitude better than v1.x. For instance, for a DataTable with PrimaryKey constraint, inserting million rows in random order takes around 45 seconds. Everett took 30 Minutes.
Maintaining DataViews in sync with underlying DataTable was very expensive in v1.x, In Whidbey it’s extremely fast – orders of magnitude better.
WebServices/Remoting for TypedDataset: It can be send across to other end without having to send its schema, significantly reducing the serialized payload and giving approx 4x+ improvement in end-end latency.

The following article (in addition to highlighting some new V2 features) also details the performance improvements.