Man vs Code : ObjectSpaces

Stop, Drop, and Roll

aconrad — Tue, 29 Mar 2005 00:32:00 GMT

A couple of weeks ago, I made a post that I thought some sort of client side data structure, which was designed to store projections of a domain model instances (both in memory and persisted in a data store), was a necessary part of any O/R framework. My conjecture was that this was required due to the current limitations of the mainstream programming languages with respect to dynamic typing in addition to the fact that most client side caches provide a common set of services which can be applied to both tuples and object graphs.

Amazingly enough, I was flamed by both camps – the dataset crowd and the O/R guys, which is quite a feat. However, surprising to me, was how polarized this issue had become.

Obviously, either I was not correct, not clear about my points, or some in the development community didn’t want to hear what I said.

This entry is an effort to clear up my original post – if you still don’t agree, please feel free to flame away.

So – my original points were:

Projections, whether over relation tables or objects graphs are a very important cog of most enterprise applications.
None of the .net languages or Java have a good way of natively exposing or storing the results of the projection. Hence, types like DataRow exist.
Existing relation caches (for example, like the DataSet) offer a number of services which are interesting to both tuples and object graphs.

There is no doubt in my mind that #2 from above will change long term. Even some of the work coming out of Microsoft shows that better solutions are being worked on. For example.

But as of today, it seems like current O/R frameworks have an inherent weakness with projections. The two most popular existing solutions I have seen is to support dynamic instantiation or return object arrays. My belief is utilizing a client side relation cache (for example – like the DataSet) is superior to the latter in almost every case or to the latter in cases where this is going to cause type inflation (MyPartialCustomer) or when clean updateability is required.

I would also push back of the DataSet crowd with some similar arguments. First, you may not believe it, but O/R frameworks are designed to take care of a lot of code that you currently have to write yourself. Second, do you really like the DataSet (i.e. The API and architecture) or do you like the rich services it provides which in most cases are the same services required by a good O/R framework? My guess is that in most cases it is the services.

Nix the DataSet??????

aconrad — Wed, 16 Mar 2005 19:50:00 GMT

Some interesting comments from my entry on new DataSet features…

Please nix the DataSet from the framework entirely and re-focus on the domain-oriented data access work you were doing prior to Tech-Ed 04. If that's not possible, please have the Visual Studio team desist from distracting .NET culture from Domain-Driven Design with that nasty, nasty typed DataSet RADule :)

I hope this helps, Scott Bellware

http://www.geekswithblogs.com/sbellware/

I don't advocate that for multiple reasons.

Sahil Malik

http://codebetter.com/blogs/sahil.malik/

Scott is an old friend from the ObjectSpaces days, so I think he is partially kidding. And I can’t say I totally disagree with him WRT the typed DataSet because I think it has been shown that O/R solutions can do better then code generation. But that is a different topic for a different time.

What I do want to talk about is projections in terms of an O/R solution – which is an interesting topic when discussing the DataSet.

Object instances in O/R solution can basically have five states:

Persisted in the data store (shredded or not).
In memory, but not materialized.
In memory, materialized as an object instance.
Partially in memory, materialized as an object instance.
Partially in memory, not materialized.

#1 is the traditional way of storing data in relational databases. Shredded means the individual properties including references have been set as column values and that the object identity is defined by a single row of storage (in fancier mappings – this row itself might be a project of base tables and/ or views). With relational database advances, there is also the ability to store the object instance wholly (i.e. Sql Server 2005 UDTs, etc) or even partially shred it and store the rest in an untyped bag (i.e. the Xml DataType)

#2 When O/R frameworks pull data out of the data store there is a transition between the rowsets being returned from the data store and the object graph returned to the consumer. Basically, this means that the object instances live in an un-materialized state. Since materialization is not cheap, many O/R solutions – particularly with client side caches, store the data in an “un-materialized” state until the specific object instances are required by the consumer. In fact, with some O/R frameworks – users can access this data.

#3 This is the in-memory materialized domain model – basically the corner stone feature of all O/R frameworks.

#4 Same as 3, but value properties and/ or references are delayed loaded. In terms of domain model consumer and for this discussion, #3 & #4 are the same. (I know there are some issues here, see my comments in the past on this).

#5 are projections of the domain model which have not been materialized as object instances – and may never get materialized. (Actually, #2 is probably a sub case of this). In fact, depending on what the projection is – it might not even be possible or even desirable to materialize the objects.

Take for example a scenario where a domain model consumer wants to search for all the customers with a given last name. From there they want to display the results to the application user and let them select the specific customer. Now, let’s assume that the Customer type is fairly complex and has 30 or so properties of which only a handful are useful for the user to visually select the “right customer”. Performance wise, this would only mean selecting a few of the type properties (a projection of the domain model) in the initial query. This design issue is quite familiar to anyone who has ever written app that binds query results to a grid. For O/R solutions this presents an interesting issue in that the properties in the project may not allow for materialization of the object instances for the retrieved type. For example, the type has several properties which either don’t have meaningful defaults, or if not initialized to persisted values leave an object instance in an illegal state.

Further, for a common last name, performance wise it might not even be desirable to materialize all the customers as object instances even if one could. So the question is, if the objects are not materialized, how is it stored/ accessible in memory. Obviously, “rowsets” are a very good idea since that is generally how relational data is exposed through data access APIs. So, a relational, client side cache (like the DataSet) would seem to be really useful. Interestingly, for this sole purpose, the DataSet is probably overkill. Really all is needed is some sort of client side collection object, even arrays could work. However, for the scenario discussed above, one is going to need other features like binding and sort/ filter/ find capabilities – which the DataSet is very good at. Ironically, these are features that are required by most applications independent of data access model.

My belief has always been that on average roughly two-thirds of all queries executed by applications requiring database persistence are projections WRT to the domain model. Unless an O/R framework has a solution for this, it either forces the domain model designer to include awkward “partial types” (i.e. PartialCustomer), utilize some sort of weak typing hack in their domain model (which kinds of goes against O/R in the first place), or translate projection queries into queries which can always generate results that can be materialized (while accepting the performance overhead).

O/R redux

aconrad — Wed, 29 Dec 2004 01:12:00 GMT

One of the big conversations that has been going on for some time in the Java world is how prescriptive container frameworks such as EJB and Spring should be. It is fairly well known, that for O/R mapping frameworks the same conversation exists since O/R persistence is really just one (albeit complicated) service that can be provided by a container. The Java folks like to refer to POJO versus non-POJO solutions. Inside Microsoft, when discussing O/R – we talk about POCO (Plain old CLR objects) and prescriptive versus non-prescriptive. Ignoring the details of implementing the various frameworks (IoC etc), it’s really all the same discussion.

The POCO/ POJO concept is fairly straight forward. The proponents suggest that a good O/R solution should allow the user to map simple domain model classes to the persistent storage without requiring the class to implement interfaces or inherit from specific base classes. In other words, there are no prescriptive requirements for the domain model developer and the types being developed could be used outside of the context of the O/R framework. They are free to develop their domain model as they like and is decoupled from the database schema. I like to call this “design time” transparency.

Nanshan Zeng – a coworker of mine for several years now in both the Xml and ObjectSpaces teams made some interesting points about this topic when we were talking a couple of weeks back. He said that in terms of O/R, the POJO/ POCO is really somewhat of a red herring and that the real conversation should be about the flexibility and extensibility of the mapping solution. In other words, if one is mapping to an existing legacy data store how much of the abstract is leaky and therefore causes the database schema to dictate the domain model type hierarchy.

For example, if my O/R framework only supported mapping a given type to single database table/ view then I would be limited to developing domain model types which were 1-1 with the database tables. Obviously, there are ways to fix this – creating a database view or mapping to a sql query instead of a table, but I think the example illustrates my point.

It seems that although there are some interesting opinions on the amount of flexibility required to make a good O/R framework, there seems to be some good solid agreement that POJO/ POCO solutions must provide non-prescriptive and flexible mapping solutions. And although the anti-EJB crowd would most likely disagree, I am beginning to think that the latter is much more important.

One crazy Italian

aconrad — Tue, 07 Sep 2004 21:06:00 GMT

Luca Bolognese, one crazy Italian, now has a blog… and he is even writing it himself. Luca was Mr. ObjectSpaces back in the day and is now working with the C# team on the sequence operator work.

Comment on a good comment

aconrad — Wed, 23 Jun 2004 17:32:00 GMT

Got a very observant comment on my ObjectSpaces and WinFS Part 2 entry:

My big scare is that merging ObjectSpaces and WinFS will restrict me from using OS as the persistence layer in places where the customer doesn't have and won't upgrade to the latest windows platform or even the latest SP... in these cases, I'm stuck rolling my own O/R mapping and persistence.

if OS+WinFS = use on any platform I can currently run .Net apps on without requiring an OS update, great. If not, elsewhere look must I -- and suddenly OS isn't that exciting anymore..

Yes. We know that is a problem that we will need to solve going forward. There are currently no specifics public details about solving that issue that I can disclose, but I want to reassure that it is a well known problem with the merger of ObjectSpaces and WinFS.

The Good, the Bad and the Ugly

aconrad — Tue, 25 May 2004 23:22:00 GMT

I will be posting more about the changes to the ObjectSpaces release plans in the near future- but in the mean time, I wanted to make sure everyone was aware of that we are willing to listen to any and all feedback about the changes. The following lists all the avenues to do so:

1. Email the feedback alias: orfeedbk@microsoft.com. This will be seen by multiple members of the WinFS program management team.

2. Comment on my blog. I review all comments posted to my blog.

3. Email me directly: aconrad@microsoft.com. I am always perpetually behind with my email, but I read everything eventually.

If you are at TechEd this week, you have a couple of additional options:

4. Come to the Data Access booth and ask for myself or Luca.

5. Come to Luca's O/R talk on Thursday.

6. Visit us in the Sql Server cabana for our scheduled times.

7. Attend the O/R BOF on Tuesday night at 6:30.

I am ready for all feedback. The Good, the Bad, and the Ugly.

Ramblings about POCO*, transparency and delayed load.

aconrad — Fri, 30 Apr 2004 18:16:00 GMT

As someone pointed out, I have been referring to ObjectSpaces as supporting true POCO – which is technically incorrect. ObjectSpaces does prescribe the usage of ObjectHolder and ObjectList for delayed load cases (more of the merits of delayed loading down below). However, beyond that – there is no prescribed type definition. So for the sake of being technically correct, I will start referring to ObjectSpaces as POCO*.

I think this leads to an interesting conversation around whether ObjectSpaces supports true transparency. In other words, can the domain model developer design types for a particular business problem without regards to the fact that the objects will be persisted to a permanent data store? To answer this question, I want to first look at the pillars of Indigo:

Service Boundaries are Explicit and Costly to traverse

Think of the data store as one very complex and powerful service. There is no way to avoid this with current relational database technology unless the store 1) is local, 2) is mostly read-only access or used exclusively and 3) has huge amounts of disk space to support a large amount of indexes. In other words, the boundary between the database and the database application is explicit and therefore all current data access technology cannot support transparent access. In then follows that ObjectSpaces cannot support true transparency. At least the developer who is designing the domain types, the mapping and/ or the database must be aware of this explicit boundary. I also doubt that the developer just utilizing the domain types can even use them transparently. There are still runtime concerns like transaction management, batching considerations and dealing with concurrency errors. I suppose one could develop a framework that abstracts away these realities from the domain model, but that would lead toward making the physical persistence transparent to the domain model – which I have yet to see a framework which completely does that and have doubts it can be done for significantly complex architectures.

So, in general - POCO* minimizes the burden of the domain model developer needing to know details about the datastore, however one must still know the datastore exists and lives across a service boundary. In other words, it offers a nice abstraction but not complete transparency.

Matt Warren has some interesting points about the evils of loading on demand. I for the most part, can’t say I particularly disagree with him. I do though suggest for value types, particularly for very large ones – this requirement justifies the means. At the end of the day I don’t want to download the entire movie when I want to see the date is was created. However that said, I would still like to drill down into what I believe the true evil is: preserving graph fidelity. Think for a second that ObjectSpaces did not support delayed loading (then we would have true POCO instead of POCO*). ObjectSpaces still would support the concept of object identity across queries via the ObjectManager. That is, if the set of results for query 1 and query 2 intersect, then ObjectSpaces is going to add the results to query 2 only for the non-intersecting subset. The rest gets thrown away since it is already in memory. So I am going to take the liberty of extrapolating what Matt said about demand loading being evil to really mean that graph fidelity should not be maintained across query executions.

Let’s table that discussion for a second, and discuss a very specific evil with delayed loading when not using a prescriptive solution like ObjectHolder/ ObjectList. Short of code injection, the only way to support delayed loading is through some sort of context that would know how to retrieve related object(s) for a given object at runtime. Then the problem becomes is what is the meaning of null?

Take for example, the case where I have type A which has a reference to type B called b. I run a query that loads A’s but does not include B in the span. In a true POCO model, b is set to a null value by the engine materializing an instance of A. What does that null mean? That A does not have a corresponding b or that b has not been materialized in memory graph? Really it is quite ambiguous unless of course one takes in account the original query. So it you buy that some sort of delayed loading support is desirable, then ObjectHolder/ ObjectList are a necessary divergence from POCO, unless of course code injection is utilized. But that is another topic for another time.

So where are we now? A limited amount of delayed loading is ok, but completely preserving graph fidelity across query executions is a very dangerous game with very bad consequences for those who embrace it’s dark side. Should preserving graph fidelity really be the job of an O/R mapping framework. Well, I can probably argue each side of the discussion. Perhaps another topic for another time.

So, to conclude – my points:

1) ObjectSpaces does not support true transparency. Unless there is a huge technological jump in datastore technology, this is not feasible. What it does support is a nice away to decouple the domain model from the data access layer, however it is not a transparent technology for persisting objects.

2) ObjectHolder/ ObjectList are a necessary divergence from the true POCO model. It allows the delayed loading of large values on demand while disambiguating the value of null. It however, can be abused (see point 3).

3) ObjectSpaces can preserve graph fidelity across query executions and therefore allow the user to materialize sections of their domain model on demand. This however can be abused and can lead to cases of the in-memory representation being out of sync with the database in a significantly complex way. Like most software, this feature is a tool which can either be used correctly or abused; it is up to the developer to do what is right for their design. Further, utilizing this feature correctly probably means even less transparency for ObjectSpaces (see point 1).

Orthogonal to Orthogonal

aconrad — Tue, 27 Apr 2004 18:34:00 GMT

I think Ralf post about “Should object persistence really be thought of as orthogonal functionality?” is orthogonal to the real problem that ObjectSpaces is trying to solve. It seems his overall point is that by allowing any domain objects to be persisted to a relational store will lead to some cases where the developer designs domain models which have poor performance for persisting to the store. In other words, he is hitting on the age old adage of to much abstraction and balck box magic can give the developer enough rope to hang themselves. Sure, I won’t argue that point, but I believe that is orthogonal to whether ObjectSpaces should be a prescriptive framework or not.

ObjectSpaces is designed to work with POCO (plain old CLR objects) since it gives the developer of the domain model the most freedom. This is designed to be a productivity gain and hopefully will supplant a large amount of the home-grown O/R mapping libraries that exist today. From that stand point, I am not sure I really understand how ObjectSpaces would introduce performance problems that don’t already exist in an applications design. IMO, the potential for performance problems comes from the flexibility in the mapping.

To persist an POCO object, ObjectSpaces roughly needs four things: the current objects, the object state, the original values (including hidden values), and the types mapping information. Currently this information is contained in the ObjectManager, but it very well could be distributed to each domain object via an base class or a transparent proxy. To me, that is almost a question of style – and each model has advantages/ disadvantages. In fact, I think the ultimate O/R framework would support all three models which would allow for the greatest flexibility. However, I believe that is orthogonal to whether the developer can use the framework to develop a poor performing design.

Think of an absurd case where the developer has hundred of types in my system each with a single property each mapping to a unique table my database. This is not going to perform under any scenario since the queries being generated for any interesting scenario are going to be quite complex. Whether or not the developer is using an O/R framework isn’t going to make any difference – independent of whether it is prescriptive or not.

However, I don’t want to be completely naive here. Developers using ObjectSpaces will have to understand that their types are being persisted to a datastore. This means making design decisions in their application roughly analogous to the type of design decisions distributed applications have to make, dealing with optimistic concurrency issues, thinking about transaction boundaries, etc … all the same decisions the typical ADO.net developer has to think about today. I’m not saying it will be exactly the same and it will take some time to figure out what the new programming idioms are when working with ObjectSpaces, but it’s going to be quite similar.

O/R BOF talk

aconrad — Mon, 26 Apr 2004 16:09:00 GMT

Scott Bellware has proposed a Object/ Relational birds-of-a-feather talk for TechEd 2004. If you are interested, please make sure you vote for the topic. Based on the strong showing for the same topic at PDC, this talk should be quite lively.

Talk ideas and dinner?

aconrad — Mon, 19 Apr 2004 17:51:00 GMT

I am giving a talk at TechEd 2004 which we have never given before called:

DAT380 Building Applications with ADO.NET ObjectSpaces

Friday, May 28 9:00 AM- 10:15 AM

Speaker(s): Andrew Conrad

Track(s): Data Management, Developer Tools and Technologies

Traditionally, object oriented business objects and relational databases have differing ways of structuring data. As a result of this impedance mismatch, developers are forced to develop a layer of Object-to-Relational "plumbing" code between their business logic and the database access layers. To solve this common data access scenario, Microsoft is developing an Object-to-Relational data access framework called ADO.NET ObjectSpaces. Learn how ObjectSpaces allows the developer to query, modify or persist their CLR business objects without being concerned with the underlying structure of the relational data store. Building on session DAT301 (Object-relational mapping Using ADO.NET ObjectSpaces), this talk will concentrate on demonstrating how ObjectSpaces can be utilized to solve real challenges faced by enterprise application developers.

I general, I have some ideas about content for the talk, but am open to suggestions. My overall goal for the talk is to discuss the challenges facing the developer in enterprise scenarios, and talk about how one can use ObjectSpaces to solve those scenarios. More importantly, I want to give example of common design patterns I see developers having to utilize as the “glue” between ObjectSpaces and their Domain Models.

In addition, let me know if there is any interest in setting up an ObjectSpaces user group dinner at TechEd.

Data Access Panel at TechEd 2004

aconrad — Fri, 09 Apr 2004 20:06:00 GMT

I will be hosting a Data Access discussion panel at TechEd 2004:

DATPNL Data Access Round Table

Thursday, May 27 11:45 AM- 12:30 PM

The following people will be participating in the panel discussion:

Mark Fussell – Lead Program Manager for System.Xml and English soccer apologist

Pablo Castro - Program Manager for ADO.net and Argentinean soccer apologist

Luca Bolognese – Program Manager for ObjectSpaces and Italian soccer apologist

Arpan Desai – Program Manager for XQuery. (No World Cup soccer affiliation I believe).

The goals of discussion are 1) to compare and contrast the current data access technologies and 2) to discuss overall vision and future direction. However, I would also like at least some (if not a majority) of the panel discussion to be customer driven so hence I am asking for topics and questions. Please email me any and all suggestions.

POCO

aconrad — Wed, 03 Mar 2004 00:14:00 GMT

Encarta definition - a little or slightly (used in musical directions)

ObjectSpaces definition - Plain Old CLR Objects

Dino on ObjectSpaces

aconrad — Wed, 25 Feb 2004 01:43:00 GMT

Dino Esposito has a nice introduction to ObjectSpaces on MSDN. Not particularly in depth, but is an excellent introduction to the API and mapping syntax.

Bare Bones Sql versus OR (?)

aconrad — Fri, 20 Feb 2004 18:46:00 GMT

I have to say I disagree with Ingo’s comments about preferring “bare bones Sql” over O/R mapping technology. IMO, the two topics are orthogonal.

Data Access is all about solving high level use cases. There are no rules out there that say one must give up the explicit, low level control just because an O/R mapping technology is utilized. In fact, I would suggest that a good O/R mapping technology needs to be easy to use AND extensible. That is it should solve all the common problems easily, but at the same time allow low level control as required. Generation of sql statements is just one possible service an O/R mapping technology can provide. Due to the wide range of transactional requirements and relational schemas currently existing, I believe that even a world class O/R mapping technology can only support 90% of the real world, Sql generation scenarios.

In regard to the scenario Ingo suggests. ObjectSpaces will provide the ability for the user to provide their own custom sql statement (including stored procedures) to both select data from the database and to persist changes. ObjectSpaces also gives the user full control over the Sql connections, thereby allowing explicit control of the transaction semantics. Sure, a majority of ObjectSpaces users will not need this sort of control, but it will be there for the ones who do. I really don’t see any reason why ObjectSpaces couldn’t solve the Ingo’s scenario.

2004

aconrad — Wed, 31 Dec 2003 20:05:00 GMT

I started working on the outline for a book I am planning to write about ObjectSpaces. My goal was to get it started before 2004, so there you go - well ahead of schedule.

Please don't tell Dare though, I am suppose to be finishing up a long over due ExtremeXml column for him today. In fact, it is part 2 of an article I wrote eight months ago. Oh well, unless distracted, I will probably finish it today also. My apologies to all the SqlXml'ers out there. Originally I said the new column would be out a few months after TechEd 2003. I guess my New Year's resolution for 2004 is to make my writing deadlines, and of course, as always - to work out more.

The articles are about using XmlSerialization and SqlXml 3.0 to turn Sql Server into an object repository. Basically, it shows how to build a basic object to relational mapping framework using existing, released technologies. In other words, a poor man's ObjectSpaces. However, please note that the ObjectSpaces' persistence engine does not utilize XmlSerialization like my sample application does. In fact, the lexical reprsentation (XML) of the data being exchanged between the client objects and the data store it really unimportant. What is being leveraged is the heirerarchical, semi-structured nature of the XML data model. In other words, XML is an excellent way to representing an object graph and SqlXml is really good at translating the relational data to the XML data model - so by utilizing SqlXml and XmlSerilization, one can create a very interesting alternative data access mechanism for .net applications.

It is no secret that going forward that the SqlXml and ObjectSpaces teams at Microsoft are sharing the core query/ persistance/ persistencemapping code base. In fact, in Whidbey the new three part mapping will only differ only on the “left hand side”. I prefer to say (with a bit of hand waving) that the only difference between the two technologies is angle brackets... but that is probably a bit of an exaggeration. I would however suggest that the two technologies are a lot more similar then not.