So thanks Danny for giving me a ping... it's been a while since I participated in one of these blog chain letters. Funny really because now I am a Program Manager, I'm not a real developer anymore ;)
How old were you when you first started programming?
I remember mucking around with my Vic20 when I was 10.
How did you get started in programming?
My university degree was in Engineering Science, i.e. Operations Research and Continuum Mechanics. It turned out it was not really for me, but I did something out my time at university: I learnt how to program. I remember mucking around trying to write an application to work out how to optimize an airline crew roster.
What was your first language?
I did a little Basic on my Vic20, but my first real programming was in Fortran, and then C.
What was the first real program you wrote?
What exactly does real mean? Every program I wrote is 'real'.
But I jest. To me real means something that has actual users, bugs, quality expectations, and a roadmap. I wrote a number of VB5 forms applications that measure up to that definition, but one really stands out, it was a Timesheeting / Invoicing application that used DCOM and actually had a couple of tiers!
What languages have you used since you started programming?
Basic, C, C++, C#, F#, Fortran, Haskell, Java, Javascript, Python, Ruby, Nermerle, T-SQL, VBA, Visual Basic 3, 4, 5, 6, Visual Basic.NET.
What was your first professional programming gig?
Olympic Software as a junior dev, working on a VB 3 point of sale integration application. The project went nowhere in the end, so I got my first canned project out of the way pretty early in my career!
If you knew then what you know now, would you have started programming?
Not sure. I like programming, but I love creating, and there are lots of ways of doing that, I kind of stumbled into computers and I seem pretty good at it... but I'm sure I could do other things as well. I definitely have the requisite ego!
If there is one thing you learned along the way that you would tell new developers, what would it be?
Learn to ask good questions. Question assumptions, question motivations, question rules, question the status quo, question the process, question the requirements...
Learn to ask good questions!
What's the most fun you've ever had ... programming?
Writing Base4.NET, without doubt. It's fun creating something that is synergistic, and more that the sum of it's parts. System's Engineering at its core, it is like playing a giant game of LEGO, except you get to decide whether to mold your own special pieces or work with what's already available.
Who’s next?
Jarek has released his sample provider for Oracle here.
In his post he talks about a few of the more notable challenges he had and how he got around them.
If you are writing an EF provider this is gold!
Just in case some of you don't know, I thought I should point out Roger's Blog.
I refer to Roger's blog at least once a day to keep up to date with what is happening in the LINQ, Data Programmability, Entity Framework and Data Services worlds.
Its a great resource.
You know you've met a visionary when they start talking about solving problems others think are too hard.
It is super easy to throw up all sort of reasons why some grand vision won't happen, here are but a few of the classic excuses:
- Its too hard
- Academics have been working on this problem for years and there are well known theoretical obstacles in your way.
- We've been trying to solve this problem since 19xx and it never works
- You are ignoring the lessons of the past
- If you can't solve the problem 100% correctly its not worth trying
- One size doesn't fit all.
Now each of these may be kinda true. But I say:
Problems tend to look worst to the people who know the most about them, they can see all the gotchas etc, but have you ever notice how things only seem to happen when you do something? Sometimes things that look hard aren't that bad once you actually try. Knowing a lot is great, but you should never use it as an excuse to not strive for improvement.
Academics are typically not as connected to users as you are, which means you can redefine the problem, into something that is both useful and, this is the key, can actually be solved.
Okay so in 19xx so and so tried to solve the same problem and failed or produced another set of 'intractable' problems, the fact is it is now 2008 and a whole lot of things you couldn't do back then are trivial now. You know the whole standing on the shoulders of giants thing.
The lessons of the past tell you how to fail, but the lesson is not: don't even try.
Sure it is hard/impossible to solve a problem 100%, but that doesn't mean the 80%, 60% or even 1% improvement you get trying isn't worthwhile!
One size doesn't fit all, sure. But life is full of patterns, not all of which have been catalogued. So don't try to stop someone from looking for more patterns, occasionally they might actually notice one that is useful.
I suppose it boils down to this, rules are all well and good, but we should never be tempted to over apply them, the world isn't black and white, ambiguity rules OK.
The real question then is Why not?
MEST stands for Multiple EntitySets per Type.
It is one of the features of the Entity Framework that is typically missing from ORM stacks.
The idea is that you can store the same EntityType in multiple EntitySets. For the purposes of this discussion you can think of EntitySets as analogous to Tables.
Now MEST works very well for very simple Entities that have no associations. For example you can easily take something like this:
And add a second EntitySet:
Which of course in ObjectServices means you can do things like this:
var good = from c in ctx.GoodCustomers
select c;
var bad = from c in ctx.BadCustomers
select c;
MEST is also why you see the name of the Set in the convenient AddToSet(...) methods on the ObjectContext, you get to choose the EntitySet that the new Entity should live in by name.
i.e.
Customer good = CreateGoodCustomer();
ctx.AddToGoodCustomers(good)
Customer bad = CreateBadCustomer();
ctx.AddToBadCustomers(bad);
Seems pretty straight forward doesn't it.
Unfortunately in V1 of the Entity Framework there are definitely some gotchas when your Entities have Associations too.
For example imagine you have this model:
What we have here is the 4 EntityTypes and 3 AssociationTypes. As you can see (or at least you could if I had taken the time to put cardinality into the picture) a Customer has Orders which have OrderLines each of which are for a particular product.
Now of course to use this in the Entity Framework you need to create 4 EntitySets (Green Boxes) and 3 AssociationSets (Green Arrows) too:
No problem. That works great.
However if you try to add a second EntitySet for Customers now, things start to get a little tricky:
The problem is that you can't create 2 AssociationSets of the same type (in this case CustomerOrders) that point to the same EntitySet on one end (i.e. Orders).
So I can't link both GoodCustomers and BadCustomers to Orders.
Why? Well the problem is that Associations are always bi-directional, unlike foreign keys, so you need to be able to navigate an Association in both directions.
Which in the above model you can't. See if you go from an Order to a Customer via the CustomerOrders association the target set, either GoodCustomers or BadCustomers, can't be determined, it is ambiguous.
The EntityFramework in V1 just can't handle this ambiguity.
Going forward we are trying to work out how to handle this situation gracefully.
Still today the only workaround is to duplicate the whole graph.
I.e. in this case we started with 4 EntitySets & 3 AssociationSets, and to make one of those EntitySets MEST, I need to clone everything, so I end up with 8 EntitySets & 6 AssociationSets:
Without doubt this occasionally makes sense, for example if the Orders, OrderLines and Products for a BadCustomer are stored in a logically different place to those for GoodCustomers.
Unfortunately often that is not the case, i.e. you don't really have two tables of Products, I have just one.
So to use MEST in these situations you end up having to lie to the EntityFramework, by creating fake tables* in the SSDL using a view (DefiningQuery) and Insert, Update and Delete Functions.
Not exactly ideal.
The moral of the story, I think, is that MEST can be useful, particularly if you have no associations or you already have two logically distinct graphs.
Otherwise MEST can be quite tricky.
Hopefully now you understand MEST a little better, and can see it for what it is, just another tool in your modeling arsenal.
Use as appropriate.
*See my post on Associations with Payloads for some tips here if you need to do this.
Okay this Fantasy Soccer thing has got me thinking.
What is this thing going to look like from about 100,000 feet? I.e. what are the big bits?
Well being a pure geek, I really want to play with new shiny stuff (i.e. the new ASP.NET MVC Framework) but that has to be balanced against a desire to be real world for the average developer 'today'.
Then it dawned on me that actually I have two Applications too write, an Administration App and a Fantasy Soccer Player App.
So I can use traditional ASP.NET for one and nice shiny MVC for the other.
Now given that requirements (i.e. my stupid ideas) are changing all the time it is a little premature to say this is a final architecture, but this is what I think things will look like today:
Exactly what is going to happen inside each of those boxes is definitely still up in the air, but the basic requirements are as follows.
The Administration Site will allow administrators to:
- Create Soccer Stars, Managers, Teams, Stadiums, Matches etc.
- Enter and Approve results
- Alter Soccer Star prices
- Award points to players
- Run reports
- Moderate team names etc.
- ...
The Player Site will allow Soccer fans too:
- Signup and Create a team
- Alter their team
- Monitor their team's points
- Monitor leaders
- Trade players
- Signup for news
- ...
My alert readers may have noticed that in my above diagram there are no N-Tier issues. I am not trying to pretend that in the real world these issues don't exist, I just want this application to grow little by little.
So don't fear I definitely plan on addressing the N-Tier issues at some point. Perhaps I'll have some sort of smart client application talking to the Administration Site, or some sort of Silverlight client to the Players site? Who knows, not me that's for sure, I'll still gathering requirements from myself, and like most Clients, I want the world, but I don't want to pay!
So for now let's just grow into this, iteratively, starting with the basics.
Next time I will probably be ready to start looking at what the Entity Data Model for our Fantasy Soccer application is going to look like...
Most of you probably know that the Entity Framework architects made a very conscious decision to make all trips to the database explicit.
The result is that this:
Person mother = person.Mother;
will return null, even if the person has a Mother, unless you either eager load (using Include) or explicitly load using Reference.Load();
i.e.
person.MotherReference.Load();
Person mother = person.Mother;
Still there are a lot of people out there who want true Lazy Loading. Which is possible but not easy today.
In steps my crazy colleague Jarek.
Jarek has put together a very interesting project that uses the EF's custom entity interfaces (i.e. IPoco) and some pretty cool magic to do lazyloading... among other things.
Do you think I think you should look at it?
If not, then just to be clear: take a look at this.
Today I spent a bit of time working on the data model for my fantasy soccer application.
I don't want to talk about the actual model today though, because I'm not really ready.
What I want to do is talk about a problem that occurred to me, namely Unit Of Measure.
Any real sport fan knows you need to know what kind of shape someone is in, did they put on too much weight in the off season? In cycling this stuff is deadly serious, Jan Ullrich was famous for coming back from winter, carrying "too much" weight, in fact so much so that it often took Jan until the 2nd to 3rd week of the Tour De France to get into optimal condition.
Anyway the point of all this is that weight has units, and if I tell you someone's weight is 120, you are none the wiser until I tell you the units (pounds or kilos).
Now if you have a database being populated from information gathered around the world you almost certainly need to deal with this sort of problem. In NZ you get Kilos, in the US pounds etc etc.
If you don't know the units in a particular row, you can't do meaningful comparisons, i.e. something like this makes no sense:
var heaviest = from athlete in ctx.Athletes
orderby athlete.Weight descending
select athlete;
Well ComplexTypes are good for this sort of thing i.e.
<ComplexProperty Name="UnitOfMeasure">
<Property Name="Value" Type="Decimal"/>
<Property Name="Scale" Type="Decimal"/>
<Property Name="Units" Type="String"/>
</ComplexProperty>
If you now define the athletes weight to be a UnitOfMeasure you know both the Weight and Units:
i.e. if you are operating in Kilos you say something like this:
athlete.Weight.Value = 90;
athlete.Weight.Scale = 2.2M;
athlete.Weight.Units = "kg";
Which is equivalent to this:
athlete.Weight.Value = 198;
athlete.Weight.Scale = 1;
athlete.Weight.Units = "lbs";
Because there are 2.2 lbs / kilo.
At this point the we can reason about the athlete' weight:
foreach (Athlete a in ctx.Athletes)
Console.WriteLine("{0} weighs {1} {2}",
a.Firstname,
a.Weight.Value,
a.Weight.Units);
Unfortunately sorting is still not possible.
However if you add a helper class like this:
public class ScaledAthlete
{
public Athlete Athlete;
public Decimal ScaledWeight;
}
And add this property to your partial ObjectContext class:
public static IQueryable<ScaledAthlete> ScaledAthletes{
get{
return from athlete in this.Athletes
select new ScaledAthlete{
Athlete = athlete,
ScaledWeight = athlete.Weight.Scale * athlete.Weight.Value};
}
}
Then you can write code like this:
var heaviest = from a in Athlete.ScaledAthletes
orderby a.ScaledWeight descending
select a.Athlete;
foreach (Athlete a in heaviest)
Console.WriteLine("{0} weighs {1} {2}",
a.Firstname,
a.Weight.Value,
a.Weight.Units);
And it will work nicely, with all the real work happening in the database.
Some key points:
- We need to use an expression to for ScaledWeight that can be converted to a database expression: i.e. ScaledWeight = athlete.Weight.Scale * athlete.Weight.Value is simple enough to be converted to T-SQL.
- We need the ScaledAthlete class because otherwise it would be an anonymous type, which of course you can't use in a method signature (i.e. IQueryable<var> is not valid).
- We project the actual Athlete entity into the Athlete field so that if we actually want the athlete it is available and more importantly it is Attached to the underlying ObjectContext, meaning of course we can make and save changes as required.
The thing that is nice about this approach is the scaling is built-in to the datasource (i.e. ScaledAthletes), so it is just there whenever you issue a query over ScaledAthletes.
At this point you might be surprise to find out there is a much easier way too.
You can re-write this using a let like this:
return from athlete in ctx.Athletes
let weight = athlete.Weight.Value * athlete.Weight.Scale
orderby weight descending
select athlete;
Or my even without the let:
return from athlete in ctx.Athletes
orderby athlete.Weight.Value * athlete.Weight.Scale descending
select athlete;
Notice you don't need a helper class, a helper method, and you can have more control over the resulting query.
So why the big detour... well as I said in my statement of intent this is supposed to be warts and all, what you just saw is how I got there and what I learnt along the way.
Happy measuring.
Yes, yes I know the real name of Soccer is Football, but I don't want there to be any confusion for any Americans, Australians or New Zealanders who might be reading this ;)
So what are the basic rules of Fantasy Soccer?
The Basic idea is that you get some fantasy money, lets say $100,000,000 to put together a fantasy team.
You do that by picking a full roster of players (typically from those that play in a league somewhere, for example the Premier League).
These players have a cost, and just like in real life the better ones cost more. The objective of course is to build the "Best" team.
The way "Best" is defined is where things get interesting.
Every week the real league has real games, and hopefully the players in your fantasy team, are involved, making strong contributions for their real teams.
I say hopefully because each player gets allocated points based on their performance for their real team. They get points for all sorts of things: scoring goals, making assists, making saves, time on the pitch, keeping a clean sheet etc.
Those points are summed for each of your players each round and added to a rolling points total for your "Team".
The "Best" team is the one with the most points at the end of the season.
Now of course you would generally expect the better players to get more points, the real challenge is to work out which players have been under priced and over priced etc, and pick accordingly. I.e. its a lot like playing the stock market.
All this is starting to sound a lot like fun.
If I've missed out something really important, don't hesitate to let me know. I don't claim to be an expert on "Fantasy" football, but I am pretty good at armchair coaching!
Next time I'll describe the application I want to build, its major parts, how it hangs together etc.
At this stage only one thing is certain, it will use the Entity Framework for data access.
Okay so it has been a while since I posted my Statement of Intent. Time to get on with things.
The main problem has been trying to come up with a good application.
I gave myself three key requirements:
- Well suited to the web.
- Ability to grow uncontrollably in scope: so as to include hooks into things like Workflow, Silverlight, Sync, WCF, Office etc.
- Fun.
Now I think (1) and (2) are pretty easy, (3) is the kicker.
Without (3) you could think of things like a fake online Bank or Insurance Company. But how do you make something like that fun?
You tell me and then we will both know.
One interesting suggestion (thanks Jarod) was a fantasy football system. Now for me Football means Soccer (you know Chelsea, Man Utd and the Gunners etc)
So the fantasy football league idea is interesting. The only problem is I've never actually played a fantasy football game, so I have only a rough idea how it would work.
Sounds like I need to do some requirements gathering...
Yesterday I was talking to a Simon, another super smart dev on the EF team, and he raised a big concern, one that I am sure you can all relate too:
We've all seen something like this:
And thought: "What is the bare minimum I can do here to save a Whatever?"
This "What properties are required ?" problem is particularly tricky when working with database Entities, because very often lots of the properties are nullable or server generated, but intellisense provides no clues.
Now my conversation with Simon reminded me of a couple of posts about intellisense planning I wrote on my old blog.
The basic idea was writing code with the aim of streamlining the programmers intellisense experience.
So yesterday Simon and I started having a think, and realised that perhaps Intellisense planning is the answer to this "What properties are required ?" problem too.
In fact all you need to do is this:
1) Create an interface for each entity called something like IRequiredForX, and then add only the properties you need to that interface:
i.e. something like this:
public interface IRequiredForPerson
{
string Firstname {get;set;}
string Surname {get;set;}
Gender Gender {get;set;}
}
2) Make your class implement that interface, and (this is the key piece) add a property to return itself cast to the interface:
public class Person: IRequiredForPerson
{
public IRequiredForPerson Required
{
get { return this; }
}
Now the programmer can use intellisense to see only what is required:
This is the sort of thing you could easily do in partial classes that extend the classes the Entity Framework creates for you. In fact, you could even write your own CodeGen to do this for you.
Never underestimate the power of intellisense, it can make all the difference.
Nifty huh?
Thanks for the idea Simon.
PS:
In case you're wondering, no I haven't forgotten about my statement of intent! I am working through a big list of possible 'real world' applications at the moment.
Expect some action on that front soon...
This blog has been a little quiet for a while now.
With good reason though.
I've been doing the hard yards answering questions on the Entity Framework forum.
To do so I had to ask literally hundreds of questions of the rest of the Entity Framework team. Guys like Colin must be completely sick of me barging into their office by now!
Now because the Entity Framework is a complicated thing, it seems no one person understands it 100%, so while answering questions I've been consciously working on building a kind of mental model to help me understand the Entity Framework.
So aside from the obvious benefit of actually answering forum customer's questions, this exercise has been hugely beneficial for me.
I have learnt a lot about two key things: the Entity Framework itself and more importantly the problems Customers are really having with the Entity Framework.
Those two things have lead to a renewed sense of purpose for this Blog.
<Intent>
I'm going to start creating 'real world' applications using the Entity Framework, and I'm going to blog about all the tradeoffs I make, the hoops I have to jump through, the gotcha's I encounter, the workarounds, the design considerations etc.
So what do I mean when I say 'real world'?
Well I mean applications that you might actually find useful or can relate to: apps with multiple tiers, with a little SOA thrown in, maybe a pinch of mid-tier caching, some data-binding, serialization, validation, auditing, work-flows, thin clients, batch processing, queuing, maybe a bit of WPF, WCF, Biztalk, Office and Sharepoint integration. Basically whatever helps to get the job done.
</Intent>
All I need to get me started, is a real world application. Now while I have lots of ideas, I really want to hear from you.
Anyone have any ideas?
Colin is a super smart colleague of mine, who works as a developer on the Entity Framework.
So it is great to see him start a blog.
His first post is super cool, because he discusses a set of extensions he put together that get around some of mapping limitations of the Entity Framework.
He does this by writing a custom entity materializer. This custom materializer supports, arbitrary stored procs, multiple recordsets etc... all very cool.
Here is a little snippet that illustrates what is possible:
var results = context
.CreateStoreCommand("GetCategories", CommandType.StoredProcedure)
.Materialize<Category>()
.Bind(categorySet);
What this does is execute the GetCategories stored proc, turning results into Category objects and binds them to the categorySet entityset so that changes can be tracked/managed by the Entity Framework.
How easy is that?
Take a look for more.
Michel Perfetti, has taken my little Maybe thingie and gone a lot further,
by using expression tree re-writting he has made it possible to express this:
string code = licensePlate.Maybe(lp => lp.Car)
.Maybe(c => c.Owner)
.Maybe(o => o.Address)
.Maybe(a => a.PostCode);
which is NULL safe but very cumbersome like this:
string code = licensePlate.Maybe2(lp => lp.Car.Owner.Address.PostCode);
which is much cleaner and still NULL safe.
Michel does this by declaring Maybe2 to take an Expression, rather than a Func i.e. it has this signature: public static V Maybe2<T, V>(this T t, Expression<Func<T, V>> expression)
Then he cracks open the expression and rewrites it to do each property access inside a call to Maybe.
Nice stuff Michel.
UPDATE: Sorry for originally getting your last name wrong Michel, and thanks Matthieu for pointing it out!
But I only just realized that extension methods are cool for avoiding NullReferenceExceptions.
We all know that if you have something like this:
LicensePlate
licensePlate = null;
Car car = licensePlate.Car;
It will throw a NullReferenceException. However if you have an extension method
public static Car GetCar(this LicensePlate
lp);
we can call it like this:
Car car = licensePlate.GetCar();
And we can still handle the fact that 'licensePlate' is null gracefully. i.e.
public static Car GetCar(this LicensePlate
lp)
{
if
(lp == null) return
null;
return
lp.Car;
}
Now imagine you wanted to chain a whole series of calls something like this:
string
code = licensePlate.Car.Owner.Address.PostCode;
That is incredibly unsafe, because any one of those properties could potentially return null.
To get around that you could use a general purpose extension method using generics, to allow you to go through a set of arbitrary translations, anyone of which could do nasty things like return null:
public static V Maybe<T, V>(this
T t, Func<T, V> selector)
where
T : class
where
V : class
{
if
(t == null) return
null;
return
selector(t);
}
Now you can use this to chain a whole load of unsafe methods together.
string
code = licensePlate.Maybe(lp => lp.Car)
.Maybe(c
=> c.Owner)
.Maybe(o
=> o.Address)
.Maybe(a
=> a.PostCode);
It looks a lot like the Maybe monad (hence the name). The interesting thing though, is that if you implement your methods as extensions methods rather than instance methods, you can wire this sort of thing in so easily, without needing to use lambdas at all.
i.e. so rather than this which is unsafe:
string
code = licensePlate.Car.Owner.Address.PostCode;
of this which is ridiculously cumbersome and hard to read:
LicensePlate
licensePlate = null;
Car car = null;
Person owner = null;
Address address = null;
string code = null;
if
(licensePlate != null)
{
car = licensePlate.Car;
if (car != null)
{
owner = car.Owner;
if
(owner != null)
{
address = owner.Address;
if
(address != null)
{
code = address.PostCode;
}
}
}
}
or this which is a little complicated for anyone uncomfortable with lambdas:
string
code = licensePlate.Maybe(lp => lp.Car)
.Maybe(c
=> c.Owner)
.Maybe(o
=> o.Address)
.Maybe(a
=> a.PostCode);
you could write this:
string code =
licensePlate.GetCar().GetOwner().GetAddress().GetPostCode();
Which is almost as easy to read as the original version. But much safer.
All you have to do is write your extension method to handle nulls.
Sometimes the coolest things are right under your nose.
