Dinesh's Cyberstation

Moving to a new location www.dineshk.net

Dinesh Kulkarni - MSFT — Thu, 28 Oct 2010 17:21:06 GMT

After a long break, I plan to restart blogging - but in a more personal location so I can blog about a few more things than would make sense under msdn.

http://www.DineshK.net

Hope to see you there

Dinesh

.NET RIA Services Resources (for March 2009 Preview)

Dinesh Kulkarni - MSFT — Fri, 27 Mar 2009 02:34:00 GMT

Some of it is covered in my previous post and on Silverlight.net but here is a summary:

Bits (prereq: Visual Studio 2008 SP1, Silverlight 3 beta tools for VS): RIA Services March Preview (unveiled last week at MIX)
Documentation / video

A PDF file with an intro and quick starts is here
If you have an hour or so, you could try the walkthrough in the RIA Services March Preview MSI

MIX talks - videos

RIA Services + Client Overview (BradA)
RIA Services Deep Dive (NikhilKo)

Samples: http://code.msdn.microsoft.com/RiaServices
Questions, comments, bug reports, feedback: .NET RIA Services Forum

Common questions and quick answers:

How is this related to ADO.NET Data Services (Astoria)?
We have a plan to give you the best of both the worlds in forthcoming CTPs. You can see slide #19 in Pablo's talk for an outline. (The March preview does not use Astoria)

Do I have to use LINQ to SQL or EF?
No you do not. You can just use DomainService base class. In fact when you access web services or XML file or some other non-ORM source, it makes sense to just write your class to derive from DomainService. For other ORMs, we are working with a few community members to get provider stories worked out. We also hope to publish a provider sample to help out.

Do I have to use Silverlight?
No you don't have to (though I hope you at least try it out). While we have a great story with Silverlight beta, you could also use RIA Services in a web app with a few early preview bits of ASP.NET DomainDataSource. There is also an SEO story. In future, we hope to bring you AJAX, publishing as a web service etc.

What is the Silverlight 3.0 RTM story?
We are planning to release an updated CTP in the Silverlight 3.0 RTM timeframe to integrate with Silverlight 3.0. We would like to release go-live CTPs later in the year before an RTW (date TBD).

Are you feature-complete? Are there more goodies coming?
There is more stuff to come. So send you feedback in - on forum or one of the team member blogs. So fire away.

What features are you considering?
That is a good topic for future blog/forum posts. I have to think about getting the candidate feature list in a reasonable shape with good description so you can give some feedback.

I tried running this on <blah> with <foo> turned on and <bar> off and <baz> showed up? Is this expected?
You bet that's by design :-). That aside, this sounds like a great question to ask on the forum where people are trying out all kinds of things way beyond the capability and imagination of the little team that built RIA Services preview (and certainly beyond my very limited knowledge).

.NET RIA Services is now public

Dinesh Kulkarni - MSFT — Wed, 18 Mar 2009 21:44:00 GMT

A few folks have asked me what I have been up to after wrapping up LINQ previews and V1. I have not been able to answer that question satisfactorily because what we were working on was non-public (well almost except the mention of now obsolete name "Alexandria" at PDC 2008).

Few other folks have wondered aloud if I was deaf to their feedback about the importance of multi-tier support in LINQ to SQL (and similar projects). I was not deaf but I couldn't answer that question satisfactorily either.

Now I can answer both by pointing to a real preview; not just blah blah.

Take a look at the .NET RIA Services March 2009 preview. It is intended to:

Simplify n-tier development (and really show you a better way to get multi-tier story for your ORM objects - LINQ to SQL or EF or whatever)
Support Rich Internet Application (RIA) development on Silverlight - there is great end-to-end story for validation, authentication, easy databinding etc.
Extend your ASP.NET development experience. This is based on what you know and use already. There is even an ASP.NET DomainDataSource and templates for SEO there (and hopefully some AJAX stuff soon - fingers crossed).

Personally another satisfying thing is that, now I can answer your questions in a straightforward fashion without worrying about what I am allowed to say.

BTW, in the good tradition of the LINQ project, we also have a great walkthrough in the MSI and a good set of quick starts on the download page.

Don't forget to see Brad's and Nikhil's sessions at MIX - either live or after the fact.

Dinesh

P.S. (Thu 3/19 update): We also have some cool additional code for you. There is a new project template that missed the March 09 preview train by a hair - it makes authentication hookup really simple. Two sample apps make it easier for you to build good looking apps with real data hookup on the server-side. I would urge you to actually go through the walkthrough steps first and not use the newly posted, completed solution ;-)

"Essential LINQ" book to be published tomorrow

Dinesh Kulkarni - MSFT — Wed, 11 Mar 2009 08:47:00 GMT

I have been underground for a while - trying to wrap up the LINQ book and working on some exciting stuff for Silverlight. For the latter, my lips are sealed until Mix 209. For the former, it is a different story ...

Last month, I finally finished all the prep work and so did my co-author Charlie Calvert. My editors from Addison Wesley tell me that the book is going to be out tomorrow. Here is the ToC and here is the Amazon listing. I am also adding the ToC to this post.

After Mix, I will have a lot more to say about how we are taking steps in the next interesting area - beyond DALs but I will keep quiet lest I divulge more than I am allowed to.

Dinesh

Table of Contents

Foreword xxi

Acknowledgments xxv

About the Authors xxix

Chapter 1 Introduction 1

Chapter 2 Getting Started 13

Chapter 3 The Essence of LINQ 39

Chapter 4 C# 3.0 Technical Overview 65

Chapter 5 Writing Query Expressions 129

Chapter 6 Query Operators 175

Chapter 7 A Quick Tour of LINQ to SQL 231

Chapter 8 Reading Objects with LINQ to SQL 247

Chapter 9 Modifying Objects with LINQ to SQL 281

Chapter 10 Using Stored Procedures and Database Functions with LINQ to SQL 319

Chapter 11 Customizing Entity Persistence and Adding Business Logic 337

Chapter 12 LINQ to Entities Overview 349

Chapter 13 LINQ to XML: Creation 369

Chapter 14 Querying and Editing XML 387

Chapter 15 XML Namespaces, Transforms, and Schema Validation 423

Chapter 16 Introduction to LINQ Patterns and Practices 465

Chapter 17 LINQ Everywhere 487

Chapter 18 Conclusion 515

Appendix A Tips for Developers 519

Index 547

.NET Framework 3.5 SP1: LINQ perf improvements (LINQ to Objects and LINQ to SQL)

Dinesh Kulkarni - MSFT — Mon, 11 Aug 2008 08:37:00 GMT

There are three perf improvements in the just released SP1. As always, I will let you run your own microbenchmarks or more meaningful app-level benchmarks.

LINQ to Objects:

Specialized enumerable: The new implementation recognizes queries that apply Where and/or Select to arrays or List<T>s and fold pipelines of multiple enumerable objects into single specialized enumerables. This produces substantial improvement in base overhead of common LINQ to Objects queries (at times 30+%).

Cast<T> breaking change: This is a bug fix and a breaking change (see this post for background). The intended use of the NET FX 3.5 Cast<T> extension method is querying over non-generic collection types, whose elements require either a reference conversion or an unboxing step to be used in a generic query context. A late change VS 2008 cycle allowed the cast to succeed in more situations than intended, such as converting float values to int, where it should instead be throwing an InvalidCastException. The breaking change reverts the beta2 behavior and improves perf by simplifying the implementation of CastIterator<T>. Value conversions and explicitly-defined user conversions cause an InvalidCastException instead of being allowed (as in RTM).

var stringList = new ArrayList { "foo", "bar" };
var intList = new ArrayList { 3, 4, 5 };

var strings = from string s in stringList
select s;

var ints = from int i in intList
select i;

The above queries compile to

var strings = stringList.Cast<string>();
var ints = intList.Cast<int>();

You can imagine a simplified implementation

static IEnumerable<T> CastIterator<T>(IEnumerable source)
{
foreach (object obj in source) yield return (T)obj;
}

LINQ to SQL:

This too is a bug fix. The original intent was to optimize id-based queries that are expected to return singletons. If an entity with a matching key value is already in the DataContext identity cache, then translating the query to SQL and executing it against the database is a pure waste of time since the retrieved row is promptly thrown away to avoid stomping on user's existing object. Now that bug has been fixed. So an id-based query will not cause a trip to the database. This results in a dramatic perf improvement (one hash table lookup instead of SQL translation + SQL query execution) in an admittedly narrow but common scenario.

BTW, as mentioned in a previous post, I haven't worked on either component for SP1. But I have been deeply involved in them for 3.5 RTM so I can't resist tracking such sweet changes. Besides, I am working on a LINQ book that keeps me very involved with the components.

Dinesh

Templated code gen for LINQ to SQL

Dinesh Kulkarni - MSFT — Fri, 08 Aug 2008 07:32:00 GMT

This is going to be a brief one. If you are using LINQ to SQL, you should check out Damien's new templated code generator sample. One of our regrets from LINQ to SQL V1 days was not providing an easy way to tailor the code-gen. This is one step in that direction.

Another major step would be to preserve "excluded" tables/columns/sprocs/... in dbml. So any change in schema can be handled appropriately as new/deleted database object (a rename could be treated as a removed column and an added column from metadata perspective).

Dinesh

Relational-Object-XML-Object with LINQ to SQL & LINQ to XML

Dinesh Kulkarni - MSFT — Mon, 28 Jul 2008 08:08:00 GMT

This is old story and I have used parts of it over the last three years to show how LINQ lets you navigate from relational (R) to object (O) or object to XML (X) etc. But for a forthcoming demo, I had to write some quick code today and it looked like it might be worth sharing. So here it is ...

A little background first: I need to show a 3-tier app where the mid-tier can use a variety of Data Acccess Layers or DALs (not just LINQ to SQL ). A good way to show that is how you can use POCO classes with some extra info. An easy way to populate it is from an XML file. But I would like to continue the demo and show a real DAL talking to a real DB and it would be good to have consistent storyboard and data. So I decided to extract a small subset of AdventureWorks data - Employee-PurchaseOrderHeader-PurchaseOrderDetail into an XML file and then use that file for the first part. So what I have is a nice trip across the three domains: R-> O -> X (file) -> O.

Here is the R-O-X part. The object model is crafted up in the LINQ to SQL designer with a bunch of unrelated foreign keys (FK) eliminated. I won't repeat it since you can easily create it with drag-drop-member delete gestures in the designer.

Caution - this is all just a quick and dirty shell to get going. It is not robust with respect to missing files, changed data, test code etc. That is left to you as an exercise

AWExtractDataContext db = new AWExtractDataContext(@"C:\Northwind\AdventureWorks_Data.mdf");
db.Log = Console.Out;

// Preselected employees based on browsing of data
var EmpList = (from e in db.Employees
              where ((e.EmployeeID > 240) && (e.PurchaseOrderHeaders.Any()))
              select e).ToList();
Console.WriteLine("\nNumber of employees: {0}\n",EmpList.Count);

foreach (var e in EmpList)
{
    // Select only five POs per employee
    e.POList = new List<PurchaseOrderHeader>(e.PurchaseOrderHeaders.Take(5));
    e.PurchaseOrderHeaders = null;
}

// Write out a single XML hierarchy with some additional denormalized data
var EmpXML = new XElement("Employees",
                from e in EmpList
                select new XElement("Employee",
                 new XAttribute("EmployeeID", e.EmployeeID),
                 new XAttribute("LoginID", e.LoginID),
                 new XAttribute("ManagerID", e.ManagerID),
                 from po in e.POList
                 select new XElement(("PurchaseOrder"),
                  new XAttribute("PurchaseOrderID", po.PurchaseOrderID),
                  new XAttribute("Status", po.Status),
                  new XAttribute("EmployeeID", po.EmployeeID),
                  new XAttribute("OrderDate", po.OrderDate),
                  new XAttribute("ShipDate", po.ShipDate),
                  new XAttribute("SubTotal", po.SubTotal),
                  new XAttribute("TaxAmt", po.TaxAmt),
                  new XAttribute("Freight", po.Freight),
                  new XAttribute("TotalDue", po.TotalDue),
                  from det in po.PurchaseOrderDetails
                  select new XElement(("PurchaseOrderDetail"),
                   new XAttribute("PurchaseOrderID", po.PurchaseOrderID),
                   new XAttribute("PurchaseOrderDetailID", det.PurchaseOrderDetailID),
                   new XAttribute("ProductID", det.ProductID),
                   new XAttribute("ProductName", det.ProductName),
                   new XAttribute("OrderQty", det.OrderQty),
                   new XAttribute("UnitPrice", det.UnitPrice),
                   new XAttribute("LineTotal", det.LineTotal)
                   )
                  )
                )
               );

EmpXML.Save(@"C:\temp\AWEmpPOs.xml");

Here is a small portion of the XML from the file.

<Employees>
  <Employee EmployeeID="241" LoginID="adventure-works\eric2" ManagerID="274">
    <PurchaseOrder PurchaseOrderID="3" Status="4" EmployeeID="0" OrderDate="2001-05-17T00:00:00" ShipDate="2001-05-26T00:00:00" SubTotal="8847.3000" TaxAmt="707.7840" Freight="221.1825" TotalDue="9776.2665">
      <PurchaseOrderDetail PurchaseOrderID="3" PurchaseOrderDetailID="4" ProductID="530" ProductName="Seat Post" OrderQty="550" UnitPrice="16.0860" LineTotal="8847.3000" />
    </PurchaseOrder>
...

Now in another project, I just took the XML file and created the object hierarchy. This time, I did NOT use any LINQ to SQL or any other DAL-generated classes. I won't bother showing all the POCO classes since the initializer shows you the members and types anyway. But the classes are hand-coded with the auto-implemented properties feature as follows:

public class Employee
{
    public int EmployeeID { get; set; }
    public string LoginID { get; set; }
    public int? ManagerID { get; set; }
    public List<PurchaseOrder> POs { get; set; }        
}

The code to populate the object graph is as follows (hierarchy first and backpatching later)

XElement Emps = XElement.Load(@"C:\temp\AWEmpPOs.xml");

var EmpFromXML =   
        from e in Emps.Elements()
        select new Employee
        {
            EmployeeID = (int) e.Attribute("EmployeeID"),
            LoginID = (string) e.Attribute("LoginID"),
            ManagerID = (int) e.Attribute("ManagerID"),
            POs = new List<PurchaseOrder>(
                from po in e.Elements()
                select new PurchaseOrder {                                
                    PurchaseOrderID = (int) po.Attribute("PurchaseOrderID"),
                    Status = (byte) (int) po.Attribute("Status"),
                    EmployeeID = (int) po.Attribute("EmployeeID"),
                    OrderDate = (DateTime) po.Attribute("OrderDate"), 
                    ShipDate = (DateTime?) po.Attribute("ShipDate"),
                    SubTotal = (decimal) po.Attribute("SubTotal"),
                    TaxAmt = (decimal)po.Attribute("TaxAmt"),
                    Freight = (decimal)po.Attribute("Freight"),
                    TotalDue = (decimal)po.Attribute("TotalDue"),
                    Details = new List<PurchaseOrderDetail> (
                        from det in po.Elements()
                        select new PurchaseOrderDetail {
                            PurchaseOrderID = (int)det.Attribute("PurchaseOrderID"),
                            PurchaseOrderDetailID = (int)det.Attribute("PurchaseOrderDetailID"),
                            ProductID = (int)det.Attribute("ProductID"),
                            ProductName = (string)det.Attribute("ProductName"),
                            OrderQty = (short)det.Attribute("OrderQty"),
                            UnitPrice = (decimal)det.Attribute("UnitPrice"),
                            LineTotal = (decimal)det.Attribute("LineTotal")
                        }
                        )
                }
              )
        };

List<Employee> EmpList = EmpFromXML.ToList();

// Backpatch the references based on collection membership
foreach(Employee e in EmpList)
    foreach (PurchaseOrder po in e.POs)
    {
        po.Employee = e;
        foreach (PurchaseOrderDetail det in po.Details)
            det.PurchaseOrder = po;
    }

// Show some of the constructed graph
ObjectDumper.Write(EmpList.First(), 1);

I found it quite easy and quick to roll this up. You might find it useful where you need to combine data from the three domains and go back and forth.

Dinesh

P.S.

In case you haven't used it before, ObjectDumper is the new incarnation of Console.WriteLine(). It shipped as a sample with Visual Studio 2008 in source form so you will need to build it into a DLL and add a reference to your project. It is well worth it.
Before you say it ... When you have a hammer, everything looks like a nail. But LINQ is an awesome hammer - it often works better than screwdrivers and spanners
I could have used a smart collection to backpatch the reference to the parent object but I just got lazy and wrote some imperative code that traverses the hierarchy again.

LINQ to SQL Tips 9: Understanding DataContext's internal caching

Dinesh Kulkarni - MSFT — Wed, 02 Jul 2008 07:33:00 GMT

Back to the "tips" series after a little break ...

One common question I get is about caching of data in LINQ to SQL. It means a lot of different things so let's start with the following baseline: LINQ to SQL was designed to get you objects from the database and to submit the changes back to the database. It is not intended to be a mid-tier caching component. It is not a cache and it does not pretend to be one. However, it does have certain behaviors that are interesting in this regard:

Object identity

If you want to be able to update the objects you retrieve from the database, it helps to avoid multiple copies of the objects. LINQ to SQL maintains an internal "identity cache" to ensure that within the scope of a DataContext, there will be at most one instance of a given type for a given key value. Think of it like a dictionary indexed by the key value (whether single column or composite key) that stores entities. When you enumerate the results of a query, DataContext looks at each row returned by the underlying DataReader, checks if an instance with the same key value already exists in the dictionary. If an entity is found, it throws away the row and just returns you the existing object from the identity cache. (For those who are more curious, you can look at DataContext.cs, IdentityManager.cs in the sources). As described in another post, we don't stomp on your copy while materializing objects (unless you ask for that using Refresh())

Change tracking

After query execution, you are out of the relational world and into the brave world of objects. There the currency for identity is an object reference rather than a key value since APIs take in objects and not rows or key values. So we also maintain a dictionary that can be looked up by the object. Among many other things, it helps us Attach() a new object by starting to track it.

Enough of the internal view. What does it mean for you?

LINQ to SQL is designed for retrieving and saving objects. Any caching it does is to that end and not for general purpose use you may have. There are other cache components you can look at.
By and large, DataContext holds a reference to an entity that it materializes for the life of the DataContext and keeps it as is unless you explicitly modify it or Refresh() it.

You can ask for it by the key value and retrieve it without remote query (this bug is getting fixed in SP1)
DataContext feels like it has an obsessive possessive disorder. You can't really take away an object from it. It would rather be disposed than give up the precious entities it holds on to. If you don't like the fact that a reference is maitained (i.e. you want to have entities garbage collected), then use the disposable pattern. See this post for more information.

Over time, the entities in the id cache get more and more stale. Fine for reference data; not OK for data that changes in the database more often.
The internal caches make DataContext a write-back cache rather than a write-through cache. We write back on SubmitChanges().
Attach() might look like an Add() on the cache (and it does behave like it in some ways), but it is designed for tracking the object for further changes.
Non-entities don't show up in the internal cache - there is no key for lookup. Most common case of non-entity is a result from a sproc that does not match the shape of an entity.
Turning ObjectTrackingEnabled off can give you a small perf boost - may be useful in a read-only app (only).

Dinesh

An account of Bill Gates' farewell

Dinesh Kulkarni - MSFT — Sun, 29 Jun 2008 00:21:00 GMT

An unusual subject for this blog that has been mostly about components I have worked on. But the event yesterday was memorable and worth sharing. The event has been covered widely so I won't even try to broadly summarize. Insted I will list what I found interesting. I was lucky enough (one of 2k lottery winners out of maybe 20+k who were interested) to get a seat for in-person viewing.

There is no substitute to hearing from the man himself and his business partner of 28 years (and a friend/associate of 35 years or so) Steve Ballmer. The range of topics chosen was very enlightening - from "How did your parents react to you dropping out" to "What was the biggest screw up and what did you learn from it". Here are three things I took away from it:

Trust and personal networks matter the most

There is no substitute for personal networks and personal trust and gut instinct. For all the talk of globalization, outcome/evidence-based hiring and rewards, creative processes like forming businesses are deeply personal rather than global; intuitive rather than analytical and grounded in optimism/aspirations rather than backward looking statistics. Maybe the takeaway suits my personality profile but I am always uneasy about decision making processes that try to reduce human judgement to an automaton. SteveB was a known quantity, his passion and intelligence was vetted long before there was any Microsoft, let alone a job opening.

How others see you differs from your self-model

Peronal motivations and self-image can be very far apart from the motivations attributed by others to you and their image of you. This gap is very important to understand. I was not an MS employee during the major anti-trust proceedings. But after hearing from BillG and SteveB about their mindset and approach and after hearing quixotic evil intentions attribute to my team and I during the last few years, I think the aforementioned gap explains a lot.I won't even bother talking about our good intentions. That is too touchy feely. But from what I have seen, my colleagues in various teams and I are so preoccupied with building what we are building and getting it out the door (before it gets killed for some bizarre internal reasons :-) ) that we have no time or smarts left to even think about doing any sinister things. Let alone plan for it years in advance and execute on it to perfection. Heck, like software engineers everywhere, we have a tough time estimating and meeting the estimates for the most basic product/component we would like to ship.

Decisions are based on aspiration and hope rather than just analytics

There is no shortage of blog posts or news items calling for radical strategy/personnel/... change. Most wonder why those in the position of power cannot see what the numbers and facts show. A lot is based on intuition about strategy/leadership/vision that can be counter to facts on the ground at a particular point in time. That doesn't validate the intuition or negate the facts. It simply means that it is futile to look for the fact-based-decision making automaton. Hear the vision, strategy and then lead. follow or get out based on your personal judgement.

Like the takeaways, my process of arriving at them was deeply personal. Take it (or not) FWIW.

Dinesh

Design of LINQ to SQL - What was I thinking or was I?

Dinesh Kulkarni - MSFT — Wed, 25 Jun 2008 08:11:00 GMT

A little break from my "LINQ to SQL tips" series of posts. A recent vote of no confidence on a related component orchestrated by community activists reminded me of many questions I have fielded and how the design team approached the design of LINQ to SQL (and also core LINQ APIs and C# language changes for LINQ). Nah, that��s for another day when it is cloudy and raining. Instead, let’s talk about my recent dream. Or rather, a nightmare!

But first turn off your flame throwers, grab a cup of coffee and don't take this too seriously ...

I had this Q&A nightmare about the component I worked on - LINQ to SQL. I am the "expert" providing the non-answers.

Q: How do I use blah pattern with LINQ to SQL (e.g. blah = ActiveRecord if you don't like abstract concepts)
A: You don't!

Q: I think I wasn't sufficiently clear. I stood on one leg and when the phase of the moon was 64% of full, it worked but now that the moon is waxing further, your foo method throws bar exception when I do baz. How do I just get that bit working with LINQ to SQL.
A: You don't

Q: (By now quite upset) Do you even understand blah?
A: (Forced to be less terse) Yes. We considered blah and decided against it for a set of reasons listed below. That pattern is not consistent with the core design assumptions and recommended usage patterns with LINQ to SQL.

Q: (Now a full-force verdict) I hearby find you guilty of violating the implicit agreement to solve the world hunger problem using blah methodology. Hence, what you produced is useless, evil and must be stopped at once. Any software built using your component will accelerate global warming and cause all glaciers to melt at once. And of course, it will irreparably damage the young and impressionable minds of generations of developers leaving them utterly useless for anything except writing some old fashioned code.
A: Thank you for your interest in LINQ to SQL err blah.

Nightmare aside, (what) were we thinking? Stay tuned for that ...

P.S.

1. This release includes backward-looking statements intended to qualify for the safe harbor from liability established by the Public Flagellation by Community Act of 2008. These backward-looking statements generally can be identified by phrases such as "did", "was", "thought" ... and by the absence of "will" "fix" "in future release".

2. I just wanted to get you objects from the table. I swear. Nothing more than that!

3. Scott B., if you are reading this, peace! I won't let you drag me on to the stage at another PDC BoF and I won't fix anything either. I can't. I don't drive LINQ to SQL anymore. I just drive righteous developers crazy J