RunOfTheMillBlog : Data

Service Boundaries, Business Services and Data

ramkoth — Tue, 09 Mar 2004 04:44:00 GMT

One of the core tenets of SOA is that the services have a clearly defined boundary. Services explicitly choose to expose certain (business) functions outside the boundary. Another tenet of SOA is that a service - with in its boundary - owns, encapsulates and protect its private data. But, there are not too many guidelines out there that talks about what should be exposed and how the boundaries should be drawn.

Keep an eye on 'Data cohesion' when you are designing the service boundary. I will give you an example. A while back, I was involved in architecting a product that has many different modules - like sales, finance, inventory, CRM, billing etc. Our natural instinct was to define each of these modules as a 'Service'. So, we went ahead and defined a Sales service, CRM service, Finance service etc. We defined the entities that are required for each one of these apps and designed the database schema etc. Also, following SOA's design principles, we defined the contracts for the services and exposed different functionalities. Finally, we used web services as a implementation mechanism to expose the services. At the end of the day, we thought we had a pretty good design.

But, things started falling apart the moment we started integrating different services. The main problem we faced was that the 'Customer' entity is required by all the systems. This entity can be updated by all the services. So, we had to implement a nasty 'merge replication' like infrastructure over web-services in order to keep the customer data in sync. To avoid building merge-replication infrastructure across service boundaries, we decided to remove the so called common entities (customer, product etc) and created a separate module or service for them - conveniently named common module. The idea here is that the common module provides services that abstracts the common entities and exposes operations for CRUD behavior (I know…CRUD is a bad word). There is another way to do that as well. For each entity, designate a service as a owner (a.k.a., System of record). You can think of Common module as a system of record for the common entities. So, in theory, both the approaches are equivalent.

So, we went down the path of building a common module (system of record) for common entities. Things worked for a while, before we got into implementing a scenario that actually required a cross-join between product entity and a related entity owned by order management service. Mind you, this is really not a business intelligence scenario. This is a simple search screen that should go to database, do the search and show the results for end-user to pick an item. Performance will go down the tube, if you implement a cross-join between entities owned by different services. So, you have to build EII (Enterprise Information Integration) like solutions, which will most probably replicate related entities, so that joins can be done efficiently in database. And, we all know that building an EII solution is a costly affair.

So, what is the solution? Here is a practical one. "Try to share the database between related services.“

But, 'wait a minute', you may ask - 'you are saying something heretic. Services are not supposed to share data between them'. Well, you are right. But, remember the problems created by simply adhering to core tenets of services. One of the problem with the core tenet of services is that the boundary tenet doesn't talk about data, it only talks about what should or should not be exposed. Since it doesn't talk about data, it is very easy draw service boundaries that may create data related problems later on.

If we still want to remain a purist, we have two choices: 1) re-interpret/relax service boundary rule 2) define a new type of service called business service. I believe option 2 is probably the best path.

Think of business service as a software system that automates one or more business capabilities. A business service may expose one or more services that correspond to each business capability. But, all those services share a logical data model. In other words, boundary is defined at the business service level and not at the individual service level. This is an important lesson we learnt.

For example, think of your favorite ERP system (SAP, Peoplesoft etc) as a business service that exposes one more services - one for HR management, one for Inventory, one for Order management, one for Accounting etc. But, they all share single logical data model. Ever wondered, why big ERP systems such as SAP have a single database that contains the tables required by all the modules? I tend to think this is the reason why they did it. But, I may be wrong.

So, here is the summary of key practical learnings

Define boundary at the business service level.
Business service may contain one or more services, but share data.
If you have legacy systems that you have to integrate, then you can't do much. Business services and their boundaries have been defined for you.

Pagination and SQL

ramkoth — Mon, 22 Dec 2003 23:17:00 GMT

Almost all applications require end-users to search through information. Search results are often shown in some sort of list controls such as Grid, Listview etc. It is very often the case that the search results contain too many records to show in one page. In those cases, search results are shown one page after another, each page containing fixed number of records.

Various solutions have been tried by solution developers. They include creating temporary tables that contain the result set to caching the result set in middle-tier for faster access. There are pros and cons to each approach which I will not delving into.

This article explains another way to do pagination. This mechanism leaves most of the work to RDBMS query execution engine. It works like this.

Presentation layer executes the initial search query based on user preferences. This query returns top <n> rows where n = pagesize.
Presentation layer code then saves information about first row and last row fetched.
When user triggers 'Previous' command, presentation layer goes back to SQL and fetches a page of matched records before the first row.
When user triggers 'Next' command, PL goes back to SQL and fetches a page of matched records after the last row

Here is an example. Let's take the example of Employees table in Northwind database. This table contains following fields (Data types are not included for brevity)

EmployeeId (PK), LastName, FirstName, Title, BirthDate and other fields.

Let's say that we want to search employees by last name. The result sets should be ordered by last name.

Here is the initial query. I am assuming the page size is 20.

Select Top 20 LastName, FirstName, Title

From Employees

Where LastName like @searchpattern --passed as input

Order By LastName

After executing this, presentation layer store the values for LastName from the first row and last row in its state. This will be used in the processing of Previous and Next commands.

Previous Query

When user wants to see the previous page, presentation layer sends a query to SQL that returns a page of matched records. Search predicate should be modified as follows

Where Clause := LastName like @lastname and LastName < <Value of last name from the first row>

Now the query looks like this

Select Top 20 LastName, FirstName, Title

From Employees

Where LastName like @searchpattern and LastName < @lastnamefromfirstrow --passed as input

Order By LastName

Next query

Processing 'Next' is very similar to previous. The search predicate would look like

Where clause := LastName like @lastname and LastName > <Value of last name from the last row>

Select Top 20 LastName, FirstName, Title

From Employees

Where LastName like @searchpattern and LastName > @lastnamefromlastrow --passed as input

Order By LastName

Are we done? Not quite. These queries will work only if the LastName is unique, which we all know is not the case. So what should we do?

In these cases, we should include additional fields that make each record in the result set unique. For example, adding EmployeeId as part of projection would guarantee uniqueness of record in the result set and order the result sets based on LastName followed by EmployeeId.

Here are the modified queries

Initial query

Select Top 20 EmployeeId, LastName, FirstName, Title

From Employees

Where LastName like @lastname --passed as input

Order By LastName, EmployeeId

Prev Query

Select Top 20 LastName, FirstName, Title

From Employees

Where LastName like @searchpattern and

( (LastName < @lastnamefromfirstrow) or

((LastName=@lastnamefromfirstrow) and (EmployeeId < @empidfromfirstrow))

Order By LastName, EmployeeId

This makes sure that even if the LastName is not unique, the previous query returns correct results. Next query is very similar to previous query.

You can see that the query gets complicated due to uniqueness issue. But, it is easy to come up with a general form to frame queries appropriately. Also, one should create appropriate indices to make sure that the performance doesn't suffer. Stored procedures can also be used for faster processing.

Issues with this approach

Main issue with this approach is that there are certain scenarios where things get difficult. For example, if you want to go to LastPage, or page <n>, then you have to stroll through one page after another to go to the page. There are ways to optimize this, but it needs little bit of additional coding and of course, there is a definite perf cost associated with this.

But if your scenarios support only Prev and Next primitives, then this would work just fine.

OR Technologies discussion contd

ramkoth — Mon, 22 Dec 2003 17:50:00 GMT

What a way to start a blog!!! Got slammed by many people for my cut-throat remarks on futility of current OR technologies. But, I love to have continued discussion on this subject. There is couple of things to start with

a) My remarks are derived from interacting with many customers who tried many different OR implementations and got disillusioned with it. Some of them are due to really ugly implementations of the technology (like J2 CMP) to some of them just didn’t really solve their business scenarios.

b) My research involved looking at *some* OR technologies offered by top vendors and I didn’t really spend time trying to evaluate *all* technologies, which would take very long amount of time. My guess would be that at least 1 out 3 competent app developers have built their own OR technology. Yours truly has built two of them in my former life.

c) I am not against abstraction of relational data. My comments are focused on what should be the right abstraction. The OR technologies of today predominantly use Static types such as classes and attributes that maps to tables/views and columns respectively. This is the abstraction that I was arguing against and I strongly believe a more amorphous data structure such as record sets or XML is probably the right way to go about.

d) I realized that I shouldn’t have used the term ‘Datasets’ as it has lot more baggage associated with it than a simple record set.

Sure enough, some of you may have implemented some thing like Field Collections for more flexible query, as one comment describes it, but I am not sure how this is conceptually different from Record Set. In fact, we are on the same page here.

Also, some of the comments imply that OR fits really well on CRUD like scenarios. Perhaps, you are building such a simple app (I can think of departmental app that has only few users wanting to do few things). Even in this case, I am not sure if OR is helpful as the devs need to go through the training to understand this or that OR technology, where as things such as ADO.NET datasets provide those functionality out of box.

Other arguments are about de-coupling database schema from application developers. I can’t agree with this more. But, it takes a leap of faith to arrive at a conclusion that the OR technologies are the better way to achieve this. I have worked with ERP vendors in the past, and I have seen them taking a different approach (IMO, more elegant approach) to solve this. They do the following

1) Define domain model for the application. Here they define business terms such as Customer ID, LastName, firstname, address etc. On top of that they define entities that maps to business concepts such as Customer, Employee etc. They store this information in a repository (usually called as dictionary).

2) Then they define some attributes for business terms for consistent usage. This may contain information such as syntax, data type etc. Then they finally define storage attributes such as column names that should be used etc. This is not very different from ER modeling.

3) Finally, they map entities to tables/views.

So far, this is not conceptually different from OR meta-data (except that the OR technologies I know have no concept of business terms. They tend to work at the level of entities). Where it is different is that the application developers (like HRMS devs) use queries that uses terms from domain model to get the data from back-end. These queries go through a query layer that translates the business terms and entities to actual fields and tables in the back-end. Again, this query layer is an infrastructure component and app devs are abstracted from translation. The whole point here is that this mechanism allows application developers to define a view of an entity and allows to extract information about that view.

So, what is the advantage of all this? I will give you a simple example. If somebody wants to change a column name for a business term called Customer’s last name from Lname to Lastname, then there is *only one* place in the dictionary that needs to be changed. You don’t have to go and change mapping for every entity. In the OR technologies that I have reviewed, I need to go and change the mapping for all the entities that include this attribute, which is not an optimal way to do this and is error prone. Not that OR meta-data can be extended to do this.

Finally, there is always a question of customization/extensibility. This is especially a big problem for ISVs. Usually customization involves extending the pre-defined entity by adding few more fields, or changing an attribute of a field from mandatory to optional and vice versa. If I use static types such as classes and structs, then it would take huge amount of customization effort to change the structure of these types, add mapping information and deploy them. On the other hand, flexible data structures make it less of a pain (you still have the pain of changing code at all the layers). This is a huge topic and should be dealt with separately.

Thanks to Paul Laudeman for summarizing my views, though I didn’t express an opinion on interop. I believe that discussion is worth a separate post. J Here is the link to his summary http://dotnetjunkies.com/weblog/plaudeman/archive/12212003.aspx

At the end of the day, I have changed my positions quite a few times in the past. May be it is time to change my opinion now. May be not.

But, I appreciate your critical comments though. As naïve as I may sound, I think I am a good learner J

Futility of Object-Relational technologies

ramkoth — Mon, 22 Dec 2003 02:28:00 GMT

OR (Object-Relational) technologies attempt to abstract OO developer from knowing intricacies of relational database. It does so by de-coupling the object definition from its persistence. Persistence layer will do the 'right thing' to persist information in the persistent store. It also knows how to fetch information from the persistent store. In general, most technologies use OO language types such as classes or in some case structs to map to tables or views with properties of the type mapped to columns or fields. This post discusses some of the short falls of such technologies.

One of the main issues with attempting to map relational tables to objects is that of flexibility. For example, think of application like Outlook which lets you choose whatever the fields you want to see (like Size, received time, subject etc). In fact this feature is called Field Chooser. I am sure most of the applications require end-users to select a view of data. Relational database lets you create views of a table through select statement. What is the equivalent in OO world? As far as I know there is no way one can create a view of an object statically, leave alone dynamic definition. OR guys may object to this, possibly saying that the application can ignore the fields that it doesn't want to see. This is a lousy argument at best. For example, think of an Employee object that has hundreds of attributes. Let's say that I want to search for an employee and all I want to see is the name, title and department. Let's say that the search returned with 1000 possible matches. If you use OR technology, you probably have 1000 employee object instances out there that contains all the attributes (even though the attributes may not be set) that the user doesn't want to see, but still occupying memory. This is not the optimal use of working-set. Again, OO guys might object to this by saying that in today's systems memory doesn't matter. In my opinion, it always matters and world class applications always use memory intelligently and spartan use of memory is always preferable.

Another related issue is that binding the result sets to presentation layer widgets such as grid or list controls. In the above said case, where the view is dynamic based on user choice, application developer has to write some code to map the object or object set to controls. Instead, if you use datasets, then you don't have to do any extra tricks to map to user controls.

Yet another issue is that of security. For example, certain users may have access to only certain fields. In this case again, application has to write additional amount of code to enforce the authorization.

There are other cases, where OR has no answers at all. For example, continuing the flexibility argument, there are cases, where an application have to send a free-form SQL query to RDBMS. I can think of situations where the users are provided with a flexible interface where users can choose what they want to see and they can define search predicate. One fine example is Query by example (QBE). The implementation might require the application to generate a dynamic sql on the fly that may involve one or more tables (again determined on the fly). I would like to call this dynamic 'aggregation' of data from many different tables. There is no way by which one can achieve this functionality using current OR technology. Objects/classes, by definition are inflexible data-structures. As of today, type system supported by the OO languages or the runtime they are based on, do not support this semantics.

Predominant implementations of OR technologies uses type system provided by the host language. For example, if you take Java or CLR or C++, classes are used to represent an object. Most of these languages or the runtime based on which the language compilers are developed, doesn't support dynamic modification of the types. The type system of OO languages is not flexible where as RDBMS is very flexible by its very nature. One can define a type in RDBMS system, define a view on it, define an aggregated view using the semantics of relational calculus. Sure enough, RDBMS requires huge amount of meta-data that one has to supply in the form of DDL. But, that is the power of RDBMS.

So, the question would then be, what are the desirable characteristics of the data-structure that represents relational information in-memory. The main characteristic, I would think of, is that the data-structure should be flexible - one should be able to make modifications to the structure on the fly. I can think of three possible data structures that fit this criteria. They are

Datasets
XML
Roll your own which will ultimately look like datasets.