RunOfTheMillBlog : Architecture

Architects...Who? - Part II

ramkoth — Tue, 14 Mar 2006 02:00:00 GMT

Let’s talk about architects in Microsoft. The architects in Microsoft I know grew threw the ranks as Software Design Engineers and are extremely technical in nature. If a normal programmer knows 10 things about a thing, then an architect for the thing knows 500 different things about the thing. For instance, Christopher Brumme probably knows 500 different things about GC than a competent .NET Programmer. Roughly, architects in Microsoft product dev team do the following things

- They usually own a specific area of a product and come up with complete design (to the level of lowest granularity). Their tremendous experience in software engineering and their ship record helps them to look around corners, optimize things based on the underlying platform and provide guidance for developers to implement a particular strategy in a most optimal way.

- They work with other feature teams and architects to make sure that whatever they design seamlessly work with other components.

- They are extremely technical. This cannot be stressed enough.

- They write code – a lot of code actually.

- They communicate really well – one thing I have appreciated and trying to learn over a period of time is the clarity these guys bring to table. It is as though they speak the language of context-free grammar. You can’t interpret a sentence in two different ways.

They have huge credibility and it is due to their proven technology depth, engineering skills and their vast knowledge.

Let me digress and talk about the other types of architects I encounter (outside product teams) on a daily basis.

Beware of VNAs

I keep hearing this new term called Vendor Neutral Architects. First time, when I heard this, I spilled the NaaN that I was eating. (NaaN is not ‘Not a Number’, this is an Indian bread type J.) From what I hear, these guys are supposed to know all the technologies provided by all the vendors at the goriest level of detail for a particular problem domain to help customers make a knowledgeable decision. For example, if the domain is Messaging, then the VNA should know Exchange, SendMail, Lotus Notes etc in all their gory details and help customers choose the technology based on whatever the customer needs are. From what I see, you can count people with deep skills in all competing technologies in any domain. For example, it must be easy to count the people who can compare and contrast CLR and JVM at the minutest level of detail (for instance how the JIT optimization works in CLR and JVM or the GC heuristics used by different runtimes) or for that matter people who knows how to compare SQL Server Query Optimization vs Oracle’s query optimization. It is of course obvious that a person who has this knowledge will be of tremendous value to any team, but it is just that they are very hard to find.

From my experience, the VNAs I have met are definitely vendor neutral – in the sense they don’t know any vendor technologies at any level of detail. These people seem to have all the other ‘right’ attributes like socializing, playing politics, playing golf, nice dress sense etc. I am not generalizing, but this has been my experience so far. Some of these architects hold a view point that they only design at high level (i.e., UML and Visio) and leave the details to developers. Hello? You can’t write code, you are not an architect. Sorry – you didn’t pass the necessary condition (at least not in MTC).

So, I am going to use two variations of architects - Architect with capital A to denote MS Software Architects type; and an ‘architect’ with small caps to represent the solution architect and VNAs as defined (or loosely defined J) by world at large.

We want Architects not architects. If we come up short, I would settle for experienced software design engineers with good customer handling skills.

PS: These are all my personal view points and doesn't reflect view point of my employer.

Architect...Who? - Part I

ramkoth — Mon, 13 Mar 2006 09:33:00 GMT

It’s been an exciting job building a Microsoft Technology Center from the scratch – all right almost from scratch. The most important job of creating and selling the business value prop behind setting up a MTC is done by my boss – Sheila Gulati. From that point, I was tasked with setting up the MTC in Bangalore and this includes logistics, hiring, marketing (internal as well as external), partner management, lab set up and budget management – basically the whole nine yards of setting up and running a business. Given that I have been a techie all along, it’s been a wonderful experience for me to learn about business side of things and needless to say, I am learning a lot these days.

Let’s talk about hiring for a moment. I have to hire a set of technology architects and a bunch of contractors for managing logistics part of MTC. Needless to say, hiring right technology architects is essential to survival. Basically, the technology architects in MTC Bangalore should own a particular practice – a practice could be Connected Systems, BI, SQL, Collaborative solutions, Infrastructure Optimization, High performance computing etc. Owning a practice means the following things

- Ability to map customer business problems and pain points to technical solutions. TA is not expected to be a domain expert and it is expected that Account Manager and field folks who are associated with that particular opportunity has to help our TA to understand business context. But, TA should be able to take the business problem and map it to MS technology/solution platform.

- Ability to design a solution (widely called as solution architecture) using Microsoft Platform. Oh, by the way, when I say Solution Architecture, I don’t just mean the routine ceremonial, hand-waving high level visio/UML type architecture, but the low level design that includes how exactly the solution be built with MS with focus on performance, security and extensibility.

- TA is considered the GOD of the practice. She is expected to be omniscient about the technology area that belongs to her practice that she owns. For example, if you are a TA for HPC, you are expected to know anything and everything about Beowulf cluster, MPI programming, HPC management, RIS etc. Customers expect you to know everything; we expect you to know everything about the area that you own.

- Needless to say, TA should have great communication skills and more importantly, great listening skills.

So, you can say it is all about technology depth, great design skills, great communication skills and customer empathy – and oh yes, little bit of experience is required (minimum 10 years).

With this idea, I set about to hire technology architects for MTC. The experience has been illuminating.

Architect…Who

We did receive whole bunch of resumes. The resumes roughly fall under following categories.

A) Project manager. Typically has been handling a mid-large sized team (25-100). The resume is all about how he/she built a practice, enabled his/her reports to do whatever they are set out to do..blah blah. They don’t have any hands-on experience of late (that means 2-5 years). Majority of the resumes belong to this category. These ends up being paper rejects because of wrong fit.

B) Developers with a difference – and the difference being that they have been developing similar apps for 10 years. They typically build in-house apps for IT dept for large customers for quite sometime. Typically (not always), they wouldn’t have designed large scale apps and hence doesn’t know some of the fundamentals related to scalability, transactions (yes sir, it is true), basic algorithms etc. Their title is architect, simply because of their experience. Most of them are paper rejects because of lack of designing large scale apps, some of them get rejected in the first round after stumbling upon cursive questions on transactions and scalability.

C) Individual contributors – These people have been designing and developing large scale apps for quite sometime. They like being individual contributors (good fit) and they want to remain technical (good fit). In this case, people with good communication and consulting skills (apart from technical skills) proceed to next round. We’ve had some success here. But, unfortunately, we don’t get a lot of these.

Here is an interesting statistics about the source of resumes. Usually, type A and type B come from System Integrators and Customer IS departments. Type C typically comes from ISVs and some SIs. So, you may wonder why we are getting type A (project management folks) when it is very clear what we want is some one with deep technical, hands-on and consulting skills. The problem is because of the nebulous definition of the word ‘Architect’ in the context of IT industry.

Let’s look at the definition of an Architect from the following sites

http://www.bredemeyer.com/pdf_files/role.pdf

http://www.agilearchitect.org/agile/role.htm

http://www.artima.com/intv/architect.html

http://www.certmag.com/articles/templates/CM_gen_Article_template.asp?articleid=1406&zoneid=225

If you look at the definition of ‘Solution Architect’ in many of these sites, they go beyond the role of solution architect (i.e., to build a solution architecture) and cross over into many of the project management functions (like political handling (for god’s sake how is this part of architect’s job. Actually every role has some politics associated with it. What’s new here), little bit of project management like managing budget, ROI calculation etc). The agile folks seem to have a much more pragmatic view of Solution architects as opposed to others. I think the problem seems to be more like the confusion of different roles such as architects, project management, portfolio management etc. It is entirely possible a single person may be doing all the roles, but it is important to cleave them apart.

And I am not sure why we take these definitions as god’s gospel. Where are the proof points that this model has worked? Given that overwhelming of the software projects are failure (http://www.agilefactor.com/stats.aspx), what you have, in fact, is the counter-proofs that these models never worked. I personally feel closer to agile approach.

There is an interesting aside. In India, most of the IT population works for system integrators. In the Indian Sis, it is (at least it used to be) very difficult for a techie to have a career as software engineer. This is not because the SIs don’t want to promote individual contributor role with technical focus, but because of the ‘time and material’ business model they all follow. In this model, ROI of a ‘resource’ is directly proportional to how much they can be billed in a specific time. If you remain as an engineer for quite sometime, the ROI (of you) will be diminished over time as your pay increases over time, while your billable amount remain constant (or near constant). I guess this is changing now as the customers also recognize the word ‘architect’ and willing to pay a little bit more. Also, there are few companies in Pune that truly value Software engineers and have built career plan for them as individual contributor role.

In the next blog entry, I will talk about architects in Microsoft and how different they are from the architects as understood by outside world.

PS: These are all my personal opinions based on my observations and it doesn't reflect my employer's view point.

SQL Server Architecture Workshop content posted

ramkoth — Thu, 28 Jul 2005 10:06:00 GMT

The content for the SQL Server Architecture Workshop (decks + samples) have been posted in a Gotdotnet workspace called .NET Architecture Workshops. Please follow the link to register yourself with the workspace to download the content.

I will post the CLR Architecture workshop content soon.

I am also looking forward to hear from you regarding the future workshops and the subject areas of interest to solution architects. Please use the 'Contact Form' to directly send an email to me. Thanks.

Entity Aggregation white paper released

ramkoth — Wed, 26 May 2004 08:24:00 GMT

In my earlier posts, I was talking some of the real world challenges that an architect face while trying to implement SOA in their enterprise. Entity Aggregation is one of those problems. You can read more about entity aggregation here.

SOA - Real World Challenges reposted

ramkoth — Tue, 16 Mar 2004 16:18:00 GMT

I guess I shouldn't post a blog during mid-night. Here is my second try. My views on real-world challenges can be found here

SOA Real-World challenges

ramkoth — Mon, 15 Mar 2004 13:30:00 GMT

This post discusses some of the challenges architects/designers face while trying to implement SOA in their Enterprise. Our team has been working with customers for the past two and half years to implement SOA. This post reflects collective experience of our team.

Most of the discussions around SOA revolves around basic issues as RPC vs Doc/Lit, Remoting vs Web Services, usage of WS-* protocols etc. These are relevant issues and has to be discussed in detail. But, there are other important issues that needs to tackled in order to implement SOA in an enterprise. In my opinion, WS-* is necessary, but not sufficient. So far we have been talking about the necessary part. This post will cover the sufficient part.

Today, SOA is mainly used in connecting applications. In other words, the sweet spot for SOA is primarily in the integration space. If you look at the solutions that are built in a typical enterprise today, they fall into two classes of solutions - Service Delivery and Information Analysis. Service delivery - as its name suggests - is about delivering services offered by business systems (like ERP) to end-users directly. Self-Service solutions such as Employee Self-Service portal falls into this category. Primary motivation for self-service type solutions is to reduce the cost by dis-intermediating the interaction between user and system.

Let's take Employee Self-service portal as an example. Building this portal involves integrating multiple systems - HR System, Payroll system and Benefits system. Usually, these are separate systems and are autonomous in nature. Some of these systems may even be outsourced. For example, Payroll capability is usually outsourced.

Now, let's look at the challenges involved in building a self-delivery application.

Identity management - Usually, every business system maintains their own identity profile. They also define their own authorization realms - they define roles specific to the business functionality they expose. So, managing identity across multiple systems is a major challenge. Some of the issues you have to resolve are

Single Sign-On (both web-based SSO and system level SSO)
Federation - For example, Payroll systems and benefits systems may be managed by a different enterprise. How do I federate identities across organization boundaries.
Provisioning - When a new identity is created (during hiring for example), what is the process involved in creating appropriate identities in other systems.
Synchronization - Usually Single-Sign on solution requires some sort of credential mapping. How do we make sure when password changes occur in a business system and how do we synchronize that SSO credential mapping store.
Authorization - How do I authorize in this world?

EII (Enterprise Information Integration)

Every system maintain its own view of employee. For example, payroll system defines the payroll specific view for the employee entity, benefits system defines benefits specific view for the employee entity etc. But, building a service-delivery application requires a unified view of employee. A better example would be 'Single view of customer' issue that every enterprise faces.
Horizontal partition - There are cases where entity instances are partitioned across multiple services. How do we present an aggregated view.
There are cases where your solution requires cross join of entities encapsulated by different systems. In that case, you have no choice but to bring those entities to a central store. Cross-join scenarios are so devastating that your mind set will shift from service-orientation to data-orientation. I have gone through that experience :)

Schema Harmonization - Every service maintains its own view of an entity. How do we harmonize the schematic differences between services? Canonical schema is one practical way to harmonize the differences. But, the real challenge is to get all the stake-holders to agree on the schema. This is where a strong enterprise architecture team can make a difference. Another challenge is - would you go for a big-bang approach with enterprise modeling or would you go for incremental approach. Current thinking is to go with incremental approach. Start with few commonly used entities such as Customer, Product etc and then add more stuff later on.

Service Management - How do I manage, bill, meter, provision, de-provision services?

There are more issues. But, these are the primary challenges one would face while trying to design a solution based on services. In my future posts, I will spend some time on each of these issues and share with you some of our experiences and key learnings.

I would love to hear your opinions on this subject. I am sure some of you may have implemented SOA in your IT and faced similar challenges. I am also sure that you may have come up with better solutions and have a better insight into this problem. Please, please share your learnings. We can all learn from each other.

Making Services Idempotent - Part II

ramkoth — Sat, 13 Mar 2004 16:27:00 GMT

I have created this as an article. Please follow this link.

Make Services Idempotent

ramkoth — Sat, 13 Mar 2004 03:31:00 GMT

In the brave new world of service orientation, service requests may fail due to different reasons that may include network connectivity, resource failures (like db failure) etc. When a client receives a failure, all it can do is to retry for a while and then give up. This implies that the request may actually arrive more than once due to un-reliable nature of protocols like HTTP. Let's take an example

[WebMethod]

ResponseMessage DoRequest(RequestMessage)

{

ProcessMessage(RequestMessage);

Return responsemessage;

}

Failure can happen at any time. Failure can happen

before the message has been processed,
during the processing of message,
after the processing of message but before returning the response.

Let's assume the third case. When a client re-sends this message again, then the message will be processed again. This may cause problems in many scenarios. For example, think of DebitTransaction as a service operation. You don't really want your debit transaction to be processed twice.

One of the important design principles of service orientation is that the service should be made idempotent. Idempotent means 'it is okay if the requests arrive multiple times'. As a designer of the service, you should expect the messages to be delivered more than once and you should process the message only once.

Unique Request Identifier

Traditional way to make a service idempotent is to create unique request identifier and pass the request identifier along with the message. When the client retries, client uses the same request identifier rather than creating new one. On the server side, service should inspect the request identifier, check if the request has been already processed and if so, return the response; if not - process the message, store the response in the persistent log along with the request identifier, indicating that the request has indeed been processed. Here is the pseudocode

[WebMethod]

ResponseMessage DoRequest(RequestMessage)

{

Extract the request identifier from the request.

Check if the request has already been processed by querying the persistent log.

If (request processed) then return response

Else

{

ProcessMessage(RequestMessage);

Store the response along with the request identifier the persistent log

Return responsemessage;

}

Will this algorithm work? Not quite. For example, consider the scenario where server crashes after processing the message and before storing the response in persistent log. This means that the request is not logged in the persistent log, even though it has been processed. So, when the request arrives again, you will process it because the request is not present in the log.

So, how do we solve the issue? The key is to use transactions. Both processing of message and log updates should happen in a single atomic transaction. Here is the modified logic.

[WebMethod]

ResponseMessage DoRequest(RequestMessage)

{

Extract the request identifier from the request.

Check if the request has already been processed by querying the persistent log.

If (request processed) then return response

Else

{

TransactionObject trans = CreateTransaction()

ProcessMessage(RequestMessage, trans);

//Store the response along with the request identifier the persistent log

UpdateLog(responsemessage, request identifier, trans)

trans.Commit();

Return responsemessage;

}

If your persistent log store is different from your application data store, then you may have to do distributed transaction - which is really costly. So, it is better to keep the log store and your app data store to be one and the same.

How do you pass the unique identifier?

Use SoapHeader to embed request identifier related information. Service can obtain the SoapHeader using the ASMX programming model. Please refer to ASP.NET documentation for using SoapHeaders.

If you are using WSE2.0 for the client, then you don't have to generate message id elements. It will be generated for you automatically.

What if the service implementation is simply a façade to legacy system? In that case, you can't really do a distributed transaction between legacy system and persistent log. How do you solve that? I will cover this in a later blog. It is not as difficult as you may think.

Do all the service requests need to be idempotent?

No. Certain requests are implicitly idempotent. For example, read requests are implicitly idempotent. Certain updates are also idempotent. For example, changing an address is idempotent - it doesn't matter how many times you process the request, end result is still the same. (think of it as an assignment statement i=2. This statement is idempotent.)

I will also cover how this topic is related to WS-ReliableMessaging in a later blog.

Mike is Modelling

ramkoth — Wed, 04 Feb 2004 18:55:00 GMT

Mike platt is running interesting posts in his blog.

He recently posted a blog on modeling. This is a must read. He says,

“I do feel very strongly that the concept of a general purpose transformation from one domain or level of abstraction to another will always yield a sub optimal solution unless there is a transform algorithm. I suspect this is why the CASE tools industry died and why O-R mapping technologies (and the upcoming O-XML mapping technologies) are also doomed to provide poor performance in the general case. I do feel very strongly that the concept of a general purpose transformation from one domain or level of abstraction to another will always yield a sub optimal solution unless there is a transform algorithm. I suspect this is why the CASE tools industry died and why O-R mapping technologies (and the upcoming O-XML mapping technologies) are also doomed to provide poor performance in the general case. “

How true. That's actually summary of my five prior posts.

Litmus test for the agillity of architecture

ramkoth — Tue, 23 Dec 2003 19:12:00 GMT

Today’s business environment is very dynamic. Things change all the time. Definition of business concepts such as customer or employee changes over a period of time. Business rules change more frequently than business concepts. Think about new federal regulations and rules that have to be adhered to.

Good architectures are flexible such that changes can be incorporated with out too much difficulty. I thought about a set of questions that you should ask about your architecture in order to assess it. Most of them stem mainly from business requirements. Here is my first cut

Apart from supporting well known abilities such as scalability, availability and performance, a Good architecture has following traits.

1) Changes to business concepts can be incorporated easily with out a huge code re-writes. For example, you may want to add few additional attributes to employee for complying with federal regulations. A good architecture allows you to do this in a systematic way.

2) Changes to business rules can be done with in a short period of time (a day to probably a week). Business analysts should be able to view and change business rules.

3) Changes to human workflow can be done with out serious re-architecture.

4) Changes to presentation layer (including changing UI flow) can be done with out serious re-architecture. Can a new channel like voice channel be added with out costly re-write?

5) Changes in security policy can be incorporated with out serious re-architecture.

6) Allows other systems in the enterprise to interact with it easily.

7) Ability to trace/audit activities in the system should be easy

If you have got 6 or more out of 7, then you probably have a good architecture. 3-5 out of 7 means you are getting there. Any thing less than that, you better get your budget aligned for a costly re-write/re-engineering exercise.

These are all business scenarios that enterprise customers demand from their IT system. It’s all about agility and ability to cope up with change.

You don’t get all these features out of box. No amount of tutorials is going to get you there. You really have to ENGINEER for it. It means real hard work. That’s not my comment. That’s John Zachman’s. I believe it is true. His site is http://www.zifa.com

I am sure there should be more to this. Your comments are appreciated.