All About Interop : Portals

Sharepoint 2007 Business Data Catalog enables Interop with LOB systems

DotNetInterop — Tue, 20 Mar 2007 23:04:00 GMT

I just listened to an on-demand webcast on the Sharepoint Business Data Catalog - aka BDC. The webcast is laboriously called "Office SharePoint Server 2007 Business Data Catalog (Part 1 of 2): Integrating Line-of-Business Data and Applications into Your Enterprise Portal (Level 200)" and is available here: http://channel9.msdn.com/ShowPost.aspx?PostID=238622

Mike Fitzmaurice listed 4 ways that historically LOB apps could be surfaced into Sharepoint Portal.

Re-use an existing HTTP/HTML interface. For example, the LOB system might have a companion web interface, and the portal could simply do HTML "screen scraping." This might be called the quick-and-dirty approach.
Point-to-point approach. The webpart does all the work to integrate with the back-end system. This approach involves the fewest layers, but the most code. The webpart might, for example, communicate with the back-end system via a built-in messaging interface supported by that back-end LOB system.
Data caching. If you've already got a data warehouse or datamart, your webpart could just access the staged data from the LOB system that is already cached in the DM or DW.
via an Integration Server. In this case, your webpart calls into a service layer, possibly hosted by an integration or process server (like BizTalk Server), and the logic in the integration server does the work to bridge into the backend system.

Mike pointed out that Sharepoint customers asked for easier and better integration. For example, maybe index the LOB data for search purposes, centrally manage the connections from a portal to various back-end systems, and so on. This was the catalyst for the BDC.

Though the Business Data Catalog is a feature of Microsoft Office Sharepoint Server 2007 (MOSS 2007), and ships as part of MOSS 2007, it can be utilized from any app, including a non-portal app or an app that isn't a traditional Sharepoint app. Though the BDC is particularly easy to use from within Sharepoint, it is also nice to be able to use it from outside systems. What does the BDC do? Essentially it is acts as a broker of back-end systems that can be accessed either via standard data interfaces (ADO.NET) or web services. People can register back-end systems once in the catalog, then any Sharepoint webpart will be able to pick-and-choose from those registered data sources to display that data. Additionally, any .NET app (with a web UI or not) can connect to the BDC to indirectly access back-end systems. In this way BDC acts as a broker for LOB data.

The BDC includes webservices for common SAP and Siebel entities, "in the box", with the Sharepoint 2007 product. Partners can create definitions for arbitrary back end systems.

There is now a third-party tool for manipulating the metadata managed by the BDC - BDCMetaMan.

BEA AquaLogic User Interface (ALUI) - write portlets in .NET

DotNetInterop — Tue, 30 Jan 2007 16:44:00 GMT

Hmm, what's ths?!?! BEA has introduced a new line of stuff, under the name Aqualogic Interface, or ALI (I think). As part of that, there is the Aqualogic User Interface (ALUI?). One of the things under that umbrella is the .NET Application Accelerator, whereby BEA is offering technical magic to enable .NET Developers to write code that runs on or adds value to an Aqualogic server. Intriguing!

The Java Portal Spec describes the model and APIs for Java developers to write portlets - fragments of Web UI that could be composed together to build a single, aggregated portal page. The goal was to enable developers to produce something like my.yahoo.com or one of any number of other web portals, but developers would be able to build these things inside enterprises, and the portlets (or fragments) could theoretically connect to any other system in the enterprise - mainframe transaction systems, databases, packaged apps, custom operational apps, and so on. The portal idea was a good one, lots of companies and vendors latched on to it, including Microsoft, with its Sharepoint products, which were at one point the fastest growing products at Microsoft.

The portal spec was good for Java vendors like BEA and IBM, because it extended the idea of the "application server" to include personalization features, membership, and so on - all extensions beyond the rapidly commoditizing J2EE specification on which their application server businesses were based. Java portal servers were like Java app servers++. And so BEA and IBM charged LOTS of money for these things. [ actually, until just now, I did not know, precisely how much money these other vendors were charging for portal servers, but I thought that it was "lots of money." Now that I looked, it knocked my socks off! IBM is charging $980 per value unit for WebSphere Portal Server. Just what is a value unit, you say? A value unit is 1/100th of a CPU, if you are running a dual-core Xeon or Opteron on your server, as near as I can tell. If you go to quad-core or some other processor, then you'll probably have to call IBM to find out how many "value units" you need to buy for your server. So for an x86 server, you will pay, I think, $98,000 per CPU to run WebSphere Portal Server. Zowie! I promise that Microsoft Office Sharepoint Server is not that expensive. ]

The model of selling pricey server licenses continues today, for both BEA and IBM, But, apparently BEA is noticing there are a whole lot of .NET developers in the enterprise today, and if they want to continue to sell high-priced servers, they better somehow allow enterprises to take advantage of all the .NET talent.

Hence, the .NET Application Accelerator.

Where does the interop come in? Well, if I have understood the product literature correctly, this widgetry allows .NET developers to build stuff that shows up as portlets in a BEA Portal server, using the WSRP standard or RSS. Hence, on-the-glass interop, via the portal server.

This looks like more of a good thing. In general, more options and choices for interop is good for customers, I think. I will be interested to hear any feedback on this interop-focused product, based on hands-on experience.

cheers!

on WSRP

DotNetInterop — Fri, 10 Jun 2005 23:55:00 GMT

I have a bunch of kids, I lost count after 3. They are all under 10 years of age, as far as I know. They move so fast, that Heisenberg effects occur. You can never really be certain exactly where they are, and exactly how fast or in what direction they are moving. And it's the same thing with their thinking, too. When you have a conversation with little ones, you can never be sure what direction they are taking you. Sometimes they spin off into outer space and I have to say, "Whoa, hold on. Take a breath. Start over, at the beginning."

I am no expert on child-rearing (though I am actively engaged in the pursuit), so I don't know whether the current psycho-recommendation is to let the kids fly with their thoughts, or to encourage them to relate more to reality. But I yam what I yam: habitually rational, so I encourage the kids to be rational, too.

What about WSRP?

Oh, I hear you, brothers and sisters. I hear you thinking, "Umm....I thought this was a geek blog? What does this have to do with WSRP?" Trust me on this.

Lots of people ask me about WSRP. Often, I'll get email citing excerpts of my posts to a newsgroup, like this one:

It is less optimal to integrate portal artifacts at the UI level. There is a WSRP standard that attempts to cover this, but it is (IMO) neither well thought out nor well-supported by either Websphere or .NET. Architecturally, WSRP is an odd bird, and the implementation itself introduces lots of challenges.

In the emails, people say, "I would appreciate if you would elaborate on this view." and so on. Ok, here 'tis.

Now We're Getting Somewhere

Customers sometimes say things like: "We want to build a WSRP-based portal for application aggregation." My response is, "Whoa, hold on. Take a breath. Start over, at the beginning." (see!? I told you the kid story was related!) The beginning for application architecture is... the business goal. Is "deploy a WSRP-based portal" the business goal? It shouldn't be, unless you are a vendor trying to sell WSRP. Just as "use web services" should not be a business goal, nor should "deploy Intel- or AMD-based hardware" be a business goal.

The business goal is usually something like,

"We want (a) to speed up a delivery of new functions by re-using existing silo-ed web applications, and to enable future re-use for new apps. Also, we want (b) to deliver such applications to a broad audience, and for them, we have little control over the mode of access to the app."

The first part of that says, Speed through re-use, now and in the future. Good. That sounds businessy. The second says, Reach. That's businessy, too. And even better, both sound like good, valid goals. We can work with this.

Last things First: Reach means Portal and More

To me the "Reach" goal implies "web", and probably "web portal". But probably very soon it will also imply "speech" and "mobile device". In other words, the PC-based web browser is not the last word in "Reach". For true broad reach, you need multi-channel access, and a web-browser centered portal is not going to deliver that. True, a web portal page can be consumed by a browser on a mobile device, but the UI is non-optimal. 1024x768 just doesn't translate well to 160x160.

The Requirement for Re-Use means Web services

Re-use is so often cited as a justification or goal, that it almost seems like a meaningless cliche, but it is still the absolute #1 best-bar-none way to reduce costs in delivering applications. The #1 cost component of applications is labor - the People that construct and maintain the app code and its surrounding support super-structure (admin scripts, db load scripts, and so on). If you don't have to incur that labor cost, the cost of the app drops to the floor. So re-use is truly a worthwhile goal.

Re-use is good. The next question is, What gets re-used, and how? For the past 5-6 years, web services has been the standard answer here. Starting with XML, SOAP, and WSDL, and building on top of that things like Ws-Security, the industry has constructed a set of standards to allow broad general-purpose interop across any variety of computing infrastructure. handhelds, PCs, servers. Java, .NET, VB, COBOL, PHP, Perl. and so on. It just works.

But web services have been over-hyped and over-applied. Would you use web services to replace database-specific networking, like Oracle's SQL*Net? You could. Should you? No! SQL*Net just works, it's mature, it's tested, it's highly optimized for what it does, so why break it? The point is, web services is a good tool, but should not be used everywhere.

Web services are best applied at facilitating the inter-connect between providers and consumers of new custom application function. What we used to call "function shipping" or "RPC". Ok, before you jump down my throat, web services is much more than an XML-based RPC, but today, that is the sweet spot. For example, using the [webmethod] attribute, any .NET programmer can expose new logic endpoints in an app. And JSR-181 aims to make it that simple in the Java world, too. Nice.

Seriously, when do we get to the WSRP Part?

Yeah, I'm getting there.

WSRP is, in my opinion, "web services gone wild". Not a pleasant thought, is it? WSRP allows user interface to be transmitted over a web services protocol. As stated in the WSRP spec, the goal is re-use of Portal UI.

Re-use is good, but it has to be re-use at the right level. The most commonly stated motivation for using WSRP is something like "well I just did a lot of work writing this portlet -- I don't want to go through that again for a different platform." This is wrongheaded. If you've invested a great deal of work in building a portlet, and the only re-use interface is the UI, you've made a poor design decision. This implies that the portlet (the UI logic) hides and obscures business logic and maybe data access logic behind the presentation code, and there's no opportunity for re-use except at the UI layer. If that's the case, that is a bad design. Bad bad bad.

What if you want to re-use that app logic in a mobile app? What if you want it in an RSS feed? What if you want a speech interface? a smart client? What if you want to build a composite app, maybe an agent with no UI whatever, that re-uses that logic? None of these things are possible if UI is the re-use layer. None of these are possible if WSRP is the re-use substrate. See? Bad, brittle design.

Yet this is exactly what WSRP does. Web services gone wild. Re-use of UI is brain-dead. Come on, people, do you really think that that glorious portal, the one you spent so much time building, is going to be the last app to inhabit your enterprise? Of course not! There will always be more, and many of them will not be web-based. That's known, right? Not a matter of opinion. That's fact. Mobile is happening, speech is happening. If you think the web portal is the last app architecture, you need to meet up with the people that built all the CICS apps that mingled 3270 screen logic inside the program logic. You can all sit around and curse progress together.

Because new apps and new app models happen, and because web is not "the last app model you will ever need," architecting systems for re-use means more than "portal UI re-use". WSRP is the wrong direction to go. Re-use should be done using application-oriented web services. Business logic. And sure, there are multiple levels of granularity. Some web services may be "top level services" while others may be foundational. But UI is NOT wrapped into these services. Never. ever ever ever. Did we learn nothing from the days of 2-tier computing? Are we not smarter after having done 3270 screen-scraping for 20 years?

This is where I try to lead customers who start with WSRP. When I tell them, "Whoa, slow down. Start from the beginning," the point is to get them to think bigger, beyond "re-use of portal UI".

Let's say you have a portal today, and you want to exploit re-use. The right way is to expose business services via XML and SOAP, and then built thin webparts/portlets/gadgets that present the data and capability of those services. The artifacts at the UI layer should be thin. If you then need that service/info in another UI channel, it should be straightforward to build the (unique) UI for that other channel, be it speech/voice, smart client, or something else. Re-use at the service layer, not the UI layer. This, by the way, is a technology-neutral approach. It applies equally well if you are an ASP.NET shop, a Java-based shop, something else, or a mix of all of the above.

Q&A

Q. Is this Microsoft's Official Position on WSRP?

A. No way, buddy, I'm not going there. This is not Microsoft's official position on WSRP. This is my opinion.

Q. Are you saying WSRP is never useful?

A. No. But it is mostly not useful. It's a niche thing. And mostly, it is either misunderstood, or deployed for the wrong reasons.

Q. So, in your opinion, when is WSRP useful?

A. The WSRP spec says, when you want to re-use portal UI. In my opinion, WSRP is useful when you have pre-existing (maybe vendor supplied, maybe custom-developed) portlets or web-parts, and they do not admit any re-use. They comingle UI and logic, and they cannot easily be re-factored. There, in that situation, WSRP is useful. But as a temporary measure only. Long term, you want to refactor.

Q. But, WSRP is an industry standard! And I thought Microsoft supported standards! How can you say such deceitful things about WSRP?!!

A. This is technology; It's not a religion for me. I thought it through, and conclude that WSRP is probably the wrong tool for the re-use job.

Q. Does Microsoft support WSRP?

A. No, not directly.

Q. Aha! Gotcha! All of this badmouthing WSRP is because you don't support it!

A. Um, if you like conspiracy theories, sure! But that reverses the actual cause and the effect: Microsoft doesn't directly support WSRP because it is so limited.

Q. Wait, then why do I see support for WSRP from so many companies, like IBM, BEA, SAP...? Hmm?!?! Answer that one, smart guy!

A. JSR-168 is the reason WSRP exists. See, JSR-168 is a portlet portability API. All those portlet vendors, after delivering their JSR-168 implementations, just realized their portlets were no longer sticky. Now someone could rip out WebSphere Portal Server at $57,000/cpu, and replace it with Apache JetSpeed (free). WSRP mitigates that risk. It provides protection from commoditization to the Java portal vendors. Remember, these are the same vendors that got scorched when Tomcat and JBoss dropped the price of the Java-based app server to zero. They don't want that to happen with portals. So now, the portal vendors tell their customers -

"Oh, that JSR-168 portability spec? No, no, don't actually use that. No. Oh my, no. Think of the mess, moving a portlet from our portal server to someone else's! Oh I guess if you want to install another vendor's portal server that would be ok, but certainly don't turn ours off. Instead, run 4 different flavors of portal server, and in each one, run a unique set of portlets. Then, allow the portlets to interop via WSRP! Much better! You never need to stop running our portal server! Won't that be nice?"

Does this make business sense to the customer? Probably not. Redundancy is never a cost-saving approach. But it does preserve the presence of multiple Java-based portals in an account. It acts as a brake to commoditization that killed the app server revenue.

Q. Now, that is what I call a conspiracy.

A. WSRP is so weird, and technically speaking, so weakly justified, the only reasonable conclusion is that it is a spec driven by the vendors' business case. It solves a technical problem in a very complex way. It is redundant, and adds no new real capability. How to explain its existence? To me it is obvious that MBAs conceived the spec.

Q. Is there no way to "do" WSRP using Microsoft technologies?

A. There are options. There's a WSRP Consumer Web Part available on GotDotNet. It's a community-supported workspace, not an officially-supported product. It may work for you. "WSRP Consumer" means, it is a Sharepoint webpart that can connect to a potentially remote WSRP artifact, and display that UI in the webpart. You can also implement WSRP producers in the form of ASP.NET Web services. An example of how to do this is available on GotDotNet, again as a community-supported, unofficial effort.

Also, there is a third party add on to .NET from NetUnity Software. It is packaged as a Visual Studio add-in and a set of classes that allow you to build WSRP producer Web services.

Any Last Advice?

Yep.

If your goal is re-use, do it primarily with business-oriented web services. Don't optimize around re-using UI. If you've already got a portlet/webpart/gadget and it is not factored for re-use, re-factor it. Now.
If there already exists a portal artifact (portlet/webpart/gadget/etc), and you want to re-use it, first inquire if you can re-use a web-service that resides behind the UI. If it is properly architected, this will be possible. If not, then you are stuck re-using at the UI layer. If the re-use is to happen within a SharePoint portal (using SPS) or a SharePoint site (using WSS), use the .NET WSRP consumer Web Part on GotDotNet and consume that remote portal artifact using WSRP.
if you are building new portal artifacts, and want to promote re-use, and for some really weird reason, despite the recommendations of reason and clear-headed thinking, you still want to allow re-use at the UI layer (maybe it's due to a misguided proclamation from a pointy-headed boss), then write a WSRP Web service. If you are doing it in .NET, either use the example on GotDotNet or make use of NetUnity Software's stuff.