George Nelson [MSFT]

Industry Technical Strategist - Financial Services

2008-10-23T18:04:00Z

I have been off the blogs for a while and recently changed roles in Microsoft to be an Industry Technical Strategist in our Worldwide Financial Services practice. I guess I picked a great time to move into our Financial Services practice. :)

One of my passions in my previous roles was around SQL Server and specifically SQL Server in Line of business applications.

Let’s give a little history

Previous to working at Microsoft, I worked with SQL Server, Sybase and Oracle back in the SQL 4.2 and 6.0 days. Anyway, back then SQL was a database that fit in between Jet(Microsoft Access and Foxpro) and Oracle. It was mainly focused at department level internal applications while Oracle and Sybase were focused at larger workloads. SQL Server was typically easier to use, manage and install than its counterparts.

Starting with SQL 7.0, we focused on larger workloads and starting competing in TPC and other benchmarks and expanded this through SQL 2000. We started catching the competition and still had the ease of use and deployment.

Then in SQL 2005 and 2008, we really expanded and are able to handle pretty much any large business workload and the largest most scalable applications. We added reporting, OLAP, and enhanced integration capabilities and moved into data warehousing in a big way. At the same time, we still kept the ease of use, manageability, deployment and installation.

A funny thing happened along the way. We learned some big lessons along the way and added features to handle a world where servers need to be online 24X7. Features such as Replication, Clustering, mirroring were added to address availability. Table partitioning and larger 64-bit servers were added to handle the normal mainframe workloads.

Today

In the past few years, almost every customer I talk to still has a leftover view of whether or not SQL Server can handle their LOB solutions. I think this is still a leftover view of SQL's early days where it wasn't viewed as a mission critical database. So as my first post in a couple of years, I wanted to start adressing these questions in Financial Services

SQL Server in Mission Critical Workloads

Here are a few proof points you may want to view when looking at SQL Server in Financial Services.

What about the really high throughput workloads like Stock quotes and lookups. Can SQL handle thousands of updates per second?

Townsend Analytics runs on SQL 2005 and handles 60,000 quotes per second inserted into a DB and 100 events per second.

http://download.microsoft.com/download/d/d/7/dd79c390-5b75-45fb-a0f5-2359a6d20643/GEMS4641/Townsend_Analytics_SQL_Server_2005_Launch%20Case_Study.doc

Citi handles 200,000updates per second

http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000001199

My point is that Microsoft SQL Server is the backend of exchanges, ticker plants, and Market research data.

What about Line of business applications with thousands of concurrent users?

Yes, we handle those as well. How about Progressive who moved their Policy Admin System to SQL 2005 with ~27,000 concurrent users? (http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=4000002133)

My point here is that we work handle the workloads of things like teller solutions in banking, Insurance (claims and policy management), and capital markets (Trading stations) we handle workloads of thousands of concurrent users who are always logged on.

What about online sites? Online Banking? Online investments? Insurance quotes?

How about Marketwatch?

http://www.microsoft.com/casestudies/casestudy.aspx?casestudyid=200969

We handle many of the online banking portals, securities portals and Insurance portals. Anyway, we have too many case studies to list, but today we handle the largest workloads with thousands of updates per second and thousands of concurrent users.

What’s next?

My next update will talk about reporting and data warehousing and finish off with how do you build solutions with SQL Server that are scalable and highly available.

PDC Event Day 2 notes

2005-09-15T20:12:00Z

I have some thoughts on day 2 from the PDC that I thought I would share.

First of all, Microsoft Office 12. This is easily going to be the best version yet. Versions 6.0, 95, 2000, XP, and 2003 had some changes, but the look and feel remained the same. The menu bars and options just grew to unmanageable levels.

The next version will use ribbons. This is a tab type concept which will show all options available for that particular concept. So you will no longer have to search menus to figure out what to do. The beta's looked awesome!

Also, the format is XML, but a zipped version of XML. In other words, a DOCX is basically multiple XML files in a zipped file. You can change styles, add in headers, and other files programmatically. The Schemas should be published soon.

Sparkle and Acrylic. These are Web Designer tools, that allow people to design sites directly from C# or VB.NET Web Forms. The demo's really looked great and the forms concept allows developers to work with the template the designer created directly.

AJAX or Project Atlas. If you have not seen this, you should get the whitepapers and the betas. This is up on ASP.NET.

http://beta.asp.net/Default.aspx?tabindex=0&tabid=47

ASP.NET “Atlas” is a package of new Web development technologies that integrates an extensive set of client script libraries with the rich, server-based development platform of ASP.NET 2.0. “Atlas” enables you to develop Web applications that can update data on a Web page by making direct calls to a Web server — without needing to round trip the page. With “Atlas”, you can take advantage of the best of ASP.NET and server-side code while doing much of the work in the browser, enabling a richer user experience.

From the developer community, the Office and ATLAS presentations were packed. They even moved to the overflow rooms. Tons of interest. Will provide links to the schemas for Office and the links to Atlas samples as they come.

The night event was at Universal Studios. Main problem was that the beer lines were long and they ran out of Beer at the tables. Sad ending to a good day….

Microsoft PDC Technologies

2005-09-14T21:53:00Z

I have been away from blogging for a while. A little too much travel to keep up with multiple blog sites. In the past 12 months, I have been to Thailand, Japan, Seattle, San Fran, LA, New York 15 time, Orlando, Dallas, Tampa and various other cities. Basically around 120,000 airline miles.

This week I am at the PDC, and I am really excited about the changes coming.

I will come back to my BizTalk blogs next week, but I wanted to talk about 4 great technologies that we are releasing soon or have already released.

Do you know the message you get when a program crashes and it asks you about sending information to Microsoft?

Do you know that when you clik yes it actually sends this information to a server that is used to be a gathering point for debugging crashes.

The server is actually winqual.

https://winqual.microsoft.com/

Did you know that as a developer, you can subscribe to this server and register your applications?

If you do, and if your application crashes and the end user chooses to send the information to Microsoft, you can go to this site and download the crash and debug it. It will also show you a list of crashes and provide you with the ability to actually send a message back with a link to a hotfix for your application after you fix it.

The LINQ project

The Language Integrated Query framework (the LINQ Project) is a set of language extensions to C# and VB and a unified programming model that extends the .NET Framework to offer integrated querying for objects, databases and XML.

LINQ’s query operators can be applied to any .NET array or collection of objects, effectively providing SQL- and XQuery-like capabilities for in-memory data. Essentially query-capabilities become a first class citizen in .NET programming languages. In addition, LINQ provides two critical APIs code-named:

· DLinq: for SQL relational data access
· XLinq: for XML hierarchical data access

Windows Workflow Foundation or Workflow in general

This is bringing BizTalk like Workflow to the developer community without having to have a copy of BizTalk Server. This is like a developer version of BizTalk without requiring XML documents. There is actual c# or VB.NET code integrated. Events, code breaks, etc.. I would highly recommend checking this out at the site below.

http://www.msdn.microsoft.com/windowsvista/building/workflow/

There is also a series of hands on labs and documentation.

AJAX or Project Atlas

Yet another new tool to get excited about. This is a set of JavaScript libraries to build client side code that will work in conjunction with ASP.NET. Basically it gives a Client-side Jscript designer so that people don't have to build the libraries to do stuff like populate a table with data from a web services.

The bits are on ASP.NET. There is a whitepaper and hands on labs as well.

http://beta.asp.net/default.aspx?tabindex=7&tabid=47

How to get Biztalk develope edition running after installing XP SP2.

2004-08-17T18:50:00Z

1) BizTalk Server 2004 will stop functioning after the installation of Windows XP SP2 due to changes in the DCOM security model. Windows XP SP2 is available for download now and will be pushed through WindowsUpdate in late August.

Modify the registry with the instructions below to resolve this issue:

http://support.microsoft.com/default.aspx?scid=kb;en-us;841893

2) The BizTalk Server 2004 Rules Engine component will stop functioning after the installation of the .NET FW 1.1 SP1 due to issues in our implementation which are now resolved. It is expected that all other existing user-level code written in the .NET FW 1.1 will continue to work in the .NET FW 1.1 SP1. The .NET Framework 1.1 SP1 is not yet publically available but will become available shortly.

Download the hotfix for the rules engine component to resolve this issue:

http://support.microsoft.com/default.aspx?scid=KB;EN-US;875540

This hotfix is supported both on the .NET FW 1.1 and the .NET FW 1.1 SP1 so you can install it today if you like.

Using Side by Side Versioning with BizTalk 2004

2004-07-27T01:32:00Z

I have been away for a week in New York again and just got back in the office and cleaned up a little of my emails. In parsing them, I wanted to pass along some information that on Side by Side assembly deployments with BizTalk and how to get this to work.

If you create another version of an already deployed Biztalk assmbly and you would like to start the new version, here is what you need to do to get this to work.

Configure any pipeline components to use a specific version of the schema.
If the schema is redeployed under a different version and this is not done it will fail because it looks up by target name space / root element not by assembly.
Deploy the new version
Bind the orchestrations.
The Manual steps are below
1. Disable the receive location
2. Unenlist the old version using the Explorer (do not stop it), this will prevent new instances from starting while allowing old ones to finish
3. Enlist the new version and start it, all new instances will be created using this one.
4. Enable the receive location

Divide and Conquer

2004-07-16T20:06:00Z

I was watching History channel last night and they were discuss the Napoleonic wars. One of the main strategies used was “divide and conquer”, and it is normally a strategy that I talk about when doing BizTalk development. One of the main purposes of this blog is to give some reusables samples and walkthroughs that can be used as a template for other BizTalk projects. I am hoping that each of the pieces that I am providing can both demonstrate a concept and be used as a template for pieces for other larger BizTalk projects.

When I talk about “divide and conquer” with BizTalk, I am trying to show that you may be tempted to build the workflow of your orchestration and then add your ports and maps and schemas and then deploy it. For a simple workflow this is easy, but when you have a more complex workflow, this can become more cimbersome and difficult to maintain. You may end up sepnding a lot of time in Health and Activity Tracking for trivial issues. What I normally suggest is drawing the workflow, create several small samples that are just pieces of the full orchestration, get each piece working, add them to the full orchestration.

For instance, if you have a workflow like the following that processes an address change for a customer across multiple systems.

1) Receive a document from an external source (SAP, mainframe, FTP etc..)

2) Use a MAP to translate it into a SQL friendly schema to do a database lookup.

3) Use a SQL Request-Response port to verify that it is a valid customer

4) If it is not a valid customer, use a map to translate a response back to the originating source that there is no customer like this.

5) If it is valid, use a scope shape and Request response port to SQL Server to update the customer information

6) Add another port to send a request to a web service which might be a wrapper to Seibel or some other CRM system.

7) Add another send port to update a contact list for an older legacy system that is still in use.

8) You can do this in a parallel shape as separate called orchestrations, but you get the general idea here.

In this case I advocate breaking each of these tasks down into separate orchestrations like the following.

1) An Orchestration that receives a document, applies a map and sends it back out. You can then view the Schema and make sure everything looks correct.

2) Create an orchestration that has a file receive port that you pass just the SQL Schema, have the SQL request-response port and then just add another send port for the repsonse. The flow for this will be receive, send request, get a response, and Send the response. Once again you can view the results and make sure everything is working.

3) Create an orchestration with one with a receive shape, Construct Message, translate, and send shape. This will be for the case of an invalid message.

4) Continue on like this with each piece until you have each piece working.

This might be a bit of overkill, but once you get each piece working you can go back to the main orchestration and simply start adding the Schemas, Maps, Shapes, and ports that worked before and hopefully will have already worked out the problems with any of the messages and schemas and maps without having to spend a lot of time using Health and Activity tracking.

This is just a suggestion that will hopefully save you some development time. This is more of an Extreme Programming technique (http://www.extremeprogramming.org/) with BizTalk, but I find it much easier to build and troubleshoot problems. You end up with your major orchestration and then various smaller pieces. You can then use the smaller pieces to troubleshoot problems as you see them and to test modifications to your maps and schemas before bring them back into the main orchestration.

Let me know if anyone else has any thoughts or other practices they use for developing BizTalk applications.

Using Scope shapes in Long running transactions and more adapter stuff

2004-07-15T18:48:00Z

A demo I sometimes show during various BizTalk presentations is one that shows how to compensate for problems with using long running transactions in BizTalk Server. This sample was originally developed by Erik Leasburgh and Raman Mahal. I created the walkthrough so we can show how to create the sample and hopefully provide details to show how it works so others can use it as a template for their designs. The problem we are trying to solve is that the atomic transaction shape in orchestration doesn’t really provide true atomic transactional semantics when used with a transactional adapter like SQL, FTP etc.

Let’s say I have an orchestration which has a send port which is bound to SQL Adapter Send port for sending the message to the SQL Database. For transactional purposes the developer has enclosed this send shape in an atomic transaction. Also in the same transaction there’s another send shape which is sending messages to another system. Both these sends should commit totally or should rollback.

Now during runtime say Send1 to SQL Server succeeds, but Send2 to the other system fails and because of that Send1 has to be rolled back. A user would think that since the 2 sends are in atomic scope in the orchestration, the orchestration will take care of the rollback process. This is not the case, you will need to compensate for this in your Orchestration. The sample and walkthrough shows you how to do this. This is a version 1.0 of my docs, so let me know if this works or if you see any problems.

Adapter Whitepaper Released

2004-07-02T21:54:00Z

For those of you who have been or are writing adapters a new white paper, "Adapters: A Developer's Guide" is now available on GotDotNET at

http://www.gotdotnet.com/team/wsservers/bts2004/BTS2004AdapterDevGuide.zip.

This was written by Kevin Smith, former developer for the BizTalk Server product team, it is great reading for after a long holiday weekend....Enjoy

HAT walkthroughs and other trivial New York information

2004-06-25T18:53:00Z

I am prepping for a trip to New York and pulling together some demos and other miscellaneous information. One of the pieces of information I usually give out is more information on using HAT to debug problems. I normally pass along a whitepaper written originally by Guy Lau which I have updated. Please feel free to use to give you examples on how to use HAT in different scenarios.

Now on to the important stuff like pizza in New York!

Since I am frequently traveling in New York, I thought it might be nice to pass along a couple of places to eat and drink if you ever find yourself in the city. I normally am either in the financial district or midtown so locations are convenient to there.

I can't mention going to New York without mentioning Pizza. If you have the time to go to Brooklyn, Grimaldi's is probably the best pizza in the world. Check it out if you have time. http://www.grimaldis.com/. If you need something more local to Manhattan John's pizza is also really good or at least a close second. There are multiple locations. http://www.newyorkmetro.com/pages/details/8512.htm. The original place is in the village off of Bleeker. I think it is slightly better than, the Times Sq location, but Times Sq is a better sit down place.

Once you have tried Pizza, try SeraFina's http://www.serafinarestaurant.com/. The 61st street restaurant is a great place. Great Italian outside of Little Italy.

Anyway, just a couple of tidbits. There are plenty of other places to check out, but these are a couple that I always have to stop by.

I hope you find the HAT walkthrough useful and if there is anything missing please let me know.

Walkthrough on using the SQL Adapter with BizTalk 2004

2004-06-22T23:42:00Z

A piece of documentation that I thought would be useful would be more information on using the SQL Adapter with BizTalk to retrieve data from SQL. I decided to write a walkthrough on this, so hopefully this will be useful. Links to whitepaper and project are in SQLTest.zip and can be found here. http://www.winisp.net/georgenenc/sqltest.zip

Creating Batch files to deploy BizTalk 2004 assemblies

2004-06-18T19:40:00Z

I wanted to focus this message on passing on some documentation on creating a deployment machanism to move from development to testing without having to run the deployment wizard over and over. Since I seem to be constantly modifying my samples and demos and redeploying them, I thought it might be nice to find a way to create a batch file to reuse my samples on different machines and even save them to a CD to take with me anywhere. Along the way I started looking into the way the BizTalk SDK samples were created and decided to copy them.

Once I got this figured out, I basically documented it and posted it here. Please use this as a guide to help you create your own mechanism to go from your development machine to QA. Will keep you form constantly have to run the deployment Wizard to remove it and add it back in.

I also have found a tool written by Erik Leasburg and Mariusz Borsa that can be used to modifying the binding files. The source and tool is located here. Usage instructions will show up when you run it without arguments form the command prompt.

I hop you find this helpful and let me know what you think.