Ryan R. Donovan's Microsoft WebLog : CS2007 Software Architecture

Wrap-Up: Commerce Server 2007 Architecture Series

rdonovan — Thu, 18 Dec 2008 19:34:00 GMT

The Commerce Server 2007 architecture series I started the other month is now complete. My hope is that this information is useful for those seeking to get acquainted with Commerce Server - and to learn some of the caveats one must consider when using the product. Additionally, hopefully some of the context shared will provide insight on why certain things work in the manner in which they are implemented within the product itself.

The suggested reading order is:

Philosophy - http://blogs.msdn.com/rdonovan/archive/2008/09/01/cs2007-software-architecture-series-part-1-philosophy.aspx
Platform Basics - http://blogs.msdn.com/rdonovan/archive/2008/10/28/cs2007-software-architecture-series-part-2-platform-basics.aspx
What's Under the Hood - http://blogs.msdn.com/rdonovan/archive/2008/10/30/cs2007-software-architecture-series-part-3-what-s-under-the-hood.aspx
Key Subsystem Overview - http://blogs.msdn.com/rdonovan/archive/2008/11/01/cs2007-software-architecture-series-part-4-key-subsystem-overview.aspx
IT Professional & Business User Component Overview - http://blogs.msdn.com/rdonovan/archive/2008/12/14/cs2007-software-architecture-series-part-5-it-professional-business-user-component-overview.aspx
Application Design Considerations - http://blogs.msdn.com/rdonovan/archive/2008/12/17/cs2007-software-architecture-series-part-6-application-design-considerations.aspx
System Usage Considerations - http://blogs.msdn.com/rdonovan/archive/2008/12/17/cs2007-software-architecture-series-part-7-system-usage-considerations.aspx
Web Site Architecture Considerations - http://blogs.msdn.com/rdonovan/archive/2008/12/18/cs2007-software-architecture-series-part-8-application-design-considerations.aspx

And, for your convenience, the entire series is tagged at http://blogs.msdn.com/rdonovan/archive/tags/CS2007+Software+Architecture/default.aspx.

For a shorter URL, try http://tinyurl.com/3mnebm.

Hope this helps!

CS2007 Software Architecture Series Part 8: Web Site Architecture Considerations

rdonovan — Thu, 18 Dec 2008 17:38:00 GMT

32-bit or 64-bit

The first thing to think about when building the end application is should it be a 32-bit or 64-bit application. Obviously, 64-bit is preferred – but there are many driving factors that will force 32-bit support to be utilized (even if running on an x64 operating system).

In short, the reason 32-bit mode would be forced is because of a piece of 32-bit legacy code running somewhere in the application. Common culprits would be older/3^rd party applications such as external content management systems or search tools. Within the realm of Commerce Server, pipeline components will typically force 32-bit mode as many ISVs have not updated their components for x64.

If 32-bit mode is utilized, up to 3GB can be utilized per worker process. Memory consumption should be planned accordingly – along with separate worker processes to host multiple sites or applications if more than 3GB in aggregate is needed.

2-tier or 3-tier

When building a Commerce Server, one of the most oft-debated subjects is whether to use a 2-tier or 3-tier architecture. However, many customers desire to utilize a 3-tier architecture. The principal factors driving 3-tier are:

· Ability to physically separate the database tier

· Better abstraction from a code perspective

· Ability to support other business channels or external punch-outs (e.g. – A site running on another platform needs to use Commerce Server as a back-end.)

The practical reality is that a 3-tier architecture will almost always be slower from a performance capacity than a 2-tier architecture. If performance and hardware footprint are a paramount concern – then it should not be considered. If it is needed from a business requirements perspective – then it is a requirement. If it is indeed needed and performance is a concern, one might want to consider having both 2-tier and 3-tier code running on top of the same Commerce Server instance so as to only incur the 3-tier performance penalties for the scenarios where they are explicitly needed.

Building a 2-tier Site

In this section, the specific design considerations that need the most thought and analysis of architecting a 2-tier application with Commerce Server shall be identified. Given that a 2-tier application written utilizing Commerce Server is fundamentally an ASP.NET application at its core; all standard best practices of architecting an ASP.NET application apply. Given that this is a well known and understood problem space, ASP.NET best practices shall not be regurgitated here. Instead, refer to http://www.asp.net/ or http://msdn.microsoft.com/en-us/asp.net/default.aspx.

Caching

When architecting a 2-tier ASP.NET application using Commerce Server, by far the single most important architectural consideration is caching. This alone will have more of an impact upon performance than almost any other single consideration.

There are several types of caching applicable to a Commerce Server application built in ASP.NET, specifically:

· Profile Cache – This is an intrinsic cache of data within Commerce Server’s Profile system.

· Catalog Cache – This is an intrinsic cache of data with Commerce Server’s Catalog system.

· Pipeline Pool – This is an in memory pool of pre-instantiated Commerce Server Order Pipelines, aimed at reducing the call time to the Pipeline.

· ASP.NET Caches – At a developer level, both data and output can be cached utilizing the standard means within ASP.NET.

With that understanding in hand, one can then apply all of these throughout the construction of an ASP.NET application based upon Commerce Server 2007. In an attempt to simplify caching philosophy to something that can be easily understood and followed, the guiding principles are:

· Always use the Pipeline Pool – it will greatly reduce call time on pages calling Pipelines.

· The Profile Cache is always on – understand how it works and plan accordingly as it cannot be disabled.

· Always cache the Commerce Server context objects in ASP.NET – this will greatly reduce the time required to instantiate and call the Commerce Server APIs

· Balance Catalog Caching with ASP.NET output caching – otherwise the same data could end up being cached in multiple locations and wasting memory

To use the Pipeline Pool, all components must be free or neutral threaded – or written in .NET. If any component does not meet this criteria, it should be upgraded or replaced if at all possible as it will negate the ability to pool pipelines in entirety. Because no two pipeline configurations are the same, memory consumption should be profiled utilizing Performance Monitor and factored accordingly into the application’s memory requirements. The pool size should be adjusted based upon the likely number of pipelines that are likely to be active at one time – so as not to overtly waste memory. This can be determined by building a projected usage profile of the site and/or examining Web log files – simply look at the number of concurrent calls to pages or paths utilizing pipelines and set the pool size accordingly. Note: The pool size is a global setting so factor it across all pipelines in use in aggregate across the site.

The Profile Cache caches profile data in memory – and then does lazy committed writes to SQL Server as data is updated. It can potentially cause data corruption unless strict session affinity is maintained, as a user can get bounced from one server to another in a Web farm and receive inaccurate data since it might not have been written to disk.

As session affinity can be complex to establish and maintain, a better practice is to force-write profile data to the database every time it is updated. But this should be done sparingly – as it is an expensive operation (e.g. – don’t do it every time Profiles are touched – just upon updates). Then, there are no requirements for considerations around session affinity. The Profile cache size should be monitored in Performance Monitor to factor memory consumption requirements and then set accordingly as a configurable parameter.

The rest of the cacheable data on the site revolves around the Catalog. The absolute best practice here is to output cache at a page fragment/user control level. Output caching in general will provide the most superior performance for caching catalog data. However, it needs to be done at a page fragment/user control level versus a page level – or other things like Profile authentication might also get cached and cause the wrong behavior to occur.

With respect to output caching, one must consider that the site’s memory consumption will grow proportionate to the size of the catalog and how much of it will actually be browsed. This is the significantly complicating factor in leveraging a strategy of output caching and represents the “price” that comes with the performance – as there is no easy or simple way to model and factor the usage considerations. This will need to be planned based upon a calculated usage profile or analysis of actual Web logs. The affects of cache flushing, cache warming, or insufficient memory to cache everything must be factored based upon the projected usage. Remediation steps may need to be taken (e.g. – scripted cache warm-up when resetting the entire Web farm, rolling updates, cache resets when staging data, etc.) to ensure proper behavior.

Alternatively, with output caching, one must set expirations accordingly to ensure data is appropriately fresh. This is why utilizing fragments is important – as data that must be fresher by necessity (such as Inventory) can have a lower expiration value versus more general catalog data, which can generally suffice with a higher expiration value.

Page Design

Writing ASP.NET code for extensibility and best practices – versus performance – are two different arts. A simple site that is lean and mean will perform best, but offer the worst extensibility and maintainability. And vice versa.

In general, if performance is of principal concern – outside of the caching practices described in the preceding section, consider:

· Do not use session state or view state

· Avoid utilizing the ASP.NET intrinsic objects such as the form controls

· Leverage JavaScript on the client whenever possible versus requiring logic at the server

· Generally avoid many design patterns (such as UIP or Web Client Software Factories) and code strictly to the actual functional requirements

International Considerations

Building an international-ready Commerce Server application is a mix of interesting considerations, largely because Commerce Server is not fully internationalized itself. A summary of the core product’s language capabilities is as follows:

· Profile: Single language, but any locale supported by the underlying data store is supported.

· Marketing: Single language, but any locale supported by SQL Server is supported.

· Catalog: Multi-lingual (e.g. – you can have French and English versions of the same attribute in the same record/row in the Catalog) in any locale supported by SQL Server.

· Inventory: Multi-lingual, as it relies upon the Catalog system.

· Orders: Single language, but any locale supported by SQL Server is supported.

The Catalog, as noted, is single currency by default. Separate fields will be needed to support multiple currencies, or a lookup table can be utilized to convert across exchange rates.

Building an International ASP.NET-based site using Commerce Server, with the aforementioned understanding, can be done by observing the following considerations:

· When building the ASP.NET site, utilize resource files, thereby separating page content from page design from page code.

· Make sure that all labels that must surround the Profile and Order data are stored in resources at the ASP.NET level – this way only the form data is going back and forth to the database and no presentation tier elements are affected.

· Configure the Catalog as multi-lingual and retrieve data based upon the desired locale – matching it to the ASP.NET site locale.

· If multiple currencies are required, the best practice is to have one currency field per required currency and set explicit pricing. This way the variations caused by managing exchange rate tables can be overcome (not that many businesses utilize this for international pricing regardless). Then pull the currency based upon the ASP.NET site’s locale.

· For Marketing, the situation is somewhat more challenging because it does not support multi-lingual data – and there are many display attributes such as showing discounts in the basket, advertisements containing text content, etc. The best way to handle the situation is to set custom targeting criteria of locale, pass that value into custom targeting context from the site, and then pull marketing campaigns based upon locale as custom targeting criteria. This unfortunately requires that campaigns be duplicated in every locale supported by the site, but it does solve the problem gracefully from a display standpoint. Alternatively, one must work around the fact that only single language data can be stored and use placeholder data in the database, then resolve that to an appropriately localized resource string at runtime (which would require a fair bit of code and be semi-awkward to implement, arguably perhaps not more awkward than the first workaround proposed here).

Security Considerations

There is nothing special about securing a Commerce Server application written in ASP.NET. It is essentially just another Web site. The critical considerations are ensuring that there are no injection attacks or other forms of malicious behavior possible through manipulation of the Web page. So, such things must be taken into account at the page design level as validating:

· Query strings

· Form posts

· HTTP Referrers (e.g. – make sure that only the correct preceding page (if in a sequence of pages) can call the current page)

· Cookies (to ensure that they cannot be easily forged)

Pages in general – including custom error pages or other exception handling sequences – should be made to be as non-CPU intensive as possible to handle a denial-of-service type attack of bulk HTTP requests.

As there is nothing specific about Commerce Server here, the general ASP.NET security best practices apply very well.

Reporting Considerations

If the Commerce Server Analytics system is to be utilized, several points of consideration must be thought about in the site design process. By default, Commerce Server Analytics pulls data from two sources – the Commerce Server databases themselves as well as the IIS Weblogs. SQL Server DTS tasks are utilized for importation of both sets of data.

If no schema customizations have been made outside of what is possible within the Commerce Server tools, then the default DTS tasks for database-based data can be utilized. If yes, then additional DTS tasks must be created and the data warehouse schema extended to facilitate importation of the customized data.

With respect to Weblogs, they are relatively easy to import – but they tell only a small portion of the story. To properly tell how a user is navigating a site, additional data points must be collected based upon specific user actions and so forth. Commerce Server’s Analytics system facilitates this through the use of Custom Events. These can be dropped into any ASP.NET page programmatically – and they will capture/log additional data as defined/collected for use in the Analytics system. These do, however, slow down page processing – and should be used with requisite care so as to not adversely affect site performance.

From a reporting perspective, no data in custom events or in custom DTS tasks will be truly integrated into the out of box reports. So, to fully leverage the newly collected data – report customizations must be made and should be factored into the overall project accordingly.

Building a 3-tier Site

Commerce Server was not designed to be a 3-tier application – at least not at the time the product was originally created in 1998-1999 (for Commerce Server 2000). However, in today’s world – 3-tier applications are inescapable.

Commerce Server 2002 Feature Pack 1 and Commerce Server 2007 debut ASP.NET (ASMX)-based Web Services. These are intended for data management only – and are utilized internally by the BizTalk Adapters and Business User tools. They are fully documented and would be appropriate for use as-is or customized for back-end data update scenarios, such as connecting with a SOA-broker different than BizTalk.

They were not, however, designed for building a runtime site. In fact, there will be many API gaps present – as only the data management aspects of the overall Commerce Server API capabilities are represented. And those that are represented are optimized for data management, and not runtime scenarios.

Building a 3-tier Web site to enable such scenarios as the following:

· Communication from other systems in a punch-out scenario, such as Site A allowing checkouts to be processed by Site B

· Enablement of other devices/channels, such as handheld, mobility, kiosks, etc.

To build a 3-tier front-end Web site, the following considerations should be examined:

· Choice of technology – the obvious choices would be ASP.NET Web Services (ASMX) or Windows Communications Foundation (WCF). Given future technology direction, a WCF-based solution will likely have far better longevity (in terms of future technology evolution) than a solution based on ASMX

· Commerce Server – All of the standard considerations from building a 2-tier ASP.NET Web site also apply equally to building a 3-tier Web site, regardless of whether or not the technology choice is ASMX or WCF.

· Service Design – This is probably the most critical piece. Web service calls are inherently quite expensive. Hence, they should be consolidated as best as possible. “Chunky” calls will always perform better than “chatty” calls. Unfortunately, Commerce Server’s API (and that of many other database-driven applications) assumes lots of small calls versus fewer larger ones. Thus, the onus becomes shifted to the developer to – within his or her own Web service design – encapsulate the Commerce Server calls to behave well in a batch mode accordingly. This will likely require a substantial amount of application level data caching at the server (and possibly session affinity) – and should be planned for accordingly in terms of both physical deployment as well as memory utilization.

Given that there is nothing Commerce Server-specific about building Web services utilizing Windows Communication Foundation, general best practices apply. For more information, please refer to http://msdn.microsoft.com/en-us/library/ms735119.aspx, which will be a good set of resources for building WCF Web Services in general – including those utilizing Commerce Server.

CS2007 Software Architecture Series Part 7: System Usage Considerations

rdonovan — Thu, 18 Dec 2008 08:05:00 GMT

Figuring out how to correctly utilize the various Commerce Server systems is mostly a matter (in most cases) of proper schema design coupled with understanding the breaking points of the various components so as not to exceed them. And this, of course, is based upon a solid understanding of the end state business requirements – as already discussed.

The other considerations and typical “gotchas” of system architecture with Commerce Server shall be enumerated throughout the balance of this section. From a logical perspective, it probably makes the most sense to attack system design in the following order:

· Profiles

· Catalog & Inventory

· Orders

· Marketing

· Analytics

By starting with the customer, then the goods being sold to the customer, then the orders to process the goods, then the discounts overlaid on top of the orders, and finally the reports against the entire experience – one can minimize changes as a result of dependencies between systems unearthed in the architecture process.

Profiles

The major consideration with the Profile system is where does the data live? Given the flexibility- it can live in a variety of different places. Some considerations:

· If the number of users is the principal concern, SQL is the best place to store the data – successful production deployments have gone to 60M+ on very moderate hardware.

o The Partitioning feature adds a lot of complexity and should be carefully weighed as to the trouble factor versus simply running on a single SQL database; keep in mind that no customer has yet come close to needing to go beyond a single database deployment given the scalability of SQL Server on hardware available today.

· Conversely, Active Directory can only store a small subset of users compared to SQL - ~10M is the practical limit.

· OLE/DB and ODBC sources – although supported, were not expressly tested during the development cycle of Commerce Server 2007 given the wide variety of combinations and potential sources possible; utilizing these capabilities can be potentially putting oneself into unexplored territory .

· Having disparate sources – although convenient – is a major performance drain. The Profile system will have to query multiple data sources and create a union of the results versus being able to execute a single query.

Because there is no “canned” schema – it is up to the user to completely define the system in whatever the systems of choice are for storing data. All standard best practices of designing relational database schema (or Active Directory, as the case may be) apply. Some particular considerations relevant to Commerce Server profiles include:

· Because this is a from-scratch defined schema, if utilizing SQL – make sure it has indexes (and in particular clustered indexes) that are relevant to the usage patterns of the data being stored.

· Design the table structure with respect to how it will be queried and updated – pulling single tables will always be cheaper than pulling multiple tables with join operations.

· Be sure to remember where/how attributes utilized in personalization for the Marketing system will be stored as these are often the most queried in a production system.

There are no specific limitations with respect to data sizing; however the product was not tested above 60M profiles (mix of anonymous and registered). That being said – going above this should not be a major concern point.

The other major aspect of Profile system design is compliance with Payment Card Industry (PCI) standards. Storing any credit card data has major implications on policies from the credit card merchants themselves. If at all possible it is best not to store credit cards – as that alleviates many aspects of PCI compliance and the resultant independent audits. If it is required, data encryption must be utilized, which is supported by the Profile system. This will, however, affect performance for queries/updates accordingly.

Catalog

The Catalog system is designed as such that one rarely has to touch the underlying database directly. Therefore, the usage considerations are far narrower in scope, but often times more subtle than straightforward SQL design.

Some of the particular schema design considerations include:

· Commerce Server stores a lot of metadata; SQL Server has a row size limitation of 8K. It is very easy to exceed this limitation when utilizing character-based data types. Using text-based data types (as opposed to character-based data types) will eliminate this problem, but make querying and accessing data slightly more complicated. It is better to go with text from the beginning however – as changes later could be potentially expensive and complex to affect against an existing system.

· Any field that needs to be free-text searchable will need to have its indexes continuously updated by SQL Server; the potential for stale data or a significant amount of processing overhead for index rebuilding is very high. In general, it is better to be judicious and find other ways to search data (if possible) than leveraging free-text searching given these factors.

· The Catalog is not truly multi-currency; plan on storing separate fields for each currency being supported or plan on leveraging an exchange-rate translation table. There is no in-between.

· Adding language support is easy from a schema perspective – so this can be easily added at any time.

· Obviously, simpler is better – keeping the structure as flat as possible will make for faster querying.

· Storing binaries (e.g. – images or other multimedia files) was not a design consideration of the Catalog system; hence this should probably be avoided and links to the file-system or other content management systems utilized.

There is no practical limit to the number of base catalogs. The product has been tested to the levels published in the performance guide, which is available at http://www.microsoft.com/downloads/details.aspx?FamilyID=E79691F0-BE0F-40A6-940C-5D3A679C5526&displaylang=en.

Going beyond this should generally not present a problem but should be tested accordingly. The principal issue that usually arises is the re-indexing time for free text searchable properties – as noted above. Therefore the fewer of those present in the schema, the better.

The Catalog Sets feature has no specific limitations on the number of catalog sets. However, fewer is better – as the list of catalogs and catalog sets will need to be enumerated on ever single request involving catalog sets to match users to particular catalogs (and there is no way possible to cache this data). Keeping this to a small number will result in far better performance. When creating target expressions for Catalog Sets, it is best to structure the expressions such that a single table within the Profile system can be queried and join operations can be avoided to ensure fastest processing. (This may, in turn, affect Profile schema design as well.)

Virtual Catalogs provide an immense degree of flexibility – and an immense potential for complications given the flexibility. The same data sizing limitations apply to catalogs. The tested limit of number of Virtual Catalogs is 10,000; going above this is possible but requires careful testing. The other big consideration of utilizing Virtual Catalogs is materialization – this will represent in the best runtime performance but requires time to rebuild from the base catalog (which takes overhead) and stale data could be presented while this is occurring.

Orders

The principal design considerations of the Order system are the schema and the Pipeline design. With respect to the schema:

· Baskets are stored in binary format.

· Orders are stored in a mix of binary format as well as normal SQL database tables; what goes where is determined by the Order Mapping XML.

· Storing binaries in baskets and orders themselves should be avoided; this was never tested and will degrade performance.

· ANY field that will be utilized after an order is captured should be stored in SQL and not in the binary field

· The data schema for SQL storage should be as simple and flat as possible – to minimize the effort required to persist an order to disk

· As the SQL schema is custom-designed by the developer, indexes must be implemented appropriately:

o Do not create clustered indexes based on fields that will slow down order storage; in fact clustered indexes may not even be appropriate at all

o Implement indexes based on the fields that will be utilized for query and analysis post-capture

With respect to Pipeline design, there are several principles that need to be considered:

· Try and keep the Basket pipeline as light as possible; this is typically run many more times than the Order calculation and capture pipelines – so having less work to do will greatly improve site throughput .

· The Order pipelines are transactional – however there may be steps in there that cannot enlist in a DTC transaction (such as Web service calls for merchant services authorization); in this case one must think about how to handle failure scenarios and accommodate manually through code.

· Pipelines are still COM components; therefore one must use COM+ transactions appropriately and ensure that the threading model supports free or neutral threading (apartment threading will not work) for everything to work properly (and allow pipeline pooling, which will greatly reduce instantiation time on the actual site).

· Long-running operations may best be handled outside of the pipeline (e.g. – if calls to credit card authorization providers are egregiously expensive) – however this then requires separate handling outside of the order capture process and may also impact Payment Card Industry (PCI) compliance.

And with the last comment – there exists a great segue to the topic of Order capture and PCI compliance. The considerations for this can be best summarized as:

· One is best off by capturing an order and NOT storing any credit card data – just the authorization number. For returns, one would have to re-input the same or a different credit card (which conceptually works just as it does in most physical store environments).

· If credit card data must be stored (such as in the scenario if it is processed offline), it must be encrypted. Unfortunately, the Order system does not support encryption as an intrinsic capability. This will require a custom pipeline component to be written to encrypt orders – which will result in a fair bit of work for initial development and possible performance degradation during the capture process (plus associated post-capture maintenance as well). This tradeoff should be weighed carefully.

Marketing

The Marketing system is a relative black box compared to the rest of the Commerce Server systems, as it is more a matter of configuration than design. That being said, each aspect of the system has its own set of unique considerations to ensure successful use from an architecture perspective – especially with regards to dependencies in other systems and upon runtime performance.

From an overall perspective, performance is the most notable consideration. The system was designed to work with between 1,000-2,000 items active at one time. Beyond that and performance will end up degrading. Thinking about scenarios such as one discount per item in the Catalog will prove to be impractical in production; instead consider customized pricing and Virtual Catalogs – as an example.

The other major performance consideration is with respect to the usage of Target Expressions. More Target Expressions (especially if compounded together) will equal worse performance. Likewise, Target Expressions that can query against a single table in the Profile system will perform best; those that require join operations internally will perform considerably worse.

It is important to keep in mind that there is no caching of the Marketing system. Expressions are evaluated upon each and every request. Hence, they should be called judiciously. In general, the discounting functionality will not represent a problem – as this is typically only called in basket and checkout operations. Although by utilizing the Runtime Discount Filtering capability it is possible to use it on the end site – this will significantly degrade performance. With respect to Advertisements, generic advertisements that utilize only impression tracking capabilities will be more practical (for cross-selling and up-selling) than using Target Expressions to target content to individual users given the overhead of evaluating the expressions. Direct Mail, because it works offline, generally will not impact end site runtime performance.

The intrinsic e-mail handling capabilities (or lack thereof) of the Direct Mail system are the last major consideration; its capabilities are very much aimed at providing basic send functionality with minimal tracking and error handling. This has proven to be a challenge point for many customers. Instead, a better alternative might be to use the List Manager feature of Direct Mail (which utilizes the Target Expressions of the Marketing system to generate the recipient list) to create lists of intended recipients and then export them for use in another mailing engine.

CS2007 Software Architecture Series Part 6: Application Design Considerations

rdonovan — Wed, 17 Dec 2008 14:42:00 GMT

With a solid understanding of exactly what Commerce Server is – and more importantly is not – one can then come up with a somewhat logical methodology to build production grade application architectures based upon the product. This section will attempt to outline the most expeditious thought process possible to get from here to there.

The most important consideration, however, comes before the technology – and that is business requirements. No two businesses run alike. In most cases, businesses will not adapt their business processes to work the way the product most naturally does. To get around this, one should generally not view Commerce Server as an out-of-box solution, but rather a toolset that can be utilized to build a solution around business requirements.

Hence, having an exceptionally solid understanding of the business requirements is the most important first step. As this book is about Commerce Server and not requirements analysis, that will not be discussed here.

The principal considerations that need thinking - beyond intrinsic idiosyncracies within Commerce Server (which will be covered in a separate post) - include:

Tenancy
Single versus Multi-Environment
Interactions by Business Users
Interactions with Back-End Systems

Tenancy
The first consideration in a Commerce Server deployment is the expected business deployment topology – specifically how many sites and how should data be shared amongst them. In the simplest scenario, one will have a single site (e.g. – www.companyx.com) and a single set of Catalog, Inventory, Orders, Profiles, Marketing, and optionally Analytics.

Rarely is life that simple. Let’s consider a few common situations:
•    Multiple Brand Subsidiaries (e.g. – www.companyx-1.com, www.companyx-.com and so forth – all belonging to the same parent)
•    International Sites (e.g. – www.companyx.ca, www.companx.com, www.companyx.co.uk and so forth – all providing the same site for different locales)
•    Mix of both!

The business requirements around this have significant impact on how Commerce Server 2007 is deployed – and which edition of the product can be utilized. The principal deciding factor is whether Catalog, Inventory, Orders, and Marketing data should be shared between end-sites – or not.

If Catalog, Orders, and Marketing data can or must be shared amongst end-site instances, an application architecture of the following must be implemented as follows... A single Commerce Server Site must be provisioned, with single instances of Site Resources such as Catalog, etc. Each end-site (e.g. – www.companyx-1.com, www.companyx-1.ca, www.companyx-2.com, www.companyx-2.ca, and so forth) must be configured as a Commerce Server Application within the site. This configuration requires Commerce Server 2007 Enterprise Edition, as the limit of one Application per Site in Standard Edition has been exceeded.

In another scenario, Catalog, Inventory, Orders, and Marketing data should not be shared between sites. But perhaps it is still desirable to leverage the same physical infrastructure. In this case, an application architecture must be implemented as follows... Each end-site must be configured as a separate instance of a Commerce Serve Site, with one set of Site Resources and Application per Site. In this scenario, up to 10 instances like what is described here can be implemented on a single physical server running Commerce Server 2007 Standard Edition; if more than 10 is required – Enterprise Edition must be utilized.

Although other permutations exist, these represent the two principal Commerce Server 2007 tenancy scenarios that can be accommodated by the product. Commerce Server itself is not truly multi-tenant (at least in this release) – so it cannot differentiate between tenants within a single Site or Application instance.

In EITHER scenario, the Profiles and Analytics resources can either be shared amongst everything or affinity established with one of the deployments – as they are global resources. Likewise, in either scenario – single or multiple physical deployments can be utilized – but databases will be shared as noted based on the chosen application deployment scenario. Besides the obvious sharing (or not) of customer data, this has impact on which business users can change and access what data as well.

Single versus Multi-Environment
The next consideration is how many environments are needed. If a single physical Web farm can be utilized, then Commerce Server 2007 Standard Edition can be utilized. However, this means that all business data changes for pricing and such will be made LIVE to the production environment. This has several potentially negative ramifications, specifically:

•    Data Inconsistency – Any data entry mistake or flakiness caused by changes as data is being updated will be immediately represented to the customer – and potentially reflect negatively.
•    Performance – Making changes to the live site will indeed negatively impact runtime performance. It’s as simple as that.
•    Security – This will require that the Web Services are exposed to the Internet on the production Web farm – which could potentially represent a point of vulnerability for a hacker.

If any of these factors are deal breakers, then Enterprise Edition must be utilized and business data changes made in a physical environment separate to production.

Interactions by Business Users
Commerce Server 2007 was designed for all business user maintenance to go through the Windows-based business user tools, specifically Catalog Manager, Marketing Manager, and Customer and Order Manager. (Or at least like approximation of those tools making similar calls to the Commerce Server Web services.)

From an architecture perspective, the first consideration is whether or not these tools should be customized. If yes, one must consider ongoing maintenance – as these tools are being revised at least as often as every Service Pack of Commerce Server. Microsoft will release DIFFs of source code between versions with Service Packs to facilitate making changes, but this must be planned and accounted for in the event that customizations are indeed made.

From a utilization perspective, these tools can tend to be somewhat processor intensive and/or destructive (given that they are modifying the underlying data upon which Commerce Server operates). They are not ever meant to be utilized on a live site. If they are to be utilized on a live site, the following considerations must be adhered to:

• Perform maintenance in off-peak hours only – as some degree of performance degradation can generally not be avoided
• Utilize caching as such that the site changes will not be noticed – and then refresh the cache utilizing some other means after changes have been made
The best practice, of course, is to perform changes in a separate environment and then stage them to production utilizing Commerce Server Staging, the BizTalk Adapters, or some other convenient means.

Interactions with Back-End Systems

When running an operational e-commerce system, there will generally be two streams of data inbound and outbound from the system. On the inbound side will be business user changes – with updates to Catalog, Inventory, Marketing, and Orders (for order status). On the outbound side will be Profiles (for customer data), Inventory (to reflect quantities purchased on the site), and Orders (to send off for fulfillment).

Commerce Server 2007 provides the BizTalk Adapters, which leverage the data management Web services, as its out-of-box mechanism for affecting line of business data updates of this nature. The data interchanges will generally have a BizTalk hub attached to Commerce Server, with at least one (if not more) Send and Receive Adapters flowing in and out of the Order, Profile, Catalog, and Inventory systems of Commerce Server.

BizTalk in general represents a good integration strategy, as data interchange can be configured within the BizTalk toolset with minimal coding to any system that can already interface with BizTalk. At the time of writing, support for BizTalk connectivity (and ergo Commerce Server connectivity via BizTalk) from Microsoft can be obtained from http://www.microsoft.com/biztalk/en/us/adapters.aspx.

Alternatively, any other SOA-broker can connect to the Commerce Server Web Services to perform the same functions as the BizTalk adapters – there are no special hidden capabilities.

Some of the best practices to consider when utilizing the BizTalk adapters (or any other integration broker) are:
•    Operate on a reasonable polling schedule for pulling data – this is generally business requirement driven but there is an artistic balance to be achieved between not enough and too much. Once per hour typically meets the needs of most customers and is generally unobtrusive. During extreme peak times, it may be best avoided and handled during off-peak times. Business rules typically need to be adjusted in this case, such as additional inventory stock-out thresholds on hot items, etc.
•    For areas where SQL indexes can be customized, tweak the SQL indexes to optimize for the queries generated by the polling operations. If one is unsure what the polling operations are doing, this can be easily discovered by utilizing SQL Profiler and indexes then adjusted accordingly. (Be careful though – especially with clustered indexes – to not inversely create a bottleneck for updating data or the runtime site.)
•    For updating data, try to keep the operations as infrequent as possible. Additionally, one must be respectful of indexes that already exist – and the impact that updating can have (especially with clustered indexes). Separately, there may be additional operations required – such as refreshing site cache – these may need to be triggered also at the time of updating.

CS2007 Software Architecture Series Part 5: IT Professional & Business User Component Overview

rdonovan — Mon, 15 Dec 2008 06:43:00 GMT

Commerce Server provides a number of application tools on top of the Development Platform to aid in the deployment. These tools can be principally broken into two categories:

• IT Professional Tools – aimed at allowing technical users to manage the operational aspects of the Commerce Server deployment.
• Business User Tools – aimed at allowing non-technical business users to manage aspects of the Commerce Server deployment on a day-to-day basis without IT professional involvement.

These are all described in detail in the remainder of this post, and comprise the remainder of the components included with Commerce Server 2007.

Business User Tools

Commerce Server provides four business user applications. They are all implemented as Windows Forms applications utilizing the product’s Agent API and calling the Web Services. A question often comes up as to why Windows Forms applications were utilized in lieu of a Web-based application. The answer is that given the technology available at the time Commerce Server 2007 shipped (mid-2006 calendar-year) – it was the only means of providing completely acceptable usability and performance for the most demanding customer scenarios.

Previous versions of Commerce Server utilized the Business Desk, which was in effect an early AJAX-style application. It was one of the most panned aspects of Commerce Server 2000 and 2002; the performance and usability when managing large amounts of data were atrocious. Web pages are simply not suited to streaming and managing multi-million item catalogs or accommodating other such challenging scenarios.

Silverlight represents a long-term answer and future direction, but that is a discussion for another day...

Source code is provided for the business user application s to facilitate customization. However, in the event that changes are made – it then becomes the developer’s responsibility to download and merge in fixes to the code-base as the Commerce Server product itself is revised (if the Microsoft out-of-box binaries and resultant user experience are not utilized).

Given that the applications are written in .NET, they will run natively on either x86 or x64 processor architectures without issue. Because they consume the Web Services, they can be utilized either over the Internet or in an Intranet scenario. They can utilize either NTLM or Basic+SSL for authentication. Microsoft Authorization Manager (AzMan) technology is utilized to provide customizable role-based security.

The tools themselves that are provided include:
•   Catalog Manager – Utilized for managing all data entities stored in the Catalog and Inventory systems.
•   Catalog Schema Editor – Utilized for defining and managing the property and entity definitions in a configured instance of the Catalog system.
•   Marketing Manager – Utilized for managing all data entities represented by the Marketing system.
•   Customer and Order Manager – Utilized for querying and – in some cases – managing the data entities stored in the Profiles and Order systems.

Separately, a Profile Schema Manager is provided – but this is as a Web application that is based on a subset of the old Business Desk technology. Why? Because this component is shared with BizTalk Server 2006 and is embedded within it for the trading partner management feature-set and compatibility would be broken if it were updated to Windows Forms.

IT Professional Tools Overview

Commerce Server 2007 provides a number of components to facilitate physical deployment. These all work separately but somewhat depend upon one another, and include:
•   Admin APIs and PuP
•   Management Console
•   Operations Manager 2005 Management Pack
•   Staging

Admin APIs, PuP, and Management Console

All of Commerce Server’s physical deployment configuration is stored in a SQL Server database called MSCS_Admin. Living on top of this data store is the Admin API, which is effectively a set of APIs that are used internally by Commerce Server (as well as callable by end developers) to configure itself at runtime as well as provision instances of the product.

Several distinct entities are managed by the Admin API, including:
•   Global Resources – A global resource is an instance of a Commerce Server system entities that can be shared amongst multiple configured instances of the product. There are only two systems in Commerce Server that can be configured in such a manner: Profiles and Analytics. What does this mean in English? One can have multiple, separately configured instances of Commerce Server (e.g. – for completely different sites for CompanyX.com and CompanyY.com) that share Profile and Analytics data.
•   Sites – A site is a single configured instance of Commerce Server. It is a collection of the Catalog, Inventory, Orders (split as Transactions and TransactionConfig to be precise), and Marketing systems that are all interrelated together. Each instance of these entities is called a Site Resource. Typically a Site has affinity with a single organization, such as CompanyX.com. Catalog, Inventory, Orders, and Marketing data cannot be shared across sites. Commerce Server 2007 Standard Edition supports up to 10 sites on a particular physical server. Commerce Server 2007 Enterprise Edition supports an unlimited number of sites – and is bounded only by physical hardware limitations.
•   Applications – A subset of a Site. It represents an ASP.NET application in IIS that has affinity with a site in Commerce Server. Each Web Service instance (Catalog, Marketing, Orders, and Profile) + each ASP.NET site built on top of a configured Commerce Server 2007 Site counts as an application. By default, there will be one instance of each Web Service + one single ASP.NET application per site (for the end Web site). Commerce Server 2007 Standard Edition supports only one application beyond the Web Services per Site. Commerce Server 2007 Enterprise Edition supports an unlimited number of Applications per Site.

The Admin API allows the provisioning and reading of configuration at runtime for all of the aforementioned entities. It also enforces the limits imposed by Commerce Server 2007 Standard Edition. In addition, it can be utilized to create custom Site Resources to allow ISVs or site developers to create new entities that can be managed as part of the Commerce Server management tools.

The PuP utility leverages the Admin API to create a single binary package of Global Resources, Sites, Site Resources, and Applications for easy deployment to a system (or a subset thereof). It can create a new binary package based on a configured deployment – or deploy an existing binary to provision a new deployment. Like the Admin API, it is extensible to handle any custom actions required needed to facilitate changes/extensions made by site developers or ISVs.

The Commerce Server Management Console is simply a Microsoft Management Console (MMC) snap-in that allows configuration of the runtime properties of all of Commerce Server’s global resources, Sites, Site Resources, and Applications.

Operations Manager 2005 Management Pack
The name of this feature could not possibly be more self-descriptive. It allows Microsoft Operations Manager (MOM) 2005 to be utilized to monitor the operational health of a Commerce Server 2007 deployment, aggregating all system events and performance counters raised by Commerce Server.

It is comprised of the Health Monitoring Service, which lives on each system utilizing Commerce Server – and the Management Pack itself (which resides on the system running MOM) – which communicates with the Health Monitoring Service to expose data inside of MOM.

If one wishes to utilize Systems Center Operations Manager 2007 (SCOM), the tools provided therein can be utilized to convert the MOM pack provided with Commerce Server 2007 for MOM.

Staging
Commerce Server 2000 and Commerce Server 2002 egregiously operated under the assumption that changes were made live to production sites. Although this may have worked sufficiently for small organizations, it caused significant operational pain for large organizations.

Most large organizations desire to make changes in one or more content development environments, copy that to a single preview environment – and then once validated as good – flip it over to production. Commerce Server 2002 Feature Pack 1 was the first release of Commerce Server to address this scenario. It did so by providing drivers to leverage Microsoft Application Center 2000 to stage Catalog and Marketing plus some Profile data.

Unfortunately, the Application Center product was not continued by Microsoft, leaving no direct successor. Enter Commerce Server Staging – a system that provides the ability to replicate between environments:
•   Content and Code
•   Catalog and Inventory Data
•   Marketing Campaign data
•   Profile Site Term Data

The Staging Service runs as its own Windows service on a machine running Commerce Server. It runs over it’s own TCP/IP protocol (leveraging the old CRS protocol from Site Server days – and the resultant IP port).

Utilizing its own MMC snap-in, a bill-of-materials can be defined, deployment targets identified, replication schedule established, and data transfer mode defined (for either full or incremental since last update).
Once defined in the MMC, the Staging service will then deploy the identified assets to the requisite deployment targets as an ACID transaction. It will do so ensuring that the operation is a single unit of work – and that if a failure occurs, a rollback will also occur to leave configuration in a consistent state.

The system is quite flexible – as multiple sourcs can be consolidated onto a single destination – or vice versa. If the source and destination cannot see each other directly – multiple relay points can be identified as interim replication steps – with the only requirement being that each relay point be running the Commerce Server Staging service and that each previous relay point can speak over TCP port 507 to the next relay point.

The Staging System provides its own event definitions and the ability to define custom pre- and post- event handlers (for both success and failure). It also provides its own COM (for use from Windows Scripting Host) and .NET API for further extensibility. Most operations can also be scripted through the command line. In short – it can be easily extended to handle most replication deployment scenarios.

The Staging service is featured only in Commerce Server 2007 Enterprise Edition; it is not available in Standard Edition.

CS2007 Software Architecture Series Part 4: Key Subsystem Overview

rdonovan — Sun, 02 Nov 2008 04:47:00 GMT

In this post, I wanted to describe the major subsystems of Commerce Server...

The Catalog System
The product Catalog system provides a managed taxonomy for storing and querying the hierarchical data that is utilized to represent goods or services for sale made available for offer to customers utilizing the system. The system is comprised of:

•   Schema – This is defined in a managed capacity to identify the item and category entities and resultant attributes that go into the catalog. Data can be multi-lingual and stored in any language supported by Windows without duplicating records for other languages. Only a single currency supported per defined attribute though – this can be translated utilizing an exchange rate table or multiple currency attributes stored per entity.
•   Base Catalog – This is an instance of a given schema
•   Virtual Catalog – This is an abstraction of one or more Base Catalogs. One can – by defining inclusion and exclusion rules – merge catalogs together, partition them apart, or override data (e.g. – keep all attribute data points the same from the base catalog item except perhaps just a price change). The main benefit – behind obvious filtering for presentation – is the ability to have data stored once but prepared for many different consumptions with only the data that needs to be changed for any given consumption scenario being changed – and all other underlying data remaining the same. Some typical real-world scenarios are:

o   Merging catalogs from various suppliers into a single catalog for presentation on a site
o   Overriding individual data attributes but leaving the rest the same for things like special customer discounts (typical in Business-to-Business (B2B) scenarios) and seasonal data changes (e.g. – marketing a coat differently in winter versus summer).
o   Or any combination thereof!

• Catalog Sets – This feature allows Catalogs to be targeted to a specific user at the time they browse the site. For example, User A will see Catalog 1 whereas User B will see Virtual Catalog 2.
Catalog data is stored in its own set of SQL Server tables. These can either live in the same database as other Commerce Server entities, or reside separately. The programming model is all based in .NET and most COM-related infrastructure from prior versions is gone in Commerce Server 2007.

The Catalog system features its own caching capabilities to help expedite querying of data when browsing a site.

The Catalog system follows the typical Commerce Server system pattern and implements a site development API, a Web Service API, an Agent API, and a BizTalk adapter.

The Catalog system does not depend on any other Commerce Server systems, with the exception of Catalog Sets – which relies upon the Profile system for targeting against customer records.

The Inventory System
The Inventory system is a close sibling of the Catalog system. It is meant to store quantity on hand and define stock-out business rules for items stored in the Catalog system. It provides its own .NET programming model and its data is also stored in SQL Server. It can live either in a standalone database or be shared in a database with other Commerce Server entities.

The site development API, Web Service API, Agent API, and BizTalk adapter support is shared with that of the Catalog system, given the close affinity between the two components.

The Inventory system depends upon the Catalog system.

The Order System
As its name descriptively implies, the Order system is all about shopping baskets and purchase orders generated at checkout. Developers can define basket and order schema as a mix of strongly and weakly typed properties. From a Basket perspective, data will be persisted in binary format. This makes searching and doing direct updates to baskets a pretty onerous task; fortunately it is not one that is typically performed every day. With respect to Orders, data storage is a mix of binary storage or what is called XML Mapped Storage – which will map data fields to underlying columns in SQL Server that can be queried directly after an order has been persisted to database.

The Order system also provides what is known as the Pipeline, which is responsible for providing a serial, linear, definable workflow for calculating basket and order totals. Pipelines are COM-based. They date back all of the way to Merchant Server 1.0 (circa 1995-1996), Site Server 2.0 (circa 1997), and Site Server 3.0 Commerce Edition (circa 1998).

What Commerce Server 2007 adds in terms of Pipeline support is x64 component support (as all previous versions of Commerce Server were 32-bit) plus the ability to write components in .NET managed code. Given the seemingly dated technology, begs the question – why on earth are Pipelines still around? The answer is simple – a lot of people use them and there are a lot of 3rd party ISV components out there. That being said, the future will likely not be bright for pipelines, with a managed multi-dimensional workflow solution being desired (such as Windows Workflow Foundation) – though some sort of backward compatibility story is still likely.

The Profile System
The Profile system is meant to store data about customers and organizations and enforcing relationships between all of the associated metadata that typically goes with such record types. It does so in a somewhat interesting manner. Rather than providing a managed schema or a black box that cannot be touched by the end developer, it is a case of bring-your-own-schema (though one is provided as a default starting point). The core assumption behind this is that profile data will live elsewhere in pre-existing database – be it SQL Server, Active Directory, or a generic OLE/DB- or ODBC-compliant data source such as Oracle or DB/2.

Conceptually, it works like this:

1.   The core attributes of customers and organizations are defined
2.   These are then mapped to underlying data sources – regardless of their technical origin
3.   A uniform view and programming model (supporting CRUD) then provides a single view of the customer or organization, regardless of where the underlying data is stored

For example, one could have user with:

•   Login information stored in Active Directory
•   Generic profile information (such as name, address, etc.) stored in SQL Server
•   Private personal statistics (such as height, weight, age, eye color, etc.) stored in Oracle

Using the Profile system this could be a single record and rowset as far as the Commerce Server developer is concerned – and it will automatically handle the data update operations to all of the various disparate data sources.

The Profile system does its work through its own custom OLE/DB provider entitled CSOLEDB. All of this is abstracted through the Profile system’s programming model. And speaking of that – the whole enchilada is delivered: site API, Agent, Web Service, and BizTalk adapter.

The Profile system has no dependences on other Commerce Server components.

The Marketing System
The Marketing system provides a personalized shopping experience for users of sites constructed with Commerce Server. It is comprised of three principal components:

•   Advertisements – The name is misleading; it really should be content blob. Here, text, image, or multimedia elements can be defined, placed on a page, and then targeted towards specific users. The system features click and impression tracking (to ensure that enough people see the content that they are supposed to see, etc.).
•   Discounts – Like its name implies, personalized targeted discounts can be created at the item level (e.g. – buy item X, get Y off item X), order level (e.g. – buy $100, get 10% off), or free/discounted shipping.
•   Direct Mail – Create personalized e-mails that can be targeted towards a specific audience (e.g. – Dear John, we know you like X and it is on sale).
Internally the marketing system utilizes a series of pre-built Pipeline components called the Content Selection Framework (for associating content with targeted users) and additional discount components to operate. In some cases, these components can be augmented with user generated code to extend and alter behavior.

Data storage is in SQL Server. The system features a Web Service and Agent API – in addition to the site API, but does not have a BizTalk Adapter. The reasoning behind no BizTalk Adapter is that these elements are represented too differently from one business to another to provide an effectively uniform means of importing and exporting data.

The Marketing system has hard dependencies on the Catalog system (for item discounts) and the Profile system and the Order system (for all discounts).

Analytics
The Analytics system collects IIS Web Log data and Commerce Server data from all of the aforementioned systems and aggregates it into a single data mart (with pre-built Extract/Transform/Load (ETL) tasks capable of working in both initial construction and ongoing incremental data population modes). Exposed on top of this report is 35 canned reports, with the provisions made to customize these reports or create new ones.

The ETL processes leverage the same OLE/DB provider utilized by profiles coupled with SQL Server 2000 Data Transformation Services (DTS) tasks to populate the data mart, which itself is based on SQL Server Analysis Services. DTS is used for compatibility with both SQL Sever 2000 and SQL Server 2005 and so as not to require two separate ETL codebases. SQL Server Reporting Services is utilized for the reporting interface on top of the data mart – which provides secure, scheduled or on-demand report generation in a variety of formats such as HTML, Plain Text, Excel, Adobe Acrobat, and e-mail. Standard SQL Server Reporting Services extensibility mechanisms can be used for customizations or new report creation.

CS2007 Software Architecture Series Part 3: What's Under the Hood

rdonovan — Fri, 31 Oct 2008 06:25:00 GMT

As noted in previous posts, Commerce Server 2007 extends the .NET Framework 2.0 and above to provide core e-commerce entities. These entities are:

•   Catalog
•   Inventory
•   Orders
•   Profile
•   Marketing
o   Advertisements
o   Discounts
o   Direct Mail Campaigns
•   Analytics

In addition, there is an administration framework to provision all of these entities that is shared horizontally across the breadth of Commerce Server. Likewise, the Analytics module – utilized for business reporting – also leverages all of the aforementioned entities.

It is important to step back and think about the system at a forest-level. Doing so is a key factor in thinking about how to successfully leverage each component and properly navigate the interdependencies between them when architecting a software solution based upon Commerce Server.

Data is stored in SQL Server. The schemas are managed and accessed through the programming model – and are generally not meant to be touched directly by the end-developer. In fact, modifying the out-of-box schema is a good way to end up in an unsupported state and have the system broken when hot fixes or service packs are issued by Microsoft.

The programming model itself provides three programming models and an adapter framework:

•   The Runtime API is meant to extend ASP.NET and provide the developer with an experience that is conducive towards building an end Web site. Plain, simple – and nothing more than that. If that’s the case WAIT – why does the programming model seem so bizarre compared to ASP.NET then? The answer lies in the past.
o   Commerce Server 2000 was the first release of the product. It at the time extended Active Server Pages (ASP) rather than ASP.NET – as .NET Framework 1.0 had yet to ship. Hence, the programming model was exclusively COM-based.
o   Commerce Server 2002 supported both ASP.NET and ASP/COM development (as backward compatibility with Commerce Server 2000 was mandatory). The ASP.NET support was implemented by implanting a two-level programming model:
•   Primary Interoperability Assemblies (PIAs) – These are simple wrappers of the COM APIs dating back to Commerce Server 2000 days. They were never meant to be consumed by end developers, though this did occur in practicality in many cases. Leveraging the PIAs in any version of Commerce Server is a bad move – as is setting the stage for broken forward-compatibility. This is because it was always intended that the product would be ported to .NET and COM support retired – therefore the PIAs were meant as a temporary stopgap measure at most.
•   Base Class Libraries (BCLs) – These are abstractions of the PIAs to provide a more .NET-friendly programming model. It is intended that the BCLs are the primary API model consumed by site developers building Commerce Server applications. Over time as the product evolved – this would be the programming model going forward and the PIAs would be retired in lieu of native .NET implementations under the BCLs. Because the BCLs evolved from the PIAs, they do appear less .NET-like than something built expressly in managed code from the get-go. Over time, it is reasonable to expect these to be gradually re-factored to drive closer look-and-feel to the core platform (also as the remaining COM underpinnings are replaced).
o   In Commerce Sever 2007, both the BCLs and PIAs are present. Some of the PIAs that were present in Commerce Server 2002 are gone and replaced with native .NET implementations. In other cases they remain – as it was less costly (in terms of both internal development effort for Microsoft and in providing backward-compatibility to the customer) to leave some COM underpinnings in place. There are some latent COM APIs, but these are NEVER meant to be called, nor are the PIAs.
•   The Web Service is meant to provide a means of performing data management CRUD (Create, Retrieve, Update, and Delete) in a secure, distributed means. It is implemented as an ASP.NET Web Service (ASMX) – as that was the newest Web services implementation available from Microsoft at the time of the product’s ship. Given the subsequent release of Windows Communication Foundation (WCF) after Commerce Server 2007 shipped, it is likely that Commerce Server 2007 will be the only major release of the product to leverage ASMX-based Web services.
•   The Agent API implements the Agent design pattern to abstract the Commerce Server 2007 Web Services. It provides a lightweight runtime that can run upon the end client and run either natively or in distributed mode. It provides a considerably more natural and friendly CRUD programming model than calling the Web Services directly. It also provides for other means such as caching, etc. to improve performance.
•   The BizTalk Adapter exposes the Web Service through BizTalk, allowing data to be interchanged utilizing the toolset and user interfaces within BizTalk Server with ANY system to which BizTalk can communicate WITHOUT writing any code. As of this writing, the list is extensive in terms of what is supported out of the box, with adapters available from Microsoft - see http://www.microsoft.com/biztalk/en/us/adapters.aspx.

One comment that is often raised is that Commerce Server 2007 will perform badly because it still has COM under the hood. Or the fact that COM is even present is brought up as a detractor. In terms of being a real issue, this is more myth than reality. First, Commerce Server is a .NET application; its only supported programming models are .NET-based. Second, COM underpinnings are not a detractor to performance at all. In fact – many parts of the Windows operating system and things that underlie parts of the .NET framework are also COM-based and follow a similar architecture.

CS2007 Software Architecture Series Part 2: Platform Basics

rdonovan — Tue, 28 Oct 2008 14:03:00 GMT

Most of the meat of Commerce Server is to be found within its programming model and underlying data storage mechanisms. Commerce Server started out with the 2000 release by providing extensions to Active Server Pages. Today, with the 2007 release Commerce Server provides a platform that extends the Microsoft .NET Framework 2.0 and above – utilizing Microsoft Visual Studio 2005 and above as the development toolset. In particular, Commerce Server 2007 is geared towards extending ASP.NET, as that represents the primary means of building Web applications on top of the .NET Framework (though certainly many other creative or legacy means are available).

Commerce Server 2007 supports both x86 32-bit as well as x64 64-bit processor architectures natively, running on top of Windows Server 2003 Service Pack 1 and above. The product does not, however, support the 64-bit IA64 processor architecture (though it can leverage SQL Server running on IA64 systems). For the most part, Commerce Sever 2007 relies upon Microsoft SQL Server for data storage. Commerce Server 2007 supports both SQL Server 2000 and SQL Server 2005. The product, because it still supports SQL Server 2000, does not leverage any specific SQL Server 2005 capabilities however. The rationale behind this was simple – most pre-existing Commerce Server customers were running on SQL Server 2000 at the time of the product’s release; the thinking was that upgrade to a new database platform (given the newness of SQL Server 2005 at the time) would be a barrier to technology adoption. BizTalk Server 2006 is supported from an application integration perspective as well. Microsoft Operations Manager 2005 is supported from a systems health monitoring perspective. Beyond those components, Commerce Server 2007 does not have any particular technical dependencies warranting consideration.

All of that being said, a Commerce Server developer should be proficient in all of these technologies to be successful: .NET Framework 2.0+ (in particular ASP.NET), SQL Server 2000+, Windows Server 2003+, and potentially Operations Manager 2005+ and BizTalk Server 2006+. This, of course, makes finding qualified Commerce Server architects and developers an at times rare breed of individual.

Going forward, Microsoft is committed to enhancing core platform support of Commerce Server 2007 throughout the product’s 5-year standard support lifecycle, which commenced upon August 1, 2006. This makes the product relatively future-proofed. So, as new versions of Windows Server, BizTalk Server, SQL Server, and Operations Manager are shipped in the 5-year period from August 1, 2006 through August 1, 2011 – it is reasonable to expect that Commerce Server will support those versions barrring any egregious breaking changes in the underlying platform - at least until replaced with subsequent versions of Commerce Server itself. Conversely, as products in the dependency stack have their standard support lifecycles end – such as SQL Server 2000 in the coming years – it may come about that platform support is removed in future service packs or other interim releases.

SP2 adds support for Windows Server 2008, Visual Studio 2008, and .NET Framework 3.5. SQL Server 2008 support on SP2 is imminent; details forthcoming.

When planning deployments and which versions of technologies to base a system upon, the bleeding edge should indeed be chosen to maximize longevity of the solution once deployed.

CS2007 Software Architecture Series Part 1: Philosophy

rdonovan — Tue, 02 Sep 2008 04:54:00 GMT

September is here. And with it, a great opportunity to resume professionally blogging. Before getting on to new stuff, I have had a bunch of older material standing around waiting for polishing and publishing - so here goes with the Architecture Series...

Software architecture is both an art form and a science – especially when designing applications for use in mission critical environments such as the typical online storefront. In e-commerce, mission critical is certainly an accurate characterization. The cost of downtime for a large retailer can be enormous – often times averaging $250,000 per hour or more. During a peak holiday season shopping spree the cost of downtime can easily extend into the millions or tens of millions of dollars per hour. Then, there is the cost of reputation damage. Bad site outages often make front page news at some level – and can in the worst case end up driving customers away to the competition.

Then, there is the cost of development and deployment. Building a mission critical e-commerce application can be an extremely costly proposition from a time and resource perspective. Deployment timeframes can range from several months to potentially several years – usually requiring the dedicated attention of a team of people. Given this, the desire is usually there to get several years of usage and utilization out of the system once it has been deployed without the need for costly retrofits of some sort.

All of this brings together the need to get the application architecture of an e-commerce site right the first time. And that task requires a fair degree of consideration over and above the average Web application. To architect a Commerce Server site effectively, one must understand not only Commerce Server and its technical considerations – but those of all of the underlying platform components as well as the requirements of the end business customer. It makes on the surface a somewhat daunting proposition; this chapter will attempt to demystify the process and provide a concise guide of the considerations needed to successfully deploy a Commerce Server solution – regardless if one is running a small online storefront or a global retailer. In addition, some insight will be provided into the how and why the product is architected in the manner it is – in an attempt to answer the annoying questions along the lines of why did Microsoft do it this particular way?…

To start with architecting a Commerce Server application, it is helpful to understand conceptually just what Commerce Server is and is not. First and foremost – it is not a server in the traditional sense. It is a combination of:

• Developer Platform – An object and data model representing typical e-commerce entities such as catalogs, inventory, orders, etc.
• Applications – Tools utilized by business users and IT professionals to deploy applications built by the user on top of the aforementioned Developer Platform.

In future posts, I shall start going through the different pieces in detail and start explaining how to think about the various pieces to develop a complete software architecture on Commerce Server 2007.

Hope this helps!