Vinayak's WebLog : New Catalog Features in CS2007

Securing the catalog system

vinayakt — Wed, 28 Jun 2006 07:20:00 GMT

The authorization model is another new feature in the Commerce Server 2007 catalog system which allows you to protect your catalog data from being accessed by unauthorized users. Since we decided to expose the functionality of the catalog system over the web service it became imperative to ensure that only authorized users can access the catalog data. The catalog system in Commerce Server 2007 has two main components: the catalog web service and the catalog database. Both these components can be appropriately secured.

Securing the Catalog web service

First let's talk about securing the catalog web service. Securing the catalog web service involves securing the web methods to ensure that unauthorized users cannot successfully invoke these methods. Instead of reinventing the wheel and introduce another authorization model for users to learn we decided to use the Windows Authorization Manager. There are several articles on Authorization manager on msdn. You can also check out the Authorization Manager Team Blog for overviews, case studies etc.

The catalog system provides various levels of authorization which can be defined by assigning the users to the different roles. Users can be Administrators, catalog editors, language translators etc. You can also create your own user model and still enforce the catalog authorization in the web service. The authorization model also allows you to control which users can view which catalogs and edit which properties. For eg you can disable certain users from editing specific product properties like list price, description etc.

The catalog system supports only the XML store which means that the authorization information is defined in an xml file. The name of the xml file is specified in the catalogWebService element in the web.config via the authorizationPolicyPath attribute. When you unpup the catalog webservice a default xml file named CatalogAuthorizationStore.xml is installed in the web service directory (typically <Drive>:\Inetpub\wwwroot\CatalogWebService). You can use the authorizationPolicyPath attribute to specify a different authorization store. If the specified path is not an absolute path then the path is relative to the catalog web service's directory.

After you install the catalog web service you need to perform the following steps to get the catalog webservice up and running.

1. Application pool: Verify that the catalog web service has been assigned to the desired Application pool. Verify the properties of the application pool especially the Identity. The identity of the application pool is the name of the account under which an application pool's worker process runs.

2. Once the identity of the catalog web service's application pool has ben configured ensure that this account has write access on the catalog authorization store. You can do this by right clicking the CatalogAuthorizationStore.xml, click properties, select the Security tab, add the catalog web service’s account and check the Allow Write checkbox. The catalog system offers catalog level security which means that you can control which users can access which catalogs. This has been implemented by means of scopes in the authorization store. Since the catalogs are dynamically created in the catalog system it becomes necessary to create a new scope in the authorization store every time a catalog is created. Creating the catalog scope needs write permissions on the xml file. This is why you need to ensure that the account running the catalog web service has Write permissions on the catalog authorization store. If this is not done then the catalog web service will throw a configuration exception at startup and none of the methods can be invoked.

3. The final step is to add the user accounts to the appropriate roles. This can be done by the Authorization Manager MMC. To do this

Start->Run->Azman.msc
Right click Authorization Manager
Click Open Authorization Store
Select the CatalogAuthorizationStore.xml
Expand CatalogAuthorizationStore->CatalogAndInventorySystem->Role Assignments
To add a user to a role, right click the role and select "Assign Windows Users or groups" and add the users to the appropriate roles.

Note that adding the user the Administrator role will allow the user to perform any operation in the catalog system.

4. As I mentioned previously we dynamically create scopes for each catalog, language and property in the catalog system to provide a finely grained authorization mode. This means that every time you create a new catalog you need to follow the above steps to ensure that appropriate users have access to this catalog. The only difference is that you need to expand the node named "CatalogScope_<Catalog Name>"->Role Assignments and add the users to the CatalogEditor role.

5. If you are unable to access the catalog web service due to the following exception: System.IO.FileNotFoundException: File or assembly name 0x1gew0n.dll, or one of its dependencies, was not found. File name: "0x1gew0n.dll" at System.Reflection.Assembly.nLoad(AssemblyName fileName, String codeBase, Boolean isStringized, Evidence assemblySecurity, Boolean throwOnFileNotFound, Assembly locationHint, StackCrawlMark& stackMark) --- LOG: Attempting download of new URL file:///C:/WINDOWS/TEMP/6hg1.dll. The resolution for this issue is to grant write permissions on the temp directory mentioned in the above URL (C:/WINDOWS/TEMP) to the catalog web service account.

The next post will be about the various operations, tasks and roles and how you can create your own tasks and roles to enforce an authorization model which is driven by your business needs. It will also discuss how to define permissions for the various user scenarios.

Improving the catalog search experience in Commerce Server 2007

vinayakt — Sun, 18 Jun 2006 08:53:00 GMT

Two years back in this post on fulltext search in the catalog system, I had explained how fulltext searches are performed in the catalog system and added the following comment

"Providing an increased number of search features/conditions is something we are looking forward to, in the upcoming releases. This should allow you to specify more complex search phrases."

Commerce Server 2007 now allows you to specify complex search phrases which allow you to customize your searches to the finest level of granularity. If the default freetext search behavior as described in the above link does not work for you, you can now use the new AdvancedFreeTextSearchPhrase property on the CatalogSearch object. An example of using this property to implement Thesaurus support is provided in this link. Note that when you set this property you should also set the UseAdvancedFreeTextSearch to true. This is to avoid ambiguity on whether the value specified for AdvancedFreeTextSearchPhrase or FreeTextSearchPhrase property should be used to perform the search.

Here are some additional search scenarios for which you can use this new property.

You can perform wild card searches by appending * to the search phrase. For eg to return all the products containing search terms beginning with "Commerce". When passing a compound phrase you need to enclose the phrase in quotes.

catalogSearch.AdvancedFreeTextSearchPhrase = "Commerce*"; catalogSearch.AdvancedFreeTextSearchPhrase = "\"Commerce Server*\"";

To perform exact searches you should enclose the phrase in quotes. For eg to return rows that contain the exact phrase "Commerce Server 2007"

catalogSearch.AdvancedFreeTextSearchPhrase = "\"Commerce Server 2007\"";

To return all the products containing the phrases "Commerce Server" and "Sql Server"

catalogSearch.AdvancedFreeTextSearchPhrase=" \"Commerce Server 2007\" AND \"Sql Server\" ";

To return all the products containing the phrases "Commerce Server" OR "Sql Server"

catalogSearch.AdvancedFreeTextSearchPhrase=" \"Commerce Server 2007\" OR \"Sql Server\" ";

To return all products beginning with Commerce Or Sql server

catalogSearch.AdvancedFreeTextSearchPhrase=" Commerce* OR \"Sql Server\" ";

You can also perform proximity searches using the NEAR keyword. For eg to return all products containing the word 2007 near "Commerce Server"

catalogSearch.AdvancedFreeTextSearchPhrase="2007 NEAR \"Commerce Server\"";

You can perform inflectional searches using the INFLECTIONAL keyword. For eg to return all products with forms of dry like drying, dried etc

catalogSearch.AdvancedFreeTextSearchPhrase="FORMSOF (Inflectional,dry)";

To return all products containing Commerce or Sql Server and not ASP

catalogSearch.AdvancedFreeTextSearchPhrase="(\"commerce \" OR \"sql server\") and not ASP\"";

The above are just some of the scenarios. To truly understand what else you can do take a look at this link. Determine the clause which best reflects your search criteria and pass it to the AdvancedFreeTextSearchPhrase as in the above example.

For a complete code sample and additional filtering using other criteria like "list price < $100 AND IsActive=1" see this post.

Implementing Thesaurus support in the catalog system

vinayakt — Mon, 12 Jun 2006 10:44:00 GMT

Sql Server 2005 now provides Thesaurus support through configurable xml files. In a nutshell the Thesaurus feature allows you to search for synonyms of the search phrase or replace the search phrase with a replacement phrase and perform the search using the replacement phrase. These synonyms and replacements can be configured using the xml files.See this link for configuring the xml files.

If you are using Sql Server 2005 as your Commerce Server database server then you can take advantage of this cool feature when performing catalog searches.

As an example, suppose that you have a Books catalog. Assume that Description is a fulltext searchable property in this catalog and that this catalog has various C# books. When performing a search on your site users might search for C# or "C Sharp". You can use the Thesaurus feature to allow searches for "C Sharp" to return C# books.

The first step is to edit the tsENU.xml file located in the SQL_Server_install_path\Microsoft SQL Server\MSSQL.1\MSSQL\FTDATA\ directory and add the following element to it

         <expansion>
                     <sub>C sharp</sub>
                     <sub>C#</sub>
          </expansion>

After you edit the xml file you will have to restart the "SQL Server FullText Search (MSSQLSERVER)" service for the changes to take effect.

The following code sample demonstrates how to perform the catalog search to allow searches for "C Sharp" to return books containing C#:

CatalogSearch catalogSearch = catalogContext.GetCatalogSearch();

catalogSearch.SearchOptions = new CatalogSearchOptions();

// Set appropriate search options

catalogSearch.SearchOptions.SetPaging(1,20);

catalogSearch.UseAdvancedFreeTextSearch = true;

catalogSearch.AdvancedFreeTextSearchPhrase = "FORMSOF(THESAURUS, \"C Sharp\")";

int totalRecords;

CatalogItemsDataSet searchResults = catalogSearch.Search(out totalRecords);

// Iterate through the search results

foreach(CatalogItemsDataSet.CatalogItem searchResult in searchResults.CatalogItems)

Console.WriteLine(searchResult.CategoryName);

New Catalogset features in Commerce Server 2007

vinayakt — Wed, 07 Jun 2006 09:39:00 GMT

The catalogsets feature is probably the least known feature in the catalog system. Unlike in previous versions, the catalogsets feature is now tightly integrated with the catalog sytsem. We have also provided a managed API to create, edit and delete catalog sets.This functionality is also available through the catalog web service. In addition you can now export and import catalog sets just as you can import and export product catalogs. In addition we have also provided appropriate Azman security for catalogsets. More importantly the ability to manipulate the catalog sets has been added in the Catalog Manager application.

What are catalogsets anyway?

Catalogsets provide a way to group together a similar set of product catalogs.You can use catalog sets to decide the appropriate list of catalogs to be displayed to your site users based on their profile. For eg you can create a catalogset containing a list of preferred product catalogs. The catalog set can then be associated to profile of runtime users and when a user browses your site you can then select the list of catalogs the user can browse.This allows you to show a different list of catalogs for anonymous users, registered users or privileged users. See this post for more information and the various fallbacks.

A new feature in Commerce Server 2007 is the ability to create dynamic catalog sets. A catalog set can now be

Static, which contains a static list of catalogs.
Dynamic, which allows you to associate an expression with the catalog set. The catalog set will now contain the list of catalogs which match the expression. A good usage of this feature would be to define a catalog set for active product catalogs. For eg "Active catalogset" can be a dynamic catalogset which has the expression "StartDate> 01/01/2006" and "EndDate<12/31/2006" associated with it. This catalog set will now contain all the catalogs which have a start and end dates that fall in the specified range.

By default the catalog system creates two default catalog sets

Anonymous User Default CatalogSet, is a catalog set for anonymous users
Registered User Default CatalogSet, is a catalog set for registered users

As I previously mentioned you can associate catalogsets to a users profile and use the users profile and the catalog sets to display the list of catalogs for that user. The newly added dynamic catalogset feature can be used here to provide a better user experience, increased flexibility and less maintainence. Once you define your expression you dont have to worry about the contents of the catalog set.

New in Commerce Server 2007 is the ability to import and export catalog sets from and to xml files. We have also added Azman security to control which users can view and edit the catalogsets.

Caching in the catalog system

vinayakt — Fri, 02 Jun 2006 07:27:00 GMT

The Commerce Server 2007 catalog system adds a new caching feature which caches frequently used datasets.

What is cached?

In order to prevent repeated calls to the sql server the catalog system caches the results of various methods in the catalog system. These results are cached in the form of datasets. The catalog system uses the System.Web.Caching.Cache object to cache the various datasets. Each dataset is inserted in the cache with the CacheItemPriority.Normal. The time for which a dataset remains in the cache is configurable. If not specified the default time is 5 minutes.

However in order to account for the relative frequency at which different items in the catalog system are changed we have classified the datasets in six different categories allowing you to specify different timeouts. The goal is to ensure allow for items which change less frequently to remain in the cache much longer that those that change more frequently.

The cache configuration can be defined when you create the CatalogContext object

CacheConfiguration cacheConfiguration = new CacheConfiguration();

cacheConfiguration.CacheEnabled = true;

cacheConfiguration.SchemaCacheTimeout = new TimeSpan(0, 10, 0);

cacheConfiguration.ItemAssociationsCacheTimeout = new TimeSpan(0, 10, 0);

cacheConfiguration.ItemHierarchyCacheTimeout = new TimeSpan(0, 10, 0);

cacheConfiguration.ItemInformationCacheTimeout = new TimeSpan(0, 10, 0);

cacheConfiguration.ItemRelationshipsCacheTimeout = new TimeSpan(0, 10, 0);

cacheConfiguration.CatalogCollectionCacheTimeout = new TimeSpan(0, 10, 0);

CatalogSiteAgent siteAgent = new CatalogSiteAgent();

siteAgent.SiteName = "SiteName";

CatalogContext cc = CatalogContext.Create(siteAgent, cacheConfiguration);

Caching is enabled by setting cacheConfiguration.CacheEnabled property to true.

The following are the different categories of datasets and their timeout configuration parameters

CatalogSchema : This applies to datasets that return schema information. These datasets will remain in the cache for the timeout defined by the cacheConfiguration.SchemaCacheTimeout property. Since the catalog schema is less likely to change once it has been defined setting the timeout to a high value can result in improved performance.

The cacheConfiguration.SchemaCacheTimeout applies to the datasets returned by the following properties/methods

CatalogContext.GetProperty()

CatalogContext.GetEntityProperties()

CatalogContext.GetProperties()

CatalogContext.GetSearchableProperties()

CatalogContext.GetPropertiesInCatalog()

ProductCatalog.PropertiesInCatalog

CatalogEnumerationProperty.EnumerationValues

CatalogContext.GetDefinitions()

CatalogDefinition.DefinitionProperties

Item Properties : Almost all the objects in the catalog system expose an Information property which is a dataset containing the properties for that object. For eg ProductCatalog.Information returns a dataset containing the properties of the specified product catalog. The duration the dataset returned by the Information property remains in the cache is controlled by the cacheConfiguration.ItemInformationCacheTimeout property.

The cacheConfiguration.ItemInformationCacheTimeout property applies to the datasets returned by the following properties/methods

CatalogSet.Information

Category.Information

InventoryCatalog.Information

InventorySku.Information

Product.Information

ProductCatalog.Information

ProductVariant.DataRow

Note that this timeout also applies to the product information stored by the QueryCatalogInfo pipeline component.

Another thing to note is that if inventory integration is enabled the inventory information also gets cached.

Catalog Hierarchy : The catalog system allows you to define parent child hierarchies between categories and products. This information is returned in the form of datasets. The duration for which these datasets are cached is determined by the cacheConfiguration.ItemHierarchyCacheTimeout property. The cacheConfiguration.ItemHierarchyCacheTimeout property applies to the following methods/properties :

ProductCatalog.GetRootCategories()

ProductCatalog.GetRootProducts()

Product.AncestorCategories

Category.AncestorCategories

Product.ParentCategories

Category.ParentCategories

Product.CanonicalCategories

Category.CanonicalCategories

Category.Products

Category.ChildCategories

Catalog Relationships : The catalog system allows you to define relationships between products and categories. The duration for which the relationships datasets remain in the cache is determined by the cacheConfiguration. ItemRelationshipsCacheTimeout property and applies to the following properties

Category.RelatedCategories

Product.RelatedCategories

Category.RelatedProducts

Product.RelatedProducts

Associations between different objects: The catalog system allows you to have various types of associations between different objects. For eg product catalogs associated to inventory catalogs, product catalogs to catalog sets, languages associated to product catalogs etc The duration for which these associations datasets remain in the cache is determined by the cacheConfiguration. ItemAssociationsCacheTimeout property and applies to the datasets returned by the following properties:

ProductCatalog.Languages

ProductCatalog.DependentCatalogs

VirtualCatalog.SourceCatalogs

CatalogSet.IncludedCatalogs

CatalogSet.NotIncludedCatalogs

InventoryContext.GetUnassociatedProductCatalogs

InventoryCatalog.AssociatedProductCatalogs

VirtualCatalog.VirtualCatalogRules

VirtualCatalog.PriceRules

Catalog Collections : The cacheConfiguration .CatalogCollectionCacheTimeout property controls how long the following datasets remain in the cache

CatalogContext.GetCatalogs()

InventoryContext.GetInventoryCatalogs()

CatalogSetsContext.GetCatalogSets

Monitoring the Catalog cache using perfmon

If you have enabled caching then you can monitor the perfomace of the cache by observing the following performance counters under Commerce : Catalog set of counters.

1. CacheTotalHits : The total number of cache hits

2. CacheTotalMissess: The total number of cache misses

3. CacheHitRate : The number of cache hits per sec

4. CacheMissRate : The number of cache misses per sec

5. CacheHitRatio : The ratio of the cache hits to the total number of cache accesses. This is equal to CacheTotalHits / (CacheTotalHits + CacheTotalMisses)

6. CacheTurnOverRate : The number of cache insert and deletes per second

Web.Config section for caching

The Caching parameters can also be set in web.config of your runtime site using the following element

<cache

enable="true"

schemaTimeout="10"

itemInformationCacheTimeout ="10"

itemHierarchyCacheTimeout ="10"

itemRelationshipsCacheTimeout = "10"

itemAssociationsCacheTimeout="10"

catalogCollectionCacheTimeout="10"

</catalog>

Here 10 is the duration in minutes, the respective datasets should be in the cache.

Since the catalog system uses the System.Web.Caching.Cache you can tune the different aspects of the catalog cache using the ASP.NET cache settings.

New Virtual catalog features in Commerce Server 2007

vinayakt — Wed, 31 May 2006 09:49:00 GMT

Virtual catalogs is a key feature we improved upon in Commerce Server 2007 to allow you to further customize your base and virtual catalogs. These features will enable additional scenarios which were not possible in previous versions. If you are not familiar with virtual catalogs, read this post.

The following new features have been added to virtual catalogs in Commerce Server 2007

1. Ability to add rules from another virtual catalog : Commerce Server 2007 now allows you to add inclusion/exclusion/pricing rules to a virtual catalog from another virtual catalog. Note that a virtual catalog can now be created by rules from

One or more base catalogs OR
Another (and only one) virtual catalog

This new feature now allows you to create a virtual catalog from one or more base catalogs (possibly coming from different vendors) and then you can create another customized virtual catalog based off this virtual catalog.

2. Ability to override values of custom properties from inherited catalogs: New in this release is the ability to override the values of custom properties from the inherited catalogs. For eg you can add a product from a base catalog to a virtual catalog and have a different description or display name for the product in the virtual catalog. In addition you can also choose not to override these values and revert back to the original values from the inherited catalogs.

3. Ability to explicitly specify prices for products in the virtual catalog: While you can use flexible pricing rules to define the prices of products and categories in a virtual catalog, you can also explicitly set the prices of products in a virtual catalog. This feature also allows you to revert back this explicit pricing and have the price rule take effect.

4. Importing and exporting virtual catalogs: Another new feature in this release is the ability to export and import virtual catalogs as virtual catalogs.

The export feature allows you to:

Export a virtual catalog as a virtual catalog. This will export the rules in the virtual catalog and values of the overridden properties.
Export a virtual catalog as a base catalog. This will export the current contents of the virtual catalog as a base catalog.
Export source catalogs. This feature allows you to export the source catalogs of the virtual catalog to the same xml file.

The import feature allows you to :

Import virtual catalogs as virtual catalogs.
Materialize virtual catalogs during import.

5. Sequencing in virtual catalog

6. In addition we have made notable performance improvements for virtual catalog rebuilds and materialization of virtual catalogs.

New catalog sequencing features in Commerce Server 2007

vinayakt — Tue, 30 May 2006 08:20:00 GMT

A while back I had written about sequencing products and categories in the feature pack and mentioned a few features that were not supported. I am happy to say that we have now addressed all the limitations I had mentioned in this post and also added a couple of new features.

The following are the new sequencing features in Commerce Server 2007

1. Sequencing products and categories:

You can now sequence child products and categories in any category.
You can also sequence root categories and root products.
You can sequence variants in a product family
Sequencing of child products/categories, root products/categories and variants is also supported in virtual catalogs. When you add a rule from a base catalog to a virtual catalog the existing sequencing for the included items will be inherited from the base catalog. We have also provided the ability to override this inherited sequencing in the virtual catalog so that you can have these items appear in a different sequence in the virtual catalog.

2. Sequencing relationships:

Just as you can sequence child products and categories you can also sequence product and category relationships. Sequencing of relationships is also supported in virtual catalogs where inherited sequencing of base catalog relationships can be overridden in the virtual catalog.

3. Sequencing enumeration values: We have also added the ability to sequence enumeration values of an enumerated property. This is particularly useful where the alphabetical ordering of the enumeration values does not confirm to the desired display order. For eg. it would make sense to display the values Small, Medium and Large for a Size property in that order rather than the alphabetical order. You can now use the sequencing feature to control the display order of the enumarated values.

The above features can be implemented programmatically or they can also be done using the Business Manager UIs. In addition the sequence values are exported to and imported from catalog xml files.

New catalog search features in Commerce Server 2007

vinayakt — Mon, 22 May 2006 09:12:00 GMT

One of my posts (Catalog classtypes and search functionality in the catalog system) described the search functionality in previous versions of the catalog system. We have now made the following enhancements to the search functionality. Incidentally, all these new features were driven by feedback from our customers.

1. Ability to search more than one category in one or more catalogs: Commerce Server 2002 allowed searching a single category in a single catalog. We have now added the ability to search multiple categories in one or more catalogs. In addition the searches can be both recursive and non-recursive. This feature is particularly useful where you have a set of similar catalogs and you want to search a specific set of categories in all the catalogs or if you want to restrict your search to a set of categories in a single catalog.

2. Improved freetext search query support: Freetext searches in CS2002 were limited to keyword searches or And/Or inflectional searches. This made it difficult to perform complex searches like "return all the products containing the phrase 'black shoes' " or "return all the products where the word 'black' is near 'shoes' ". In addition to providing the default search behavior of CS2002 we now allow you to specify an expression that is a valid for the ContainsTable function thereby exposing the full power of Sql Server fulltext search engine.Anything you can do using Sql Server fulltext search is now possible from the catalog system search. If you want to perform more complex searches you can use this feature to constuct the search phrase that can return more detailed and targetted search results. If you are using Sql Server 2005 then this feature will also allow you to implement the Thesaurus feature when performing catalog searches.

3. Inventory integration: One of the compelling features in Commerce Server 2007 is the addition of a new inventory resource and its tight and flexible integration with the Catalog and Orders systems. This allows you to manage inventory for products in your product catalog and also control the display of products on your site based on their inventory status. For eg you can use the search functionality to prevent out of stock products from being displayed on the site.

Performance improvements:

4. Updating fulltext catalogs: We have now fixed the way fulltext catalogs are updated. Once you set up your fulltext catalogs correctly, incremental updates will now trigger incremental updates on the fulltext catalogs. This will no longer cause your CPU to get busy everytime you update your fulltext catalog.

5. Performance improvement when performing freetext search

6. Additional performance improvement when performing freetext searches on virtual catalogs

Microsoft Commerce Server 2007 Release Candidate is now available

vinayakt — Sat, 20 May 2006 07:14:00 GMT

After several years of hardwork the Release candidate for Microsoft Commerce Server 2007 is now available for public download. The details of the download location and the name change from Microsoft Commerce Server 2006 to Microsoft Commerce Server 2007 can be found in Ryan's post.

The team had been pretty busy for the past several weeks in making this possible and the biggest achievement after we shipped Beta was performance. After several weeks of performance and stress testing, I can definitely say that we now have a product which is not only feature rich and secure but also highly performant. Now that we have delivered on an important milestone I will start blogging regularly on the several cool features in the Commerce Server 2007 catalog system.

The new catalog system in Commerce Server 2007 ... Part III

vinayakt — Tue, 07 Mar 2006 08:28:00 GMT

6. New features in Import

Just as we provided a number of options in Export , we have provided a similar number of options for Import.

1. Importing from client machines: The Commerce Server 2007 catalog system now allows you to import data from client machines. The client side import functionality has been built in the catalog system and you don't need to install additional external assemblies. The import can be done over the catalog web service in a secure and performant manner.

2. Import Modes: There are two modes for import

a. In the Full mode if the catalog already exists in the catalog system then it will be deleted and its contents will be replaced by the contents of the catalog in the xml file. This operation is equivalent to deleting the existing catalog and creating a new one from the xml file.

b. In the Incremental mode if a category, product or variant exists in the xml file and the catalog system then the catalog item will be updated. If the catalog item does not exist in the catalog a new one will be created. All other items will be preserved. Items marked for deletion in the xml file will also be deleted from the catalog system.

3. Operations: In this release we have simplified way xml files can be imported and validated. You can

a. Validate, an xml file in which case the file will be parsed, validated and errors reported through the status reporting mechanism.

b. Import, an xml file in which case the file will be imported and errors reported through the status reporting mechanism.

c. ValidateAndImport, allows you to first validate the xml file and if the file is valid it will be imported. If there are errors in the file then the import will not happen.

4. Importing selected catalogs: You can now specify a selected list of catalogs you want to be imported from the file.

5. Importing selected properties: Just as you can specifiy a list of catalogs to import, you can also specify a set of properties to import. A typical scenario is where you want to import a file but don't want the list price of the products to be imported.

6. Transactioning the import process: You can now transact some of the key elements during import. There are two modes

a. Non transactional: In this mode individual elements in the xml file like catalog, category, product, variant etc will be transacted. However failure in importing a single category will still cause the rest of the catalog iyems to be imported.

b. Transactional: If you want to ensure that all elements in a catalog should be imported successfully or none at all then you can use the transactional mode. In this mode importing an entire catalog element (all category, product and variant elements ) will be transacted. A failure in importing a single category, product and variant element will rollback all imports on that catalog. Note that this option applies to individual catalog elements and not the entire xml file.

Irrespective of the above modes importing the CatalogSchema element is transactional. failure to import a single property, definition will abort the import and rollback any changes.

7. Importing schema changes: We now allow you to import an xml file where the catalog schema in the xml file does not match the catalog schema in the catalog system. In this mode any changes made to the catalog schema in the xml file will be made to the schema in the catalog system. For eg suppose that you have a Definition named book in the catalog system containing properties name and author. If you import an xml file containing a definition named book with properties name and writer then importing the xml file with schema changes will change the book definition in the catalog system be deleting the author property from and adding the writer property to the book definition. When doing this operation values assigned to the author property of the book definition in existing catalogs will be lost.

Note that not all types of schema changes are permitted. For eg a language sensitive attribute of a property cannot be changed. Similary the data types of the properties should be compatible. For eg changing the data type of a property from integer to datetime is not permitted. However changing the datatype from integer to biginteger and changing the datatype from biginteger to integer (provided that there is no loss of existing catalog data) is permitted.

8. Restricting the number of errors and aborting import: You can now control the number of errors that can occur before the import operation is terminated. The default value is 100 which means that if the import process encounters 100 errors the import will be terminated.

9. Updating fulltext indexes: This option allows you to control if full text indexes on the catalog should be updated after the import is done.Since updating the full text indexes is a CPU intensive operation you can delay these updates to occur during downtime.

10. Maintaining existing hierarchy and relationships: When importing a catlaog in an incremental mode, you now have an option to retain existing hierarchy and relationships for the catalog items being imported. For eg consider a product "My story" which has a parent category "Biography" and is related to another product "My autobiography" in the books catalog . Now if you are importing the books catalog from the xml file and the product "My story" has a parent category "Autobiography" and is related to "My autobiography 2" you have an option to specify that importing the product will delete or retain the existing hierarchies and relationships for the product. If you choose the delete option the the imported product will have the parent category "Autobiography" and will be related to "My autobiography 2". If you choose the retain option the the imported product will have the parent categories "Biography" and "Autobiography" and will be related to "My autobiography" and "My autobiography 2".

11. Filtering products to import: When importing in incremental mode, you can also control which products to import from the xml file and which products to update in the catalog system. This is achieved by using the LastModifiedDate option. When you set the LastModifiedDate option then the catalog system will not attempt to import products and categories in the xml file having a lastModifiedDate attribute which is less than the specified LastModifiedDate option. Also the catalog system will not update a product if it has been modified after the specified LastModifiedDate option. This is to ensure that changes made to the products in the catalog system are not overwritten inadvertently by the import process.Note that when using this option all product and category elements in the xml file should have the lastModifiedDate attribute. Absence of this attribute will be treated as an error.

12. Importing the last modified date values from the xml file: When importing an xml file the import process will set the lastmodifieddate property of the products and categories to the value of the corresponding lastmodifieddate attribute in the xml file. If the lastmodifieddate attribute is missing then the current datetime will be used.

13. Status reporting: This is a new addition in CS 2007. All long running operations like import, export and virtual catalog rebuilds now report current status of the operation in terms of the percentage of the operation completed.

14. Cancelling an import operation: You can now cancel an import operation.

15. Marking a catalog item as ready for export : In LOB adapter and staging scenarios where you are using import/export to synchronize changes in your catalog systems you need to control if importing a catalog item should signal the product to be ready for export to another system. You can use the ExportReady option to signal if the exportready flag of a catalog item should be set.

As you can see there are a number of options which allow you to control how your information is imported and updated.

While we added a number of cool features in CS2007 we also dropped a couple of features from CS2002.

1. If you remember CS2002 allowed importing xml files in the elementcentric format. We have now dropped importing catalog data in the elementric centric format. We will however provide a script which will allow you to import these types of xml files. You should however note that the element centric format is now deprecated.

2. We no longer support importing catalog data from CSV files. This might be provided as an SDK tool in the future.

The new catalog system in Commerce Server 2007 ... Part II

vinayakt — Mon, 06 Mar 2006 10:08:00 GMT

5. New features in Export

Import/Export is one of the features we took a new look at in Commerce Server 2007, studied customer feedback and came up with a number of configurable options which should address most of typical scenarios like staging, LOB adapters, trading partners etc. These are the things you can do with export

1.Exporting to client machines: The 2007 catalog system now allows you to export data to client machines. In addition to catalog data you can now export catalogset and inventory information. The client side export functionality has been built in the catalog system and you don't need to install additional external assemblies.

2. Modes: The catalog system now allows you to export all your catalog data (Full mode) or you can specify which data should be exported by specifying an expression(Expression based mode). The expression based export can be further fine grained based on other options discussed later.

3. Exporting empty values: By default empty (null or empty) values for a property are not exported. You can use this option to specify that properties whose value is null or empty should also be exported.

4. Exporting deleted items: The catalog system keeps track of products and categories deleted from individual catalogs. You can use this option to export these deleted items to your xml file. This option is particularly useful if you want to use import/export for staging data from staging to production servers and delete catalog items deleted from your staging server on your production server.

5. Specify catalogs to export: You can now specify a list of catalogs to be exported. If not specified all base and virtual catalogs will be exported.

6.Specify properties to export: Similar to specifying catalogs to export you can also specify which of the category and product properties you want to be exported to the xml file. This option is particulary useful if you want to send xml files to your vendors but dont want competitive data like list price, quantity etc to be exported to the xml file.

7. Controlling data to be exported:As I noted in point 2, you can now use expressions to control which data should be exported. If you look at the CatalogExportOptions class, you will see that there are three properties on this class which allow you to control this behavior. The ProductsAndCategoriesClause allows you to specify an expression which will be applied on each catalog that being exported. For eg this clause can be something like "IsActive=1 and Lastmodified > '2006/03/03' " which means export all the catalogs items which are active and which have been modified after '2006/03/03'.

The FilterDescendantsClause allows you to specify an additional clause which will be applied to the results of the ProductsAndCategoriesClause. Extending the above example and specifying "cy_list_price> 99" for the FilterDescendantsClause means export all the catalog items which are active and have been modified after '2006/03/03' and which have a list price greater than 99 dollars.

The FilterDescendantsClause allows you to specify an additional clause which will be applied to the results of the ProductsAndCategoriesClause. There is a third option which allows you to control the set of records on which the FilterDescendantsClause will be applied. This is the Descendants option and it has three values

a. None : means that the FilterDescendantsClause will be applied on the records that are obtained by applying the ProductsAndCategoriesClause.

b. ImmediateChildren : means that the FilterDescendantsClause will be applied on the records that are obtained by applying the ProductsAndCategoriesClause and the immediate ( first level ) children of these records.

c. All : means that the FilterDescendantsClause will be applied on the records that are obtained by applying the ProductsAndCategoriesClause and all the (descendant) children of these records.

The algorithm for this is something like:

1. Apply the ProductsAndCategoriesClause on the catalog.

2. If Descendants = ImmediateChildren then get all the immediate children of the records obtained in step 1 else if Descendants = All then get all the descendants of the records obtained in step 1.

3. Apply the FilterDescendantsClause on the records obtained in step 2. These will the items that will be exported

Let's consider an example to make this less confusing. Suppose that you have the following hierarchy in a base catalog:

Category	ProductId	VariantId	IsActive	cy_list_price	Parent
Category1			1	10
Category2			1	99	Category1
	ProductFamily1		1	55	Category2
	ProductFamily1	Variant1	0	57	ProductFamily1
	Product1		0	89
	Product2		1	85	Catgeory1
	Product3		0	45	Category2

1. ProductsAndCategoriesClause = "cy_list_price <59" , FilterDescendantsClause = "IsActive=1", Descendants=None. The ProductsAndCategoriesClause will select Category1, Productfamily1 and Product3. Since descendants is none the FilterDescendantsClause will be appied to the above three items and since Product3 is not active only category1 and Productfamily1 will be exported.

2.ProductsAndCategoriesClause = "cy_list_price <59" , FilterDescendantsClause = "IsActive=1", Descendants=ImmediateChildren. The ProductsAndCategoriesClause will select Category1, Productfamily1 and Product3. Since descendants is ImmediateChildren the immediate children of Category1, Productfamily1 and Product3 viz. Category2 and Variant1 will also be selected as records to apply the FilterDescendantsClause on. This will result in Category1, Productfamily1 and Category2 to be exported.

3. ProductsAndCategoriesClause = "CategoryName='Category1'" , FilterDescendantsClause = "IsActive=1", Descendants=All. This means export all the descendants of Category1 which are active.

As you can see a combination of these three options provide for any set of records to be exported from the catalog.

8. XSD support: In addition to exporting catalog data the catalog system now allows you to export an xsd representation of the catalog xml schema. You can use this option to export an xsd. This will help you generate your own catalog xml files that confirm to the catalog xml schema. This will help you transfer data from external tables to the commerce server catalog system.

9. Controlling exporting of the catalog schema: The catalog xml file contains a CatalogSchema element which contains the properties and definitions used in the catalog system.By default the entire schema will get exported to the xml file. However you can now control how the schema is exported to the xml file using the SchemaExportType option

a. All, is the default and will export the entire schema to the xml file

b. None, will not export the CatalogSchema element to the xml file.

c. Relevant, will export only the schema contained in the catalogs being exported to be exported to the xml file. For eg if there are two product definitions Books and CDs in your system and you only want to export the Books catalogs then using the Relevant option will export only the Books definition in the xml file.

10. Exporting virtual catalogs as virtual catalogs: Unlike CS 2002, you can now export a virtaul catalog as a virtual catalog. In this mode the XML file will contain a representation of the inclusion, exclusion and price rules in the virtual catalog. In addition you also have an option to export a virtual catalog as a base catalog wherein the contents of the virtual catalog will be exported in a base catalog format. This effectively converts your virtual catalog to a base catalog.

11. Exporting dependent base catalogs: Complementing option 10 is snother option for virtual catalogs which allows you to specify that when you export a virtual catalog, export the virtual catalog and also the base catalogs included in the virtual catalog.

12. Exporting catalogsets: You now have an option to export catalogsets to your xml file.

14. Cancelling an export operation : New in CS 2007 is the ability to cancel an export operation.

As you can see there are a number of options which allow you to control how your information is exported. This opens a number of scenarios like staging data from staging to production servers, sending xml files to trading partners, allowing customers to download your catalog information using the web service in a secure and performant manner.

As you will notice the new Staging and LOB adapters that we introduced in CS2007 heavily rely on the import/export features of the catalog system.

While we added a number of cool features in CS2007 we also dropped a couple of features from CS2002.

1. If you remember CS2002 allowed exporting catalog data in elementcentric and attributecentric formats. We have now dropped exporting catalog data in the elementcentric format. The reason behind this was that the elementcentric format was used mostly in CS2000 and does not work well with schema validators.

2. We no longer support exporting catalog data to CSV files.

The new catalog system in Commerce Server 2007 ... Part I

vinayakt — Sun, 05 Mar 2006 10:32:00 GMT

Commerce Server 2007 offers the following major enhancements to the catalog system since Commerce Server 2002.

1. Programming model :

The catalog system is one of the components which has been completely rewritten in managed code. The COM objects will no longer be supported. The entire catalog functionality will now be available as web service methods. Since it is not easy to program against the web service methods directly, we have provided a managed assembly which will contain a developer friendly object model to program against. More importantly these methods can be invoked in the web service mode or in inproc mode (for improved performance). We are also introducing strongly typed collections instead of datasets. The exception model is also very flexible and developer friendly.Compared to the COM objects you will find this model rich in functionality, performant, developer friendly and more importantly consistent.

2. Unified Property Store

Commerce server 2002 allowed extending products and categories by adding properties to the definitions. We also supported extending catalog and property definitions by adding user defined attributes.

In Commerce Server 2007, we added a new Inventory component and also rearchitected the usage of the property definitions and the extension model in the catalog system. You can now create a property in the catalog system and use it to extend the property definitions, catalog definitions, product and category definitions, inventory catalogs and inventory skus. For eg you can create a property named VendorId and use it to extend the Catalog definitions. This will allow you to associate a vendor id for each catalog. In addition you can use the same property to extend the inventory catalogs and associate vendor ids to your inventory catalogs. This model now allows you to defined your properties in the catalog system and use it to extend other entities in the catalog and inventory systems in a consistent manner. This should greatly improve the maintainability, extensibility and consistency of the new extension model.We have also added support to add languages to property definitions to allow display names to be defined in each language.

3. New features in Base catalogs

Ranking : You can now rank child products and categories in a category, variants in a product family and root products and categories in base catalogs. This will allow you to control how products and categories are displayed on your runtime site. For eg you can rank your products in such a way that important products are displayed at the top and less important products at the bottom. In addition to catalog items you can also rank related products and categories thereby allowing you to control display of cross relationships.You can also rank the enumeration values for an enumeration property. This is important for a property like Size which has values Large, Small and Medium and you want to control how they are displayed to the user.

Display Names: We have now added the ability to provide displaynames for categories, products and variants.

4. New features in virtual catalogs

Given that virtual catalogs was a V1 feature in Commerce server 2002 it was only natural that we improve upon it in Commerce server 2007.

Ranking : Just as in base catalogs you can now rank child products and categories in a category, variants in a product family and root products and categories in virtual catalogs. In addition you can either inherit the ranking from base catalogs or override ranks in the virtual catalog. The same goes for ranking relationships.

Overriding prices and properties in virtual catalog : You can now override property values and prices of base catalog items in virtual catalogs. For eg if a product has a description "foo" in a base catalog and you include this product in a virtual catalog you can now change the description of this product to "bar". In adition to properties you can also override prices of products in the virtual catalog. For eg if a product has a price of 200 dollars in the base catalog and you include it in a virtual catalog with a 50% discount the product will now have a price 100 dollars in the virtual catalog. The new virtual catalog system now allows you to override this price and set it to 75 dollars. Once you override a property value or a price you can also reset it so that it inherits these values from the base catalog.

Adding rules from virtual catalog : The new virtual catalog system now allows you to add rules from another virtual catalog. There is however a limitation to this.A virtual catalog can contain rules from (one or more) base catalogs or one virtual catalog. This opens a numer of scenarios. You can now aggregate one or more base catalogs (possibly from your vendors or distributors) in one virtual catalog and then create another virtual catalog (off this virtual catalog ) for your customers.

What's new in Commerce Server 2007?

vinayakt — Sat, 04 Mar 2006 09:11:00 GMT

Twenty months back when we started planning for the next release of Commerce Server I had written a post asking for feedback on Commerce Server 2007. Now twenty months later we are on track to ship Commerce Server 2007 sometime soon. Commerce server 2007 beta was released in December, '05 and is available for download. The first drop of the Starter Site and the documentation are also available.

Given that this is the first major Commerce Server release since Commerce Server 2002 the obvious question is "What's new in Commerce Server 2007"?

These are just some of the new features in Commerce Server 2007

1. The biggest feature is the architectural change in the major commerce components to use the web service model.What this means is that the functionality of the catalog, orders, marketing and profiles systems is now available as web services. Some of these components have been fully written in managed code and provide a developer friendly object model to program against.

2. A flexible role based security model based on Windows Authorization manager which defines which users can perform which operations based on their role has been built in the catalog , orders, marketing and profiles systems.

3. Performant Winforms UI for managing catalog schema, managing catalogs, marketing and orders.

4. A C# based production ready starter site which demostrates how to use the commerce components.

5. A new Inventory component, tightly integrated with the catalog system which will allow you to store, update and maintain inventory information for products and variants and also control how they are rendered on the site based on their inventory levels.

6. Staging data from staging servers to production servers was a big issue in previous releases. We now have a product team tested staging module which will allow staging your commerce data from staging to production servers.

7. LOB adapters for inventory, catalog, orders and profiles.The will help you build Connected Commerce applications that will easily tie your Commerce Server deployment into LOB systems, trading partners, or other Commerce Server deployments.

8. Revamped orders systems.

9. Major improvements in the catalog system especially virtual catalogs, search functionality, security and ranking. More on this later.

10. MOM pack for monitoring the state of your commerce server deployment.