Maxime Bombardier - SharePoint Architecture & Development : Web Content Management

BlobCache, Style Library, and anonymous users

MaximeB — Tue, 11 Aug 2009 14:36:24 GMT

As a follow-up to Chris O’Brien’s posts regarding BlobCache and 304s, which can be found here and here, I wanted to clarify the situation as to when you may have 304s.

By default, the Style Library (and Master Pages gallery for that matter) is breaking permission inheritance. However, during its creation, the SPList.AllowEveryoneViewItems property is set to true – thus giving anonymous users access – remember, the list is created before anonymous access is given, this property thus simplifies giving access to everyone during creation.

If you enable anonymous access (i.e.: Entire Web Site) on the root permission, and you access the site anonymously, you will not be prompted for credentials. However, when the BlobCache goes through the list’s permissions, it doesn’t find anonymous access and will not serve the request.

As a result, while the file is saved in the BlobCache directory but not served to the user from that cache, the cache-control Http Header will still contain “private, max-age=0”. At the next user request, the browser will validate its cache against the file on the server and return a 304.

Workaround

The workaround is simple, the first step is to explicitly give anonymous access to the Style Library – either by resetting to inherit from parent or by giving anonymous access. The 2nd step is to clear the BlobCache on all zones and all servers.

The ‘Clear BlobCache’ checkbox in the ‘Object Cache’ page will only clear it for that zone for that server – you should clear it either through the web application property. Changing the BlobCache directory (in the web.config of all servers for all zones) the BlobCache will also work.

While Chris’ last post regarding this did essentially provide the workaround, there was some confusion due to the nature of the tests (which can bring false positive or false negative results if you don’t follow steps rigorously), and the cache clearing that may be inaccurate if you do it on the wrong zone/server.

Final note

If your WCM portal use files from the Style Library (or any other list you created and set the AllowEveryoneViewItems property) and allows anonymous access, make sure you update the list’s permission and clear the BlobCache on all zones/servers. This will allow user to cache the files and reduce the server impact (i.e.: the 304s) considerably. This should be part of your “provide anonymous access” procedure.

A correctly configured BlobCache is essential to a high performing web site.

Experiences on using a single farm for developers (WCM)

MaximeB — Mon, 06 Jul 2009 22:42:34 GMT

Scenario:

5-8 developers
WCM portal with lots of sites (SPWeb) and pages, but not a large database
Development is focused on Fields, Content Types, Layouts, Web Parts, Controls, HttpModules

Typical environments:

The general recommendation is for developers to have their own virtual machine (or physical machine) that contains a complete SharePoint installation. There’s a variant where the SQL Server is shared but it will still be several SharePoint farms with a single server in each.

The problem with these is that you end up with multiple farms to manage. This can work good for the first deployment but will be difficult for incremental releases. It can also be good for developers working for different production environments.

However, when developers are working for a single production environment, other requirements will come in:

Download content artefacts from Production to Development (especially for WCM or light collaboration scenarios)
Update all developer environments with new artefacts
Update all developer environments with new or updated layouts & styles
Update all environments with a new CU or service pack

Developers already had those requirements without impacting anyone else:

Update file system files in the web folder (web.config, browser file, etc.)
Execute an IISRESET
Update assemblies

What impact would a single farm have for development?

After living with multiple farms using shared resources, I implemented a single farm at a customer. While there are identifiable caveats, it also bring a lot of value while diminishing overall time spent on maintenance and on bug due to non-synchronized environments.

For WCM, you will have 2 types of development : 1) content related and 2) pure development artefact. The first is easy to handle, your web designer can simply work like before and it will generally not impact anyone else, in fact, it’s beneficial since page layouts will be updated for everyone at the same time.

The only drawback is that I’ve often seen web designers doing check-ins or publishing layouts to test them. This isn’t optimal for anyone as it uses more resources on SharePoint and if the layout has a display issue, everyone will have it. Therefore, some assistance (how-to’s) should be planned to ensure that web designers are working optimal for everyone.

For Development artefact, it can be a little bit more complicated depending on what type of artefact. For example, if you are developing a custom SharePoint Field, results will be inconsistent if you add a new field instance to a content type that is used for everyone. In the same fashion as the previous case, working on a separate site collection (or at least content type) might be a more manageable solution for the few times when you are developing those artefacts.

In fact, for most cases, it’s the visibility that is more important than the actual change. The reason for this is that a single developer working on a custom Field or a Web Part will only work on his development machine for obvious reason. If the farm + web application + site collection are shared, then we may have to create an unused sub-site for testing purposes during the development.

While this may seem cumbersome, at least on “paper”, it’s actually quite simple and is much simpler to manage than multiple environments. This allows to have less requirements on the development workstations and resources are shared – the database/disk/backups footprint is lessen.

Mixed topology

At the next level, you can have the bulk of your development done in a single shared farm but this doesn’t mean you cannot have 1 or 2 other farms for specific development artefacts if they prove too cumbersome. For example, you may want to have a specific farm (with content) used for various special operations such as testing new CU/Service Packs, 3rd party tools and solutions, etc. All without affecting your standard integrated testing environment – it’s still easier to maintain 2 farms than 8-10.

I had a scenario where we created a development farm for support. It’s purpose was to test anything else that wasn’t in the standard development line.

Conclusion

So far, I’ve had good success with the single farm topology. There is some added governance to be done while implementing – primarily for developers and web designers – but this ensures that you have a good definition of what type of work is being done and which one you wishes to support.

On a daily basis, developers didn’t have a negative impact due to improper updates by someone else. On a longer basis, management was much easier, especially with content updates and patches – we are now 100% positive that everyone sees the same thing.

While this may not be ideal for some situations, it certainly proved to be working well for WCM development.

Fixing absolute URLs for all Alternate Access Mappings (AAM) of Content Editor Web Part with a Control Adapter

MaximeB — Tue, 23 Dec 2008 16:56:21 GMT

Hi everyone and sorry for the no-post for several months. My ‘Draft’ list keeps piling but I didn’t take the time to unpile it much.

Anyhow, I came across this problem again last week with the out of the box Content Editor Web Part (CEWP). Basically, if you use the rich html text editing capability, all URLs are absolute.

I came across a similar issue last year while using the HtmlField and Content Deployment where the URLs were still pointing to the authoring web site. The CEWP was also renowned to have the same issue. Back then, we had fixed the general issue with a quick HttpModule installing an HttpFilter which was essentially rewriting URLs. In terms of architecture, I didn’t like the idea but it was a quick fix done in the early hours. It did the job. Since then, we removed Content Deployment so we didn’t have the issue anymore, thus we removed the HttpModule.

While adding new Alternate Access Mappings URLs to the web site, we noticed some link errors with only the CEWP this time. I looked at past support cases to see if the issue had been resolved in any Cumulative Update (up to December 2008) but no, it’s not. Apparently, the fix involves so many changes in the source code that it’s too high a risk. As of right now, it’s not scheduled to be fixed. The official solutions were :

Update your CEWP for Publishing Html Fields. Anyone doing publishing knows that it’s not possible in all scenarios. If you know all the Html content types, using Html fields is better and recommended, however, in most landing pages, it’s difficult to do.
After using the Rich Editing capability, go in the source code and update all the links manually. This is beyond practical that I won’t discuss it.
Look at 3rd party components. The only one I found that was readily SharePoint enabled was the one from Telerik : RadEditor. It’s a great control that I would recommend for whatever else it does or if you start a new web site. Unfortunately, the Lite (and free) edition doesn’t seem to support the properties StripAbsoluteAnchorPaths and StripAbsoluteImagesPaths. The complete version allows you to use these and it’s 1,000$ USD per developer.

While I’d definitely recommend the RadEditor control, it wasn’t fixing my currently existing pages that uses the CEWP. My next thought was back to the HttpModule but then I remembered looking at ASP.NET Control Adapters.

ASP.NET Control Adapters is basically a way to modify the Render of a control. It’s a little bit like inheritance but only for Render AND it allows to inherit Sealed class such as the ContentEditorWebPart class. Since I was a bit rusted with Control Adapters, I typed in a search for “ContentEditorWebPart ASP.NET Control Adapter” and came across this post from Waldek Mastykarz. He lays out most of the essential but he essentially strips all absolute URLs and this was impractical in my scenario.

In most scenario, you’ll want to strip a list of URLs. In the case of Content Deployment, you’ll want to supply the list of AAMs from the authoring environment (if it’s in the same farm, you could do this programmatically much like what I’m doing in this code but you supply a different URL to the SPWebApplication.Lookup method, if it’s a different farm, supply the list in a config file). In the case of a single site with multiple AAMs, this code will work great:

        1: using System;

       2: using System.Collections.Generic;

       3: using System.Text;

       4: using System.Web;

       5: using System.Web.UI;

       6: using System.Web.UI.WebControls;

       7: using System.Web.UI.Adapters;

       8: using System.IO;

       9: using System.Text.RegularExpressions;

using Microsoft.SharePoint.Administration;

  

  

namespace MaximeBBlog

{

    public class ContentEditorWebPartAdapter : ControlAdapter

    {

        protected override void Render(System.Web.UI.HtmlTextWriter writer)

        {

            StringBuilder sb = new StringBuilder();

            HtmlTextWriter htw = new HtmlTextWriter(new StringWriter(sb));

            base.Render(htw);

            string output = sb.ToString();

  

            List<Uri> alternateUrls = GetAlternateUrls();

            foreach (Uri alternateUrl in alternateUrls)

            {

                output = output.Replace(alternateUrl.ToString(), "/");

            }

  

            writer.Write(output);

        }

  

        private List<Uri> GetAlternateUrls()

        {

            List<Uri> alternateUrls = (List<Uri>)HttpContext.Current.Cache["AlternateUrls"];

            if (alternateUrls == null)

            {

                alternateUrls = new List<Uri>();

  

                SPWebApplication webApp = SPWebApplication.Lookup(System.Web.HttpContext.Current.Request.Url);

                foreach (SPAlternateUrl alternateUrl in webApp.AlternateUrls)

                {

                    alternateUrls.Add(alternateUrl.Uri);

                }

  

                HttpContext.Current.Cache.Add("AlternateUrls", alternateUrls, null, DateTime.Now.AddHours(12), System.Web.Caching.Cache.NoSlidingExpiration, System.Web.Caching.CacheItemPriority.Normal, null);

            }

  

            return alternateUrls;

        }

    }

}

In a nutshell, I’m doing the following:

Create a class inheriting from System.Web.UI.Adapters.ControlAdapter
Override the Render method and read the stream’s text (Html)
Get the list of AlternateUrls from the current Web Application
Keep the list in caching (I did it for 12 absolute hours, you can change that to your liking)
Do a simple String.Replace() of the AlternateUrls for a relative url

In terms of performance, it’s negligible and it won’t be reprocessed if you have Page Output Caching enabled. The SPWebApplication.Lookup is also very fast and I’m keeping the list in cache. As for the architecture, it ensures you are only stripping the correct URLs for only that control.

You’ll notice that there is no mention of the ContentEditorWebPart so far and that’s because we aren’t inheriting this Web Part. That’s where the Control Adapter magic starts and you could even attach it to multiple controls if there’s a need! To bind it, you need to add (or update if you have one already) a compat.browser file in the App_Browsers folder of your web site. This file requires the following:

       1: <browsers>

       2:     <browser refID="Default">

       3:         <controlAdapters>

       4:             <adapter controlType="Microsoft.SharePoint.WebPartPages.ContentEditorWebPart" adapterType="MaximeBBlog.ContentEditorWebPartAdapter, MaximeBBlog, Version=1.0.0.0, Culture=neutral, PublicKeyToken=febc2e2cb2c3d564" />

       5:         </controlAdapters>

       6:     </browser>

       7: </browsers>

Note: All you need to update is the namespace in both the code and compat.browser file to match your own; as well as signing it with your key.

You’ll notice that the DLL is signed, this allows me to deploy it in the GAC (through a WSP for example) and NOT to require Full trust. I’m still using WSS_Minimal in fact. There is NO requirement for a SafeControl entry in the Web.Config either. Once your DLL is in the GAC and the compat.browser file in the App_Browsers directory of your web site, that web site will render the ContentEditorWebPart through your class.

Note: if you application domain is already loaded, you may need to recycle it for the update to start working, especially if the page was in the page output already.

Thanks again to Waldek for his early start on the issue.

Cheers!

How to sort the list of sites in the Create Sites page

MaximeB — Wed, 20 Aug 2008 19:51:55 GMT

In a customer deployment, we had several custom Site Definitions to create (because they have variations) and we started noticing that the list of sites available in the Create Page is sorted in an odd way. In fact, it's using neither the creation order, the configuration ID, nor the title (which is okay since it's multilingual).

Using Reflector, I noticed that the page was using the GetAvailableWebTemplates() method of the SPWeb class in order to get its list. I tried sorting the list like I wanted and then using the SetAvailableWebTemplate() but the list was still being returned in the original sorted way.

Digging deeper with Reflector, I noticed that it uses the web property "__WebTemplates" and that returns a list of 'All' web templates as well as web templates by LCID.

Here's the method I created to sort the web's site creation list. You need to pass an SPWeb, a flag to determine if you want to be recursive, and the LCID of your web. In my case, I only had All + 1033 in the English, and All + 1036 in French. If you have multiple languages available in the same site, you may need to update the code a little bit.

        1: private void SortWebTemplates(SPWeb web, bool recursive, uint lcid)

       2: {

       3:     SPWebTemplateCollection webTemplates = web.GetAvailableWebTemplates(lcid);

       4:     StringBuilder sb = new StringBuilder();

       5:  

       6:     Collection<SPWebTemplate> collection = new Collection<SPWebTemplate>();

       7:     foreach (SPWebTemplate template in webTemplates)

       8:     {

       9:         bool itemAdded = false;

  

        for (int i = 0; i < collection.Count; i++)

        {

            if (template.Title.CompareTo(collection[i].Title) < 0)

            {

                collection.Insert(i, template);

                itemAdded = true;

                break;

            }

        }

  

        if (!itemAdded)

            collection.Add(template);

    }

  

    sb.Append("<webtemplates><lcid id=\"all\">");

    foreach (SPWebTemplate webTemplate in collection)

    {

        sb.Append("<webtemplate name=\"" + webTemplate.Name + "\" />");

    }

    sb.Append("</lcid><lcid id=\"" + lcid.ToString() + "\">");

    foreach (SPWebTemplate webTemplate in collection)

    {

        sb.Append("<webtemplate name=\"" + webTemplate.Name + "\" />");

    }

    sb.Append("</lcid></webtemplates>");

  

    web.AllProperties["__WebTemplates"] = sb.ToString();

    web.Update();

  

    if (recursive && web.Webs.Count > 0)

    {

        foreach (SPWeb subWeb in web.Webs)

        {

            SortWebTemplates(subWeb, recursive, lcid);

            subWeb.Dispose();

        }

    }

}

Maxime

The power of deleting Content Deployment Jobs & Paths

MaximeB — Tue, 10 Jun 2008 15:07:40 GMT

For the past few weeks, I have been in parental leave (we just had our first baby girl) and unable to write much. However, it seems that there was some issues with Content Deployment at work.

Actually, the installation of SP1 and Blackout had been scheduled for a while but it didn't resolve as many issues as we had hoped. As such, we were still plagued with some "Object Reference Not Set" and "Primary Key constraint violation ..." during the Export phase.

By calling support, my team was directed to another hotfix (950279), http://support.microsoft.com/default.aspx/kb/950279 that had been released just after Blackout and we hadn't noticed it. This is the 3rd hotfix that I remember seeing that fixes "Primary Key constraint violation...".

What you need to know about these errors is that they are mostly generic. Also, it's not by exporting the content and importing it in your dev environment that you will be able to reproduce the issue. The error is "Content Deployment" specific (I didn't check which database hosts the Content Deployment jobs and paths but it's likely to be Configuration, or otherwise Central Administration).

By deleting the Jobs & Paths, you will likely remove the issue that causes the Export to fail. It seems that issues that were still there before the upgrade were still there after. Also, the thing to know is that, after deleting/recreating the jobs & paths, the next incremental deployment will not have any knowledge of what has been deployed previously and will deploy everything.

For some reason, the first incremental gave us some issues where we had several 'file already exists with the same name' issues. I went ahead and deleted the Jobs/Paths again, did a Full Content Deployment once, and all other deployments have been Incremental and running fine.

So unless you have no problem installing hotfixes (privates and/or publics), I would try this simple no-risk step before installing anything in production. While, in some situation, it may only be a temporary solution, it may buy you enough time to test the hotfix deployment in your environments and hopefully wait for a public fix or even a Service Pack.

Maxime

Updating site parameters using the API may freeze the variation job

MaximeB — Sat, 26 Apr 2008 16:32:02 GMT

As you probably know if you read my blog, we like to automate as much as we can our WCM portal deployments. One of the thing we do is to update the Publishing Navigation to hide some sites/pages or simply re-order them correctly. Unfortunately, when you have variations and you want to automate this for all languages, you have to specify some kind of "sleep" time so that it waits for the Variation Timer Job to kick in and create the site in all labels.

Obviously, the sleep time period is guess work and is different on all servers (even on the same one!). During the first attempts, our sleep timer was rather low and we noticed that if we tried opening a site in a target variation/label while the variation process was doing its work, the process was simply stopping! I tried restarting the OWSTIMER service, rebooting the server, wait a day ... but no luck, it simply wasn't picking up again. The Timer Job definitions were still 'running' every minutes, they simply didn't check if there was any updates to propagate.

Unfortunately, so far, I haven't found a way to fix it and to be honest, we simply delayed running our updates on all labels to a later stage. While I did take a look at trying to 'kick start' the process, I didn't go very deep and if I have the issue again, I'll try Gary Lapointe's variation relationship fix (which is VERY interesting and practical even if you don't have the issue I outlined here). If that doesn't work, I'll go see if the relationship are created anyway but it's the Timer Job that cannot read them anymore. I'll update the post when I have the time to reproduce the error and debug it.

Maxime

Deployment lifecycle when using Site Definitions and Features (including post production day)

MaximeB — Sat, 23 Feb 2008 16:14:22 GMT

Summary

As you have noticed, I've changed my latest SharePoint focus to look at Site Definitions (and obviously Features but I played with them a lot prior to this). The reason for this is that I was looking at the "best" way to deploy artifacts for SharePoint WCM portals. While I had a very good working solution prior to me playing more with Site Definitions, I now think that I am ready to handle the task much better. This is also a follow-up to a previous article : http://blogs.msdn.com/maximeb/archive/2008/01/13/real-life-team-based-development-for-web-content-management-with-sharepoint-2007.aspx

The reason for this is that there is no single way to deploy SharePoint artifacts. Worst, once you deployed the first time, your original mechanism may not work anymore or if it does, it's probably not what you should use! What I'll attempt to do here is to define a few SharePoint artifacts that we deploy in a Web Content Management scenario for both production day and post production day, as well as the deployment lifecycle for the developers pre-production-day and post-production-day.

SharePoint artifacts we'll cover:

Master pages
Page Layouts
CSS / JavaScripts / Images (for the masterpages and layouts)
Site Columns & Site Content Types
Library content types association
Custom Lists
Event Handlers on libraries
Feature Receivers
Timer jobs

SharePoint topics discussed:

Site Definitions
Features
Variations
Feature Stapling
Timer Jobs
Content Deployment

Why Site Definitions?

The timing of me digging Site Definitions more and this post, http://www.andrewconnell.com/blog/archive/2008/02/15/You-dont-need-to-create-site-definitions.aspx, from Andrew Connell was interesting. First of all, he's right, you do not need to create a Site Definition all the time. However, there are also general guidelines. When you look at the Publishing portal that is available out of the box, it really looks like a good sample for reverse-engineering (and believe me, I opened it often to learn stuff!). How often have you kept the original "Press Releases" sub-site or used "Blueband" or any other master for your site? What Andrew mentions is that you should create your own minimal publishing site or you can take a look at this one here on CodePlex but I suggest you still take it apart with your requirements.

Also, I'm almost always in a multilingual scenario (i.e.: SharePoint Variations) and then Site Definitions takes its full meaning. Why? well the Variation mechanism is really just a system that provision sites. It is not a copy of the source site. When the variation timer job kicks in, it will detect a new site in the source variation. When it reads it, it takes the title, the description, the URL and the Site Template (i.e.: site definition). Then it simply does a standard Create Site... with those parameters. Once that is done, it will replicate only the publishing fields of the welcome page. Last, it will provision the other publishing pages the same way and replicate only the publishing fields on those pages. Note that any customization done manually at the source, beside publishing pages and fields, will NOT be brought to the other variations. That includes libraries undefined in the site definitions, documents or images, versioning settings, workflows, and everything else you see in the user interface that isn't a publishing field!

I think my last sentence was obvious enough that if you customize something at the source, you will customize it again at the destination. While there may be scenarios when you might want this, it'll be unusual and the solution is to use Site Definitions so that the provisioning brings as much as possible.

I'll finish this section with an example: today, if you have a Publishing site with the OOB site definition and that your welcome pages aren't "WelcomeLinks", what do you do? First of all, at the source, you created a new page with your layout. Then you updated the welcome page of your site to your new page and deleted the old "default.aspx". Now all your welcome pages are not "default.aspx" but it's no big deal. When you created a 2nd variation label, all the default.aspx/WelcomeLinks are back and they are the welcome pages of your sites in that variation!! So you again updated the welcome page and deleted the default.aspx. That's not very user friendly. Also, if you wanted to ended up renaming your custom page to "default.aspx", the variation mechanism would break when trying to provision the site since it's the name for the WelcomeLinks.

As you can see, Site Definitions becomes a must when using variations.

What defines a Site Definitions?

First of all, there is no golden rule for this but here's what I personally follow to determine if I use the same definition or a new one:

A different welcome page layout
I want to pre-provision a default set of web parts in a welcome page layout
(f) I want to associate specific content types to libraries
(f) I want to allow specific page layouts creation (i.e.: a "news" only section, a "product" only section, etc.)
(f) I want to add special Site Actions for some sites
(f) I want to have a special workflow for some sites
(f) I want to update library settings for some sites

Note that the items prefixed by "(f)" are items where the delivery mechanism is Features but they are associated in some site definitions.

Also, one note that will affect your site definition architecture are Feature Stapling. That functionality is one of the very practical mechanism to apply a feature to a site definition. Unfortunately, it's also "only that". What I mean is that if you have multiple sites with the same definition, it will apply the feature on all those sites. This is great inside a single Site Collection but when your SharePoint farm contains multiple site collections (possibly for different customers), you may not want to apply a feature to all of them. Unfortunately, I do not personally know of a better solution than to create your Site Definitions "per site collection" only (i.e.: in the naming). They will still likely use the same features though.

And really, when I design Site Definitions, I created a Feature that contains all the Features that I use for ALL sites. That way, my Site Definitions are very small and only contains the customization for that site. Also, I suggest taking a look at this tool to read your sites definitions : http://blogs.msdn.com/maximeb/archive/2008/02/16/site-definition-viewer-a-windows-net-2-0-tool-to-read-site-definition-and-features-configurations-version-0-1.aspx. I also created some CMD files to automate development of site definitions and they come in handy to maximize your time. I will post them soon.

Before taking a look at the SharePoint artifacts, I need to give you a bit of background on how we get the artifacts: We set up a MOSS server only for "web integrators". The Site Collections in there contains the files directly in the database edited through SharePoint Designer. We hooked up an event on the libraries that, when a check-in is done, it brings the file in a Team Foundation Server. Then, we have someone maintaining the build process and when he get the "go" from the web integrators, he builds the feature containing the files and that is sent to the developers for updates and eventually to production.

SharePoint artifact deployment - Master Pages - Page Layouts - CSS/JS/Images

1) Pre-production "day 1"

You should create a site-scoped Feature containing your master pages and other files (pages layouts and css/js/images). Take a look at the PublishingLayouts for ideas. When this feature is activated on a site collection, it will add the file in the libraries in a ghosted format which means that there are no customization on them. Note that if you ever unghost it at the destination, your feature will never update the file after.

2) Post-production "day 2+"

We'll have to define some sub-scenarios but first, let's start that, yes, you may be able to uninstall the feature and reinstall it while using the -force command but I do not recommend it. First, as I just mentioned, if you unghosted a file, it will not be updated. So if you plan on using this technique, make sure you test first. Also, I had some issues with BlobCache with CSS/JS/GIF/JPG but only when they were ghosted, never when unghosted.

Scenario #1 Updates to some/all the files

I would suggest creating an STSADM extension that update files defined in your Feature. This extension will unghost the file and it will act as your mechanism to deploy updates. What I use is a simple extension with an Xml input file that defines the files to update. This file simply grows over time since I only append to it. It gives me an added benefit with "change management" since it's not simply a build that picked up all files and sent it in the feature which may contain an update.

Scenario #2 New files

If you have new files, you probably have a "business reason" behind. For a new master pages or page layout, is it only for a new type of site (i.e.: defined in a site definition) or would you like to apply it automatically on some/all sites? That you have a new site definition or not, you are likely to have a new feature that brings the file. This feature may contain a Feature Receiver that will do some actions with those files such as :

Assigning the master page to some sites
Allowing the page layout to be created in some/all sites
If you have a new site definition, you may want to add it in the list of available web templates

Once you have your feature, you may want to create another one to staple the first feature with your site definition (some or all of them).

For new CSS/JS/images, they are probably used by a new page layout or master to being with so they will go in the same new feature.

3) Developers Pre/Post production day

During the initial development phase "pre-production day", developers should update their environment when they need it. Since I have everything in Team Foundation, I created special CMD/Bath files for developers that helps them clean/create a site definition with some sub-sites already pre-populated. This is enough for most cases. It's clean and fast and they are up and running quickly.

"Post-production day", the site definitions will not be updated (although they can be appended), nor the features, so they should run the same script that add files in the database and install the new features.

SharePoint artifact deployment - Site Columns and Site Content types + library association

1) Pre-production "day 1"

Feature is the best mechanism to bring columns and content types. Note that the feature has to be Site scoped so it will not bring the content types to a library unless you manually create a page through the GUI or that you use a content type association feature in your site definition. You can take a look at Andrew Connell's STSADM extension for a good sample of exporting your columns/content types.

2) Post-production "day 2+"

Well we learned the hard way, since the MSDN documentation wasn't available back then, that if you add a column to a content type (in a feature), a "reinstall" of the feature will only bring the new column at the Site Collection level and it will not be replicated down to each libraries using that content type. Fortunately, the user interface works great so if you do not have many columns to add, you may simply want to document the manual update.

But you know I like automation ... so yes, we are also fortunate with the API and it replicates down the change as well. An STSADM extension that adds column + associate it to a content type can be done easily so that it reads an Xml input file with all the updates since Day 1. When it runs, if the column's already there, it simply skips it and go check if the content type contains the association to it.

3) Developers Pre/Post production day

The same process will apply for developers. However, since the "STSADM extension" isn't created until Day 1, they may either recreate their site collection or manually add the column/content types when a change occurs. In my case, since creating a site definition is scripted, it goes very fast.

SharePoint artifact deployment - Custom lists

1) Pre-production "day 1"

A custom list is defined in a feature. The definition contains the columns and types. An instance of a list is defined in a Site Definition or through the user interface. You can also populate rows of data for an instance of a list defined in the Site Definition.

2) Post-production "day 2+"

That one is tricker, you can update and reinstall the feature. However, that will only affect new list creation. I personally haven't gotten the need to add columns and data to a list post-production but I would likely need another little STSADM extension to update it.

3) Developers Pre/Post production day

Same process as the content types, if it's only a feature, then reinstall it. If it's in the site definition, run the creation script.

SharePoint artifact deployment - Event Handlers, Feature Receivers, and Timer Jobs

1) Pre-production "day 1"

All of these are class items. You may want to run code for a specific event on a library, run code when a feature's activated, or run code through a timer job at a specific time. All of them are also originally brought through a feature.

2) Post-production "day 2+"

Now, this can be either very easy or much harder depending on how "politically correct" you want to be. Technically (or less politically correct), the classes are either in the BIN or the GAC, if you need to update the code, you can simply deploy them without touching anything in SharePoint (beside an application recycling or an IISRESET). That is, only if you kept the same assembly version.

Unfortunately, if you want to be more politically correct, you'd have to change the feature, since it contains the assembly version, and reinstall it but there are cases when it seems that some references are "ghosted" somewhere in a database so it may not work. If you want to be politically correct (and normally, we should) with assembly versions, I would strongly suggest doing tests.

3) Developers Pre/Post production day

For pre-production day, it's easy, the version isn't likely to change much so you are good to deploy the DLL and reset IIS/app pool. Post production day will depend on what you chose in scenario 2.

Content Deployment with all this

As long as your site definitions, features, and assemblies are all deployed at the destination, Content Deployment will work great with your artifacts. Sounds simple but even with all my run-ins with Content Deployment, it works good in that scenario.

Hope it helps a bit. I know it's a bit sad that there isn't a "single scenario" and I'm still using STSADM extensions to do some updates, but remember, most of the updates are doable from either SharePoint Designer or the web user interface. These 2 tools works good for the updates. Unfortunately, this version of the product doesn't seem to include the fact that field WCM deployments often have a more rigid and secure environment and SPD users aren't likely to play in production or even staging. That's when we, developers, want to automate the process and that's when we create STSADM extensions to fill the gap.

Maxime

Debugging Content Deployment issues

MaximeB — Tue, 19 Feb 2008 04:56:20 GMT

(Last updated June 17th with the Object Reference Not Set to an Instance)

SharePoint's Content Deployment feature is a very complex functionality that, unfortunately, does not work all the time. I've had scenarios where it ran very smoothly and I've had other scenarios (albeit similar in terms of architecture and design) that simply seems to have a different issue everyday.

What I'll attempt here is to include a few ways of debugging Content Deployment as well as links to fixes (or at least, known issues).

Is the error reproducible?

First of all, is the error reproducible. If it's not, then it'll make it harder to determine the issue but I've had cases where we could still work it out. If it is reproducible, I'd suggest 2 things:

Use STSADM -o export [...] and STSADM -o import [...] These commands are essentially a Full Content Deployment but it's also a little bit more verbose and you may be able to get the exact error and item.
Open the exported CAB file and look at the item that's failing. For example, if you successfully 6872 objects, look at the 6873e object in the CAB file. (note that this also works if the error is not reproducible)

Look at the ULS logs

The Unified Logging System (ULS) logs may contain more information regarding your issue. Look at the logs at the same "time" and see if you have more information depending on the type of error and how verbose you are logging.

Look at the Windows Event Log

For some reason, since we have SharePoint logs, we tend to go less often in the Windows Event Logs. It's actually surprising how much more relevant the information can be there and it's also easier to search on an event ID than a string, especially if you have a localized error message.

Unfortunately, a default installation of SharePoint may also bring a lot of errors in the event log (that could be .NET errors, DCOM errors, etc.) so I would suggest taking a look at it before going to production and apply any other fixes that would help.

Run SPSReport, especially if you plan on opening a support call

You can download it here : http://www.codeplex.com/spsreport.

Try between Incremental & Full Content Deployments

Okay, that's not a funny one but I've had cases where I had a reproducible error during incremental and we couldn't figure out why. We ran a Full Content Deployment and the error disappeared for a couple weeks. I had the opposite cases too though :)

While this may seem trivial, I often forget to look at the CAB file myself so I guess it's possible that not everyone knows about it; same with the Event Log. Hopefully, they will give you enough information to find out which item is breaking your deployment or if a fix is available.

Read the exported Manifest.xml file

The manifest file contains all deployed objects and may help determining which of them broke. However, you need it to break during Import because the manifest file is only completed at the end of the Export phase. Plus, the Incremental jobs almost always delete the file when the content deployment fails (and succeeds). You can read about it here.

List of issues I've worked with (with or without fixes, I will update with comments & new issues/fixes)

The given key was not present in the dictionary. at System.ThrowHelper.ThrowKeyNotFoundException() at System.Collections.Generic.Dictionary`2.get_Item

Solution: The Post-SP1 rollup fixes (WSS, MOSS) are fixing this.

Save Conflict Your changes conflict with those made concurrently by another user

Solution: None yet, often case, a following Content Deployment may work. I have to check at the CAB file for this one but I didn't get the chance

The specified file is already used

Solution: Delete the file and create it under another name; or delete it, deploy, and recreate it. Also, the new post-SP1 WSS roll-up may fix this as it seems to fix it for lists (hopefully list items as well) : KB941422.

I also documented this one here : http://blogs.msdn.com/maximeb/archive/2007/11/20/incremental-content-deployment-completes-successfully-with-errors-mentioning-that-the-name-is-already-is-used.aspx.

Transaction logs are growing fast, especially when using Full Content Deployment

Solution: While this seems to be by design, you can limit the transaction log file by using a SQL "Simple" recovery mode. However, it also means that you can only recover your database with a backup (not with a backup + log backup). Also, Database Mirroring requires a Full recovery mode.

Object reference not set to an instance during export

Solution: The .NET Framework 2.0 SP1 fixes an instance of that issue (related to memory issues). However, there's another case when you have a Quick Deploy job running while you flag a page to be quick deployed. You can read it here.

I noticed "Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt." messages in the ULS logs and Event Logs. See KB923028.

Full Content Deployment may be slower after installing SP1

Solution: None yet, not a "known issue"

I documented this one here : http://blogs.msdn.com/maximeb/archive/2008/02/13/make-sure-you-test-your-content-deployment-after-installing-sp1.aspx. I basically ran some performance tests (after noticing a performance degradation) after installing SP1 and got some concerns. Update: Note that in some cases, Anti-Virus software may be the cause to this as well (pre and post SP1).

HTTP (400) Bad request, Invalid Hostname or "the file type is blocked" ... error message

Solution: Configure your SharePoint Alternate Access Mappings (AAM) for your site so that the default zone for the Central Administration isn't load balanced; or configure IIS so that it responds when the server name and port number are used instead of the load balanced URL.

I documented this one here : http://blogs.msdn.com/maximeb/archive/2008/01/28/central-administration-and-alternate-access-mapping-aam-issues-with-content-deployment-and-sp1.aspx.

The URL "Style Library/en-us" is invalid

Solution: Delete your site collection at the destination, create a Publishing Site at the destination, delete it, and now create a Blank Site.

I documented this one here : http://blogs.msdn.com/maximeb/archive/2007/10/10/content-deployment-the-url-style-library-en-us-is-invalid.aspx. Update: In fact, doing this also creates the correct Timer Jobs for other operations such as Variations to work so it's a requirement.

The TCP connection was forcibly closed

Solution: Run the "Netsh int ip set chimney ENABLED" command; KB912222

I documented this one here : http://blogs.msdn.com/maximeb/archive/2007/11/17/slow-content-deployment-transportation-tcp-connection-was-forcibly-closed.aspx.

Links are updated at the destination and are still pointing on your authoring farm

Solution: None officially nor planned so far. I ended up creating a simple "StringReplacement" HttpModule for a customer.

I documented this one here : http://blogs.msdn.com/maximeb/archive/2007/11/20/content-deployment-does-not-fix-links-at-import-when-the-source-has-been-deployed-with-features.aspx.

You cannot perform this action on a checked out document

Solution: Someone probably checked out a document at the destination. Unfortunately, the logs do not tell you which document is checked out. I ended up creating an STSADM extension that would list all checked out documents by anyone. If anyone knows a CAML query to do this, it would also be an easy way to do it in the Content Structure Reports.

Master pages are not honoured

Solution: Install SP1 or create an STSADM extension to update it. However, note that incremental content deployment will not "erase" your update so you can update it manually at the destination after the initial content deployment.

Violation of Primary Key

Solution: Install SP1.

Special characters are not deployed correctly

Solution: Install SP1.

Update : Error 4958: The stream does not support concurrent IO read or write operations

Solution: Set the FileMaxSize property to 2MB in the Content Deployment Configuration. Thanks to Jian Gao for this one.

Update : Violation of Primary Key on #ExportObjects

Solution: This one occurs sometimes when you modified some files in the Style Library and an exception is thrown early in the Export stage. If you have this, go in the Style Library of your source web site, click on List settings and then Versioning settings and update the versioning so that it only accepts Major versions (NOT major & minor). Execute the Content Deployment and it execute fine after.

Maxime

Site Definition Viewer - a Windows (.NET 2.0) tool to read Site Definition and Features configurations (version 0.1)

MaximeB — Sat, 16 Feb 2008 03:15:23 GMT

I spent a little bit on time understanding Site Definitions lately, particularly the Publishing ones like BLANKINTERNET. One very frustrating thing is to make sense of all the WEBTEMP* configurations, the site definitions, the features, the features dependencies, and the stapled features ... When you start doing Windows search on a "text containing this Feature GUID" and then you open up a bunch of Notepads, you get lost :)

So I created this little Windows application with .NET 2.0. It basically ask you for which Site Definition folder you want to read and it will then create a complete TreeView showing the following information :

All Site Definition Configurations
All Site Features as well as Web Features
All manifests files in any feature used by the site definition configuration
All receiver classes in any feature used by the site definition configuration
All stapled features to the site definition configuration
All properties for any feature used by the site definition

Other features enabled:

On the feature GUID/NAME, a tooltip will appear with its description
Double-clicking on the site definition configuration will open ONET.XML with your default XML launcher
Double-clicking on a manifest file will open the file with its default launcher (usually XML)
Double-click on a feature name will open the file with your default XML launcher
There's a configuration file if you installed your 12 folder elsewhere, you can also specify the site definition to automatically select (practical if you launch the tool often and want to read the same definition :))

** UPDATE ** : I updated the application to not show an error when the feature.xml file doesn't load. Some installations have a few empty folders and I was showing up lots of message boxes. They have been removed.

Here's what it looks like for BLANKINTERNET:

You can see here the all files (manifests) defined with the Publishing feature:

You can see here the properties for the NavigationProperties feature:

You can see that the Publishing feature also defines a receiver class (when you are doing reverse engineering of a feature to understand it, the receiver class is very practical):

Last, you can see that the PublishingSite and PublishingWeb are stapled to all BLANKINTERNET definitions (even though they are defined in the site definition anyway) :

This tool is simply there to help you read all the parameters quickly. It's working fine for me but I haven't done extensive testing and it's not doing everything. Also, I read the WEBTEMP*.xml files in 1033 (english) only but that's also configurable in the configuration file.

Maxime

To Content Deployment or not to Content Deployment, that is the question

MaximeB — Thu, 14 Feb 2008 04:37:47 GMT

As you probably have read, I ran into a couple issues lately with Content Deployment (Slow with SP1?, major/minor versions, AAM on Central Administration, specified file already in used, links not fixed at destination, TCP connection closed, the requirement to create a publishing site + delete + create a blank site) along with a few others that I haven't post on yet (since I don't know how to fix them yet!). Errors such as :

Save Conflict Your changes conflict with those made concurrently by another user
You cannot perform this action on a checked out document
Timed out (at importing)
Object reference not set (during export, I noticed "memory corrupt" exceptions close to the same time it fails; that problem might be fixed with .NET Framework 2.0)
Cannot delete file since it does not exist
Cannot create the file since it already exists at the destination
The SQL Transaction logs are growing up very fast (I had a development environment with a 100 MB web site running a Full Content Deployment every day, yes I forgot I had activated it!, after a month, the transaction log was at 356 GB <-- yes GB and no I don't have backups for that database so that's why the log was so big). it turns out that the Full Content Deployment runs so many queries that the transaction log grows pretty fast if you have Full Recovery (which is required for database mirroring).
Web parts added on a page may not always be there at the destination
and then some more issues that I forgot over time

Most of these are either errors that you can happily forget, some are errors that may or may not happen if you restart the deployment, some are "user created" but shouldn't happen, and some I had to create workarounds since no fixes are scheduled. When you look at this, I cannot say that Content Deployment is reliable for that environment. Unfortunately, it's not only for that environment. I was able to experience these random issues with an OOB publishing site that didn't have any added content!

I'm in great need of inspiration from the product group on this one because I have a hard time saying that there won't be any issues with Content Deployment to a customer. So I started musing on why we are using Content Deployment in the first place, after all, it's for an Intranet scenario. Normally, you need Content Deployment because you want to have a staging site collection (or farm). Especially for an Internet scenario, you probably want to put the authoring in a DMZ or even internally with your authenticated users secured. Otherwise, you may simply want to limit your authors so that they cannot bring have an error in production easily and you want to validate everything before a content deployment.

While these are still true, one of my environment is for an Intranet portal so the DMZ point isn't important as the users are authenticated everywhere on the site (and it's personalized). All that's really left are the authors "screwing up" the web site! However, I've seen scenarios where Incremental content deployments are running every 30 minutes ... that doesn't let you validate everything before now, does it? Also, you can always approve a page right before a deployment and still have the issue. So, if you don't need a DMZ, why should it be a best practice to have a staging farm?

Let's check what an author can really do to "break" a web site?

The author forgot to approve images or linked documents used in the published page.

First of all, they can still run into that issue if they approve the page right before the Content Deployment. 2nd, is content approval required for those images & documents? often case, it's probably not. If it is , then there isn't much to do but have a more rigorous approval process. Otherwise, Solution : don't use the default site definition and create yours where content approval is not required on documents & images (but still is on pages). If you are already deployed, create a feature to set the parameters and staple it to the definitions you used.

The author needs to see the page as a visitor prior to approving it.

Solution : there is an option available through the toolbar under where you can view the page even if it's not approved.

What if the author adds something that shouldn't be there?

Well that's why "Approval" is still required. If you activated the Review Workflow because it's the right thing to do but that approving simply means clicking the button, something's not right. I often case have the question "why can't we approve multiple pages at once?". Reality is that they might need "Publishing" instead of approving if they aren't going to check the pages. To get back on track, the solution is to define better workflows and to have more rigorous approvers. It's also good to know that a workflow with multiple & sequential approvers do not actually require "Approvers" permissions, only the last one in the list who will do the actual transition from archived to published. Don't forget that authors are also accountable. While that may not be much once a "negative" page has been sent online, simply knowing this will likely make them think twice before approving or publishing content.

Can the performance be impacted since authoring pages are usually much bigger?

While it's true that editing will use more resources on the servers, the solution is simple : dedicate one of your farm servers for your editors where their DNS directs them there directly and this server isn't returned by the load balancer. If they are on the same DNS server, then you could also create an Alternate Access Mapping to that site, thus a different DNS, that only the authors access.

There's no need for quick deploy anymore

Well that's not a problem but directly a solution where Quick Deploy only works for publishing pages ... if that page references an image or a document, it won't be deployed at the same time. While I'm guessing the feature's meant for updating text on an existing page, it's still annoying for authors to be able to deploy a page but not its images.

Food for thoughts

While I obviously "feel safer" to have a staging/authoring farm, in an intranet portal scenario, reality is that those portals are often case close to a collaboration portal without collaboration. What I mean by that is that you are sharing information and documents while you don't need users to collaborate on it. If you thought of it as Collaboration, you wouldn't consider using Content Deployment so why should it be different for a WCM intranet portal with no DMZ requirements? I'm still taking about a portal that's viewed by 10-50k people a day. Any comments?

Maxime

Make sure you test your Content Deployment after installing SP1...

MaximeB — Thu, 14 Feb 2008 02:47:45 GMT

I have been testing WSS/MOSS Service Pack 1 on a large-scale WCM portal for the past couple weeks and I ran into some issues with Content Deployment. First of all, I do not want to say there's necessarily a widespread bug with Content Deployment and SP1. Alright, you are warned :) My goal here is that I do indeed have an issue and we are trying to go through support to eventually have some help from the product group to enlighten us. Since I have this issue and that I do not see anyone opening support cases for the same thing nor people writing about it on the web, I'm assuming that it's working in most scenarios.

In summary, make sure you test your Full Content Deployments after you install SP1 in your staging environment.

Now that it's said, let's see why I'm thinking we have an issue. First of all, we experience much more often the "Timed out" issue when using the Content Deployment issue. Sometimes, it keeps going, sometimes, it's not. The environment we are testing currently has been running a SINGLE job of Content Deployment for the last 8, yes EIGHT, days. Since it's importing, the cancel button's not available and we cannot kill the job since it's running. What are we importing? about 100 MB of content with 21,000 objects. It stopped doing anything significant after around 5400 objects. While I would try rebooting the server or playing with a few jobs, I am purposely leaving the server as-is so that the support engineer might be able to realize that there's an issue.

While we had that issue, the customer (obviously) is asking me if we should install SP1 in production and would we have the same issue. The staging environment is a little bit different (barely) where it's running in a VMWare virtual environment. It has the same content as the production (we did a manual export/import from production to staging). The STSADM extension that limits versions (described here) hasn't been ran in staging (it was thought that the export was done after it was ran in production but actually wasn't so we had multiple versions of each pages/documents). So I started working on those 2 differences to see if they had an impact.

I did the test with 2 machines:

My development virtual environment running in a VMWare with 2 GB RAM. The SQL server is centralized on a physical killer box and it's running smoothly.
My personal Virtual PC on my laptop (dual-core centrino with 3 GB RAM but 1.7 RAM allocated to the VPC).

Since Content Deployment usually takes longer than "import" (and I didn't want to export everytime) since it's often "freezing" on my VMs since SP1, I decided to test with export/import. SO I did an export of a "vanilla site" similar to the production (with much less content) that is about 100 MB (actually the CAB file is compressed at 17MB) and 11,000 objects. I deleted the destination collection on both environments and started the imports.

on VMWare, the first full import took 1h15min
on VPC, the first full import took 45min

So far, not too bad although I would be curious as to why it's taking THAT long for so few elements. I was a bit surprise on the VMWare where it took much longer than my VPC while I was still working with Outlook & IE & Word at the same time. VMWare 0, VPC 1 :)

Right after that, I started a 2nd import of the same "export" file again:

3h30 minutes on the VMWare
2h15 minutes on VPC

this is a bit depressing. I have the same content re-imported and it's taking a much longer time. While I understand it's checking the file ... in most content deployment scenarios, we don't really care, we simply want the same thing at the destination.

The 3rd time I ran it, I had the following:

4h on VMWare, 3h on VPC

it's going up again! The only differences this time is that there was 2 versions and it's adding a 3rd version!!!

I did some SQL profiling and couldn't see any request taking over 40ms (and there was very few of these). I was only checking queries & stored procedures ... and I had over 100,000 lines in less than 5 minutes. While everything seems to be running fine in Network Monitor on all environments, (CPU, disks, memory, network are all green), it's simply not going fast. Maybe it's the sheer amount of queries that are running, I don't know and I do not have answers from the product group yet, but I'm getting more concerned with the feature now. Unfortunately, I haven't run the same type of tests before SP1 so I don't know how bad it was, but I know it wasn't going up exponentially.

Last, I noticed that it "freezes" every time, at least on the VMWare development boxes (I have over 12) I have. I wondered if it was any customization we'd done so I created an OOB publishing site with barely anything (not even variations which are known to add LOTS of objects in Content Deployment (i.e.: a new site can take 140 objects instead of 3). I have about 1,100 objects and it takes 20+ minutes for a few MBs!!!! The export & transport phases are fast, but the import is really slow. Unfortunately, not only it's slow, it also stops at times. I simply do not see much SQL traffic going on at those times and the objects imported aren't going up. It might do that for 10 minutes and start again!

Hopefully I'll have better news in the future regarding this. Until then, I'm hoping it's a problem with my environments only or maybe a VMWare issue that we now have. Just make sure you test with your staging environment before production ...

Maxime

Setting Major/Minor versions and Content Deployment issues

MaximeB — Thu, 14 Feb 2008 02:19:05 GMT

Just a quick heads-up when you want to limit versions in a Web Content Management scenario and you have Content Deployment set up.

First of all, let me explain why you would like to do this: Basically, if you happen to execute routinely a Full Content Deployment, a new version of every document (pages, images, document, etc.) at the destination. Since the default library settings are set to accept unlimited major and minor versions, you will likely run into a disk space issue before long. I noticed that on a WCM portal that was planned for about 100 MB of web content that turned out to have 600 MB quickly because not all content had been "web-ified" and was still in PDF and document. That 600 MB was being added at each Full Content Deployment, that a modification occurred or not!

So now you have a fully deployed portal with 150+ sub-sites that you want to limit the versions on at least "/documents" and "/images" (and likely "/pages" as well). At the time, we didn't know enough (or anything really) about Feature Stapling to use them so we created an STSADM extension that updates a library's versioning settings (i.e.: does it require approval, the major & minor versions supported, etc.). I would suggest using Feature Stapling for yourself (with the same codebase really) since it will also affect new sites.

To get back to the primary subject: if you set a major or a minor version limit in the GUI, apply the change, and deploy the update to the publishing farm; then you go back and update to only have a major limit and do not support minors, the GUI will work correctly when you apply. In fact, the "staging or authoring" farm will work correctly. Unfortunately, when you use content deployment, it will say that you cannot set minor version limit on a library that only supports major versions!

I happened to know that error exactly since, when creating the STSADM extension, I got that error when playing with the properties in the "wrong" order. It seems that Content Deployment do use the right order either. You will have to remove the version limits, use content deployment, and then update to have only major versions.

Maxime

Beware of SPWebProvisioningProvider when using Variations

MaximeB — Tue, 05 Feb 2008 20:22:20 GMT

I was doing some research lately on how to have a different "Approvers" group depending on business rule. To make it simple, the following statement had to be fulfilled:

Pages under subsite "/site1" have to be approved by group "Approvers1"
Pages under subsite "/site2" have to be approved by group "Approvers2"
and so on...
We are in a multilingual scenario (i.e.: Variations)

we also have custom site definitions for modifying the default welcome page to something else than WelcomeLinks as well as bringing our content types in the library. So the first thing we looked at was to use SPWebProvisioningProvider that allows to execute a class when provisioning a site. You can take a look at these URLs for more explanation on SPWebProvisioningProvider:

So basically, a site definition reside in the "onet.xml" file and you cannot define a class in that file. The WEBTEMP*.xml (defined in all "available languages", i.e.: 1033 folder for english) file is the configuration used by the "Create site" interface. This is where you specify your custom class for provisioning a web. The first thing your custom class does is to use ApplyWebTemplate() so that the web site is created with the correct template. Then I was simply modifying the "AssociationData" of the only workflow (Parallel Approval) to update its "Approvers" to my custom one depending on the URL.

So far, so good. Unfortunately, when you create this under a variation, it only works at the source. The reason for this is that the variation process is only a provisioning mechanism; furthermore, it's only provisioning based on the site definition, not any customization done after. That includes customization done through a SPWebProvisioningProvider. So basically, while the "onet.xml" file is called by the variation process, the "WEBTEMP*.xml" isn't.

What I ended up doing was to create a feature that contained the same code and added it to my site definition. If the sites are already deployed, you can look at at using Feature Stapling to do the same thing.

In my opinion, the SPWebProvisioningProvider should be a more accurate place to do this code as I want to customize another Feature through business logic. Of course, I could copy the ReviewWorkflow and create as many of them as I need plus as many Site Definitions for them, which might be okay for SharePoint, but wouldn't be a very reusable approach. I also find curious that we have a great customizable solution with the provider but it's simply not being called through the variation mechanism.

Maxime

SPWebProvisioningProvider and ApplyWebTemplate on publishing site definition (i.e.: BLANKINTERNET or custom)

MaximeB — Fri, 01 Feb 2008 21:34:28 GMT

While I am planning to soon blog on real life SPWebProvisioningProvider and Site Definitions in regards of Variations & Content Deployment, I wanted to comment on a little issue that had been plaguing me the last couple days. First of all, my goal is to create a custom publishing site with :

More content types in the pages library
Update the default Parallel Approval workflow to have a different approvers group
Add a Delete Approval workflow
Have a custom default page that isn't a WelcomeLink

So that leads me to creating a custom Site Definitions. Since the approval workflow parameters that I need are variable, I am working with the SPWebProvisioningProvider to update the workflow association after I created either a BLANKINTERNET#2 or a custom site definition that's similar to that one. Note that BLANKINTERNET#2 is the standard Publishing Site with Workflow definition. I found this blog (in French), http://blogs.developpeur.org/themit/archive/2007/10/18/sharepoint-deployment-feature-feature-solution-mais-aussi-webprovisioningprovider.aspx, that talks about how to do this and, unfortunately, it doesn't support comments so I'm commenting here.

Basically, you do a "web.ApplyWebTemplates("BLANKINTERNET#2");" followed by a "web.Update();". Unfortunately, if you do an Update() right after applying a definition that activates the Server Publishing feature, it will give you an odd behaviour. Basically, you end up having a "half publishing site" where you can edit the page, create a sub-site, but cannot create a new page or update navigation from the Site Actions. I noticed that deactivating and reactivating the Server Publishing was working but the Navigation parameters were still off.

I then tried, by mistake :), commenting the "web.Update();" line and everything was working fine. So beware if you use an Update() method call after doing an ApplyWebTemplate (you might require it depending on what you need to modify). I didn't try that yet but you might be better off to do a "SPWeb updatedWeb = web.Site.OpenWeb(web.ServerRelativeUrl);" to get a new SPWeb variable.

Maxime

Real-life Team-Based Development for Web Content Management with SharePoint 2007

MaximeB — Sun, 13 Jan 2008 21:33:21 GMT

It took me a while to come up with everything I needed for this post as I want to make it as accurate as possible with the experience I had on the field. The goal of this post is to provide information on the different ways to work as a team for a WCM web site with MOSS 2007.

I believe that MOSS is large enough that a "one size / fit all" approach is unlikely. There are simply too many variables. Fortunately, I believe that what myself and awesome people I've worked with have overcome most issues and we have automated most deployment procedures. I also believe that most of this knowledge will likely apply to other MOSS scenarios such as Document Management.

This post applies to : Microsoft Office SharePoint 2007 (based on real-life deployments pre-SP1), Microsoft Office SharePoint Designer 2007, Microsoft Visual Studio 2005 Team System, Microsoft Team Foundation Server (not required but nice to have).

Contents

Overview
IT oriented lifecycle with multiple environments
HTML oriented lifecycle
A primer on SharePoint's Content Deployment
Solutions for deploying SharePoint artifacts
A primer on SharePoint Features
A primer on SharePoint Solution Packages (WSP)
A primer on SharePoint APIs
Features versus SharePoint APIs
A note on using Virtual Machines with SharePoint

Overview

What this post will contain is directions and guidance on how to achieve a working multi-environment team-based development lifecycle for WCM with MOSS 2007. Unfortunately, for legal purposes, I cannot offer all STSADM extensions, Event Handlers, Features, and other code artifacts we developed for specific entities. While I may eventually create a more "blank code version", this isn't available at the moment. However, the how-to knowledge should be relatively easy for developers.

In my experience, there are 2 primary types of development topologies in a MOSS WCM project:

IT oriented lifecycle with multiple environments such as :
1. Individual development (while team-based)
2. Integrated testing
3. Staging
4. Production
HTML oriented lifecycle with 1 (production only) or 2 environments (staging + production)

The main differences between the 2 is where your "base environment" is and where you allow "SharePoint Designer users" to act. In an IT oriented lifecycle, every deployment follows a strict cycle of testing, staging, and scheduled deployment to production for all artifacts. In an HTML oriented lifecycle, that may be true for "code artifacts" but you may want to have web integrators use SharePoint Designer directly in the staging environment and/or production.

Obviously, there are advantages and disadvantages for both sides :) I'll try to describe them to the best of my knowledge, but first, I have to explain what Content Deployment is and why it's not meant for development environments:

IT oriented lifecycle with multiple environments

What I call an IT oriented lifecycle is when you already have a development lifecycle in place. Often case, this is for secure or critical environments where downtime is not acceptable. A secure close-to-production Staging environment is usually available in order to test deployments before doing them in Production. You will want minimal steps to deploy as the deployment may be done by a 3rd party who may not have as much SharePoint experience and who's goal is to maintain the environment, not debug it. This lifecycle is more typical for Intranet web sites.

Developers

When you have this environment in place, are you are likely to have separate developer environments as well as an integrated environment. First of all, let's talk about the separate developer environments. You can take a look at this general Team-Based Development document on MSDN : http://msdn2.microsoft.com/en-us/library/bb428899.aspx for more information. Basically, in order to develop MOSS artifacts such as:

ASP.NET Web Parts
MOSS library Event Handlers
Custom Workflows
ASP.NET User or custom controls to use in MOSS
STSADM extensions
Expression Builders
BDC related code
Feature or SharePoint solutions (WSP) event handlers

... a developer requires to have MOSS installed on the same machine as Visual Studio. That will allow him to develop and debug his artifacts without impacting anyone else. Note that, often case, in order for a developer to be effective, he will require the same page layouts, master pages, and other "graphical artifacts" to completely test his development. The deployment process has to allow a developer to receive graphical updates without scraping his complete site collection.

Web integrators

For the web integrators doing graphical work and layouts, they do not require a separate development environment. The best solution is for them to have SharePoint Designer (and their graphical design application) on their workstation and all of them should access the same MOSS server. If working on the same Site Collection, SharePoint Designer may be challenging at times and requires some quirks (at least, pre-SP1).

Sometimes, for web integrators who already had their tools prior to developing for SharePoint, I found it easier for them to design their layouts outside first with the tools they know best. Obviously, knowledge of ASP.NET, Master Pages, and SharePoint will help having an easier transition to SharePoint after. Once their layout is as desired in their respective tools, they can then move that content in SharePoint Designer and apply the required updates to work in a SharePoint world.

The MOSS server they work on should also receive development updates when they are at least unit-tested. This update should occurs outside of their working hours if possible and possibly be automated.

Staging

The Staging environment is where you want your customers to first see the end results of their web site. This environment has to be stable and is probably not updated frequently. This is also where you test your complete deployment process. It's not unusual that non-SharePoint resources execute the deployment or it may even be managed by a 3rd party. The process has to be as transparent and automated as possible. This will allow to validate how the deployment to production will occur.

This environment is also often where you have all components similar to the production and hopefully a multi-server MOSS farm. That allows other type of testing (i.e.: deploying WSP across multiple servers, deployment of MOSS hotfixes in a farm, etc.). The network may also be more constrained/secured.

Production

This is the environment where everything has to run smoothly. The deployment process should be the same as the Staging environment. Authoring occurs here, either on the same site collection or a different one (the site collection could even be in a different farm if desired).

Summary

It's clear that, with that many environment, an automated solution to deploy to all of these environments has to be elaborated. Unfortunately, the only one available with the product is not meant for these deployments. SharePoint Features and SharePoint Solution projects are a big key to the solution but may be incomplete depending on your scenarios (as they have been in my experience). I will cover Features, Solutions, and alternatives further.

The main observation on what I call IT focused lifecycle is that the "SharePoint Designer" role is not a heavy role and updates to page layouts are streamlined through tested IT deployments. Designers aren't able to use Designer directly in Staging or Production. Let's see the HTML oriented lifecycle.

HTML oriented lifecycle

What I call an HTML oriented lifecycle is where your web site requires frequent layout updates. Your designers are likely to be close to the authors rather than the developers. This can be typical for highly frequented Internet web sites. Essentially, the same roles seen with the IT lifecycle has to be fulfilled, however, since developers are higher up in the chain, the "bubbling down" of their work may not be as obvious and less controlled.

Developers

Developers essentially do the same thing as in the IT lifecycle. Integrated testing is done similarly as before on a centralized server that receives more frequent updates from the designers. The integrated testing environment acts as a staging for the development artifacts.

Web integrators

The main difference of this lifecycle reside here. Web integrators are doing daily updates to layouts on an Authoring Site Collection. They might even be "power authors" that also manage content. They often work directly in the production environment albeit a different Site Collection that may or not be in a different MOSS farm than the Publishing Site Collection.

This allows them to easily deploy layout updates without requiring an IT deployment that is usually more complicated or constrained.

Staging

The real staging environment is really the Authoring Site Collection since this is the origin for all layouts.

Production

This environment will be the same as before with the exception that SharePoint Designer is allowed on the Authoring Site Collection.

Summary

As you can see, there are similarities in the 2 lifecycles, however, from an architecture perspective, there are differences when you want to extract designer updates. For example, in the IT lifecycle, we can link the integrated environment with a Source Control (Team Foundation Server for example) and generate builds. The same does not easily apply to the HTML lifecycle as your source control environment is unlikely to be in the same network environment and you will rather want to schedule extracts of the graphical artifacts, copy them to the development environment, and then create a build for developers.

Also, while I've seen both topologies in the field, one could debate why would designers need to update layouts so frequently. In my opinion, if you design your layouts correctly and provide the correct level of customization, you should probably not need updates frequently and you might be able to work out with the IT oriented lifecycle. I wanted to discuss about the 2 main scenarios I've experienced and provide solutions for both.

Now that we have seen the 2 lifecycles, I want to discuss Content deployment quickly to assert why it isn't a viable solution for deploying on all environments.

A primer on SharePoint's Content Deployment

Content Deployment is a SharePoint feature that allows a complete site collection to be deployed from a source to a destination. These 2 environments can be in the same farm or a different farm. This is done through the Operations tab of the Central Administration web site and requires simple configuration. Basically, you create a path between 1 or 2 Central Administration servers (i.e.: 1 or 2 farms), and then you can create jobs to copy content on a scheduled or manual basis.

The primary requirement is that the target or destination has to be a site collection with the Blank Site template (of the same language of the source) and that all activated features to be installed but not activated at the destination. When the first content deployment runs, it will copy all libraries and items to the destination and then activate the same features as the source.

This is very important and to explain it easily, take a look at the Publishing feature. If you activate it, it creates the out of the box master pages, page layouts, and styles for a Publishing Web Site. If you activate it before doing the first content deployment, it will create a "Style Library" with a different GUID than the source. If you were to use content deployment, it would fail as there would be an item with the same name at the source with a different GUID and it won't accept to deploy it. This is the same if you create custom features.

So in essence, since Content Deployment is used to have 2 sites that will be exactly be the same, it's primary use is for having a "staging authoring environment" where you author and test everything. Then Content Deployment is used to update the production environment according to a schedule or a manual request. Note that there are 3 variants of Content Deployment:

Through the Central Administration web site
Through the STSADM -o import and STSADM -o export commands
Through the APIs (Stephan Gossner wrote a great article about the APIs on his blog : http://blogs.technet.com/stefan_gossner/archive/2007/08/30/deep-dive-into-the-sharepoint-content-deployment-and-migration-api-part-1.aspx.

A deployment job consists of 3 primary steps :

Export (at the source). This operation may be computer intensive.
Transport (between the 2). This is simply the file copy, however, there's a related article that can affect performance and I had to apply the fix for a customer. It made the transport much faster.
Import (at the destination). This operation is not so computer intensive, yet very long to process. I have cases where it takes less than 5 minutes to export and transport, and then 1h30min to import.

The Central Administration web site's Content Deployment interface allows you to schedule deployment jobs easily. Unfortunately, you cannot separate the 3 steps so it's good if you only need to deploy to one environment.

Unfortunately, this is usually not an acceptable scenario to update all environments such as Development or even Integrated Testing for the following reasons:

You do not want multiple exports;
The import process is long;
It's not meant for destinations where you are also authoring on it (i.e.: developers testing their new web parts);
It's not easy to "label" (as in Source Control labelling) a deployment.

Because of these concerns, other scenarios such as SharePoint Features, SharePoint Solution (WSP) projects, and the SharePoint APIs have to be explored and the following sections will elaborate on my findings for these technologies.

Note, Chris O'brien also have great posts on Content Deployment and Features.

Solutions for deploying SharePoint artifacts

Now that we have a better understanding of what we are up against, let's look at what solutions are available to us. Here are the possible solutions :

Content Deployment
Backup/Restore
Export/Import through SharePoint Designer (i.e.: Personal Web Package)
Publishing through SharePoint Designer
SharePoint Solution Generator
SharePoint Features
SharePoint Solution Packages (WSP)
SharePoint APIs

There may be other possibilities that I'm unaware of but these are the ones I've tested with. Note that this was done pre-SP1. I'll work by elimination (and reduce the list dramatically :)):

Content Deployment : As stated previously, it's good only for equal environments so it's not practical for our scenarios;
Backup/Restore : essentially the same as Content Deployment albeit much faster but less user friendly, not "scheduled", but usually with less errors/issues;
Export/Import through SharePoint & Personal Web Package : My experience with it was ... very buggy. I simply could not make it work correctly; for example, it created a 2nd Master Page Gallery (after renaming the original) at the destination when I tried it. It didn't seem like a reliable solution, at least, before SP1;
Publishing through SharePoint Designer is similar to Content Deployment. I'm not sure if it uses the same Web Services or not, but the end result is the same with a confusing user experience;
SharePoint Solution Generator : This tool is available through the WSS 3.0 Tools. It's great to create site definitions, however, it doesn't work with the Publishing templates and isn't meant for WCM.

So we are left with 3 plausible solutions:

SharePoint Features
SharePoint Solution Packages (WSP)
SharePoint APIs

Let me start by summarizing each of them:

A primer on SharePoint Features

SharePoint Features are basically what SharePoint uses to deploy functionality to a SharePoint site. For example, the Publishing feature is activated on a Blank Site to create a Publishing Web Site. This will add the Style Libraries and the Master Pages gallery along with modifying the look & feel of your web site. For example, the Site Actions will contain different actions than for a Collaboration web site.

A SharePoint Feature consists of a manifest Xml file that is deployed in the 12 hive. That manifest in turn specify what actions need to be done to the web site. These actions can be from simply copying files (i.e.: page layouts, master pages, CSS, images, etc.) for creating the baseline of your web site, it can create site columns and content types, as well as even executing any .NET code you want through Event Handlers. On paper, it's the solution and SharePoint itself uses it.

Unfortunately, creating features can be challenging as it requires knowing CAML. CAML is the language definition for configuring any features. While MSDN provides documentation for lots of the Xml elements available, MSDN is meant more as a reference, not for training. Also, while I still create SharePoint Features today, I do not use is exclusively anymore as I found some bugs & quirks that I couldn't workaround well. Particularly, using it to provision the baseline files. For some other scenarios (Timer Jobs or Site Actions for example), Features are still the best solution.

A primer on SharePoint Solution Packages (WSP)

WSP files are really CAB files that SharePoint knows what to do with it. The goal of SharePoint Solution Packages are to deploy a package once to the MOSS Farm and then it will deploy and execute what's required for each server of the farm. It includes even more features such as an interface in the Central Administration where you can schedule when you want a solution to be deployed or even retracted! And to sugarcoat it, if you add servers to a farm, it will apply the solution packages automatically.

** Update ** : A colleague pointed me to the fact that solution will even automatically deploy after a server crash. His scenario was that a solution was deployed in a farm and one of the servers had crashed. When they fixed the servers, they simply had to bring it back online and the solution was automatically deployed to that server without any manual operation (no wizard execution). ** /end Update**

Common scenarios include copying file to the GAC, modifying the web.config (using SPWebConfigModification class), copying resources files, and copying physical files to the web site. SharePoint Solution Packages are a must to most SharePoint deployments.

A primer on SharePoint APIs

SharePoint APIs might seem to be a curious addition to this list as it's not really a deployment mechanism, or at least, not yet a user-friendly one. However, while there may be bugs with SharePoint Features and Content Deployment in some cases, the SharePoint APIs is very robust and complete. More so, strong developers can find more code samples as well as using tools such as Intellisense to guess possible solutions.

In your deployments, you are likely to have to use APIs for Event Handlers or web.config modifications, at the least, but my reason for mentioning it as a deployment alternative is that we ended up replacing some "Feature" scenarios that we had issue with for a complete SharePoint API solution. Here's some operations that we automated, by creating custom STSADM extensions, for deploying complete Publishing web sites:

Create Site Columns including Lookup columns
Create and update Content Types
Create and update Master Pages, Page Layouts, CSS, Javascript, images, and other "baseline files" for a web site
Create Publishing Sites and Pages with content (great for creating content-less (or near content-less) landing pages as well. We also create an input file to create complete files with content so that, at a click, we can create a fully working development environment with content)
Create SharePoint lists and referenced by Lookup columns
Update Available Page Layouts for a publishing site
Update Default page for a publishing site
Create SharePoint Permission set and groups
Update Versioning information for publishing site (i.e.: number of major & minor versions, activate/deactivate workflow requirement, etc.)
Update a site's master page
Update Publishing navigation settings (hiding sites/pages, inheriting from parent (or not) navigation, etc.)
and more!

Since lots of these cannot or should not be done by Features, we had to drill the APIs extensively. When strong (showstopper) issues came up for some Features scenarios, we shifted our focus to the APIs. Let me go into more details on the pros & cons of Features versus APIs.

Features versus SharePoint APIs

Features can be a great reusable way of deploying new functionality to a SharePoint site. A SharePoint Feature is a set of functions, settings, and files to be added to a site (there are different levels of features but I'll keep it simple for now) when activated. It's a great reusable mechanism for lots of scenarios such as adding a Custom Site Action, adding a Timer Job, and to some level, adding columns and content types. You can take a look at this MSDN article for starting to work with Features : http://msdn2.microsoft.com/en-us/library/ms460318.aspx.

On one of my projects following the IT Lifecycle, we tried to use Features as much as possible since it was, for us, how reusable deployment artifacts should have been. Unfortunately, we ran into some severe issues when using them for Columns/Content Types and files such as Page Layouts (ASPX), Master Pages, CSS, JavaScripts, and images. While we have successfully used features for other artifacts as mentioned in the previous paragraph, our feature that creates the "web site's baseline" with all the right master pages using custom CSS, images, and scripts; is being phased out. Here are the issues that we ran into:

Sometimes, the BlobCache may corrupt some files that are deployed through a feature. This requires some explanation as the first goal of the BlobCache is to physically cache a file on the Web Front-End instead of fetching it from the database all the time. Since the feature already brings the file on the disk (in the 12 hive), BlobCache wouldn't be necessary. Unfortunately, the BlobCache is also the mechanism by which the Cache-Control Http header is set and that header is the one that let a browser (or proxy) cache a file instead of making it expire right now. We experienced 2 types of corruptions, one where the files were half-correct, and half-blacked-out; one where JS and CSS were simply cropped and were missing content. Note that all files were correct in the feature but simply not in the BlobCache folder. Luckily, we use Content Deployment from Authoring to Publishing and that makes it so that the files are now in the database and there is no corruption then. While MOSS itself uses files in features (i.e.: _layouts or styles library), there is a special Virtual Directory for them and the BlobCache doesn't treat them the same way.
If you want to export (i.e.: content deployment) a site collection (the production/publishing environment) and import them in a test environment, a feature update will not work since the Import process will put the files in the database (similar to unghosted files, but really, it's not reading the feature at that point).
If you update a Content Type and add a new column to it, the column will not be propagated to all libraries (this is not related to the checkbox to force inherit settings on child content types). Let me explain a bit more, when you create a Site Column or a Content Type at the site collection level, and then you create a page that uses that content type in a sub-site's Pages library, the content type and columns are copied to the library. When you add a column to a content type through the GUI or the APIs, it will automatically add the column in all libraries where the content type is used. Unfortunately, a Feature simply does not do it.

While number 1 was annoying for the authoring environment, and number 3 wasn't happening often enough; number 2 is the real issue where we hadn't found a workaround yet. Plus we knew that the APIs work great for all of them. What we ended up creating was an STSADM extension that reads the same CAML files that this feature used and:

Creates the site columns
Create the content types
Add page layouts and master pages
Add files (CSS, GIF, JPG, PNG, JS) in any libraries

We still package a WSP that runs the commands, actually, we could still use a Feature that adds and removes the files when activated/deactivated (through code) instead of using the CAML definition of provisioned files. In my experience with Publishing Web Site, it's much safer in the database than in physical files. Also, learning how to create columns and content types in .NET, or to add files in the database, is very easy and you'll have access to Intellisense. Learning CAML and try all the scenarios (updating an environment, updating an environment that uses Content Deployment, versioning of documents deployed through features, etc.) is likely to be more challenging and time consuming.

The next section is for detail's sake and because I have lots of questions on Virtual environments for developing in SharePoint.

A note on using Virtual Machines with SharePoint

While SharePoint can work correctly in a virtual environment, you have to be aware that the solution is not officially supported by Microsoft unless it's on Virtual Server. What Microsoft Support will offer is what they call "Best effort support" where they'll help until they cannot reproduce the error and may ask to reproduce the error on a non-virtual environment.

Also, in my experience, I wouldn't recommend virtual environment for production web sites.

For development and up to staging however, I have seen both successful and very sluggish environments. Personally, if going VM, I recommend the following parameters:

Have at least 2 GB of RAM
You can use a single SQL for all developers
SQL will offer better performance if not in a VM
SQL's disks should be physical if possible as I've seen saturated SAN scenarios for having a few MOSS developers

In a coming article, I'll detail how we used Team Foundation Server to synchronize SharePoint Designer users with the main source database which allowed us to easily create WSP packages.

Maxime