Maxime Bombardier - SharePoint Architecture & Development : Content Deployment

The power of deleting Content Deployment Jobs & Paths

MaximeB — Tue, 10 Jun 2008 15:07:40 GMT

For the past few weeks, I have been in parental leave (we just had our first baby girl) and unable to write much. However, it seems that there was some issues with Content Deployment at work.

Actually, the installation of SP1 and Blackout had been scheduled for a while but it didn't resolve as many issues as we had hoped. As such, we were still plagued with some "Object Reference Not Set" and "Primary Key constraint violation ..." during the Export phase.

By calling support, my team was directed to another hotfix (950279), http://support.microsoft.com/default.aspx/kb/950279 that had been released just after Blackout and we hadn't noticed it. This is the 3rd hotfix that I remember seeing that fixes "Primary Key constraint violation...".

What you need to know about these errors is that they are mostly generic. Also, it's not by exporting the content and importing it in your dev environment that you will be able to reproduce the issue. The error is "Content Deployment" specific (I didn't check which database hosts the Content Deployment jobs and paths but it's likely to be Configuration, or otherwise Central Administration).

By deleting the Jobs & Paths, you will likely remove the issue that causes the Export to fail. It seems that issues that were still there before the upgrade were still there after. Also, the thing to know is that, after deleting/recreating the jobs & paths, the next incremental deployment will not have any knowledge of what has been deployed previously and will deploy everything.

For some reason, the first incremental gave us some issues where we had several 'file already exists with the same name' issues. I went ahead and deleted the Jobs/Paths again, did a Full Content Deployment once, and all other deployments have been Incremental and running fine.

So unless you have no problem installing hotfixes (privates and/or publics), I would try this simple no-risk step before installing anything in production. While, in some situation, it may only be a temporary solution, it may buy you enough time to test the hotfix deployment in your environments and hopefully wait for a public fix or even a Service Pack.

Maxime

SharePoint Content Deployment Manifest Reader

MaximeB — Fri, 16 May 2008 20:50:59 GMT

I ran into some more Content Deployment issues lately and I was going through the manifest (see http://blogs.msdn.com/maximeb/archive/2008/02/19/debugging-content-deployment-issues.aspx for more examples) with some difficulties.

First of all, let me give you a quick background. When you use Content Deployment, it will create a series of CAB files that will contain a file called manifest.xml. This is the file that drives the show. Unfortunately, especially with Incremental deployments, the file is deleted at the end of the deployment, even if it failed.

Basically, right after the "transporting" phase, the "importing" phase kicks in. This is the best time to go, with Windows Explorer, on the destination server and make a copy of the Content Deployment's files (that are in the last modified folder).

If you open ExportedFiles.cab (not ExportedFiles1.cab or any other cab files), you will be able to get the Manifest.xml file.

What this tool will require is simply that file as well as the last number of imported objects that you have in the Content Deployment report. Type in the same number and it will show you the next object that was trying to be deployed and failed.

As you can see, I simply get the right object and create an Xml document with only that object. You can then navigate it as if you would in Internet Explorer. The 2nd tab contains the actual text only of the object and you can do a 'select all / copy' if you need:

Note that the manifest location and number of objects imported are saved in the application configuration so that you don't have to enter them all the time.

Last, the 'Solution Helper' tab will be the one that I update over time (or you can send me manifest samples with solutions so that I can test + develop + document the solution in the tab for others.

The first example of this is that I had an issue in an environment where all columns were being deleted and recreated. The Incremental deployment was failing with a message stating that you cannot delete a column that's in use. The Solution Helper, when it sees that the object is of 'DeploymentFieldTemplate' (a field), it will attempt to get all the document libraries that uses the field (according to the manifest) and show them there. (In our case, it turned out that a RetractSolution + AddSolution had occured and was deleting + recreating the same columns. A Full Content Deployment will 'fix' the error.

While it's in no way a complete solution, I hope it may help you see which objects are failing and give you ideas on what to fix. You can download it here :

Maxime

Deployment lifecycle when using Site Definitions and Features (including post production day)

MaximeB — Sat, 23 Feb 2008 16:14:22 GMT

Summary

As you have noticed, I've changed my latest SharePoint focus to look at Site Definitions (and obviously Features but I played with them a lot prior to this). The reason for this is that I was looking at the "best" way to deploy artifacts for SharePoint WCM portals. While I had a very good working solution prior to me playing more with Site Definitions, I now think that I am ready to handle the task much better. This is also a follow-up to a previous article : http://blogs.msdn.com/maximeb/archive/2008/01/13/real-life-team-based-development-for-web-content-management-with-sharepoint-2007.aspx

The reason for this is that there is no single way to deploy SharePoint artifacts. Worst, once you deployed the first time, your original mechanism may not work anymore or if it does, it's probably not what you should use! What I'll attempt to do here is to define a few SharePoint artifacts that we deploy in a Web Content Management scenario for both production day and post production day, as well as the deployment lifecycle for the developers pre-production-day and post-production-day.

SharePoint artifacts we'll cover:

Master pages
Page Layouts
CSS / JavaScripts / Images (for the masterpages and layouts)
Site Columns & Site Content Types
Library content types association
Custom Lists
Event Handlers on libraries
Feature Receivers
Timer jobs

SharePoint topics discussed:

Site Definitions
Features
Variations
Feature Stapling
Timer Jobs
Content Deployment

Why Site Definitions?

The timing of me digging Site Definitions more and this post, http://www.andrewconnell.com/blog/archive/2008/02/15/You-dont-need-to-create-site-definitions.aspx, from Andrew Connell was interesting. First of all, he's right, you do not need to create a Site Definition all the time. However, there are also general guidelines. When you look at the Publishing portal that is available out of the box, it really looks like a good sample for reverse-engineering (and believe me, I opened it often to learn stuff!). How often have you kept the original "Press Releases" sub-site or used "Blueband" or any other master for your site? What Andrew mentions is that you should create your own minimal publishing site or you can take a look at this one here on CodePlex but I suggest you still take it apart with your requirements.

Also, I'm almost always in a multilingual scenario (i.e.: SharePoint Variations) and then Site Definitions takes its full meaning. Why? well the Variation mechanism is really just a system that provision sites. It is not a copy of the source site. When the variation timer job kicks in, it will detect a new site in the source variation. When it reads it, it takes the title, the description, the URL and the Site Template (i.e.: site definition). Then it simply does a standard Create Site... with those parameters. Once that is done, it will replicate only the publishing fields of the welcome page. Last, it will provision the other publishing pages the same way and replicate only the publishing fields on those pages. Note that any customization done manually at the source, beside publishing pages and fields, will NOT be brought to the other variations. That includes libraries undefined in the site definitions, documents or images, versioning settings, workflows, and everything else you see in the user interface that isn't a publishing field!

I think my last sentence was obvious enough that if you customize something at the source, you will customize it again at the destination. While there may be scenarios when you might want this, it'll be unusual and the solution is to use Site Definitions so that the provisioning brings as much as possible.

I'll finish this section with an example: today, if you have a Publishing site with the OOB site definition and that your welcome pages aren't "WelcomeLinks", what do you do? First of all, at the source, you created a new page with your layout. Then you updated the welcome page of your site to your new page and deleted the old "default.aspx". Now all your welcome pages are not "default.aspx" but it's no big deal. When you created a 2nd variation label, all the default.aspx/WelcomeLinks are back and they are the welcome pages of your sites in that variation!! So you again updated the welcome page and deleted the default.aspx. That's not very user friendly. Also, if you wanted to ended up renaming your custom page to "default.aspx", the variation mechanism would break when trying to provision the site since it's the name for the WelcomeLinks.

As you can see, Site Definitions becomes a must when using variations.

What defines a Site Definitions?

First of all, there is no golden rule for this but here's what I personally follow to determine if I use the same definition or a new one:

A different welcome page layout
I want to pre-provision a default set of web parts in a welcome page layout
(f) I want to associate specific content types to libraries
(f) I want to allow specific page layouts creation (i.e.: a "news" only section, a "product" only section, etc.)
(f) I want to add special Site Actions for some sites
(f) I want to have a special workflow for some sites
(f) I want to update library settings for some sites

Note that the items prefixed by "(f)" are items where the delivery mechanism is Features but they are associated in some site definitions.

Also, one note that will affect your site definition architecture are Feature Stapling. That functionality is one of the very practical mechanism to apply a feature to a site definition. Unfortunately, it's also "only that". What I mean is that if you have multiple sites with the same definition, it will apply the feature on all those sites. This is great inside a single Site Collection but when your SharePoint farm contains multiple site collections (possibly for different customers), you may not want to apply a feature to all of them. Unfortunately, I do not personally know of a better solution than to create your Site Definitions "per site collection" only (i.e.: in the naming). They will still likely use the same features though.

And really, when I design Site Definitions, I created a Feature that contains all the Features that I use for ALL sites. That way, my Site Definitions are very small and only contains the customization for that site. Also, I suggest taking a look at this tool to read your sites definitions : http://blogs.msdn.com/maximeb/archive/2008/02/16/site-definition-viewer-a-windows-net-2-0-tool-to-read-site-definition-and-features-configurations-version-0-1.aspx. I also created some CMD files to automate development of site definitions and they come in handy to maximize your time. I will post them soon.

Before taking a look at the SharePoint artifacts, I need to give you a bit of background on how we get the artifacts: We set up a MOSS server only for "web integrators". The Site Collections in there contains the files directly in the database edited through SharePoint Designer. We hooked up an event on the libraries that, when a check-in is done, it brings the file in a Team Foundation Server. Then, we have someone maintaining the build process and when he get the "go" from the web integrators, he builds the feature containing the files and that is sent to the developers for updates and eventually to production.

SharePoint artifact deployment - Master Pages - Page Layouts - CSS/JS/Images

1) Pre-production "day 1"

You should create a site-scoped Feature containing your master pages and other files (pages layouts and css/js/images). Take a look at the PublishingLayouts for ideas. When this feature is activated on a site collection, it will add the file in the libraries in a ghosted format which means that there are no customization on them. Note that if you ever unghost it at the destination, your feature will never update the file after.

2) Post-production "day 2+"

We'll have to define some sub-scenarios but first, let's start that, yes, you may be able to uninstall the feature and reinstall it while using the -force command but I do not recommend it. First, as I just mentioned, if you unghosted a file, it will not be updated. So if you plan on using this technique, make sure you test first. Also, I had some issues with BlobCache with CSS/JS/GIF/JPG but only when they were ghosted, never when unghosted.

Scenario #1 Updates to some/all the files

I would suggest creating an STSADM extension that update files defined in your Feature. This extension will unghost the file and it will act as your mechanism to deploy updates. What I use is a simple extension with an Xml input file that defines the files to update. This file simply grows over time since I only append to it. It gives me an added benefit with "change management" since it's not simply a build that picked up all files and sent it in the feature which may contain an update.

Scenario #2 New files

If you have new files, you probably have a "business reason" behind. For a new master pages or page layout, is it only for a new type of site (i.e.: defined in a site definition) or would you like to apply it automatically on some/all sites? That you have a new site definition or not, you are likely to have a new feature that brings the file. This feature may contain a Feature Receiver that will do some actions with those files such as :

Assigning the master page to some sites
Allowing the page layout to be created in some/all sites
If you have a new site definition, you may want to add it in the list of available web templates

Once you have your feature, you may want to create another one to staple the first feature with your site definition (some or all of them).

For new CSS/JS/images, they are probably used by a new page layout or master to being with so they will go in the same new feature.

3) Developers Pre/Post production day

During the initial development phase "pre-production day", developers should update their environment when they need it. Since I have everything in Team Foundation, I created special CMD/Bath files for developers that helps them clean/create a site definition with some sub-sites already pre-populated. This is enough for most cases. It's clean and fast and they are up and running quickly.

"Post-production day", the site definitions will not be updated (although they can be appended), nor the features, so they should run the same script that add files in the database and install the new features.

SharePoint artifact deployment - Site Columns and Site Content types + library association

1) Pre-production "day 1"

Feature is the best mechanism to bring columns and content types. Note that the feature has to be Site scoped so it will not bring the content types to a library unless you manually create a page through the GUI or that you use a content type association feature in your site definition. You can take a look at Andrew Connell's STSADM extension for a good sample of exporting your columns/content types.

2) Post-production "day 2+"

Well we learned the hard way, since the MSDN documentation wasn't available back then, that if you add a column to a content type (in a feature), a "reinstall" of the feature will only bring the new column at the Site Collection level and it will not be replicated down to each libraries using that content type. Fortunately, the user interface works great so if you do not have many columns to add, you may simply want to document the manual update.

But you know I like automation ... so yes, we are also fortunate with the API and it replicates down the change as well. An STSADM extension that adds column + associate it to a content type can be done easily so that it reads an Xml input file with all the updates since Day 1. When it runs, if the column's already there, it simply skips it and go check if the content type contains the association to it.

3) Developers Pre/Post production day

The same process will apply for developers. However, since the "STSADM extension" isn't created until Day 1, they may either recreate their site collection or manually add the column/content types when a change occurs. In my case, since creating a site definition is scripted, it goes very fast.

SharePoint artifact deployment - Custom lists

1) Pre-production "day 1"

A custom list is defined in a feature. The definition contains the columns and types. An instance of a list is defined in a Site Definition or through the user interface. You can also populate rows of data for an instance of a list defined in the Site Definition.

2) Post-production "day 2+"

That one is tricker, you can update and reinstall the feature. However, that will only affect new list creation. I personally haven't gotten the need to add columns and data to a list post-production but I would likely need another little STSADM extension to update it.

3) Developers Pre/Post production day

Same process as the content types, if it's only a feature, then reinstall it. If it's in the site definition, run the creation script.

SharePoint artifact deployment - Event Handlers, Feature Receivers, and Timer Jobs

1) Pre-production "day 1"

All of these are class items. You may want to run code for a specific event on a library, run code when a feature's activated, or run code through a timer job at a specific time. All of them are also originally brought through a feature.

2) Post-production "day 2+"

Now, this can be either very easy or much harder depending on how "politically correct" you want to be. Technically (or less politically correct), the classes are either in the BIN or the GAC, if you need to update the code, you can simply deploy them without touching anything in SharePoint (beside an application recycling or an IISRESET). That is, only if you kept the same assembly version.

Unfortunately, if you want to be more politically correct, you'd have to change the feature, since it contains the assembly version, and reinstall it but there are cases when it seems that some references are "ghosted" somewhere in a database so it may not work. If you want to be politically correct (and normally, we should) with assembly versions, I would strongly suggest doing tests.

3) Developers Pre/Post production day

For pre-production day, it's easy, the version isn't likely to change much so you are good to deploy the DLL and reset IIS/app pool. Post production day will depend on what you chose in scenario 2.

Content Deployment with all this

As long as your site definitions, features, and assemblies are all deployed at the destination, Content Deployment will work great with your artifacts. Sounds simple but even with all my run-ins with Content Deployment, it works good in that scenario.

Hope it helps a bit. I know it's a bit sad that there isn't a "single scenario" and I'm still using STSADM extensions to do some updates, but remember, most of the updates are doable from either SharePoint Designer or the web user interface. These 2 tools works good for the updates. Unfortunately, this version of the product doesn't seem to include the fact that field WCM deployments often have a more rigid and secure environment and SPD users aren't likely to play in production or even staging. That's when we, developers, want to automate the process and that's when we create STSADM extensions to fill the gap.

Maxime

Debugging Content Deployment issues

MaximeB — Tue, 19 Feb 2008 04:56:20 GMT

(Last updated June 17th with the Object Reference Not Set to an Instance)

SharePoint's Content Deployment feature is a very complex functionality that, unfortunately, does not work all the time. I've had scenarios where it ran very smoothly and I've had other scenarios (albeit similar in terms of architecture and design) that simply seems to have a different issue everyday.

What I'll attempt here is to include a few ways of debugging Content Deployment as well as links to fixes (or at least, known issues).

Is the error reproducible?

First of all, is the error reproducible. If it's not, then it'll make it harder to determine the issue but I've had cases where we could still work it out. If it is reproducible, I'd suggest 2 things:

Use STSADM -o export [...] and STSADM -o import [...] These commands are essentially a Full Content Deployment but it's also a little bit more verbose and you may be able to get the exact error and item.
Open the exported CAB file and look at the item that's failing. For example, if you successfully 6872 objects, look at the 6873e object in the CAB file. (note that this also works if the error is not reproducible)

Look at the ULS logs

The Unified Logging System (ULS) logs may contain more information regarding your issue. Look at the logs at the same "time" and see if you have more information depending on the type of error and how verbose you are logging.

Look at the Windows Event Log

For some reason, since we have SharePoint logs, we tend to go less often in the Windows Event Logs. It's actually surprising how much more relevant the information can be there and it's also easier to search on an event ID than a string, especially if you have a localized error message.

Unfortunately, a default installation of SharePoint may also bring a lot of errors in the event log (that could be .NET errors, DCOM errors, etc.) so I would suggest taking a look at it before going to production and apply any other fixes that would help.

Run SPSReport, especially if you plan on opening a support call

You can download it here : http://www.codeplex.com/spsreport.

Try between Incremental & Full Content Deployments

Okay, that's not a funny one but I've had cases where I had a reproducible error during incremental and we couldn't figure out why. We ran a Full Content Deployment and the error disappeared for a couple weeks. I had the opposite cases too though :)

While this may seem trivial, I often forget to look at the CAB file myself so I guess it's possible that not everyone knows about it; same with the Event Log. Hopefully, they will give you enough information to find out which item is breaking your deployment or if a fix is available.

Read the exported Manifest.xml file

The manifest file contains all deployed objects and may help determining which of them broke. However, you need it to break during Import because the manifest file is only completed at the end of the Export phase. Plus, the Incremental jobs almost always delete the file when the content deployment fails (and succeeds). You can read about it here.

List of issues I've worked with (with or without fixes, I will update with comments & new issues/fixes)

The given key was not present in the dictionary. at System.ThrowHelper.ThrowKeyNotFoundException() at System.Collections.Generic.Dictionary`2.get_Item

Solution: The Post-SP1 rollup fixes (WSS, MOSS) are fixing this.

Save Conflict Your changes conflict with those made concurrently by another user

Solution: None yet, often case, a following Content Deployment may work. I have to check at the CAB file for this one but I didn't get the chance

The specified file is already used

Solution: Delete the file and create it under another name; or delete it, deploy, and recreate it. Also, the new post-SP1 WSS roll-up may fix this as it seems to fix it for lists (hopefully list items as well) : KB941422.

I also documented this one here : http://blogs.msdn.com/maximeb/archive/2007/11/20/incremental-content-deployment-completes-successfully-with-errors-mentioning-that-the-name-is-already-is-used.aspx.

Transaction logs are growing fast, especially when using Full Content Deployment

Solution: While this seems to be by design, you can limit the transaction log file by using a SQL "Simple" recovery mode. However, it also means that you can only recover your database with a backup (not with a backup + log backup). Also, Database Mirroring requires a Full recovery mode.

Object reference not set to an instance during export

Solution: The .NET Framework 2.0 SP1 fixes an instance of that issue (related to memory issues). However, there's another case when you have a Quick Deploy job running while you flag a page to be quick deployed. You can read it here.

I noticed "Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt." messages in the ULS logs and Event Logs. See KB923028.

Full Content Deployment may be slower after installing SP1

Solution: None yet, not a "known issue"

I documented this one here : http://blogs.msdn.com/maximeb/archive/2008/02/13/make-sure-you-test-your-content-deployment-after-installing-sp1.aspx. I basically ran some performance tests (after noticing a performance degradation) after installing SP1 and got some concerns. Update: Note that in some cases, Anti-Virus software may be the cause to this as well (pre and post SP1).

HTTP (400) Bad request, Invalid Hostname or "the file type is blocked" ... error message

Solution: Configure your SharePoint Alternate Access Mappings (AAM) for your site so that the default zone for the Central Administration isn't load balanced; or configure IIS so that it responds when the server name and port number are used instead of the load balanced URL.

I documented this one here : http://blogs.msdn.com/maximeb/archive/2008/01/28/central-administration-and-alternate-access-mapping-aam-issues-with-content-deployment-and-sp1.aspx.

The URL "Style Library/en-us" is invalid

Solution: Delete your site collection at the destination, create a Publishing Site at the destination, delete it, and now create a Blank Site.

I documented this one here : http://blogs.msdn.com/maximeb/archive/2007/10/10/content-deployment-the-url-style-library-en-us-is-invalid.aspx. Update: In fact, doing this also creates the correct Timer Jobs for other operations such as Variations to work so it's a requirement.

The TCP connection was forcibly closed

Solution: Run the "Netsh int ip set chimney ENABLED" command; KB912222

I documented this one here : http://blogs.msdn.com/maximeb/archive/2007/11/17/slow-content-deployment-transportation-tcp-connection-was-forcibly-closed.aspx.

Links are updated at the destination and are still pointing on your authoring farm

Solution: None officially nor planned so far. I ended up creating a simple "StringReplacement" HttpModule for a customer.

I documented this one here : http://blogs.msdn.com/maximeb/archive/2007/11/20/content-deployment-does-not-fix-links-at-import-when-the-source-has-been-deployed-with-features.aspx.

You cannot perform this action on a checked out document

Solution: Someone probably checked out a document at the destination. Unfortunately, the logs do not tell you which document is checked out. I ended up creating an STSADM extension that would list all checked out documents by anyone. If anyone knows a CAML query to do this, it would also be an easy way to do it in the Content Structure Reports.

Master pages are not honoured

Solution: Install SP1 or create an STSADM extension to update it. However, note that incremental content deployment will not "erase" your update so you can update it manually at the destination after the initial content deployment.

Violation of Primary Key

Solution: Install SP1.

Special characters are not deployed correctly

Solution: Install SP1.

Update : Error 4958: The stream does not support concurrent IO read or write operations

Solution: Set the FileMaxSize property to 2MB in the Content Deployment Configuration. Thanks to Jian Gao for this one.

Update : Violation of Primary Key on #ExportObjects

Solution: This one occurs sometimes when you modified some files in the Style Library and an exception is thrown early in the Export stage. If you have this, go in the Style Library of your source web site, click on List settings and then Versioning settings and update the versioning so that it only accepts Major versions (NOT major & minor). Execute the Content Deployment and it execute fine after.

Maxime

To Content Deployment or not to Content Deployment, that is the question

MaximeB — Thu, 14 Feb 2008 04:37:47 GMT

As you probably have read, I ran into a couple issues lately with Content Deployment (Slow with SP1?, major/minor versions, AAM on Central Administration, specified file already in used, links not fixed at destination, TCP connection closed, the requirement to create a publishing site + delete + create a blank site) along with a few others that I haven't post on yet (since I don't know how to fix them yet!). Errors such as :

Save Conflict Your changes conflict with those made concurrently by another user
You cannot perform this action on a checked out document
Timed out (at importing)
Object reference not set (during export, I noticed "memory corrupt" exceptions close to the same time it fails; that problem might be fixed with .NET Framework 2.0)
Cannot delete file since it does not exist
Cannot create the file since it already exists at the destination
The SQL Transaction logs are growing up very fast (I had a development environment with a 100 MB web site running a Full Content Deployment every day, yes I forgot I had activated it!, after a month, the transaction log was at 356 GB <-- yes GB and no I don't have backups for that database so that's why the log was so big). it turns out that the Full Content Deployment runs so many queries that the transaction log grows pretty fast if you have Full Recovery (which is required for database mirroring).
Web parts added on a page may not always be there at the destination
and then some more issues that I forgot over time

Most of these are either errors that you can happily forget, some are errors that may or may not happen if you restart the deployment, some are "user created" but shouldn't happen, and some I had to create workarounds since no fixes are scheduled. When you look at this, I cannot say that Content Deployment is reliable for that environment. Unfortunately, it's not only for that environment. I was able to experience these random issues with an OOB publishing site that didn't have any added content!

I'm in great need of inspiration from the product group on this one because I have a hard time saying that there won't be any issues with Content Deployment to a customer. So I started musing on why we are using Content Deployment in the first place, after all, it's for an Intranet scenario. Normally, you need Content Deployment because you want to have a staging site collection (or farm). Especially for an Internet scenario, you probably want to put the authoring in a DMZ or even internally with your authenticated users secured. Otherwise, you may simply want to limit your authors so that they cannot bring have an error in production easily and you want to validate everything before a content deployment.

While these are still true, one of my environment is for an Intranet portal so the DMZ point isn't important as the users are authenticated everywhere on the site (and it's personalized). All that's really left are the authors "screwing up" the web site! However, I've seen scenarios where Incremental content deployments are running every 30 minutes ... that doesn't let you validate everything before now, does it? Also, you can always approve a page right before a deployment and still have the issue. So, if you don't need a DMZ, why should it be a best practice to have a staging farm?

Let's check what an author can really do to "break" a web site?

The author forgot to approve images or linked documents used in the published page.

First of all, they can still run into that issue if they approve the page right before the Content Deployment. 2nd, is content approval required for those images & documents? often case, it's probably not. If it is , then there isn't much to do but have a more rigorous approval process. Otherwise, Solution : don't use the default site definition and create yours where content approval is not required on documents & images (but still is on pages). If you are already deployed, create a feature to set the parameters and staple it to the definitions you used.

The author needs to see the page as a visitor prior to approving it.

Solution : there is an option available through the toolbar under where you can view the page even if it's not approved.

What if the author adds something that shouldn't be there?

Well that's why "Approval" is still required. If you activated the Review Workflow because it's the right thing to do but that approving simply means clicking the button, something's not right. I often case have the question "why can't we approve multiple pages at once?". Reality is that they might need "Publishing" instead of approving if they aren't going to check the pages. To get back on track, the solution is to define better workflows and to have more rigorous approvers. It's also good to know that a workflow with multiple & sequential approvers do not actually require "Approvers" permissions, only the last one in the list who will do the actual transition from archived to published. Don't forget that authors are also accountable. While that may not be much once a "negative" page has been sent online, simply knowing this will likely make them think twice before approving or publishing content.

Can the performance be impacted since authoring pages are usually much bigger?

While it's true that editing will use more resources on the servers, the solution is simple : dedicate one of your farm servers for your editors where their DNS directs them there directly and this server isn't returned by the load balancer. If they are on the same DNS server, then you could also create an Alternate Access Mapping to that site, thus a different DNS, that only the authors access.

There's no need for quick deploy anymore

Well that's not a problem but directly a solution where Quick Deploy only works for publishing pages ... if that page references an image or a document, it won't be deployed at the same time. While I'm guessing the feature's meant for updating text on an existing page, it's still annoying for authors to be able to deploy a page but not its images.

Food for thoughts

While I obviously "feel safer" to have a staging/authoring farm, in an intranet portal scenario, reality is that those portals are often case close to a collaboration portal without collaboration. What I mean by that is that you are sharing information and documents while you don't need users to collaborate on it. If you thought of it as Collaboration, you wouldn't consider using Content Deployment so why should it be different for a WCM intranet portal with no DMZ requirements? I'm still taking about a portal that's viewed by 10-50k people a day. Any comments?

Maxime

Make sure you test your Content Deployment after installing SP1...

MaximeB — Thu, 14 Feb 2008 02:47:45 GMT

I have been testing WSS/MOSS Service Pack 1 on a large-scale WCM portal for the past couple weeks and I ran into some issues with Content Deployment. First of all, I do not want to say there's necessarily a widespread bug with Content Deployment and SP1. Alright, you are warned :) My goal here is that I do indeed have an issue and we are trying to go through support to eventually have some help from the product group to enlighten us. Since I have this issue and that I do not see anyone opening support cases for the same thing nor people writing about it on the web, I'm assuming that it's working in most scenarios.

In summary, make sure you test your Full Content Deployments after you install SP1 in your staging environment.

Now that it's said, let's see why I'm thinking we have an issue. First of all, we experience much more often the "Timed out" issue when using the Content Deployment issue. Sometimes, it keeps going, sometimes, it's not. The environment we are testing currently has been running a SINGLE job of Content Deployment for the last 8, yes EIGHT, days. Since it's importing, the cancel button's not available and we cannot kill the job since it's running. What are we importing? about 100 MB of content with 21,000 objects. It stopped doing anything significant after around 5400 objects. While I would try rebooting the server or playing with a few jobs, I am purposely leaving the server as-is so that the support engineer might be able to realize that there's an issue.

While we had that issue, the customer (obviously) is asking me if we should install SP1 in production and would we have the same issue. The staging environment is a little bit different (barely) where it's running in a VMWare virtual environment. It has the same content as the production (we did a manual export/import from production to staging). The STSADM extension that limits versions (described here) hasn't been ran in staging (it was thought that the export was done after it was ran in production but actually wasn't so we had multiple versions of each pages/documents). So I started working on those 2 differences to see if they had an impact.

I did the test with 2 machines:

My development virtual environment running in a VMWare with 2 GB RAM. The SQL server is centralized on a physical killer box and it's running smoothly.
My personal Virtual PC on my laptop (dual-core centrino with 3 GB RAM but 1.7 RAM allocated to the VPC).

Since Content Deployment usually takes longer than "import" (and I didn't want to export everytime) since it's often "freezing" on my VMs since SP1, I decided to test with export/import. SO I did an export of a "vanilla site" similar to the production (with much less content) that is about 100 MB (actually the CAB file is compressed at 17MB) and 11,000 objects. I deleted the destination collection on both environments and started the imports.

on VMWare, the first full import took 1h15min
on VPC, the first full import took 45min

So far, not too bad although I would be curious as to why it's taking THAT long for so few elements. I was a bit surprise on the VMWare where it took much longer than my VPC while I was still working with Outlook & IE & Word at the same time. VMWare 0, VPC 1 :)

Right after that, I started a 2nd import of the same "export" file again:

3h30 minutes on the VMWare
2h15 minutes on VPC

this is a bit depressing. I have the same content re-imported and it's taking a much longer time. While I understand it's checking the file ... in most content deployment scenarios, we don't really care, we simply want the same thing at the destination.

The 3rd time I ran it, I had the following:

4h on VMWare, 3h on VPC

it's going up again! The only differences this time is that there was 2 versions and it's adding a 3rd version!!!

I did some SQL profiling and couldn't see any request taking over 40ms (and there was very few of these). I was only checking queries & stored procedures ... and I had over 100,000 lines in less than 5 minutes. While everything seems to be running fine in Network Monitor on all environments, (CPU, disks, memory, network are all green), it's simply not going fast. Maybe it's the sheer amount of queries that are running, I don't know and I do not have answers from the product group yet, but I'm getting more concerned with the feature now. Unfortunately, I haven't run the same type of tests before SP1 so I don't know how bad it was, but I know it wasn't going up exponentially.

Last, I noticed that it "freezes" every time, at least on the VMWare development boxes (I have over 12) I have. I wondered if it was any customization we'd done so I created an OOB publishing site with barely anything (not even variations which are known to add LOTS of objects in Content Deployment (i.e.: a new site can take 140 objects instead of 3). I have about 1,100 objects and it takes 20+ minutes for a few MBs!!!! The export & transport phases are fast, but the import is really slow. Unfortunately, not only it's slow, it also stops at times. I simply do not see much SQL traffic going on at those times and the objects imported aren't going up. It might do that for 10 minutes and start again!

Hopefully I'll have better news in the future regarding this. Until then, I'm hoping it's a problem with my environments only or maybe a VMWare issue that we now have. Just make sure you test with your staging environment before production ...

Maxime

Setting Major/Minor versions and Content Deployment issues

MaximeB — Thu, 14 Feb 2008 02:19:05 GMT

Just a quick heads-up when you want to limit versions in a Web Content Management scenario and you have Content Deployment set up.

First of all, let me explain why you would like to do this: Basically, if you happen to execute routinely a Full Content Deployment, a new version of every document (pages, images, document, etc.) at the destination. Since the default library settings are set to accept unlimited major and minor versions, you will likely run into a disk space issue before long. I noticed that on a WCM portal that was planned for about 100 MB of web content that turned out to have 600 MB quickly because not all content had been "web-ified" and was still in PDF and document. That 600 MB was being added at each Full Content Deployment, that a modification occurred or not!

So now you have a fully deployed portal with 150+ sub-sites that you want to limit the versions on at least "/documents" and "/images" (and likely "/pages" as well). At the time, we didn't know enough (or anything really) about Feature Stapling to use them so we created an STSADM extension that updates a library's versioning settings (i.e.: does it require approval, the major & minor versions supported, etc.). I would suggest using Feature Stapling for yourself (with the same codebase really) since it will also affect new sites.

To get back to the primary subject: if you set a major or a minor version limit in the GUI, apply the change, and deploy the update to the publishing farm; then you go back and update to only have a major limit and do not support minors, the GUI will work correctly when you apply. In fact, the "staging or authoring" farm will work correctly. Unfortunately, when you use content deployment, it will say that you cannot set minor version limit on a library that only supports major versions!

I happened to know that error exactly since, when creating the STSADM extension, I got that error when playing with the properties in the "wrong" order. It seems that Content Deployment do use the right order either. You will have to remove the version limits, use content deployment, and then update to have only major versions.

Maxime

Central Administration and Alternate Access Mapping (AAM) (issues with Content Deployment and SP1)

MaximeB — Mon, 28 Jan 2008 16:10:47 GMT

A few months ago, I had to debug an issue where Content Deployment was not functioning correctly. As soon as the Import process was starting, we were either experimenting an HTTP 400 error (Bad Request) or a "...this file type is blocked..." error. We couldn't figure out what was going on as everything was functioning correctly across the farm.

The "blocked file type" error simply happens when you try to contact the web service on a different web site than the Central Administration while an Http 400 error occurs (usually) when IIS cannot determine which web site should respond to the request.

I was able to figure out when I looked at the Alternate Access Mapping (AAM) configuration. While everything looked right at first, the only thing I could see that was different than my test environments were that the Central Administration web site's default zone was configured with a load-balanced URL. What that does is set the IIS configuration so that the Central Administration responds to the Fully Qualified Domain Name that you specified. Unfortunately, Content Deployment works with the server name and no IIS web site was responding to this name. You can "guess" this when you set Content Deployment general parameters and specify which server should receive incoming requests (only server names appear).

So it's all good, we added an intranet zone with the server name + port number and that was the end of the story. Then came along SP1 ... While installing the WSS SP1 Language Pack (at least, the WSS & MOSS SP1 worked), we got the following error :

Exception: System.ArgumentException: The IncomingUrl is already present in the collection in a different zone. A url may only map to one zone at a time. Use a different url for requests from this zone.

at Microsoft.SharePoint.Administration.SPAlternateUrlCollection.Add(SPAlternateUrl alternateUrl, Boolean fUpdate, Boolean throwIfExists)
   at Microsoft.SharePoint.Administration.SPAdministrationWebApplication.Provision()
   at Microsoft.SharePoint.Administration.SPWebServiceInstance.Provision()
   at Microsoft.SharePoint.PostSetupConfiguration.CentralAdministrationSiteTask.ProvisionAdminVs()
   at Microsoft.SharePoint.PostSetupConfiguration.CentralAdministrationSiteTask.Run()
   at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

At least, the error was leading me to the AAM configurations again. It turns out that the default zone has to be the "server name + port" or else the SP1 Language Pack process will fail. As soon as you modify the default zone, the process will pick up and you'll be fine.

Maxime

Real-life Team-Based Development for Web Content Management with SharePoint 2007

MaximeB — Sun, 13 Jan 2008 21:33:21 GMT

It took me a while to come up with everything I needed for this post as I want to make it as accurate as possible with the experience I had on the field. The goal of this post is to provide information on the different ways to work as a team for a WCM web site with MOSS 2007.

I believe that MOSS is large enough that a "one size / fit all" approach is unlikely. There are simply too many variables. Fortunately, I believe that what myself and awesome people I've worked with have overcome most issues and we have automated most deployment procedures. I also believe that most of this knowledge will likely apply to other MOSS scenarios such as Document Management.

This post applies to : Microsoft Office SharePoint 2007 (based on real-life deployments pre-SP1), Microsoft Office SharePoint Designer 2007, Microsoft Visual Studio 2005 Team System, Microsoft Team Foundation Server (not required but nice to have).

Contents

Overview
IT oriented lifecycle with multiple environments
HTML oriented lifecycle
A primer on SharePoint's Content Deployment
Solutions for deploying SharePoint artifacts
A primer on SharePoint Features
A primer on SharePoint Solution Packages (WSP)
A primer on SharePoint APIs
Features versus SharePoint APIs
A note on using Virtual Machines with SharePoint

Overview

What this post will contain is directions and guidance on how to achieve a working multi-environment team-based development lifecycle for WCM with MOSS 2007. Unfortunately, for legal purposes, I cannot offer all STSADM extensions, Event Handlers, Features, and other code artifacts we developed for specific entities. While I may eventually create a more "blank code version", this isn't available at the moment. However, the how-to knowledge should be relatively easy for developers.

In my experience, there are 2 primary types of development topologies in a MOSS WCM project:

IT oriented lifecycle with multiple environments such as :
1. Individual development (while team-based)
2. Integrated testing
3. Staging
4. Production
HTML oriented lifecycle with 1 (production only) or 2 environments (staging + production)

The main differences between the 2 is where your "base environment" is and where you allow "SharePoint Designer users" to act. In an IT oriented lifecycle, every deployment follows a strict cycle of testing, staging, and scheduled deployment to production for all artifacts. In an HTML oriented lifecycle, that may be true for "code artifacts" but you may want to have web integrators use SharePoint Designer directly in the staging environment and/or production.

Obviously, there are advantages and disadvantages for both sides :) I'll try to describe them to the best of my knowledge, but first, I have to explain what Content Deployment is and why it's not meant for development environments:

IT oriented lifecycle with multiple environments

What I call an IT oriented lifecycle is when you already have a development lifecycle in place. Often case, this is for secure or critical environments where downtime is not acceptable. A secure close-to-production Staging environment is usually available in order to test deployments before doing them in Production. You will want minimal steps to deploy as the deployment may be done by a 3rd party who may not have as much SharePoint experience and who's goal is to maintain the environment, not debug it. This lifecycle is more typical for Intranet web sites.

Developers

When you have this environment in place, are you are likely to have separate developer environments as well as an integrated environment. First of all, let's talk about the separate developer environments. You can take a look at this general Team-Based Development document on MSDN : http://msdn2.microsoft.com/en-us/library/bb428899.aspx for more information. Basically, in order to develop MOSS artifacts such as:

ASP.NET Web Parts
MOSS library Event Handlers
Custom Workflows
ASP.NET User or custom controls to use in MOSS
STSADM extensions
Expression Builders
BDC related code
Feature or SharePoint solutions (WSP) event handlers

... a developer requires to have MOSS installed on the same machine as Visual Studio. That will allow him to develop and debug his artifacts without impacting anyone else. Note that, often case, in order for a developer to be effective, he will require the same page layouts, master pages, and other "graphical artifacts" to completely test his development. The deployment process has to allow a developer to receive graphical updates without scraping his complete site collection.

Web integrators

For the web integrators doing graphical work and layouts, they do not require a separate development environment. The best solution is for them to have SharePoint Designer (and their graphical design application) on their workstation and all of them should access the same MOSS server. If working on the same Site Collection, SharePoint Designer may be challenging at times and requires some quirks (at least, pre-SP1).

Sometimes, for web integrators who already had their tools prior to developing for SharePoint, I found it easier for them to design their layouts outside first with the tools they know best. Obviously, knowledge of ASP.NET, Master Pages, and SharePoint will help having an easier transition to SharePoint after. Once their layout is as desired in their respective tools, they can then move that content in SharePoint Designer and apply the required updates to work in a SharePoint world.

The MOSS server they work on should also receive development updates when they are at least unit-tested. This update should occurs outside of their working hours if possible and possibly be automated.

Staging

The Staging environment is where you want your customers to first see the end results of their web site. This environment has to be stable and is probably not updated frequently. This is also where you test your complete deployment process. It's not unusual that non-SharePoint resources execute the deployment or it may even be managed by a 3rd party. The process has to be as transparent and automated as possible. This will allow to validate how the deployment to production will occur.

This environment is also often where you have all components similar to the production and hopefully a multi-server MOSS farm. That allows other type of testing (i.e.: deploying WSP across multiple servers, deployment of MOSS hotfixes in a farm, etc.). The network may also be more constrained/secured.

Production

This is the environment where everything has to run smoothly. The deployment process should be the same as the Staging environment. Authoring occurs here, either on the same site collection or a different one (the site collection could even be in a different farm if desired).

Summary

It's clear that, with that many environment, an automated solution to deploy to all of these environments has to be elaborated. Unfortunately, the only one available with the product is not meant for these deployments. SharePoint Features and SharePoint Solution projects are a big key to the solution but may be incomplete depending on your scenarios (as they have been in my experience). I will cover Features, Solutions, and alternatives further.

The main observation on what I call IT focused lifecycle is that the "SharePoint Designer" role is not a heavy role and updates to page layouts are streamlined through tested IT deployments. Designers aren't able to use Designer directly in Staging or Production. Let's see the HTML oriented lifecycle.

HTML oriented lifecycle

What I call an HTML oriented lifecycle is where your web site requires frequent layout updates. Your designers are likely to be close to the authors rather than the developers. This can be typical for highly frequented Internet web sites. Essentially, the same roles seen with the IT lifecycle has to be fulfilled, however, since developers are higher up in the chain, the "bubbling down" of their work may not be as obvious and less controlled.

Developers

Developers essentially do the same thing as in the IT lifecycle. Integrated testing is done similarly as before on a centralized server that receives more frequent updates from the designers. The integrated testing environment acts as a staging for the development artifacts.

Web integrators

The main difference of this lifecycle reside here. Web integrators are doing daily updates to layouts on an Authoring Site Collection. They might even be "power authors" that also manage content. They often work directly in the production environment albeit a different Site Collection that may or not be in a different MOSS farm than the Publishing Site Collection.

This allows them to easily deploy layout updates without requiring an IT deployment that is usually more complicated or constrained.

Staging

The real staging environment is really the Authoring Site Collection since this is the origin for all layouts.

Production

This environment will be the same as before with the exception that SharePoint Designer is allowed on the Authoring Site Collection.

Summary

As you can see, there are similarities in the 2 lifecycles, however, from an architecture perspective, there are differences when you want to extract designer updates. For example, in the IT lifecycle, we can link the integrated environment with a Source Control (Team Foundation Server for example) and generate builds. The same does not easily apply to the HTML lifecycle as your source control environment is unlikely to be in the same network environment and you will rather want to schedule extracts of the graphical artifacts, copy them to the development environment, and then create a build for developers.

Also, while I've seen both topologies in the field, one could debate why would designers need to update layouts so frequently. In my opinion, if you design your layouts correctly and provide the correct level of customization, you should probably not need updates frequently and you might be able to work out with the IT oriented lifecycle. I wanted to discuss about the 2 main scenarios I've experienced and provide solutions for both.

Now that we have seen the 2 lifecycles, I want to discuss Content deployment quickly to assert why it isn't a viable solution for deploying on all environments.

A primer on SharePoint's Content Deployment

Content Deployment is a SharePoint feature that allows a complete site collection to be deployed from a source to a destination. These 2 environments can be in the same farm or a different farm. This is done through the Operations tab of the Central Administration web site and requires simple configuration. Basically, you create a path between 1 or 2 Central Administration servers (i.e.: 1 or 2 farms), and then you can create jobs to copy content on a scheduled or manual basis.

The primary requirement is that the target or destination has to be a site collection with the Blank Site template (of the same language of the source) and that all activated features to be installed but not activated at the destination. When the first content deployment runs, it will copy all libraries and items to the destination and then activate the same features as the source.

This is very important and to explain it easily, take a look at the Publishing feature. If you activate it, it creates the out of the box master pages, page layouts, and styles for a Publishing Web Site. If you activate it before doing the first content deployment, it will create a "Style Library" with a different GUID than the source. If you were to use content deployment, it would fail as there would be an item with the same name at the source with a different GUID and it won't accept to deploy it. This is the same if you create custom features.

So in essence, since Content Deployment is used to have 2 sites that will be exactly be the same, it's primary use is for having a "staging authoring environment" where you author and test everything. Then Content Deployment is used to update the production environment according to a schedule or a manual request. Note that there are 3 variants of Content Deployment:

Through the Central Administration web site
Through the STSADM -o import and STSADM -o export commands
Through the APIs (Stephan Gossner wrote a great article about the APIs on his blog : http://blogs.technet.com/stefan_gossner/archive/2007/08/30/deep-dive-into-the-sharepoint-content-deployment-and-migration-api-part-1.aspx.

A deployment job consists of 3 primary steps :

Export (at the source). This operation may be computer intensive.
Transport (between the 2). This is simply the file copy, however, there's a related article that can affect performance and I had to apply the fix for a customer. It made the transport much faster.
Import (at the destination). This operation is not so computer intensive, yet very long to process. I have cases where it takes less than 5 minutes to export and transport, and then 1h30min to import.

The Central Administration web site's Content Deployment interface allows you to schedule deployment jobs easily. Unfortunately, you cannot separate the 3 steps so it's good if you only need to deploy to one environment.

Unfortunately, this is usually not an acceptable scenario to update all environments such as Development or even Integrated Testing for the following reasons:

You do not want multiple exports;
The import process is long;
It's not meant for destinations where you are also authoring on it (i.e.: developers testing their new web parts);
It's not easy to "label" (as in Source Control labelling) a deployment.

Because of these concerns, other scenarios such as SharePoint Features, SharePoint Solution (WSP) projects, and the SharePoint APIs have to be explored and the following sections will elaborate on my findings for these technologies.

Note, Chris O'brien also have great posts on Content Deployment and Features.

Solutions for deploying SharePoint artifacts

Now that we have a better understanding of what we are up against, let's look at what solutions are available to us. Here are the possible solutions :

Content Deployment
Backup/Restore
Export/Import through SharePoint Designer (i.e.: Personal Web Package)
Publishing through SharePoint Designer
SharePoint Solution Generator
SharePoint Features
SharePoint Solution Packages (WSP)
SharePoint APIs

There may be other possibilities that I'm unaware of but these are the ones I've tested with. Note that this was done pre-SP1. I'll work by elimination (and reduce the list dramatically :)):

Content Deployment : As stated previously, it's good only for equal environments so it's not practical for our scenarios;
Backup/Restore : essentially the same as Content Deployment albeit much faster but less user friendly, not "scheduled", but usually with less errors/issues;
Export/Import through SharePoint & Personal Web Package : My experience with it was ... very buggy. I simply could not make it work correctly; for example, it created a 2nd Master Page Gallery (after renaming the original) at the destination when I tried it. It didn't seem like a reliable solution, at least, before SP1;
Publishing through SharePoint Designer is similar to Content Deployment. I'm not sure if it uses the same Web Services or not, but the end result is the same with a confusing user experience;
SharePoint Solution Generator : This tool is available through the WSS 3.0 Tools. It's great to create site definitions, however, it doesn't work with the Publishing templates and isn't meant for WCM.

So we are left with 3 plausible solutions:

SharePoint Features
SharePoint Solution Packages (WSP)
SharePoint APIs

Let me start by summarizing each of them:

A primer on SharePoint Features

SharePoint Features are basically what SharePoint uses to deploy functionality to a SharePoint site. For example, the Publishing feature is activated on a Blank Site to create a Publishing Web Site. This will add the Style Libraries and the Master Pages gallery along with modifying the look & feel of your web site. For example, the Site Actions will contain different actions than for a Collaboration web site.

A SharePoint Feature consists of a manifest Xml file that is deployed in the 12 hive. That manifest in turn specify what actions need to be done to the web site. These actions can be from simply copying files (i.e.: page layouts, master pages, CSS, images, etc.) for creating the baseline of your web site, it can create site columns and content types, as well as even executing any .NET code you want through Event Handlers. On paper, it's the solution and SharePoint itself uses it.

Unfortunately, creating features can be challenging as it requires knowing CAML. CAML is the language definition for configuring any features. While MSDN provides documentation for lots of the Xml elements available, MSDN is meant more as a reference, not for training. Also, while I still create SharePoint Features today, I do not use is exclusively anymore as I found some bugs & quirks that I couldn't workaround well. Particularly, using it to provision the baseline files. For some other scenarios (Timer Jobs or Site Actions for example), Features are still the best solution.

A primer on SharePoint Solution Packages (WSP)

WSP files are really CAB files that SharePoint knows what to do with it. The goal of SharePoint Solution Packages are to deploy a package once to the MOSS Farm and then it will deploy and execute what's required for each server of the farm. It includes even more features such as an interface in the Central Administration where you can schedule when you want a solution to be deployed or even retracted! And to sugarcoat it, if you add servers to a farm, it will apply the solution packages automatically.

** Update ** : A colleague pointed me to the fact that solution will even automatically deploy after a server crash. His scenario was that a solution was deployed in a farm and one of the servers had crashed. When they fixed the servers, they simply had to bring it back online and the solution was automatically deployed to that server without any manual operation (no wizard execution). ** /end Update**

Common scenarios include copying file to the GAC, modifying the web.config (using SPWebConfigModification class), copying resources files, and copying physical files to the web site. SharePoint Solution Packages are a must to most SharePoint deployments.

A primer on SharePoint APIs

SharePoint APIs might seem to be a curious addition to this list as it's not really a deployment mechanism, or at least, not yet a user-friendly one. However, while there may be bugs with SharePoint Features and Content Deployment in some cases, the SharePoint APIs is very robust and complete. More so, strong developers can find more code samples as well as using tools such as Intellisense to guess possible solutions.

In your deployments, you are likely to have to use APIs for Event Handlers or web.config modifications, at the least, but my reason for mentioning it as a deployment alternative is that we ended up replacing some "Feature" scenarios that we had issue with for a complete SharePoint API solution. Here's some operations that we automated, by creating custom STSADM extensions, for deploying complete Publishing web sites:

Create Site Columns including Lookup columns
Create and update Content Types
Create and update Master Pages, Page Layouts, CSS, Javascript, images, and other "baseline files" for a web site
Create Publishing Sites and Pages with content (great for creating content-less (or near content-less) landing pages as well. We also create an input file to create complete files with content so that, at a click, we can create a fully working development environment with content)
Create SharePoint lists and referenced by Lookup columns
Update Available Page Layouts for a publishing site
Update Default page for a publishing site
Create SharePoint Permission set and groups
Update Versioning information for publishing site (i.e.: number of major & minor versions, activate/deactivate workflow requirement, etc.)
Update a site's master page
Update Publishing navigation settings (hiding sites/pages, inheriting from parent (or not) navigation, etc.)
and more!

Since lots of these cannot or should not be done by Features, we had to drill the APIs extensively. When strong (showstopper) issues came up for some Features scenarios, we shifted our focus to the APIs. Let me go into more details on the pros & cons of Features versus APIs.

Features versus SharePoint APIs

Features can be a great reusable way of deploying new functionality to a SharePoint site. A SharePoint Feature is a set of functions, settings, and files to be added to a site (there are different levels of features but I'll keep it simple for now) when activated. It's a great reusable mechanism for lots of scenarios such as adding a Custom Site Action, adding a Timer Job, and to some level, adding columns and content types. You can take a look at this MSDN article for starting to work with Features : http://msdn2.microsoft.com/en-us/library/ms460318.aspx.

On one of my projects following the IT Lifecycle, we tried to use Features as much as possible since it was, for us, how reusable deployment artifacts should have been. Unfortunately, we ran into some severe issues when using them for Columns/Content Types and files such as Page Layouts (ASPX), Master Pages, CSS, JavaScripts, and images. While we have successfully used features for other artifacts as mentioned in the previous paragraph, our feature that creates the "web site's baseline" with all the right master pages using custom CSS, images, and scripts; is being phased out. Here are the issues that we ran into:

Sometimes, the BlobCache may corrupt some files that are deployed through a feature. This requires some explanation as the first goal of the BlobCache is to physically cache a file on the Web Front-End instead of fetching it from the database all the time. Since the feature already brings the file on the disk (in the 12 hive), BlobCache wouldn't be necessary. Unfortunately, the BlobCache is also the mechanism by which the Cache-Control Http header is set and that header is the one that let a browser (or proxy) cache a file instead of making it expire right now. We experienced 2 types of corruptions, one where the files were half-correct, and half-blacked-out; one where JS and CSS were simply cropped and were missing content. Note that all files were correct in the feature but simply not in the BlobCache folder. Luckily, we use Content Deployment from Authoring to Publishing and that makes it so that the files are now in the database and there is no corruption then. While MOSS itself uses files in features (i.e.: _layouts or styles library), there is a special Virtual Directory for them and the BlobCache doesn't treat them the same way.
If you want to export (i.e.: content deployment) a site collection (the production/publishing environment) and import them in a test environment, a feature update will not work since the Import process will put the files in the database (similar to unghosted files, but really, it's not reading the feature at that point).
If you update a Content Type and add a new column to it, the column will not be propagated to all libraries (this is not related to the checkbox to force inherit settings on child content types). Let me explain a bit more, when you create a Site Column or a Content Type at the site collection level, and then you create a page that uses that content type in a sub-site's Pages library, the content type and columns are copied to the library. When you add a column to a content type through the GUI or the APIs, it will automatically add the column in all libraries where the content type is used. Unfortunately, a Feature simply does not do it.

While number 1 was annoying for the authoring environment, and number 3 wasn't happening often enough; number 2 is the real issue where we hadn't found a workaround yet. Plus we knew that the APIs work great for all of them. What we ended up creating was an STSADM extension that reads the same CAML files that this feature used and:

Creates the site columns
Create the content types
Add page layouts and master pages
Add files (CSS, GIF, JPG, PNG, JS) in any libraries

We still package a WSP that runs the commands, actually, we could still use a Feature that adds and removes the files when activated/deactivated (through code) instead of using the CAML definition of provisioned files. In my experience with Publishing Web Site, it's much safer in the database than in physical files. Also, learning how to create columns and content types in .NET, or to add files in the database, is very easy and you'll have access to Intellisense. Learning CAML and try all the scenarios (updating an environment, updating an environment that uses Content Deployment, versioning of documents deployed through features, etc.) is likely to be more challenging and time consuming.

The next section is for detail's sake and because I have lots of questions on Virtual environments for developing in SharePoint.

A note on using Virtual Machines with SharePoint

While SharePoint can work correctly in a virtual environment, you have to be aware that the solution is not officially supported by Microsoft unless it's on Virtual Server. What Microsoft Support will offer is what they call "Best effort support" where they'll help until they cannot reproduce the error and may ask to reproduce the error on a non-virtual environment.

Also, in my experience, I wouldn't recommend virtual environment for production web sites.

For development and up to staging however, I have seen both successful and very sluggish environments. Personally, if going VM, I recommend the following parameters:

Have at least 2 GB of RAM
You can use a single SQL for all developers
SQL will offer better performance if not in a VM
SQL's disks should be physical if possible as I've seen saturated SAN scenarios for having a few MOSS developers

In a coming article, I'll detail how we used Team Foundation Server to synchronize SharePoint Designer users with the main source database which allowed us to easily create WSP packages.

Maxime

Incremental content deployment completes successfully with errors mentioning that the name is already is used

MaximeB — Wed, 21 Nov 2007 02:23:31 GMT

I often get the question as to why we have this error when using an incremental Content Deployment: The specified file is already in used.

Simply, that message appears if the authors have created a file, deleted it, and recreated another file with the same name. All that before a single incremental content deployment (or Quick Deploy for that file) has occured. Basically, the incremental will process the "delete" action after the "create" action and that's the standard error for that. Fortunately, the deployment job will keep going as it's not a fault error; unfortunately, you'll have a 404 when accessing the document.

If you do not want that issue, simply overwrite files instead of delete+recreate. Otherwise, you'll be stuck having to use Full Content Deployment all the time.

**UPDATE** : The new WSS post-SP1 rollup, KB941422, may contain the fix as it seems to fix a similar issue when you recreated a deleted list. I'm hoping it works for list items as well. **/UPDATE**

Maxime

Content Deployment does not fix links (at import) when the source has been deployed with features

MaximeB — Wed, 21 Nov 2007 02:20:08 GMT

I ran into this last week when making a spot check on the "content deployment destination", the links and images that were supplied by authors in the Publishing Html fields were still pointing at the "source". It turns out that Content Deployment doesn't work correctly when using features.

Here's a quick rundown of the issue: The "source" or authoring farm is a single-server SharePoint farm where we have deployed site columns, content types, page layouts, master pages, along with images & stylesheets. The destination is a multiple server farm that simply receives the unactivated feature on a blank site and we use content deployment to bring the content along with activating the feature.

So far, the issue is only on links in RichHtmlField columns for both links and images. Unfortunately, there is no fix for this yet from Microsoft and it's not planned (yet, we are pushing!). The only OOB workaround is to create your links, in your authoring environment, to point directly for your destination. Obviously, that's less than ideal. I'll work on an HttpModule to do a RegEx to replace "source" to "destination" and the issue will disappear in a not-so-elegant-but-works-great way :) I'll post it when I'll have the time.

Maxime

Slow Content Deployment transportation / TCP connection was forcibly closed

MaximeB — Sun, 18 Nov 2007 02:08:02 GMT

We were having issues with slow Content Deployment for the transporting phase and it was sometimes failing with this error:

Publishing: Content deployment job failed. Error: 'System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags) at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

A collegue of mine sent me something about TCP chimney (first time I heard about this!) and a simple command to run in the command line : Netsh int ip set chimney ENABLED. After this, the transporting phase was much faster and without errors. You can find the information about this command from knowledge base 912222.

Maxime

BlobCache woes, how to work (clean) correctly with it, and what to expect when using it with SharePoint features

MaximeB — Sat, 17 Nov 2007 12:59:10 GMT

I ran into an issue with BlobCache lately where it was sending incorrect Http headers. Basically, all files were sending 2 http headers:

Exires : (15 days behind)
cache-control: private, max-age=0

Yes, "exires", not "expires", so not only we have a bug but also a typo :) Anyway, this was ensuring that all static content wasn't properly cached by proxies and browsers and it was impacting performance big time. Now, before going into the details of this, let's explain what is BlobCache first.

The main function of the BlobCache is to cache large objects from the database to the web front-ends. Its configuration is simple, you specify which extensions you want it to trap, a local folder to drop the cached files on the WFEs, an expiration (in seconds), and whether or not to enable it. Sounds fairly simple and it is (until it doesn't work correctly). Now BlobCache has a secondary function which is to send "better" Http Headers to clients so that they can also cache the files along the way. Basically, it will send a "Cache-Control: public, max-age=86400" to the browser and by having it public, it means that anything along the way (proxies) can cache it. The 86400 is the default value for max-age if you don't specify it and it means that the file will expire in 1 day.

In my case, I am using features to deploy the static CSS, JS, GIF, JPG (along with master pages & page layouts); this means that they are already on disk at each WFEs. Technically, I don't care to have BlobCache for those and would only need it for images that authors add to their publishing web site. What I really need for those files is the "cache-control" header that the BlobCache sends. I also have a setup with 2 SharePoint farm: 1) for authors only, and 2) we use content deployment to publish the first farm to the 2nd for visitors.

First problem encountered: BlobCache doesn't always work correctly with the authoring farm. For some (still unknown) reasons, files that are coming from a feature, which means that they aren't in the database, are sometimes corrupted. I had 2 different behaviours: 1) images were half working where I could see parts of the images and the rest was blacked out; and 2) JavaScripts files are simply cropped. I haven't resolved that one yet; I was able to determine that it only was acting like this with files that are already on disk. For example, I unghosted a corrupted file and it was correct at that moment. I also did a "revert to definition" and it kept working (so I'm assuming that it's still in the database but to its original version).

Fortunately, when you use Content Deployment, any files that are in Style Library will be sent to the destination, even if they come from a feature. Of course, it requires the feature to be installed (but not activated) at the destination but it will only activate it at the end of the import process. That means that, at the destination, the files are in the database and I do not have a BlobCache issue when files are in the database. So normally, in the production environment, my cache-control header should be fine.

And it was until it stopped working after a deployment. When it started failing, it was showing the 2 headers I wrote in the beginning. I tried flushing it through the GUI (Site Settings - Site Collection Object Cache - Force flush on this server), deleting the files manually on disk, doing an IIS reset; but I kept having the issue. Then I found the following :

If you have multiple zones (i.e.: you extend a web application), each of them has its own BlobCache folder. It also means that the "Force flush" only works for the zone that you are pointing to in the URL. Also, it only flushes it on the server you are on (in the URL). So if you have a load-balanced URL, play with a host file :)

In my tests, the ideal way to make it work is to, for all servers:

Use the GUI to flush the BlobCache for all zones
Optionally, delete the content of your C:\BlobCache (by default) folder
Use IISRESET

I tried this several times and it was fixing my issues with the incorrect Http Headers and now SharePoint had much better performance. I still have to find out why it's not working with items that aren't in the database (for my authoring environment).

Maxime

Content Deployment : The URL "Style Library/en-us" is invalid

MaximeB — Thu, 11 Oct 2007 01:00:08 GMT

I came across this error yesterday while testing a Content Deployment for a customer and had this error "The URL "Style Library/en-US" is invalid ..." with the stack trace mentioning that it was updating a File or Folder. The kind of message where you say "yeah I know, I'm deploying this so you should bring it up!".

I couldn't find what was causing the issue. For a recap, when you use first use Content Deployment (for a Publishing Site), your destination must be a "Blank Site" of the same language than the source's site collection language. However, a coworker mentioned to me another issue he had with content deployment and the solution was to add another step prior to the first deployment :

For a successful Publishing Content deployment, your destination must have first created a Publishing Site collection, deleted it, and then created the Blank Site. That will add elements to the Web Application and your Content Deployment will complete successfully after that.

I had heard of issues where Resource files weren't fully brought by the Content Deployment, that would also fix it.

Maxime

My STSADM extension toolbox

MaximeB — Sat, 06 Oct 2007 21:34:14 GMT

Here's a listing of custom STSADM extensions myself and my coworkers have created. Feel free to ask questions regarding the namespaces for these. I'd like to thank those who contributed to this : Stéphane Perron, Louis Fournier, and Vincent Trépanier at Orckestra, an outstanding partner for SharePoint, Biztalk, and Commerce in Montreal; and a co-worker Philippe Bédard.

BaseStsAdmExtension

The basic class for all my extensions. It adds up default classes for a common output, verbose output switch, Url switch, and a wait switch to allow a developer to attach to the process.

ContentBuilder

Generates all columns, content types, page layouts, style sheets, site collection images, and master pages in a Feature. The "files" have to be exported (and we have another mechanism for that) in order to import them in another site. This was originally taken from http://andrewconnell.com/blog/articles/MossStsadmWcmCommands.aspx and then updated to fix a few issues and add more elements. You can also find information on Features here : http://msdn2.microsoft.com/en-us/library/ms460318.aspx and here http://www.codeproject.com/spoint/ExtendingSPS.asp.

PreloadWebSites

A simple extension that reads an input Xml file and hits all the specified Urls with HttpWebRequests. This helps loading specific content types & page layouts in the JIT after you ran the Warm up scripts.

PublishAllItems

An extension that will approve items in SharePoint. It has 2 main switches, one to approve "web integrators" libraries (Master pages, Style Library, Site Collection images); and one to approve "Pages" libraries starting a given path and downward. It will also cancel any workflow associated with the element's it's approving. You can also take a look at the one available at http://stsadm.blogspot.com/2007/08/stsadm-commands_09.html as it ended up being very similar to mine.

ResetMasterPages

This one I had to create to automate the master pages settings when using Content Deployment. There is a known bug where, when doing the first Full Content Deployment, the master page parameters aren't kept at the destination. This will bring them over.

ListBuilder

This will read an input Xml file and create custom lists with custom columns. It can also add Lookup site columns and add those columns to content types. This was done because we used Features to add content types but found out that a list was always recreated and used a different GUID. We couldn't provision Lookup columns to them and if we used a Feature to bring a list over, then the GUID was changing every time and causing issues with Content Deployment. It can also provision a BDC column in the list.

SiteBuilder

This extension will create a whole lot of things for a portal : sites with title/description, content pages with content (fields) values, set master pages, set available page layouts, set the content types per libraries (to fix a little bug), and set security (coming soon). It's great when you have features for the containers/artefacts and you want to add a default

AddLocalFile / DeleteLocalFile

We needed a little help to add and remove files other than through Features. Basically, we first had the issue when deploying updates to ItemStyles.xsl and other OOB files. This extension was deleting the OOB files from that web site and then applying the ones from our Feature (and we needed it in ItemStyles and couldn't have a custom file).

AddPublishingWorkflow/RemovePublishingWorkflow

This one is in development;I want to be able to add and remove the Publishing workflow for all sites so that we can switch between requiring a "publish" or an "approve" through the extension. We often want publishing only in development but an approval requirement for other environments; this extension would permit this.

There's also a bunch of custom extensions available by Gary Lapointe at the following address : http://stsadm.blogspot.com/2007/08/stsadm-commands_09.html.

I'll update this list with blog links when I create them and add new extensions.

Maxime