Microsoft SharePoint Developer Documentation Team Blog : content migration

Overview of Backing Up and Restoring Data in Windows SharePoint Services

RickK — Thu, 24 Jan 2008 01:11:00 GMT

Introduction

This post provides some basic facts about backing up and restoring data in Windows SharePoint Services 3.0. It will serve as background information for several developer-oriented posts that I will create in the next couple of weeks.

Note: Unless explicitly stated otherwise, all classes and members referenced in this post are in the Microsoft.SharePoint.Administration.Backup namespace.

What Can Be Backed Up and What Can Be Restored

There are six types of content components built-in to Windows SharePoint Services 3.0 that can be backed up and restored through either the Central Administration application's UI, the Stsadm.exe Command-line Tool or a custom application that uses the Windows SharePoint Services 3.0 backup and restore object model.

· Site collections, each of which might contain multiple Web sites.

· Content databases, each of which might contain multiple site collections.

· Web applications, each of which might contain multiple content databases.

· Content publishing "Web services," each of which might contain multiple Web applications.

Note: This refers to content publishing "Web services" (which are really partitions of content) that are represented in the object model by Microsoft.SharePoint.Administration.SPWebService objects. It does not refer to the functional Web services in the more common sense of "Web service," such as the Alerts (websvcAlerts) service or the Meetings (websvcMeetings) service. For more information about content publishing "Web services," see Server and Site Architecture: Object Model Overview and The High Level Object Model of Windows SharePoint Sevices 3.0 (to be published on MSDN).

· Search Windows service including its databases and indexes.

Note: This does not refer to the Search Web service websvcSPSearch.

· A whole Windows SharePoint Services farm.

If an enhanced functionality product such as Office SharePoint Server 2007 has been installed in addition to Windows SharePoint Services 3.0 and that product includes Shared Service Providers, then they can also be backed up and restored with Windows SharePoint Services 3.0.

In addition, you can create new types of content objects that can be backed up and restored by implementing the IBackupRestore interface.

Limitations

There are some limitations on what can be backed up and restored through either the Central Administration application's UI, the stsadm command line utility or a custom application that uses the Windows SharePoint Services 3.0 backup and restore object model.

· You cannot backup (or restore) an individual Web site, list, or list item except by backing up (or restoring) the entire site collection to which it belongs.

· You cannot back up a Windows SharePoint Services farm's configuration database or the content database of the Central Administration application except by backing up the whole farm.

· You cannot restore a farm's configuration database or the content database of the Central Administration application. The backups of these components that are included in a backup of a farm provide a snapshot of these components at the time of the backup. Such snapshots might be useful for troubleshooting because they can be used to compare with the present state of the components using SQL Server tools. (There is an exception to this point: you can restore the configuration database and the content database of the Central Administration application if you are using the Volume Shadow Copy Service – VSS - of Windows Server 2003/2008).

· You cannot back up the Internet Information Server (IIS) metabase.

The following kinds of content cannot be backed up with either the Central Administration application's UI or the stsadm command line utility, but you can create custom backup solutions with the Windows SharePoint Services SDK that include these types of content.

· Registry keys.

· Files that live on the front-end servers; that is, outside any content database, such as certain master pages, .ascx files, web.config files, and other configuration files.

Types of Backups and Restorations

Backups of a given component can be either full or incremental. In the latter case, only parts of the component that have changed since the last full backup are backed up.

Note: A Windows SharePoint Services Search index cannot be incrementally backed up. If a search index is included in an incremental backup job, the index will get a full backup.

Restorations can either overwrite the original backup source or they can be to a new location. This means that the backup and restore functionality in Windows SharePoint Services 3.0 can also be used as a method of migrating content components.

Two Kinds of Custom Backup Applications

There are two ways to use the Windows SharePoint Services object model to create custom backup applications.

The Main Backup and Restore Object Model

You can create a backup and restore application by using the main backup and restore object model, sometimes called "catastrophic" backup/restore. This is located mainly in the Microsoft.SharePoint.Administration.Backup namespace; but backups and restores of individual site collections are performed with the Microsoft.SharePoint.Administration.SPSiteCollection.Backup() and Microsoft.SharePoint.Administration.SPSiteCollection.Restore() methods.

Interface to the Volume Shadow Copy Service

Windows SharePoint Services 3.0 deployments can also take advantage of the Volume Shadow Copy Service (VSS) in Windows Server 2003/2008. Windows SharePoint Services 3.0 includes a Windows SharePoint Services VSS Writer service that will create shadow copies of native and custom content in the deployment. The service contains a VSS writer that will write shadow copies of all native Windows SharePoint Services 3.0 databases and all custom databases. Non-database custom components can also be registered with the service by using the SPVssComponentDefinition and SPVssDiscoveryHelper classes. It is also necessary that you create a VSS writer for any such non-database custom components.

Note: With the VSS service, you can target only the whole farm or individual content databases for database shadow copy. Individual Web applications and individual content publishing "Web services" cannot be set for shadow copying independently of making a shadow copy of the whole farm. (For information about the meaning of "content publishing Web service" see the first note of this post.)

Content Migration in SharePoint (Post 2):

NickG (msft) — Thu, 06 Dec 2007 02:40:00 GMT

Key Concepts in Selective Migration

Picking up where the first post left off, today's foray explores the basic concepts of SharePoint content migration. Most of these concepts come into play in nearly every migration scenario where you use the SharePoint.Deployment namespace APIs. This post covers the following concents:

Content selection (also known as "cherry picking")
The content migration package (.cmp) file
Object identity (that is, retaining object identity on import)
Reparenting objects
Handling content types when migrating a site collection
Handling workflows when migrating

Content selection for export

Once the initial full migration has occurred and the source and destination sites are established, selective migration relies on logic in your custom code for selection criteria that determines which items are selected for migration, and when. Also known as "cherry picking," this is the process through which specific items on the source site collection are migrated to the destination site collection based on selection criteria that is specified in the custom code.

In the basic scenario, logic in a code module associated with a timer job runs on a recurring schedule (for example, every 24 hours at 3:00 a.m.); the module examines date stamps or change logs (or both), then selects and exports only those files that have changed since the last migration.

The operation relies on the identity of objects in the source and destination site collections. At import time, the destination recognizes objects by GUID identifiers and then updates their corresponding files accordingly.

Content migration package (.cmp) file

The content migration package is a compressed cabinet (.cab) file that uses the .cmp file extension. The migration package (.cmp) file aggregates data for export from the source site, along with site structure and dependency data, and stores it as compressed, serialized XML files.

While the export operation compresses site data by default, you can override this behavior and export uncompressed files by changing the SPDeploymentSettings.FileCompression property to false. This property is on the SPDeploymentSettings object.

When using file compression, you must must specify the name of the compressed content migration (.cmp) file using the BaseFileName property on the SPExportSettings object. The FileLocation property specifies where the compressed file is located on the source server.

By default, the .cmp files are limited to 24 MB in size, although you can change this using the FileMaxSize property. When site data exceeds the 24 MB limit, your site data is separated in to two or more migration files. However, where a single file exceeds the maximum file size, the operation resizes the .cmp file to accommodate the file. You can have any number of .cmp files.

Object identity

SharePoint objects are identified by GUID on the source server. Upon import, by default, the objects are assigned new GUIDs on import to the destination. To support selective migration scenarios, however, you must change the default and retain the identity of the object that you are exporting. This is very important, because the only way that files can successfully update their corresponding versions on the destination site is for the destination to recognize the object by its identifying GUID.

You can retain object identity by setting the RetainObjectIdentity property on the SPImportSettings object to true.

Note To use the RetainObjectIdentity property to support selective migrations, you must also set the ExportMethod property to the value ExportChanges.

Retaining object identity should be used carefully, and in limited scenarios. It most commonly supports a publishing scenario in which source and one or more destination servers are mirror images — for example, where a development or test server is migrating updates to one or more production servers.

If you are creating a new site that you know in advance will receive file updates on a regular basis (for example, from a development or test server), you can enable a smooth link between the two by first doing a full migration from the source to a completely blank site, but with RetainObjectIdentity property set to true. It is very important, then, that you never modifythe destination. The initial migration established the mirrored hierarchy and mappings; any modifications to the destination can break the linkage. If this relationship gets broken, your only recourse is to delete the destination and then reestablish the linkage as originally done.

Note In the scenario described above, you can use Stsadm.exe to complete the export portion of the migration, since it does not support the retention of object identity. However, you must use the migration APIs to complete the import portion of the migration in order to retain object identity.

SharePoint content databases do not permit duplicate GUIDs or file names, so you must be careful when implementing this property. You should not retain object identity when you are simply updating files on the destination server, because the import operation cannot determine whether a file is new or an update, and will therefore place a duplicate copy of the file on the destination.

This happens because the migration process uses GUID identifiers, but only down to the level of the item collection. Beneath that level, that is, for individual files, the system uses the file name. Consequently, the import simply copies over the updated version of the existing file and leaves the original intact, potentially creating duplicates. When the destination server then encounters duplicate file names, it cannot resolve the ambiguity and becomes unstable.

Important When deleting files, it is very important to delete the files on both the source and destination servers. Not doing so can cause duplicate files names and an unstable site.

To support field deletion (a feature that deletes previous versions of items on the source server when the migration updates and deletes the same file on the destination), the RetainObjectIdentity property must be set to true.

Reparenting

Now let's consider a much different scenario. Instead of a publishing scenario, in which objects on the source server have to map to their equivalent files on the destination server, consider instead a scenario in which a specific subweb or list item is migrated and you want to place it at a different place in the destination hierarchy. This is a very common requirement and you handle this by reparenting the object (or objects) on import.

It's important to note that, as discussed in the section on retaining object identity, any time that you do not retain object identity, a migrated object will need a new parent specified on import. It doesn't matter if the destination site collection is the same site collection as the source or even if it is in the same database as the source site collection. It also doesn't matter if the destination site collection is in the same or a different farm as the source site collection.

If you export an item from a database without exporting its parent, then the item that you've exported will become orphaned in the content migration package, but that's not necessarily a problem. A package can contain multiple orphaned objects.

Having orphaned objects in the migration package is not a problem because the import method allows us to define a new parent for each of these orphaned objects. This is what is meant by "reparenting." However, objects that are exported with their parent objects keep their relationships during migration. So, for example, if you export Web A and Web B is a sub web of web A, then you can only change the parent of web A. The parent of Web B will remain as Web A, because Web B.

There are two ways to reparent orphaned objects when importing. In the first, you import all of your orphaned objects into the same subweb, while in the other method you assign parent objects individually to each orphaned object.

Reparent by importing all orphans to the same subweb

You can reparent your orphan objects by simply importing all of them into the same subweb. However, this only works if all of the orphan objects are destined for the same subweb. For example, this is the preferred method if you have exported two or more list or document library objects and you want to import them all into the same subweb. This works even if the export objects are from different source sites. Here is some sample code for doing this. Of particular note are the WebUrl and RetainObjectIdentity properties.

SPImportSettings settings = new SPImportSettings();

settings.SiteUrl = "http://localhost:2001";

settings.WebUrl = "http://localhost:2001/MyDestinationWeb";

settings.FileLocation = @"c:\export";

settings.FileCompression = false;

settings.RetainObjectIdentity = false;

SPImport import = new SPImport(settings);

import.Run();

In the above example, the WebUrl property defines the site (the SPWeb) that becomes the new parent for all orphaned objects in the export package.

Note You cannot use this method to reparent orphaned objects if the migration package contains orphaned documents; SharePoint site cannot become the parent of a document document object (only list or folder objects can parent documents).

Note, too, that the RetainObjectIdentity property is set to false. This is important because keeping this property as false (which is the default) cases the operation to assign a new GUID to the orphaned objects, which is essential for reparenting those objects.

Reparent by assigning new parent objects individually

Assigning new parents to orphaned objects individually is much more flexible than the previous method, but also requires more coding. In this approch, you must intercept the import operation to assign a new parent to each orphaned object after the operation has accumulated a list of all orphaned objects in the migration package. You can do this by implementing a custom event handler, as in the following example:

static void OnImportStarted(object sender, SPDeploymentEventArgs args)

{

SPSite site = new SPSite("http://localhost:2001");

SPWeb web = site.RootWeb;

SPImportObjectCollection rootObjects = args.RootObjects;

foreach (SPImportObject io in rootObjects)

{

io.TargetParentUrl = web.Url;

}

web.dispose();

site.dispose();

}

This approach uses the RootObject collection of the event arguments to aggregate all and then reparent all of the orphan objects. Actually, we do much the same thing that we did when we imported all of the objects to the same subweb, that is, we defined a specific site (SPWeb) as the new parent of all included orphaned objects. However, notice how you can extend the logic in the event handler to, for example, assign different parents to different orphans based on object type, as shown here:

static void OnImportStarted(object sender, SPDeploymentEventArgs args)

{

SPSite site = new SPSite("http://localhost:2001");

SPWeb web = site.RootWeb;

SPList list = web.Lists["MyDocLib"];

SPImportObjectCollection rootObjects = args.RootObjects;

foreach (SPImportObject io in rootObjects)

{

if (io.Type == SPDeploymentObjectType.ListItem)

{

io.TargetParentUrl = list.RootFolder.ServerRelativeUrl;

}

if (io.Type == SPDeploymentObjectType.List)

{

io.TargetParentUrl = web.Url;

}

...

}

web.dispose();

site.dispose();

}

There is a great deal of flexibility here. In addition to reparenting based on the object type, you could instead look at the original TargetParentUrl, for example, to obtain the location of the source and then include that as part of your reparenting logic.

The code example below shows us how to hook up the event handler to the import operation. You can do this as follows:

static void ImportDocLibItem()

{

SPImportSettings settings = new SPImportSettings();

settings.SiteUrl = "http://localhost:2001";

settings.FileLocation = @"c:\deployment5";

settings.FileCompression = false;

settings.RetainObjectIdentity = false;

SPImport import = new SPImport(settings);

EventHandler<SPDeploymentEventArgs> eventHandler = new EventHandler<SPDeploymentEventArgs>(OnImportStarted);

import.Started += eventHandler;

import.Run();

}

Notice that you need to register the event handler with the import class before you start the import operation.

There are several more event handlers that you can register and use during import; these are explained in more details at help topics for SPImportEvents and for SPExportEvents. Similar events can also be registered for export, if required:

Content types (handling)

Migration operations that use the export and import commands include your content type definition schema files in the export package. However, in cases where content type definitions hold references to custom SharePoint features, you must manually install and configure on the destination compute the feature for which the content type holds a reference. Features that are native to the SharePoint server environment will have their type definition references restored automatically.

On starting the import operation, you will receive a warning message that alerts you with the identity of custom features for which references must be recreated. You must resolve these references after completing the import.

For more information about content types, see Content Types. For more information about SharePoint features, see Working with Features.

Workflows

Workflows are not included in export packages so when you initially set up a migration target, you must copy any custom workflows that you have onto the destination server and reconfigure the workflow association table. This includes creating (or restoring on the destination) the workflow definitions files.

Of course, manually restoring your custom workflows on the migration destination is only necessary when you initially set up your destination server. Subsequently, when doing selective migrations in a publishing scenario, it is advantageous that workflows are excluded from the migration package.

For more information about managing workflows, see the following:

· Introduction to Workflows in Windows SharePoint Services

· Workflows in Windows SharePoint Services

Content Migration in SharePoint

NickG (msft) — Sat, 01 Dec 2007 01:20:00 GMT

Around the time work started on a long whitepaper about migrating content files from SharePoint server "A" to SharePoint server "B" using the migration and deployment APIs in the Microsoft.SharePoint.Deployment namespace, I came across a series of blog posts by Stefan Goβner that covered the subject matter in detail. Many of you have probably seen Stefan's work: Deep Dive into the SharePoint Content Deployment and Migration API, Parts 1-5.

I contacted Stefan and he agreed to collaborate. He's allowed me to use some of his code examples, and to expand on some of his points. Additionally, I've collaborated with the feature PM and lead developer on the migration/deployment APIs here at Microsoft, Patrick Simek, so the whitepaper treats the subject quite a bit more broadly than does Stefan's blog posts.

I'll chunk this article up into segments for posting to this blog. Comments are DEFINITELY welcome. I'll be converting much of the content of this article (and these posts) into Help content in the SharePoint SDK documentation, so anything that helps to improve the quality of the content is highly desired. Here's the first section:

The ABCs of Content Migration (Post 1)

Many scenarios require moving content from one SharePoint site to another. When the scenario calls for a full migration (that is, moving the entire contents of a SharePoint site or site collection), the task is relatively simple. Typically, you use either of two main approaches:

Use the export and import operations in the Stsadm.exe utility to migrate data from one site to another.
Alternatively, use SOAP calls to the Sites.ExportWeb and Sites.ImportWeb methods that are implemented on the Sites Web service to migrate data.

However, both of these methods have limitations. Both limit you to migrating only a full SharePoint site, or site collection. In addition, neither allows you to retain object identity during the migration operation. Retaining object identity is an essential feature of selective migration. Using Stsadm.exe and Sites Web service have other limitations as well.

Consequently, content migration scenarios that require you to export only selected content, or that require you to automate or customize migration operations, there is only one approach: you must write a custom solution that uses the APIs in the Microsoft.SharePoint.Deployment namespace.

Note Selective migration operates when there is an established source and a recognized destination for a known quantity of content. That is, selective migration requires that a full migration was initially performed so that the destination is a mirror image of the source.

Selective migration applies, typically, to content that needs to be migrated from server to server based on factors such as content version (current vs. future), time stamp, and content state (approved vs. in review, for example). Selection criteria provides a high degree of granularity from the scope of the site collection down; that is, you have selection control at the scope of the Web, the list, folder, and list item.

You can have any number of .cmp files in a migration operation, and you can also have multiple destinations. However, the objects contained in a given content migration package (.cmp) file must originate from a single site collection.

Typical Migration Scenarios

The APIs in the Microsoft.SharePoint.Deployment namespace provide a rich migration toolbox that gives you an enormous degree of flexibility to support wide ranging migration scenarios. Following is a list of migration and deployment features that are supported in Windows SharePoint Services 3.0. This list is only a high-level, generalized summary of supported migration scenarios. The Deployment APIs are sufficiently rich to support any number of special circumstances that you might face.

Export an entire site collection (that is, do a full migration).
Export a specific site inside a site collection, including or excluding content in subsites as needed. The deployment APIs allow you to include or exclude dependencies.
Export a SharePoint list or document library, or even of a specific folder inside a document library.
Export a single list item or document from a document library. This provides the extreme migration granularity that enables so much flexibility.
Export object dependencies like images or attached files.
Generate your export file as a compressed export (.cmp) package, or as uncompressed files.
Throttle exports by specifying a maximum size for the compressed export package file (multiple export files will be created if required).
Support incremental exports of items that are selected for export based on change tokens. This allows you to automate exporting all items that have been created, changed, or deleted after the timestamp specified in the change token.
Import the exported content with or without preserving object identity—that is, you can configure objects in the export package to retain their object GUID. This is a requirement for selective migrations.
Import the migration content under the same or a different parent in the destination content database. Moving items to a different position in the site hierarchy is an operation is known as reparenting.
Do link fixup during import.

In my next post we'll continue with this high-altitude look at the subject of content migration. There, we'll talk extensively about key concepts in the migration/deployment feature area.