Microsoft SharePoint Developer Documentation Team Blog
The Official Blog of the SharePoint Developer Documentation Team

January, 2008

  • Microsoft SharePoint Developer Documentation Team Blog

    Synchronizing with Windows SharePoint Services, Part 1

    • 8 Comments

    Introduction

    There may be occasions where you will need to synchronize third party programs with Windows SharePoint Services. GetListItemChangesSinceToken, introduced in Windows SharePoint Services version 3.0, makes this synchronization much more efficient.

    This post explores using GetListItemChangesSinceToken to synchronize with Windows SharePoint Services version 3.0. The article details the queryOptions used by GetListItemChangesSinceToken.

    The follow on post will discuss conflict detection issues; and document best practices for clients who want to synchronize with calendar items, documents, and tasks in Windows SharePoint Services version 3.0.

    Please forgive the length of this entry, the various tables and descriptions I've included will be in future updates of the SDK.

    Synchronization Using GetListItemChangesSinceToken

    You can use GetList to download a list; however, the most efficient way to synchronize information is to download only those items that have changed since the last synchronization occurred. In Windows SharePoint Services version 3.0 this can be done by calling the GetListItemChangesSinceToken Web method.

    Sending a GetListItemChangesSinceToken request without including a token returns the list schema, the full list contents and a token. The token represents the moment when those changes were requested. By including this token the next time you call GetListItemChangesSinceToken, the server returns only those changes that have occurred since the token was generated.

    If the list schema itself has changed, GetListItemChangesSinceToken returns the entire list schema, the full list contents, and a token.

    #GetListItemChangesSinceToken Request and Response

    Here is a listing of the GetListItemChangesSinceToken Request and Response and an explanation of the very useful queryOptions parameter.

    Request

    public SoapXml.SoapXmlElement GetListItemChangesSinceToken(

                string listName,

                string viewName,

                SoapXml.SoapXmlElement query,

                SoapXml.SoapXmlElement viewFields,

                string rowLimit,

                SoapXml.SoapXmlElement queryOptions,

                string changeToken,

                SoapXml.SoapXmlElement contains)

     

    Name

    Description

    listName

    This can be the list title, but using the list id (GUID) results in better performance.

    viewName

    Not supported.

    query

    A CAML query similar to the CAML query used in GetListItems and SPQuery. Refer to GetListItems and SPQuery in the SDK for more information.

    Note   Not intended to be used with the contains parameter.

    viewFields

    Set of field references for each field required in the response. <ViewFields /> returns all fields in the list. A Properties='true' attribute will separate the MetaInfo field into its separate decoded properties.

    rowLimit

    The maximum number of values to return.  Used for paging.

    queryOptions

    Set of options related to the query (see details below).

    changeToken

    An opaque token used to determine the changes since the last call.  This token should never be parsed or constructed, as its format may change in the future.

    contains

    A CAML filter applied to the query results.  This is the Where clause in a SPQuery.

    Note   Not intended to be used with the query parameter.

     

    Query Options Parameter

    The queryOptions element can contain a variety of tags which modify the query.  For Boolean options the default is FALSE, only a value of TRUE (must be uppercase) enables them.

     

    Available Query Option Tags

    Query Option Tag

    Description

    <Paging ListItemCollection

    PositionNext="X" />

    X is an opaque token used to determine the page of items to return.  Like the changeToken value, this should never be parsed or constructed.

    <IncludeMandatory

    Columns>

    Ensures that fields defined as required are included, even if not specified in the viewFields. This option may be misleading because Windows SharePoint Services actually has a separate set of mandatory fields that are always returned independently of this option.

    <RecurrenceOrderBy>

    This is a requirement for some calendar programs. For each recurring series, the master item is returned first and then all exceptions. This is a special internal ordering that is applied ahead of any other ordering.

    Note   This should not be used unless your program explicitly requires it.

    If the view has a field of type Recurrence, the list will be ordered by fields of reference type UID, EventType and StartDate in the definition of the recurrence field.

    <ExpandUserField>

    Special rendering for the user field values that makes them include the login name, email, SipAddress, and the title when present. This causes a user field to behave as a multi lookup field.

    The lookup fields used in the expansion are "Name", "EMail", "SipAddress" and "Title". The values are separated by ,#. Any commas in the lookup field name are encoded as ,,.

    These values occur in the normal field data for each item. 

     

    Examples of ExpandUserField

    <ExpandUserField>FALSE</ExpandUserField> looks like:

    ows_Author="1;#Admin AdminName"

     

    <ExpandUserField>TRUE</ExpandUserField> looks like:

     ows_Author="1;#Admin AdminName,#login\name,#email@address,#sip@address,#Admin AdminName "

     

    More Query Option Tags

    Description

    < DateInUtc>

    Date fields are returned in UTC format and zone.

    UTC is in GMT time zone an ISO6801 format: 2006-10-04T10:00:00Z

    Normal is in local (server) time zone and the same as above with the T and the Z replaced by spaces.

    <ViewAttributes

    foo=bar />

    All attributes of this option will be used as view attributes of the view schema. The most commonly used is Scope="RecursiveAll" that defines the view as flat instead of scoped to a particular folder.

    <Folder>

     

    Set the root folder scope of the view. This is a server relative URL.

    <Folder>Shared Documents/foldername</Folder>

    <MeetingInstanceId>

     

    Defines which meeting instance to sync with if this list is part of a meeting workspace. -1 should be used for all instances unless the client wants to filter for a particular instance.

    <IncludePermissions>

     

    One way a client can request individual item permissions.

    <IncludeAttachmentUrls>

     

    Changes the value returned for the Attachments field from a Boolean to a list of full urls separated by ;#

    <IncludeAttachment

    Version>

    Used in conjunction with IncludeAttachmentUrls, IncludeAttachmentVersion also returns the GUID and version number used for conflict detection on update.

    <RecurrencePattern

    XMLVersion>

    v3

    </RecurrencePattern

    XMLVersion>

     

    Used to maintain backwards compatibility, RecurrencePatternXMLVersion changes the value of a RecurrenceData field to NOT return <V3RecurrencePattern /> when it contains elements only present on a version 3 pattern.

    Without this tag, recurrence patterns that were not present in Windows SharePoint Services version 2 are sent as <V3RecurrencePattern />. Including this tag means that recurrence patterns new to Windows SharePoint Services are sent correctly.

    <ExtraIds>

    1,4,23

    </ExtraIds>

    Request extra items to be included on the returned set regardless of whether they changed or not. The common use of ExtraIds is to specify the IDs of the folders you're syncing if you were in a doclib and chose "connect to..." on a folder rather than on the entire doclib.  This way you get the folder name and can tell when it is renamed.

    Note   This should only be used with a change token.

    This allows a client to sync to one or more folders and detect if any folder above the hierarchy was deleted or renamed.

    Folder names are not returned unless some changes are done to the list and the query to fetch changed items also uses IDs.

    <OptimizeFor>

    The two values supported are:

    • ItemIds
    • FolderUrls

    ItemIds is the default as long as a query or recurrence order is not requested and optimizes our SQL query with an ID order.

    FolderUrls optimizes a sync filtered to the flat contents of one or more folders by optimizing the SQL query with a DirName, LeafName order.

    <OptimizeFor>ItemIds</OptimizeFor>

     

    Default Query Options

    If the queryOptions parameter is absent, the following default options are used:

     

    <RecurrenceOrderBy>TRUE</RecurrenceOrderBy>

    <ViewAttributes Scope="Recursive" />

    <DateInUtc>TRUE</DateInUtc>

    <IncludePermissions>TRUE</IncludePermissions>

    <IncludeAttachmentUrls>TRUE</IncludeAttachmentUrls>

    <IncludeAttachmentVersion>TRUE</IncludeAttachmentVersion>

    <RecurrencePatternXMLVersion>v3</RecurrencePatternXMLVersion>

    <ExpandUserField>TRUE</ExpandUserField>

    <MeetingInstanceID>-1</MeetingInstanceID>

     

    Note   Clients should always specify a query option; failing to do so will negatively impact scale performance.

    Response

    A GetListItemChangesSinceToken response is delivered in the following format:

     

    <listitems [list and global properties] [namespace declarations]>

      <Changes [MoreChanges="TRUE"] [LastChangeToken="X"]>

        [Changes]

      </Changes>

          <rs:data ItemCount="N" [ListItemCollectionPositionNext="X"]>

      [Data]

      </rs:data>

    </listitems>

    Note   X is an opaque token used to determine the page of items to return.  Like the changeToken value, this should never be parsed or constructed.

     

    Namespace declarations

    Any XML namespaces used below are declared here.

    List and global properties

    These include configuration properties, alternate URL information, and list permissions.

    List and global properties table

    Property

    Definition

    MinTimeBetweenSyncs

    A server parameter that represents the minimum amount of time between user-initiated or automatic synchronization. The value represents a time in minutes.

    Note   Clients should respect this value even if the user initiates synchronization manually.  That is, if this is set to 5 minutes, clients should only send one request per 5 minutes even if the user repeatedly clicks "send/receive".

    RecommendedTime

    BetweenSyncs

    The recommended minimum amount of time between synchronizations. This should specifically be respected for automatic syncs. Clients should never automatically synchronize more often than this.  User-initiated synchronizations can override this interval.

    MaxBulkDocumentSyncSize

    The total size of content to be synchronized to the client. The default is 500 MB. You get the URL and metadata for each document when you call GetListItemChangesSinceToken, but you then need to do an HTTP GET to retrieve the actual document contents. Setting this value to high may degrade performance.

    AlternateUrls

    Alternate URLs are listed in the following zone order, delimited by commas: Intranet,Default,Extranet,Internet,Custom

    EffectiveBasePermissions

    The permissions on the list as returned by SPList.EffectiveBasePermissions.ToString().

    Changes

    Changes consist of a list of change events that need to be specially handled by the client.

    These are taken from our internal change log when a change token is supplied. For a full synchronization (no change token) we still return the Changes tag with the current change token.

    The limit on the number of updates returned from a change token is 100. The MoreChanges attribute indicates that the last change token is not current. More changes were done on the list and the client should call this method again with the new change token.

    Change

    Description

    <List></List>

    If the list schema has changed, or if no change token was provided, we return the list here. The format is the same as returned by GetList.

    <Id ChangeType=

    "InvalidToken" />

    The token is either invalid or old.  You must do a full sync.

    <Id ChangeType="Restore" />

    The list has been restored from the recycle bin or from a backup.  You should do a full sync.

     

    In both of the cases above, the client should ignore other changes and do a full reconciliation of the list.

     

    Change Type

    Description

    <Id ChangeType=

    "Delete">ID</Id>

    This item is no longer present.  Note that Delete changes are sent even if the item was filtered out by the query.

    <Id ChangeType="MoveAway"

     AfterListId="ID"

    AfterItemId="ID">ID</Id>

    Treat in the same manner as a delete.

    <Id ChangeType="Restore">

    ID

    </Id>

    This item and any items beneath it were restored.

    <Id ChangeType=

    "SystemUpdate">ID</Id>

    Some clients may use hidden version, version history or modified time to determine whether to update an item. A SystemUpdate means Windows SharePoint Services has made changes and that you need to update all properties on that particular item.

    <Id ChangeType="Rename">ID</Id>

    Just like SystemUpdate, renamed items may retain hidden version information.

    Data

    The ItemCount attribute contains the number of items returned. Each Item is returned as a <z:row> tag.

    A ListItemCollectionPositionNext attribute is returned only for a full sync (no change token) when a row limit parameter was used to limit the number of items returned in one call. This type of paging is blocked on an incremental sync by restricting the number of updates processed from the change log to the row limit if smaller than our internal maximum.

    The contents of the items are returned as attributes with an "ows_" prefix and the internal name of the field. The values are encoded to be valid XML attribute values. A set of fields is always returned.

    Some fields are marked as required and returned if the IncludeMandatoryColumns option is set. If <ViewFields /> is sent, all fields are returned.

    When Properties="TRUE" is set in the ViewFields tag, the property bag (MetaInfo field) is separated into an attribute per property. These have a prefix of "ows_MetaInfo_".

    Most of the column values are simply converted from their internal representation, while others are constructed specially for a client. Several attributes are not represented by a field in the list schema.

     

    Table: List of attributes not represented by a field in the list schema.

    Attribute

    Description

    MetaInfo

    This is the property bag container, SPListItem.Properties. For more information, refer to the property bag in the SPListItem object model.

    Fields of type "Attachments"

    This is a bit column in the database, but query options modify it to return attachment data.

    Recurrence

    Data

    This is the XML definition of a recurrence.

     

    See http://blogs.msdn.com/sharepoint/archive/2007/05/14/understanding-the-sharepoint-calendar-and-how-to-export-it-to-ical-format.aspx for more details on this XML

    Acknowledgements

    I would like to acknowledge the following persons for their gracious help in technical reviews for this article: Matt Swann (Microsoft Corporation), Bill Snead (Microsoft Corporation).

    See Also

    I will be continuing this discussion in part 2.

  • Microsoft SharePoint Developer Documentation Team Blog

    Synchronizing with Windows SharePoint Services, Part 2

    • 7 Comments

    Introduction

    In my last blog post I introduced GetListItemChangesSinceToken and discussed how using GetListItemChangesSinceToken can make synchronization more efficient. In this post I'll talk some more about synchronization; take a quick look at GetList and UpdateListItems, and property bags. I'll finish up by discussing conflict detection, and performance best practices.

    Other Web Services

    In addition to GetListItemChangesSinceToken, Lists.asmx defines several other web services which allow clients to query for changes and make updates. There are several good examples in the Lists web service topic in the Windows SharePoint Services SDK.

    GetList

    GetList returns field schemas and other list properties.  Clients typically call this before syncing a new list, and then parse the response to match WSS fields to their client-side representation. 

    If the field exists as an out-of-box site column, it can be matched by its field ID.  In other cases, the field's internal name can be referenced.

    UpdateListItems

    UpdateListItems adds, modifies, or deletes list items.  Clients typically call this to keep server items in sync with changes made on the client.

    Request

    public SoapXml.SoapXmlElement UpdateListItems(string listName, SoapXml.SoapXmlElement updates)

    This is the format of the updates parameter:

     

    <Batch [update options]>

       <Method ID="X" Cmd="CMD">

          <Field Name="InternalName">VALUE</Field>

       </Method>

    </Batch>

     

    A batch is a collection of methods, each of which specifies the following value for the Cmd attribute:

     

    List of batch methods

    Method

    Description

    New

    Create a new item with the specified field values.

    Update

    Update the specified field values for an item.

    Moderate

    Change the moderation status for an item (used in the same manner as Update). The ModerationStatus field can only be changed by a Cmd=Moderate call. 

    Delete

    Delete the item with the following field values

    Note   Setting ModerationStatus must be done using a Cmd=Moderate call. Attempting to set ModerationStatus using Cmd=Update or Cmd=New has no effect. For more information about ListItem, refer to the SDK.

     

    The ID attribute of the Method tag is only used to correlate the method in the batch with the right item in the result.

     

    Update Options

    Option

    Description

    OnError="Continue"

    Continue processing the batch if errors are encountered

    LockSchema="TRUE"

    ListVersion="N"

    Only process the update if the list version matches

    ViewName="VIEW'

    Return columns present in this view with the item's data

    RootFolder="FOLDERURL"

    Perform the update in the context of this folder

    Properties="TRUE"

    Return the properties in the item property bag as separate fields

    DateInUtc="TRUE"

    The dates updated and returned are in UTC

     

    Fields

    In updates and deletes, the ID field needs to be supplied in order to identify the item.  If the update is being made on a document library, the FileRef field is also required to identify the document being updated.

    When updating an item, only the changed fields need to be supplied.

    When adding or changing certain types of items, certain fields may be required. 

     

    Clients may update the property bag in two ways:

    Update

    Description

    <Field Name="MetaInfo">

    XXX

    </Field>

    Add or update the specified name/value pairs in the property bag

    <Field Name=

    "MetaInfo" Property=

    "Name">

    VALUE</Field>

    Add or update this specific name/value pair in the property bag

     

    Note   There is no way for a client to delete individual name/value pairs in the property bag. See Property Bag below for more information.

     

    Response

    The return value is:

    <Results>.

       <Result ID="X,CMD">

          <ErrorCode>HR</ErrorCode>

          <ID>ID</ID>

          <z:row ... />

       </Result>

    </Results>

     

    Tag

    Description

    Error Code

    0x00000000 if no error. A hex HR if there was an error.

    ID

    Not sure if, when and why this shows. Maybe only for deletes?

    Z:row

    For all commands except Delete, the updated item with all its fields is returned.

    Property Bag

    Property bags are mechanisms for developers to add their custom data to corresponding objects inside of SharePoint.  Property bags let developers transform simple lists into rich data stores, and webs into full applications. The property bag is a virtual container that can store almost any typed of value. The SPListItem Properties property returns a property bag for the specified object.

    Note   If you use anything besides a String, int, or DateTime for the value, you will get a SPUnsupportedPropertyDataTypeException, with the message of "Only String, int, and DateTime datatypes can be used as the value in Properties."

    A call to the Update method on the object persists the values set in the property bag. All values can be stored and retried by using Web service methods.

    You must send the entire property bag, you cannot update just part of it.

    For more information about property bags in Windows SharePoint Services, refer to http://msdn2.microsoft.com/en-us/library/ms480101.aspx.

    Fields and Properties

    SharePoint has very extensible list schemas.  A client that has fixed content types (calendars, tasks, etc) needs to be able to relate its own item properties to specific fields on the list.  Typically this is done by calling GetList to get the field schema before a client syncs to a SharePoint list for the first time.

    Item properties in a client are best matched to fields on the server by the field GUID in the list schema.  The internal name of SharePoint fields can also be used to associate client-side properties with their server-side counterparts.  If a client cannot find an appropriate field on the server, it should store the data in the property bag for other clients to consume.  The client should cache these associations so that it doesn't have to fetch the list schema on every call.

    GetListItemChangesSinceToken is a very fast call when no changes were done on the list. GetList is not quite as fast.

    When a client calls GetListItemChangesSinceToken, it will receive a new list schema if the schema has changed. Unfortunately, if the schema change indicates a change in the internal name of one of the fields which are requested by name, the results of the call must be discarded. This is because the field you requested by name may not be included in the result set.

    There are a few snags with respect to client use of the property bag.  If, when making an update, a client doesn't know which particular property has changed, it must update every property (even the ones not set) so that it is sure to clear the ones recently emptied.

    Note   A field with an empty value is equivalent to an absent field; while an empty-value property is not.

    If a new field is added to the list, values stored by the client in an equivalent property are not automatically promoted into the field value. The client must decide whether to respect the value in the property and the value in the field, possibly losing user data in the process.

    Document Sync

    Clients should use HTTP/DAV to sync document content. Although some SharePoint web methods support document content fetch and update (for attachments), this is not the preferred way of transferring binary document content. Because SOAP is based on XML, binary documents need to be encoded into an XML-compliant form. This is generally accomplished using hex encoding, which roughly doubles the bandwidth used.

    A core field in document libraries (and also in generic lists in wssversion3short) is FileRef, which is basically a combination of two other fields associated with columns in the SQL table: DirName (server path to the containing folder) and LeafName (name of the document).

    Document libraries also have two other fields computed from FileRef - ServerUrl and EncodedAbsUrl.

    One of these can be used to reference a document.

    On a generic list, another field (Attachments) is used to determine if the item has attachments.

    Although there is a separate method to determine what these attachments are, by default there is no way to quickly determine if any attachments of an item have changed. This is what the IncludeAttachmentUrls and IncludeAttachmentVersion query options are for.

    When these options are used, the value of this field should contain; #[AttachmentUrl];#[AttachmentGuid],[AttachmentVersion] for each attachment. The client can compare these values with what they stored to determine which attachments need to be re-fetched.

    Because document contents can be quite big, a client can support a header-only mode that lets the user decide which document contents should be off-lined. MaxBulkDocumentSyncSize is a property that can be set on the server to guide the client when to automatically sync all contents.

    Note    Some generic lists have content that can be quite large. Discussion Boards are an example. The content of a discussion item is stored in a couple of rich text fields. The header-only concept could also be used here by separate the call for the generic fields from a separate GetListItems or call for the contents. We have no examples of the best way doing this and we have not analyzed it for performance.

    Conflict Detection

    Besides performance considerations, discussed below, this is arguably the most important sync topic: How to detect and deal with an object that was modified on both sides.

    Fetch-Send-Refresh

    It is better to have conflicts detected and resolved by the client. The client can better detect that the changes didn't actually conflict, it can raise an alert for the user to manually correct the conflict, and it can store a copy of the changes applied by the local user in the user's storage.

    Thus it is best if a sync operation consists first of fetching the changed data from the server, then detecting any conflicts with changes in the client copy, and finally uploading those changes if there is no conflict.

    SharePoint objects may have some business logic applied at the point of the update. Because of that, the UpdateListItems method returns the updated values of all the fields and properties of the updated items.

    owshiddenversion Field

    wssversion3short uses this field to detect conflicts. If the field value is not supplied on update, the server will overwrite any changes. A client should always supply it on update to prevent data loss. This number should be whatever the server last sent.

    This is used so that the server can tell if you're updating a stale copy of the item.  For example, a client syncs an item and gets a value of '2' for this attribute.  Someone changes the title of the item on the server, so the value increments to become 3.  When the client sends a change with value '2', the server complains because the item has been modified since the client last requested it.

    When there is a conflict, the server will return a TP_E_VERSIONCONFLICT (0x81020015) error and the current contents of the item.

    vti_versionhistory Property

    The hidden version field is sufficient for simple conflict detection when all clients are synchronizing with a central server however; peer to peer synchronization presents further challenges. You want to avoid raising unnecessary conflicts when the change was synchronized by a peer client.

    This situation may also happen in a non-peer-to-peer scenario. If a client successfully uploads a change to the server but does not receive an acknowledgment (response from UpdateListItems), the client needs a way to know that its changes were uploaded on next sync.

    ETag DAV Header

    Document and Attachment fetches and updates are done through HTTP/DAV. For that protocol, we have a separate mechanism for conflict detection. In every http get of a file (be it a document in a list, an attachment or a page outside a list) we return an ETag, which is supposed to be another blob that contains a guid and a version number. When uploading a document with http put, a client should request that the ETag matches the one supplied.

    Note   Version history is not supported for this protocol.

    Attachments

    Although updating attachments can be done through HTTP/DAV, adding attachments requires using the AddAttachment method from the lists.asmx web service which takes a binary array and returns the URL of the attachment.

    For more information about AddAttachment , refer to lists.asmx web service.

    Performance

    There are some performance issues to keep in mind when you are dealing with syncing with a server.

    Latency

    The amount of time it takes for an action to complete is what is important to a user. Also, there are usually limits on the amount of time allowed to process a request on a database, on a front end and by the entire request, so extremely long requests can turn into denied requests. Even despite this, you want to be able to give the users feedback on the action.  This is why paging is required.

    Using the row limit property on GetListItemChangesSinceToken to limit the amount of data requested each time is crucial for the above reasons, but it should be clear that it will also increase the total amount of time to complete a sync process.

    Throughput

    Obviously, reducing the total amount of cycles required to process a request helps performance by reducing latency. However, with multiple clients, it is more important to reduce the adverse effects one client has on the others. Most of the time, it is easier, cleaner, safer and more effective to make the server do some processing than to implement the same processing on a multiple number of clients. However, to increase throughput, it is almost always better to do that work on the client. Although the server will likely be a lot more powerful, the client will likely have more available CPU time. A sync client should make the data request to the server be as little and simple as possible.

    Bandwidth

    We target high-bandwidth scenarios, but even then, it is important to try to minimize the amount of data sent across the wire.

    It a client is not going to require a piece of information, it should avoid requesting it.

    Paging

    When performing a full sync (no change token), the client should request a maximum number of items returned per page using the rowLimit parameter. If the filtered number of items in the list is greater than that number, the server will return a ListItemCollectionPositionNext attribute to be used to request the next page.

    We will only return the current change token of the list on the first page to prevent the loss of any changes being made to the first page. The client should store the change token from the first page for a subsequent incremental sync.

    Secondary pages will also not include list and global properties like permissions, alternate urls and TTL.

     

    rowLimit is also supported on incremental syncs (change token supplied), but on an incremental sync this will limit the processing of our internal change log, and we have an internal limit of 100. Although the client can be sure the number of items returned will never be greater than that limit, in certain circumstances not all changes may have been synchronized even if the number of items returned is smaller than the limit. This is because we will stop processing the change log as soon as we reach a number of updates equal to the limit. When that is the case, we return the MoreChanges attribute to indicate there are more changes in the change log. Instead of waiting for the next sync period, the client should request more changes immediately using the returned change token.

     

    The limit works this way on incremental sync for a few reasons:

    • We have an internal limit of 100 because we don't have a modified time index we can use to filter the items returned and SQL has a limit of 160 on the number of ors in a query and starts performing badly close to that number. 100 gives us a potential extra 60 as part of the filter requested by the client.
    • We could have made several separate SQL queries, but that would imply supporting all ordering and filtering on the middle tier.

    Because of this, we needed to return a change token that is not current, so that extra changes could be processed on a separate call. We could have still looked at the entire change log to better determine the latest point at which the number of items returned would be smaller than the limit, but even this wouldn't be accurate without filtering on the middle tier.

    Filtering and Ordering

    Filtering is a way to allow the user to only get a certain set of items in a list. The two most common usages of this are for folder sync, where the user only gets the items inside a folder, and for certain Group Board scenarios where the user only gets the items associated with him/her.

    Filtering can be done using the contains parameter or the query parameter. Contains is more restrictive since it is basically the Where clause of a SharePoint CAML query, while query is the full query. Contains is safer to use because we can optimize certain scenarios. Query is more powerful and flexible, but the caller must understand its performance effects.

    A client should avoid filtering by a non-indexed column. Otherwise, fetching a page will require a scan of the entire list until it finds the number of items requested.

    A client should also avoid requesting an order unless the column of the order is indexed. Otherwise, fetching a page will at a minimum require a sort on the entire filtered dataset.

    Finally, if the filter is not on the same indexed column of the order, then SQL may still scan the entire list to avoid sorting the filtered dataset.

    An incremental sync has an implicit filter. We will request items with a certain ID. In this case, the client should never order by something other than ID. Filtering by something else is OK, for the dataset is restricted to a maximum of 100.

    Filtering by folder can be done using the Folder query option, but the list should be ordered by the FileLeafRef. For a recursive query, it should first be ordered by FileDirRef as well.

    There is also a way to filter by multiple folders using something like

    "<Or><BeginsWith><FieldRef Name="FileRef"/><Value Type="Note">Shared Documents/folder1/</Value></BeginsWith><BeginsWith><FieldRef Name="FileRef"/><Value Type="Note">Shared Documents/folder2/</Value></BeginsWith></Or>".

    This will synchronize the full contents of folder1 and folder2.

    The client should use this in the contains parameter and add the following query option:

    "<OptimizeFor>FolderUrls</OptimizeFor> "

    This will make sure the SQL query is optimized appropriately by ordering it by FileDirRef, FileLeafRef and constraining the right columns.

     

    Conclusion

    The most efficient way to synchronize with Windows SharePoint Services is to download only those items that have changed since the last synchronization occurred. In wssversion3short this can be done by calling the GetListItemChangesSinceToken Web method.

    GetListItemChangesSinceToken allows clients to track changes on a list.  Changes, including deleted items, are returned along with a token that represents the moment in time when those changes were requested. By including this token the next time you call GetListItemChangesSinceToken, the server looks for only those changes that have occurred since the token was generated.

    We discussed considerations that the developer must keep in mind to obtain the best possible performance.

    Acknowledgements

    I would like to acknowledge the following persons for their gracious help in technical reviews for this article: Matt Swann (Microsoft Corporation), Bill Snead (Microsoft Corporation).

    See Also

    http://blogs.msdn.com/sharepointdeveloperdocs/archive/2008/01/21/synchronizing-with-windows-sharepoint-services-part-1.aspx

    See Also

  • Microsoft SharePoint Developer Documentation Team Blog

    Overview of Backing Up and Restoring Data in Windows SharePoint Services

    • 5 Comments

     

    Introduction

    This post provides some basic facts about backing up and restoring data in Windows SharePoint Services 3.0. It will serve as background information for several developer-oriented posts that I will create in the next couple of weeks.

    Note: Unless explicitly stated otherwise, all classes and members referenced in this post are in the Microsoft.SharePoint.Administration.Backup namespace.

    What Can Be Backed Up and What Can Be Restored

    There are six types of content components built-in to Windows SharePoint Services 3.0 that can be backed up and restored through either the Central Administration application's UI, the Stsadm.exe Command-line Tool or a custom application that uses the Windows SharePoint Services 3.0 backup and restore object model.

    ·         Site collections, each of which might contain multiple Web sites.

    ·         Content databases, each of which might contain multiple site collections.

    ·         Web applications, each of which might contain multiple content databases.

    ·         Content publishing "Web services," each of which might contain multiple Web applications.

    Note: This refers to content publishing "Web services" (which are really partitions of content) that are represented in the object model by Microsoft.SharePoint.Administration.SPWebService objects. It does not refer to the functional Web services in the more common sense of "Web service," such as the Alerts (websvcAlerts) service or the Meetings (websvcMeetings) service. For more information about content publishing "Web services," see Server and Site Architecture: Object Model Overview and The High Level Object Model of Windows SharePoint Sevices 3.0 (to be published on MSDN).

    ·         Search Windows service including its databases and indexes.

    Note: This does not refer to the Search Web service websvcSPSearch.

    ·         A whole Windows SharePoint Services farm.

    If an enhanced functionality product such as Office SharePoint Server 2007 has been installed in addition to Windows SharePoint Services 3.0 and that product includes Shared Service Providers, then they can also be backed up and restored with Windows SharePoint Services 3.0.

    In addition, you can create new types of content objects that can be backed up and restored by implementing the IBackupRestore interface.

    Limitations

    There are some limitations on what can be backed up and restored through either the Central Administration application's UI, the stsadm command line utility or a custom application that uses the Windows SharePoint Services 3.0 backup and restore object model.

    ·         You cannot backup (or restore) an individual Web site, list, or list item except by backing up (or restoring) the entire site collection to which it belongs.

    ·         You cannot back up a Windows SharePoint Services farm's configuration database or the content database of the Central Administration application except by backing up the whole farm.

    ·         You cannot restore a farm's configuration database or the content database of the Central Administration application. The backups of these components that are included in a backup of a farm provide a snapshot of these components at the time of the backup. Such snapshots might be useful for troubleshooting because they can be used to compare with the present state of the components using SQL Server tools. (There is an exception to this point: you can restore the configuration database and the content database of the Central Administration application if you are using the Volume Shadow Copy Service – VSS - of Windows Server 2003/2008).

    ·         You cannot back up the Internet Information Server (IIS) metabase.

    The following kinds of content cannot be backed up with either the Central Administration application's UI or the stsadm command line utility, but you can create custom backup solutions with the Windows SharePoint Services SDK that include these types of content.

    ·         Registry keys.

    ·         Files that live on the front-end servers; that is, outside any content database, such as certain master pages, .ascx files, web.config files, and other configuration files.

    Types of Backups and Restorations

    Backups of a given component can be either full or incremental. In the latter case, only parts of the component that have changed since the last full backup are backed up.

    Note: A Windows SharePoint Services Search index cannot be incrementally backed up. If a search index is included in an incremental backup job, the index will get a full backup.

    Restorations can either overwrite the original backup source or they can be to a new location. This means that the backup and restore functionality in Windows SharePoint Services 3.0 can also be used as a method of migrating content components.

    Two Kinds of Custom Backup Applications

    There are two ways to use the Windows SharePoint Services object model to create custom backup applications.

    The Main Backup and Restore Object Model

    You can create a backup and restore application by using the main backup and restore object model, sometimes called "catastrophic" backup/restore. This is located mainly in the Microsoft.SharePoint.Administration.Backup namespace; but backups and restores of individual site collections are performed with the Microsoft.SharePoint.Administration.SPSiteCollection.Backup() and Microsoft.SharePoint.Administration.SPSiteCollection.Restore() methods.

    Interface to the Volume Shadow Copy Service

    Windows SharePoint Services 3.0 deployments can also take advantage of the Volume Shadow Copy Service (VSS) in Windows Server 2003/2008. Windows SharePoint Services 3.0 includes a Windows SharePoint Services VSS Writer service that will create shadow copies of native and custom content in the deployment. The service contains a VSS writer that will write shadow copies of all native Windows SharePoint Services 3.0 databases and all custom databases. Non-database custom components can also be registered with the service by using the SPVssComponentDefinition and SPVssDiscoveryHelper classes. It is also necessary that you create a VSS writer for any such non-database custom components.

    Note: With the VSS service, you can target only the whole farm or individual content databases for database shadow copy. Individual Web applications and individual content publishing "Web services" cannot be set for shadow copying independently of making a shadow copy of the whole farm. (For information about the meaning of "content publishing Web service" see the first note of this post.)

     

Page 1 of 1 (3 items)