Microsoft Enterprise Content Management (ECM) Team Blog

Building up your Organization’s Value with Taxonomy

Ryan Duguid — Thu, 08 Dec 2011 17:53:27 GMT

Getting the most out of SharePoint 2010 should involve the new Managed Metadata Service. For those of you who aren’t familiar with this service and the support it provides for modeling and deploying a rich corporate taxonomy, I’d start by reading Pat’s post Introducing Enterprise Metadata Management. By knowing how the documents and concepts could be grouped together you can start shaping the user experience with your SharePoint instance. Once you decide that you want to take the organization of your enterprise content to the next level, you need to decide how to organize your documents. If you are lucky, you already have a well defined taxonomy that can be imported into the Term Store with SharePoint’s import capabilities. Even if you have something that starts to help organize your data the power of leveraging other views of the subject you already have will help find improved ways to access and organize your data for the users. If your organization hasn’t started to organize, or wants to improve their taxonomies, or wants to compare their data to make sure there aren’t areas where you haven’t covered, you now can find pre-built foundational taxonomies in a variety of fields and expertise.

The SharePoint team has teamed up with WAND, a leading provider of Enterprise Taxonomies, to provide a starting point and make their General Business Taxonomy available as a freely available download. The General Business Taxonomy consists of common functional areas that exist in most businesses. The General Business Taxonomy can be imported in to the SharePoint 2010 term store within minutes and provides a great starting point for customers looking to build out term sets and take advantage of the Managed Metadata Service. In addition to this freely available download, WAND provide a range of taxonomies covering a variety of domains including Products and Services, Jobs, Skills, Medical, Finance, Records Retention, Legal, Insurance, Manufacturing , Information Technology, and Mining

Download the General Business Taxonomy today and start to explore the benefits that taxonomy can bring to your business and your people.

If you’re new to taxonomy and the benefits it can bring to your business, take a look at the following sites:

Introducing Document Sets

Ryan Duguid — Tue, 18 Oct 2011 22:20:09 GMT

Hi there, this is Quentin again. In today’s post I would like to introduce an exciting new content type that is used as an architectural component for organizing multi document based work products. Document sets fit between documents and sites. You can use document sets when you have a group of related documents to collaborate on, but don’t need an entirely new document library or site. Document sets provide a collaborative workspace where users can work on a related group of documents. Since the release of SharePoint 2010, I have seen document sets being used for a wide variety of scenarios internally here at Microsoft and with some of our customers. Some examples include research projects, software development specifications, knowledge management repositories, and usability studies.

Document sets are similar to folders, but when you access a document set you get a web part page that can be customized. When you upload documents to a document set they are displayed in a web part that works just like a document library view. Several document set features make it faster and easier to work on projects of related documents. Here are a few of these features:

Shared metadata enforces the same metadata on all items inside the document set
Default documents can be automatically provisioned when a document set is created
A web part page called the welcome page displays the contents of the document set and can also be customized to display more information than just documents

Document sets fit in with our overall document management vision by helping manage the unmanaged and bringing collaboration and metadata into document creation and management. Document sets can be used to organize and find content and they make it easy to group and relate documents. Document sets encourage using columns for metadata on the document set itself and the documents within. Shared columns put document set metadata on all of the documents inside and it is useful to display the properties of the document set on the document set welcome page. Document sets make it faster and easier to work on groups of related documents, saving time and money.

We built document sets based on scenarios that we saw both internally at Microsoft and from our customers. For example on the Office team we organize the parts of the products we build into features and each feature has a team that works on it. These features have specifications, development plans, and test plans. Document sets make it easier to create the necessary documents and manage a collection of documents for a feature.

We also saw customers with similar scenarios, such as sales proposals that require presentations, spreadsheets, and sales contracts. Rather than manage each item individually, in these cases there may be metadata that should be shared and workflow processes that should happen on the collection of items. We expect partners and customers to take document sets and customize them for specific vertical scenarios to support important processes such as managing cases and research projects.

Now that I have given a quick intro into what document sets can be used for, allow me to give an overview of the main features of document sets.

Welcome Page

When you go to a document set you will notice that it isn't your typical folder view. Document sets have a customizable welcome page. By default this page has a document set properties web part. From the document set settings page of the content type you can specify which document set columns show up in this web part. There is also a Document Set Contents web part which displays the documents that have been added to the Document Set.

This is an example of a non-customized document set welcome page with a few documents.

This is an example of a customized welcome page that has had the image changed and a note board added. The welcome page is shared by all document sets of a particular content type in a list so any changes will be reflected for all instances. This makes it fast and easy to make changes to the welcome page because the updates will be instantly available to existing document sets.

Allowed Content Types

You can select the content types that are available to the document set, allowing you to control the types of content that are available to be used inside the document set.

Default Content

Documents can be specified to be automatically created when a new document set is created. Default documents can be specified, allowing you to control the templates people use for a document set. This saves the time of having to find and upload templates to get started with. Of course users can still upload their own documents to a document set as well.

Shared Columns

Document set columns can be shared to all the documents inside the document set. This will make the column read only so the value can only be changed by editing the properties of a document set. This saves users time because they only need to update metadata in one place and it will then be updated on all documents inside the document set. It also helps ensure that metadata is consistent. You can use shared columns to make sure all of the documents have certain metadata.

Send To

Send To locations that have been specified in Central Administration will be available to document sets so you can send them to a configured content organizer. You can also use workflow actions to send document sets to a content organizer. The content organizer is a new SharePoint 2010 feature for moving content based on metadata to folders, libraries, or other sites.

Customizable Ribbon

One of the major improvements with SharePoint 2010 is the ribbon interface that is similar to the ribbon interface used in the Office client applications. Document sets have their own ribbon to perform actions such as send to, delete, and capturing versions. This ribbon can also be customized to add new commands.

Versioning

Document sets have their own versioning that is used to capture a snapshot of checked in versions of documents and the properties of the document set.

Workflow

Document sets support workflows on a collection of documents. Several workflow actions specific to the document set are available in SharePoint Designer including capture a version and start an approval process.

Summary

Document sets are a great way to organize related documents for project based work. They provide a customizable and collaborative environment that you can use to meet the needs of multi document work products. They are also a central information architecture component that can be used to represent concepts such as research notebooks, sales proposals, and product specifications. Document sets make it easier to automatically provision templates, collaborate on a group of documents, and share information about documents (such as metadata properties).

Be sure to subscribe to the ECM blog RSS feed and keep reading the blog. I will be posting more blog posts about document sets where I will cover creating a custom document set content type and customizing the welcome page.

Quentin Christensen

Program Manager, Document and Records Management

Policy and Retention in SharePoint Server 2010

Ryan Duguid — Wed, 12 Oct 2011 22:55:00 GMT

In Microsoft Office SharePoint Server 2007 we released Information Management Policy. This allowed auditing and expiration on a per content type basis. With SharePoint Server 2010 retention flexibility has been greatly increased. There are now several retention actions that can be performed out of box and it is easy to trigger custom workflows. The biggest new feature is the ability to do location based retention (set retention by folder), which opens new options for hierarchical file plans. You can also do multi stage retention as well.

Configuring Policy

The Information Management Policy settings page can be accessed by going to the site content types gallery, selecting a content type, and clicking on the Information management policy settings link. On the settings page you can fill in an administrative description, this is what is displayed to administrators when configuring policies. A policy statement can also be displayed, this is what will be shown to end users for example when they open a Word document they will get an information bar. There are for categories of policy: retention, auditing, barcodes, and labels. Retention allows you to specify actions that occur after a certain date. Auditing gives you options to select of operations to items that will be tracked. Barcodes and labels can be used to track documents.

Retention Actions

The retention actions include:

· Move to recycle bin

· Permanently delete

· Transfer to another location – You can select from configured Send To locations that will send the document to a content organizer

· Start a workflow – select from available workflows that are associated with the content type (or list in the case of location based policy)

· Skip to next stage

· Declare record – this will mark the item a record using the new in place records feature. At the site collection level you can specify whether a record should be treated as a normal item, block delete, or block edit and delete. In order to use this action the In Place Records site collection feature must be activated.

· Delete previous drafts

· Delete all versions

In the events section you will notice that you can select columns, and there is a disabled option to set a custom retention formula. This option is available if a custom retention formula has been installed on the server. You can check out the article Creating a Custom Expiration Formula Based on Metadata in SharePoint Server for more information about creation a custom retention formula.

Content Type Based and Location Based Retention

With SharePoint Server 2010 you can configure content type or location based retention. Content type based retention lets you specify that all items of a particular content type will follow a retention schedule. To specify a retention stage go to the information management policy settings page, which can be accessed from the main settings page for a content type. From there you can click on the enable button for retention and add new stages. It is important to note that pages, blogs, and wikis are content types that can have policy as well as documents.

Retention can also be specified by library and folder. To do this navigate to the list you would like to configure retention for, go to the list settings page and then click the link for information management policy settings. By default items will follow content type based retention. To change the retention schedule source click the change source link. Now all items in this list will follow the retention schedule that is specified for this list, rather than whatever is specified for the content type. To the left of the screen there is a tree control that can be used to navigate the folders in the list. You can use this control to select different folders to specify a retention schedule for that particular folder and all folders beneath it.

Folders inherit location policy from their parent. At the list level I can set a retention schedule for all items to expire 5 years after created date. All items in the list will receive then expire after 5 years. For the Confidential folder I need to retain items for 7 years, so now any items inside the confidential folder (including any new folders) will now expire at 7 years.

You can also set multi stage retention schedules. Let’s say that after a document has not been modified for more than 90 days previous versions will not be needed, so to save space I want all previous drafts to be deleted (because each version takes is the full size of the document, 10 versions means 10 times the space may be taken up). Also I do not want documents cluttering up this library because it is a collaborative space. After 1 year documents are no longer needed, but they do need to be retained for 7 years due to legal compliance reasons. I can set a second stage to send all documents to a record center after 1 year.

SharePoint Server 2010 provides greatly improved capabilities and flexibility for retention of documents. A few of the major improvements include multi stage retention actions, more out of box retention actions, and location based retention. Also improvements with creating workflows with SharePoint Designer make it much easier to create custom workflows to create retention actions to solve your needs.

Quentin Christensen

Program Manager, Document and Records Management

Metadata Defaults in SharePoint Server 2010

Ryan Duguid — Wed, 12 Oct 2011 22:53:00 GMT

SharePoint Server 2010 unleashes new features such as managed metadata and metadata navigation that make metadata even more important. But a big problem for many SharePoint projects is getting metadata onto documents. It is important to consider the tradeoff of metadata vs. user tax. As the number of metadata columns that must be filled in increases it becomes less likely that users will fill in metadata because it is that much additional work to go through and see which columns actually apply. If a large amount of required columns are used then user adoption may be slow because it is so taxing to upload content. In a very open and collaborative scenario this can be detrimental. But as the value of the content and effort to create that content increases, it becomes more likely that users will take the time to fill in the appropriate fields, especially when this operation is not frequent.

For any SharePoint project you should carefully consider what metadata will be needed to perform required operations and for users to find content. Evaluate how long it will take users to fill in that metadata, and evaluate the user impact. If lots of metadata is required but end users do not adopt the system because the overhead for creating content is high it will be difficult to have a successful implementation.

Metadata defaults help with this problem because you can automatically fill in metadata for users. If a particular field will have the same value 50% or more of the time then you should specify a metadata default to help users fill in forms faster. Metadata defaults can be specified for each column at the site level, list level, and folder level. Metadata defaults inherit from each other and the child default will always override. For example if a default is set on a folder and there are defaults for the list and the site column, the default for the folder will be applied. If there are defaults for a column and a list, but no default for a folder then when an item is added to that folder it will get the list default. When users create or upload a new item the metadata default will be applied and be displayed in the edit properties form. The user can then change the property as needed. By planning for and setting metadata defaults early it will be easier to evaluate what the metadata defaults should be and how they can be used to help make it easier to get metadata applied to items.

Metadata defaults are supported on the following column types:

· Single Line of Text

· Choice

· Number

· Currency

· Date and Time

· Yes/no

· Managed Metadata

Setting metadata defaults when editing columns:

When creating or editing a column at the site or list level there is a field to specify a default value. You can specify defaults at the site level, and then override them at the list level.

Setting metadata defaults with the tree control:

In libraries you can use a tree control to set metadata defaults for the library and per folder. To do this go to library settings and click on the link “Column default value settings”. This will give you a tree control that you can use to navigate the folders in the library. You can set defaults at the root of the library as well as for each individual folder. Folders will inherit default values unless you specify a particular default value for the child folder.

Metadata defaults are a great way to help place metadata on columns. You can even use it to apply defaults on hidden columns so users never even have the option to edit the field, but they can use the metadata to navigate and retrieve content.

Quentin Christensen

Program Manager, Document and Records Management

Document ID in SharePoint Server 2010

Ryan Duguid — Wed, 12 Oct 2011 22:46:29 GMT

Document ID Overview

A common content management problem is documents getting moved to different locations and links no longer working. Another problem is submitting content to a repository and being able to find it at a later time. These are the types of problems the Document ID feature handles. Document ID is a site collection level feature that when activated adds columns to all document content types. Whenever an item is created or uploaded and is set to a content type that inherits from document it will get a Document ID. The document sets content type is a special case non document content type that also gets Document IDs.

Quick Tip: You may want to activate the Document ID feature on site collections that already contain a large amount of documents. Because this event triggers a content type pushdown on the Document content type the process can take a while. As a result, Document ID activation is done through a timer job that by default is run nightly. If you would like to activate the Document ID feature right away please run the Document ID enable/disable job in Central Administration. There is a separate time job that assigns Document IDs to existing documents.

Document ID Settings

Document IDs consist of two parts, a prefix (that is randomly generated per site collection by default) and two sets of numbers. An example Document ID looks like this: J2W3DN6QF6XW-2-10. The first number is the ID of the list that the document was first created in and the second number is the ID of the item in that list. The prefix can be specified per site collection and can easily be changed from the Document ID settings page.

The Document ID settings page can be accessed in Site Settings at the root of the site collection, and will be in the Site Collection Administration category. On the Document ID settings page you can specify whether or not Document IDs are assigned and the prefix that is used. By default this is randomly generated so Document IDs will be unique across a farm. You can change this to a more readable prefix such as an abbreviation. If you change the prefix there is a check box that you can select to reset all Document IDs. If you do reset the IDs of existing documents then you will break the intended functionality of Document IDs (the ID will no longer work because no document with previous IDs will be found). Reset Document IDs should only be done fairly soon after IDs were initially assigned (before people use Document IDs.) On the settings page you can also specify the search scope to use, by default the All Sites scope, to find Document IDs across other site collections.

Document ID (linked to document) Column

In list views and in view properties of documents the Document ID column can be displayed. The Document ID column is a URL with a format like: http://site/_layouts/DocIdRedir.aspx?ID=J2W3DN6QF6XW-2-40. When this URL is clicked search will be used to find the document. This allows the URL to work even if the document has been moved to another location.

Document ID Send To Hint URL

In Central Administration you can configure send to locations that allow you to submit content to another site and a content organizer will automatically route the document to its final destination. In this case a Document ID lookup URL is returned (If the Document ID feature is activated on the destination) which also contains a HintURL on the end so users can find the document immediately even though search may not have indexed it yet. Here is an example of a link to the HintURL:

http://sites/_layouts/DocIdRedir.aspx?ID=J2W3DN6QF6XW-2-96&hintUrl=DropOffLibrary/A1.docx

Document ID in Documents

One scenario you can use Document ID for is to track the electronic version of a document with a printed one. For example you may print out a document and give it to people, but they have no way to find the electronic version to see if there have been updates. You can use a Word quick part to insert the Document ID into a document.

You may even want to create a template so whenever users create a new instance of a content type it has the Document ID right in the header or a page of the document. To do this upload a document into SharePoint where it will get a Document ID. Then download the document, customize the template as you want to use it for your content type and insert the Document ID Value property using quick parts from the Insert ribbon of Word. Then save the file as a Word Template (dotx) and create your content type. In advanced settings for the content type, upload your template. Now when people use the New Document drop down to create new items of your content type they will have the Document ID Value embedded as a quick part.

Find By Document ID Web Part

There is a Find by Document ID web part that can be used where users can enter Document IDs to lookup a document. You can find it in the Search category of web parts. You can use the Document ID web part in combination with Document IDs that are printed out on physical documents so users can look them up.

Of course users can also just copy the Document ID into a normal search box to find the document as well. DocID is a managed property so you may want to use the following search syntax: docid:doc id value, for example in search type – docid:J2W3DN6QF6XW-2-96.

Document IDs are a new SharePoint Server 2010 feature that makes it easier to find documents when they have been moved to other locations. When documents are copied a new Document ID will be assigned and when Documents are moved or cut and pasted they will retain their Document ID. You can also use Document IDs to track physical documents with the electronic version.

Quentin Christensen

Program Manager, Document and Records Management

Taxonomy–The Challenge of Starting from Scratch

Ryan Duguid — Mon, 07 Mar 2011 04:35:37 GMT

One of the most talked about capabilities since the launch of SharePoint 2010 is the Managed Metadata Service. For those of you who aren’t already familiar with this service and the support it provides for modeling and deploying a rich corporate taxonomy, I’d recommend reading Pat’s post Introducing Enterprise Metadata Management. For those of you who are familiar with the great taxonomy capabilities in SharePoint 2010, I’m sure many of you have spent time looking at an empty term store wondering where to start. If you’re lucky, you already have a well defined corporate taxonomy and should by now have leveraged our import capabilities to pre load SharePoint with the vocabulary you want your users to leverage for tagging and finding content. On the other hand, you could be like many customers I talk to who don’t even know where to start when it comes to developing a taxonomy, or have spent years in conference rooms debating what the right taxonomy should be. You’ve probably even head someone say “I’m sure someone has already solved this problem”, and if that’s the case, that someone was the smartest person in the room for two key reasons. The first is that there are professional taxonomists who have already modeled most business domains and the second is that the people responsible for creating content in your company have already developed a community vocabulary or folksonomy that they use extensively.

If you happen to be one of those customers who is stuck looking at an empty term store then I’ve got great news for you. The SharePoint team have teamed up with WAND, a leading provider of Enterprise Taxonomies, to make their General Business Taxonomy available as a freely available download. The General Business Taxonomy consists of around 500 terms describing common functional areas that exist in most businesses. The General Business Taxonomy can be imported in to the SharePoint 2010 term store within minutes and provides a great starting point for customers looking to build a corporate vocabulary and take advantage of the Managed Metadata Service. In addition to this freely available download, WAND provide a range of taxonomies covering a variety of domains including Products and Services, Local Search, Enterprise, Jobs, Travel, Medical, Lifecycle, Finance and Records Retention.

Download the General Business Taxonomy today and start to explore the benefits that taxonomy can bring to your business and your people.

If you’re new to taxonomy and the benefits it can brings to your business, take a look at the following sites:

Ryan Duguid
Senior Product Manager
Microsoft Corporation

Announcing the Release of the CMIS Connector for SharePoint

Ryan Duguid — Fri, 16 Jul 2010 07:13:28 GMT

I’m pleased to announce that we have Released to Web, the Content Management Interoperability Services (CMIS) Connector for SharePoint. The CMIS Connector for SharePoint ships as part of the SharePoint 2010 Administration Toolkit, providing a CMIS interface over the top of SharePoint as well as a CMIS Consumer Web Part that can be used to display content from other CMIS enabled repositories.

You can download the SharePoint 2010 Administration Toolkit today and start to take advantage of this new set of capabilities within SharePoint Server 2010 by building your own Composite Content Applications that talk to SharePoint through CMIS or configuring SharePoint to interoperate with other ECM repositories through the CMIS Consumer Web Part.

Microsoft has been involved in defining the CMIS specification since the beginning and has invested significant resources to ensure that our customers are able to take advantage of support for CMIS in SharePoint 2010 just months after releasing the latest version of our platform. We are excited about the opportunities that the CMIS standard will open up within the industry and look forward to seeing more ECM vendors deliver support for CMIS in their upcoming product releases.

For further reading on CMIS, visit these sites:

Ryan Duguid
Senior Product Manager
Microsoft Corporation

Introducing Enterprise Metadata Management

Ryan Duguid — Tue, 22 Jun 2010 21:31:23 GMT

Hi there, my name is Pat Miller, and I am the development lead for the Enterprise Metadata / Taxonomy features in SharePoint 2010. I've been working on the ECM team and its fore-bearers for the better part of 11 years now, first with NCompass Labs which was acquired by Microsoft in 2001, then on the Content Management Server team, then with the CMS team as part of MOSS 2007. This is the first of many blog posts on the Enterprise Metadata Management (EMM) system in the 2010 release. This will be the overview of the system, and future posts will drill into specific areas like event receivers, field editing and search refinements.

First, some background. At one point during the development of Content Management Server 2002, we spent some time with the folks that run the Microsoft.com set of websites. One of the things they were very keen on was this taxonomy system that they had built. It seemed fairly useful, and we considered implementing something like it, but didn't have the time, and there was a general concern that no one would actually do the work of tagging data. During the development of MOSS 2007, we were spending most of our time rewriting our feature set to run on top of SharePoint, and once again, taxonomy fell off the list of things we were willing to tackle (and still, people would consistently say that people just don't tag).

Around this time people started tagging things in their own world. The rise of digital cameras and mp3 players brought a huge amount of data that for the most part, had to be marked up with metadata in order to be searchable. Some metadata was added to the files automatically (things like date, size, camera model, etc.), but specific user information wasn't there. You quickly learned that if you categorized the images (either through folder location or tags) you could navigate your way through 10's of thousands of files (images, music, etc.) the way that works for you personally, rather than relying on default information like date the picture was taken. People became more familiar with the concept of navigating their content via metadata - "Let's listen to all my Pearl Jam albums, I feel like listening to Electronica, find me photos of Dad". It's only a small step from that to wanting to impose some sort of hierarchy - find me photos of my whole family, my extended family, I want to listen to all classical music, or perhaps just from the Baroque period. Tagging all that data really unlocked a lot of potential.

Perhaps the landscape had changed...

We decided to run with it in the 2010 release. There were a few main tenets that we tried to let guide us:

No one (well, almost no one) apply metadata for the shear joy. It's always for a purpose.
#1 means that the reason for the system has to be for the end user benefit. What can you do if you have this rich metadata applied?
In order for #2 to come to realization, the metadata has to be present, which means that applying consistent metadata needs to be as easy and ubiquitous as possible.

To that end, we set out to enable a bunch of new user scenarios for SharePoint 2010.

We started out the release with a blank sheet of paper and some very knowledgeable people in the information management space. We also found that most people started twitching uncontrollably when the word "ontology" was mentioned. 'Tagging' was fine, 'metadata' was OK, at 'taxonomy' they started looking for an exit. Telling people that a taxonomy was just a hierarchy calmed them down, but the whole ontology thing was too much of a stretch. It also complicated things considerably, and we could still get a huge amount of value out of a taxonomy, so this was our starting point.

Some features were very obvious - filtering list views based on hierarchy inclusion, search refinement, etc. Some were a small step from this - if you have a consistent vocabulary across an enterprise, you can start to do some interesting things. You can match areas of expertise to specific content or workflows. You can start to relate content in totally different systems based on something with more context than a simple string. What if you could relate your analytics content to your taxonomy system and get a real-time view of what topics people are viewing instead of simply guessing based on their position in a URL namespace? How about overlaying your security model with your metadata so that certain people had rights to view content based on the metadata applied to it? How about we get down to business and focus our resources and ship a compelling collection of features.

To that end, we came up with the following components in the system:

The taxonomy repository itself, we call it the Term Store. Some companies have very top down strict taxonomies, so some term stores might have a very few people allowed to edit them. We'll have to support having multiple term stores.

The taxonomy system needs to be able to support a complex enterprise. A simple flat list of strings isn't going to be sufficient. To that end, we support the following concepts and behaviors:

Terms - A term is the central object in the taxonomy system. It's the concept itself. It's very hard to come up with a name for a concept and have it be sufficiently descriptive and not too vague. Term is what we came up with.
Labels - Terms have to be known by a bunch of different names. When someone types "check" it should be the same thing as someone that types "cheque". "USA" and "United States" and "United States of America" are all referring to the same term. We call these names labels.
Default Label - It's a whole lot easier if one label is the default. You can find it through any of its synonyms, but we'll display the default label in most circumstances.
Termset - A collection of related terms in a hierarchy is a termset. Things like "locations" and "products".
Term Reuse - This is a key point to the system. If you have two termsets "Capitol Cities" and "Locations", the term "London" and all of it's synonyms, etc. should be the same in both. We don't allow a term to have two parents in the same termset, but it can have two parents in different termsets.
Homographs - A homograph is a word that is spelt the same, but has a different meaning. You should be able to have a hierarchy that has "Paris" existing in both France and Texas. To keep things a bit more sane for the user, we don't allow homographs to have the same parent.
Multiple language support - A given term has a bunch of meaning associated with it. The translations belong to the term in the same way that synonyms do. If a term doesn't have a translation, we use the default language.
Groups - Groups in the taxonomy system are simply collections of termsets that share a common security assignment. Termsets and terms aren't ACL'd, groups are.
Deprecated terms - if a term shouldn't be used any more, it can be deprecated. This doesn't remove it from the system, you just can't apply it to new content moving forward.
Terms that are unavailable for tagging - this is slightly different from deprecated terms. A deprecated term is deprecated in all occurrences in the taxonomy and isn't shown to the user when tagging. Unavailable terms are only unavailable in a specific termset, and are still displayed when browsing the hierarchy at tagging time. The purpose of this is to allow things to be hierarchical without allowing people to tag with the wrong term. For example, in the Capitol Cities termset, you might have continents in it so that people can find a particular city, but they would be marked as unavailable for tagging (with respect to Capitol Cities) because they should not be selectable at tagging time.
Merging terms - at times, you might get multiple terms in the system that really are the same thing. They might be in the same termset, or they might be in different termsets. When you merge them, you get a single term with all of the properties, and this new term will be reused in all termsets that the original terms existed.
Open Termsets - There are times when a highly managed taxonomy makes sense. You shouldn't be able to add random countries to the list of known countries. However, you probably don't want to give taxonomy editing permissions to everyone that is creating a new codeword. Open termsets allow content editors to add new terms to a hierarchy at content authoring time. It's a bit of a meeting point between bottom up folksonomies and top down taxonomies.
Keywords - The degenerate case of a folksonomy is a simply flat list of strings. They have no extra semantic meaning. This is the enterprise keywords termset. Terms here don't have a hierarchy, definitions, synonyms or translations. However it is possible to move a keyword into a managed termset and add this additional data.
Local termsets - The taxonomy field type gives you all sorts of useful features, but you probably don't want "places to order food from" to wind up in your enterprise taxonomy. Local termsets are only visible within a single site collection.

OK, that's a nice set of features in the taxonomy system. What do we want to do with all those terms and termsets?

The next set of features involve integrating the taxonomy system with SharePoint. The primary place this happens is in the new managed metadata field type. Think of it as a choice field that went to the gym. It's much more powerful. The metadata field type is a normal field that can be applied to any content type (list or document library). However it has a few nice things associated with it:

Termset binding - You can specify what termset a field should be bound to. You can have lots of fields bound to the same termset. When you update the termset, all of the bound fields use the changes immediately.
Path or node display - You can choose to display the default label of the term by itself "Paris" or its path "Europe > France > Paris".
Multi-lingual rendering - If a given term has been translated to a given language, when your UI is set to that language, the term translations are displayed.
Content type syndication - This isn't a taxonomy feature per se, but it's part of the enterprise metadata feature set. We allow a term store to have a site collection defined as it's "hub". On that hub you can publish content types, and these content types will be pushed out to all consuming site collections. This means that in addition to having a consistent vocabulary across your enterprise, you can have a consistent set of content types using all that goodness.
Rich editing - when you are applying a term to an item, you can search across the entire termset (including synonyms) or view the tree itself. It makes it possible to choose from thousands of choices, which would normally break lookup and choice fields.
Editing support in the rich client applications - the document information panel in the Office client applications allows for applying terms.
Offline editing in the rich client applications - when you edit in the rich client applications, a copy of the bound termsets is cached locally. You can tag on the plane.

Once data is in SharePoint, other SharePoint features can deliver additional goodness:

Better listview filtering - not only can you filter in the normal "everything with value X" but you can also do inclusive filtering, displaying everything tagged with X or a child of X.
Better metadata navigation behavior - The metadata navigation feature allows you to navigate through libraries using hierarchies other than the folder hierarchy. The termset is one of the allowed hierarchy types, meaning that you can browse your libraries along multiple axes. You can now free your data from the tyranny of the URL or folder namespace.
Routing and policy - The document routing feature can direct your content based on the metadata applied to it. Taxonomy fields can even be used to create folder hierarchies at the routing destination. Retention policies can be driven off of taxonomy fields as well.
File open / save - Can't remember exactly where your document is stored in a large library? You can use the taxonomy field to filter the open dialog display.

Now that we have all that nice consistent metadata on our content, we can do a few more things:

Content by query Web Part enhancements - You can configure the CBQ to filter based on taxonomy fields, including descendent inclusion.
Automatic search refinement - The search system is aware of all taxonomy fields, and if a result set has a sufficient amount of data with the same taxonomy fields, a search refinement will appear, allowing users to filter their data.
Power user profile and social tagging - it doesn't make much sense to have a corporate taxonomy and then do your social tagging using just string matching. All of the social properties are actually sourced from the taxonomy system, meaning that you won't get people asking you where a good place to stay in Paris, France when you are an expert on Paris, Texas.

And since we know that we can't possibly implement every feature that everyone would want, everything is accessible through our API. In future blog posts, we'll go over how to use this API to deliver some compelling features.

Hopefully this is a nice introduction to the work we did around taxonomies and enterprise metadata. We had a lot of fun coming up with the design and implementation, and hope that it resonates with you.

Thanks for reading.

Pat.Miller at Microsoft.com

Variations: Propagate Pages on Your Terms

Ryan Duguid — Tue, 22 Jun 2010 20:06:00 GMT

Hi everyone,

I'd like to answer a common question about how to modify the behavior the Variations feature in SharePoint 2010 uses when propagating pages. That is, how pages in the source variation site are copied and appear on target variation sites as minor draft versions.

Page propagation is triggered by publishing a page on the source variation site by default. Each time you publish a source page, the Variations Event Receiver adds a work item to the Variations Propagate Pages timer job queue. When the timer job runs, it will begin executing the first 100 page propagation work items. For each work item, Variations will copy the source page to all target sites, creating the page if it does not yet exist, or appending a draft minor version if the target page does already exist.

In some cases, users might not want changes to a page on the source to necessarily propagate to all targets. That is, users might want to make source-local changes and have the option to make changes globally applicable when they want. This often takes the form of a question like "How can I stop variations from overwriting my target pages every time I publish a source page?" Variations in SharePoint 2010 helps you do this.

In SharePoint 2010, we've worked to improve the Variations feature's server citizenship by moving all Variations operations to the timer service. This way, server administrators can control the frequency with which operations run and better manage server load. The "Update Variations" button now adds a work item to the same Variations Propagate Page timer job queue as does publishing a page when "Automatic Creation" is enabled. What differentiates "Update Variations" is that you can also use this button to propagate source draft versions without publishing them on the source variation site.

When you run this PowerShell script to enable "On-Demand Page Propagation," all Variations Propagate Pages timer job work items are filtered and discarded except those added to the queue by the Update Variations button:

Enable On-Demand Page Propagation:

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$site = new-object Microsoft.SharePoint.SPSite("http://yourserver/sites/abc")
$folder = $site.RootWeb.Lists["Relationships List"].RootFolder
$folder.Properties.Add("DisableAutomaticPropagation", "True")
$folder.Update();

Disable On-Demand Page Propagation:

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$site = new-object Microsoft.SharePoint.SPSite("http://yourserver/sites/abc")
$folder = $site.RootWeb.Lists["Relationships List"].RootFolder
$folder.Properties.Remove("DisableAutomaticPropagation")
$folder.Update();

"Update Variations" in SharePoint 2010 can be used to propagate the current version of a page on-demand provided that the Variations Propagate Page Timer Job is enabled

In MOSS 2007, some users disabled the "Variations Propagate Pages" timer job in Central Admin as a workaround. With the timer job disabled, publishing a source page would not cause SharePoint to copy the source page to any target page. Authors on the source variation site could then use the "Update Variations" button to propagate the current version of the source page on-demand.

Clicking "Update Variations" in MOSS 2007 immediately propagated the current version of the source page to all target pages, "skipping the line" and bypassing the Variations Propagate Pages timer job. So when the timer job was disabled, "Update Variations" could still be used to propagate pages.

However, when the timer job is disabled, publishing pages on the source continues to add work items to the timer job queue if the "Automatic Creation" option is enabled, as it is by default. Over time, the queue can grow and contain hundreds or thousands of work items, all of which would begin to execute if the Variations timer job were re-enabled in the future. If you upgrade to SharePoint 2010 with a backlog of work items, SharePoint will discard these.

With the new “On-Demand Page Propagation” functionality, you can achieve this content distribution model out of the box.

"Update Variations" in MOSS 2007 works differently under the hood from its counterpart in SharePoint 2010

Things to keep in mind:

“On-Demand Page Propagation” affects the entire site collection; that is, if you enable this setting, no source page will be copied to any target page when the source page is published. Only the "Update Variations" button will cause pages to propagate when the timer job is enabled.
Source pages will be copied as draft minor versions to all target variation sites when you use the "Update Variations" button.

Thanks for reading. Happy propagating!

Josh Stickler
Program Manager

Email Management and SharePoint

Ryan Duguid — Tue, 15 Jun 2010 05:59:18 GMT

My colleagues in the Exchange team have introduced a wealth of new capabilities in Exchange 2010 to support email archiving, retention and discovery but I’m often asked how an organization should think about managing emails in SharePoint as part of an overall collaboration and content management strategy. While there are no hard and fast rules, it pays to think about four distinct scenarios:

Personal email management
Project and case management
Email archiving
Records management

Each of these scenarios has a set of desired outcomes and set of capabilities that best meet those outcomes so let’s take each one in turn.

Personal email management

Personal email management is all about empowering end users to take control of their inbox, making it easier to organize, find and take action on email. Users want a mail client that makes it easy to manage email on a day to day basis and expect their IT department to take care of backup and restore.

Project and case management

Project and case management is all about sharing information and managing a group of related artifacts in a single location with a common security model, metadata model and information management policy. Users are looking for a solution that makes it easy to collaborate and find information while leveraging workflow to drive common business processes.

Email archiving

Email archiving is all about taking control of the proliferation of email within an organization, driving down the cost of provisioning ever increasing inbox requirements and applying broad brush time based disposition. Email archiving is typically driven by IT who implement rules and retention policy that is typically transparent to end users.

Records management

Records management is all about identifying business critical content, driving appropriate classification and then applying relevant retention management policies. Accurate classification of content and applying appropriate metadata ensures that information is easy to find and use throughout the enterprise. At the same time, appropriate use of retention policies ensure that businesses can gracefully age content that is no longer of value while adhering to relevant government and industry regulations. Email is a critical part of any modern records management strategy and so businesses need to make it easy for end users to identify and classify email that is considered to be business critical content.

Outlook 2010 and Exchange 2010 provide great capabilities to deal with personal email management and email archiving while SharePoint 2010 provides an ideal platform for storing email that is part of project and case management or an effective and encompassing records management strategy.

Of course there is a natural flow or continuum as email may start by being well managed in a user’s inbox, it may have an email archiving policy attached to it but a user may decide to manage it as part of a project and then finally declare the email as a record upon project completion. As I said at the start, there are no hard and fast rules but hopefully I’ve given you a better frame of reference for working out what systems are required to support email from creation to disposition depending on the required business outcomes.

If you want to hear more about this topic, I’ll be presenting a webinar with Colligo, one of our partners who provide an add-in for Outlook that makes it easy for users to drag and drop email in to SharePoint, applying the appropriate Content Type and metadata attributes as part of the process. The webinar is on June 17th so sign up now.

Ryan Duguid
Senior Product Manager – ECM and Compliance
Microsoft Corporation

What’s New with the Content Query Web Part

Ryan Duguid — Fri, 14 May 2010 22:30:20 GMT

Howdy, my name is Dustin Anglin and I’m a Program Manager on the Enterprise Content Management team. Today I’m here to talk about the new things we’re doing with one of most popular SharePoint web parts, the Content Query Web Part, or as it’s better known amongst acronym-o-holics, the CQWP.

If you haven’t already checked it out, be sure to check out both the ECM overview by Jim Masson and the WCM overview by Sangya Singh.

CQWP for Newcomers

For those of you who are new to the CQWP, here’s a brief explanation of what it does and what kinds of things you can use it for.

The Content Query Web Part is a tool that site designers can use to aggregate interesting and relevant slices of information on web page by letting you build queries through an easy to use UI and then display that content in unique, configurable ways.

The CQWP is designed to return roll-up content over several different scopes, anywhere from a single a list or library, to all lists or libraries across an entire Site Collection. In SharePoint 2010, the CQWP single list query is also optimized to work over large single libraries, taking advantage of smart indices and other tools designed to improve query performance over large document libraries.

For a more in-depth overview of the CQWP, I’d recommend checking out the SharePoint 2007 blog post by George Perantatos.

Creating a Related News Stories Web Part with CQWP

I find the best way to learn about something is to give an example, so let’s go through archetypical scenario that uses the CQWP to show news stories related to the current news story I’m reading. Also, since we don’t want older stories, we’ll make sure our web part is set up to show only recent stories, say stories created in the last week.

Building the Query: Driving the Query based on Context

The Content Query Web Part is awesome for displaying queried content on a page, but I don’t want to have to add a new CQWP to a web part zone EVERY time I create a new news story, especially when I’m just going to set up the same web part over and over again.

In SharePoint 2007, you could add a CQWP to a page layout so it appeared on all pages of a certain type (for instance, in this scenario you could create a “News Article” page layout), the problem was, you could only set one query on that web part, which means each page using the “News Article” layout would return the same set of results.

What we want to do is have each “News Article” page show articles that are related to the current article I’m reading, like related sports stories if I’m reading a sports article.

We also need a metadata field to describe what kind of “News Article” the current article is, like Sports or Business, so we will add a new Managed Metadata field called “News Category” which we can link to the term store where we can create a managed taxonomy of news categories for our authors to pick from.

So here’s the conceptual outline for what our “News Article” page layout will look like:

I mentioned that we want our web part to be driven by the context of the current article, specifically the current articles “News Category.” In SharePoint 2010, we can accomplish this by setting up our query filters with two nifty new tokens:

PageFieldValue

This token lets you specify a field that is on the current page layout and dynamically replaces the value for the filter with the current page’s field value.

For Example: I’ve created a custom Managed Metadata field “News Category” which specifies what kind of story my “News Category” is (Business, Politics, World, Sports, etc…) and added it to the “News Article” page layout. By using the PageFieldValue token, the query will change based on the current pages News Category. So if the current page’s category is “Sports” it will return pages tagged with “Sports.”

PageQueryString

Just like the PageFieldValue token, you can also choose to look at the URL query string for a value to use in the Query.

For Example: You could add a Query String value to the URL like “&NewsCategory=Sports” and set the CQWP’s query to return pages where the News Category field was equal to “Sports” or whatever value was currently in the Query String.

Using the query string parameter is better when you want to show different sets of results in a web part without having to make a new page for each different results set.

For this example, we will use the PageFieldValue token to setup our CQWP to return news articles whose “News Category” field contains the same metadata value in the current article’s “News Category” field. And by modifying the custom “News Article” page layout, the CQWP will automatically appear on all new “News Article” pages, already set up and ready to display related news articles.

Looking a bit closer at the query, you can see that I’ve added two other filters. The first one filters out any article with the same name as the current article (essentially making sure the current article isn’t duplicated in the query results), and a filter that only returns news stories created in the past 7 days, making sure the results are always fresh.

Also, in this instance, the “News Category” metadata field is one of the new Managed Metadata fields, which the CQWP fully supports, including the single & multi-value varieties. While not shown in this example, the CQWP also supports filtering over the Enterprise Keywords columns & the “All Tags” column which lets you look for any managed metadata tag on a queried item, regardless of which metadata field on the item contains the tag.

Displaying the Results: Mapping Columns from the Tool Pane

Some people love writing XSLT, and to them I say write all the XSLT you want and more power to you. However, for the rest of us, and some might argue the majority of us (we still love you XSLT coders), figuring out how to show field X, Y, and Z by writing XSLT is last thing we want to do.

In SharePoint 2010, one of the coolest new features to the CQWP is the ability to choose what fields you want to display directly in the tool pane UI. So now that the query is defined, let’s open up the Presentation section and check it out:

Going to the Presentation section of the tool pane, there is a new section called Fields to display which lets you type in the fields you want to display for any defined XSL style. So think of the Item Style as a blank template, defining how many things can be displayed and where they go, but not which fields get displayed. Once you’ve selected your template, you just need to plug in the fields that you want to display by typing them into the slots that show up for the given item style.

I chose to use a style that had an image on the left, a link to the page that was represented by the articles title, and a brief excerpt of the article (a Custom Field I defined for my page layout). By typing in the fields I want to display in each of those slots, I get the following results:

Total time spent messing with XSLT: 0.0 seconds! Woohoo!

Wrapping it up

So to recap things, here’s what we accomplished:

Added a CQWP Web Part to all my news story pages by modifying the page layout in one place
Configured the Web Part query to be driven dynamically based on the current page’s context (sports articles return related sports articles, business articles return other business articles, etc…)
Configured the presentation of the web part to show custom fields without writing custom XSLT.

And here’s a look at the final product:

Now when my content authors write news stories, every article will have a web part showing the most recent related articles, and my content authors don’t have to mess around with web parts and queries.

In Summary

This is just one example of some of the cool new things you can do with the Content Query Web Part by leveraging power of context driven queries, and the easy-to-use, code-less way to configure how you display your content. Check back often for more info on the CQWP and other exciting new ECM features in the SharePoint 2010.

View Changes Makes In-Browser Content Localization Easier

Ryan Duguid — Wed, 05 May 2010 08:00:00 GMT

View Changes is a new SharePoint 2010 Variations feature that compares two versions of a source page relevant to the corresponding target page. View Changes provides a report highlighting the differences between the source version that has most recently propagated to the target and the prior source version that propagated to the target and was published on the target. By highlighting differences, the View Changes button simplifies in-browser content editing using the Variations feature.

I’m Josh Stickler, the Program Manager responsible for Variations. In this post, I will explain:

Where to find the View Changes button
When the View Changes button is available
Our envisioned content editing process using View Changes

The most common application of the Variations feature is in multi-language sites. Let's look at View Changes from the perspective of Anders, an English-to-Danish translator working with the Danish subsidiary of AdventureWorks, an international camping goods retailer.

AdventureWorks is set up with an English (EN-US) site as its source label and target labels for international markets, each corresponding to a different language. Pages from the source label automatically propagate to the target labels when they are published so AdventureWorks' global web presence is in sync. Translators at each of the targets then process the English-language content for localized consumption. AdventureWorks’ Variations hierarchy looks like this:

Initial Localization

Automatic Creation

Let's imagine that content authors at AdventureWorks in the United States have just published a new page with a sneak peek of this winter's new product lineup. Since “Automatic Creation” is enabled (this is the case by default), the page is picked up by the Variations Propagate Pages timer job and copied to all target labels, including Danish (DA-DK).

E-mail Notification

As the designated owner of the new page, Anders gets an e-mail informing him that this page has been copied to the target label by the Variations feature and is ready for processing.

Target Translation and Publication

Anders navigates to the page on the Danish (DA-DK) variation of the AdventureWorks website and sees the English language content. Since it's all new, he translates all of this content into Danish and submits the page for approval. The page is approved and published and now appears on the Danish variation of the website.

Since Anders received an entirely new page to translate, there were no changes to view; hence, the View Changes button is not available.

Subsequent Modification

Source Modification and Publication

Back in the United States, AdventureWorks decides to announce a new product in its sneak peek lineup. English language content authors add a paragraph describing this new product, an ultra light sleeping bag, and publish the page. The page now propagates to the Danish variation.

E-mail Notification

Anders receives an e-mail notification that new content is ready for processing. He visits the appropriate page on the Danish variation site and the English content appears and is waiting for translation.

But wait, there is a lot of English content here, and Anders has already translated most of it. Only one paragraph has been added. How will Anders know that he doesn’t need to re-translate the whole page?

It’s at this point that the View Changes button comes to the rescue and is available.

Please note that View Changes requires the Variations Propagate Pages timer job to be enabled. View Changes only compares changes between a source version of a page and a target version that has been copied using the Variations Propagate Pages operation.

Version Comparison

Anders clicks the button and a version differential window pops up, highlighting the new paragraph that has been added. Now, Anders knows that only this paragraph has been added and doesn't have to scan through the new and old versions of the English content to determine what he has to translate.

Restore Previous Version

Anders decides he prefers to revert back to the translated Danish version of the page as a basis for adding the new paragraph. With the View Changes window open, Anders knows exactly which paragraph to translate and where it goes. He adds the new content in Danish, submits for approval, and it's published live on AdventureWorks’ Danish variation site. Fantastisk.

In addition to providing target variation site translators with insight into what content has changed when pages are copied from the source, SharePoint 2010 also enables authors on the source to decide when to propagate content to targets. By default in MOSS 2007, when content authors published pages in the source variation site, that page would automatically propagate to all target variation sites, even for small changes that are relevant only to the source variation site.

SharePoint 2010 provides the ability to disable automatic page propagation; source variation site content authors can then use the Update Variations button to propagate content on demand. See my previous post, “Site and Page Propagation” for more information on how to enable this setting.

Thanks for reading! Keep checking back for new blog posts.

Regards
Josh Stickler
Program Manager

Web Analytics in SharePoint 2010: Insights into Reports and Metrics

Ryan Duguid — Mon, 03 May 2010 10:41:42 GMT

As part of SharePoint 2010, we have created a set of features to help you collect, report, and analyze the usage and effectiveness of your SharePoint 2010 deployment. These features are a part of the Web Analytics capabilities of SharePoint 2010. The overview of the Web Analytics features in SharePoint 2010 was presented in this blog post.

This blog post delves deeper into the various metrics available to analyze the site usage data. There are three categories of the SharePoint Web Analytics reports: Traffic, Search, and Inventory. The reports are aggregated for various SharePoint entities like Site, Site Collection, and Web Application for each farm. Further, reports are also aggregated per search service application. By default, the reports show the data for a period of 30 days but you can change the time period to view data for up to 25 months by going to ‘Analyze’ tab in the Ribbon UI.

Visually we show the metrics in one of the two ways: trend reports and rank reports. A trend report shows how a particular metric is doing over a period of time. While a rank report, shows the top 2000 results for a particular metric. Figures 1 and 2 show examples of a trend and rank report respectively. That’s not all; you can further analyze the reports by applying filters like string match in the URL, user name, queries, browser and others.

Figure 1: Example of a Trend Report showing Number of Page Views for each day for a default period of 30 days.

Figure 2: Example of a Rank Report showing the Top Pages sorted on the Number of Page Views for a default period of 30 days.

What follows is an overview of each type of the report and the associated metrics. Also, summarized are the kind of reports available for each level of aggregation i.e. Site, Site Collection and Web Application and Search Service Application.

Traffic Reports

The traffic reports capture the user behavior information related to total clicks, frequent users, popular pages, and information about navigation to and from the current SharePoint component.

Trend Reports

Number of Page Views: Total number of page views each day.
Number of Daily Unique Visitors: Total number of unique visitors each day. SharePoint authenticated users and anonymous users (using IP address) are counted as visitors.
Number of Referrers: Total number of unique URL’s external to the current entity (parent entity is treated as external as well), from where the users navigated to the current entity.

Rank Reports

Top Pages: Most viewed pages in the current entity.
Top Visitors: Most frequent visitors of the current entity.
Top Referrers: Top URL’s external to the current entity from where users navigated to the current entity.
Top Destinations: Similar to Referrers, these are the top external URL’s that the user visited from the current entity.
Top Browsers: Top browsers being used to visit the current entity.

Report Scope	Site	Site Collection	Web Application	Search Service Application
Number of Page Views
Number of Unique Visitors
Number of Referrers
Top Pages
Top Visitors
Top Referrers
Top Destinations
Top Browsers

Table 1: Summary of the traffic reports availability at different SharePoint hierarchy levels

Search Reports

The search reports capture the user behavior information related to the queries on the site.

Trend Reports

Number of Queries: Total number of queries each day.

Rank Reports

Top Queries: Most issued queries per day.
Failed Queries: Most issued queries for which either there were no results or the user did not click on any results.
No Result Queries: Most issued queries for which no results were returned.

Other Reports

Best Bet Suggestion Report: Recommends URLs as most likely results for particular search queries based on analysis of usage patterns. The site administrators can accept or reject these suggestions. If they accept, the corresponding query-URL pair is added to the search keywords list.
Best Bet Usage: Shows how Best Bet suggestions are doing over time. For every Best Bet query issued, the report shows the percentage of clicks on the Best Bet URL compared to other URLs.
Best Bet Action History Report: Tracks the actions performed by the site administrator on the ‘Best Bet Suggestion’ Report.

Report Scope	Site	Site Collection	Web Application	Search Service Application
Number of Queries
Top Queries
Failed Queries
No Result Queries
Best Bet Usage
Best Bet Suggestions
Best Bet Suggestion Action History

Table 2: Summary of the search reports availability at different SharePoint component hierarchy levels

Inventory Reports

The inventory reports are targeted to help the site administrators in managing the site by keeping track of the site structure and storage and version issues.

Trend Reports

Number of Site Collections: Total number of site collections for each Web Service Application for each day.
Storage Usage: Total storage used in Megabyte (MB) for a site collection and the ‘Maximum Storage Allowed’ in MB for each day.
Number of Sites: Total number of sites within each Site Collection for each day.

Rank Reports

Top Site Product Versions: The ‘Site Product Version’ sorted in the order of ‘Number of Sites’ or ‘Percentage of Overall’ sites using the corresponding version for this site collection.
Top Site Languages: The ‘Site Product Language’ sorted in the order of ‘Number of Sites’ or ‘Percentage of Overall’ sites using that language for this site collection.

Report Scope	Site	Site Collection	Web Application	Search Service Application
Number of Site Collections
Storage Usage
Number of Sites
Top Site Production Versions
Top Site Languages

Table 3: Summary of Inventory Reports availability at different SharePoint component hierarchy levels

Look out for more to come

Keep an eye out for more blogs on customizing the reports using Excel, using workflow feature to scheduled reports and alerts and adding the ‘What’s Popular’ Web Part to your pages.

Announcing the CMIS Connector for SharePoint

Ryan Duguid — Wed, 21 Apr 2010 16:58:00 GMT

Today at the SharePoint 2010 Summit @ AIIM Expo, Eric Swift (@eswift), General Manager of SharePoint Marketing announced that Microsoft with be shipping the CMIS Connector for SharePoint as part of the SharePoint Administrator Toolkit by the end of June 2010. The CMIS Connector for SharePoint provides a CMIS interface over the top of SharePoint as well as a CMIS consumer Web Part that can be used to display content from other CMIS enabled repositories.

CMIS (Content Management Interoperability Services) is a specification that Microsoft developed in along with IBM, EMC, Alfresco, OpenText, SAP and Oracle to enable greater interoperability between content management repositories and to enable a whole new range of Composite Content Applications that can be build agnostic of the underlying repository.

We see CMIS as a great solution to help our customers more effectively leverage content maintained in a heterogeneous environment but more importantly, we see the specification as a way to enable a whole new range of Composite Content Applications that can be build agnostic of the underlying repository.

For further reading on CMIS, visit these sites:

Public voting on the CMIS specification ends on April 30th and we expect that the specification will be ratified as a standard shortly afterwards. We are excited that our work on the specification alongside the other leading ECM vendors is coming to fruition and are looking forward to providing support for the standard in SharePoint 2010.

Update: The OASIS international consortium has announced the approval CMIS v1.0. We are excited that the standard has been ratified and look forward to a new era of interoperability in the ECM industry.

Ryan Duguid
Senior Product Manager
Microsoft Corporation

Get Ready for the SharePoint 2010 Summit @ AIIM Expo

Ryan Duguid — Fri, 16 Apr 2010 06:01:04 GMT

Over the course of the next few days, a team from Redmond will be making their way to Philadelphia. Our goal? To bring the SharePoint 2010 story to the East Coast of the USA through a series of educational sessions and our Customer Immersion Experience. We’ll be delivering this content at the AIIM Expo + Conference and we hope you can join us to learn from the people behind the product. If you register for a main conference pass, you’ll get access to 28 sessions covering product capabilities and best practices. In addition, if you register (for FREE) for entry to the Expo Hall, you’ll have access to our Customer Immersion Experience where you can get hands on with the latest release of Office 2010 and SharePoint 2010.

We’re excited about the upcoming release of Office 2010 and SharePoint 2010 and are looking forward to meeting with you in Philadelphia. Before we head out East, I’d like to introduce you to our speakers:

Microsoft Executives
	Eric Swift	As General Manager of Product Management for SharePoint, Eric Swift is responsible for managing customer and industry requirements, product positioning, licensing, and marketing strategies for Microsoft’s Collaboration Platform for the Enterprise and Internet. Swift has been with Microsoft for nine years. Previous to his current position, he had roles as General Manager of the Unified Communications Group and Director of Product Management in Microsoft’s Application Platform Group. Prior to joining Microsoft, Swift held Vice President positions at Enterprise Application Integration and CRM software vendors where responsibilities included product management, CRM, Data Warehouse implementations, and technical support operations. Swift has an MBA from Columbia University in New York, NY focused on marketing of information technology and has studied at the school of public administration and business at Fundação Getulio Vargas in Sao Paulo, Brazil.
	Tricia Bush	As Director of the Microsoft SharePoint Internet business, Tricia Bush oversees the SharePoint For Internet Sites and FAST Search for Internet Sites product management. This group is responsible for the foundation driving Microsoft’s digital marketing strategy. Bush joined Microsoft in March, 2005, and has over fifteen years of experience in technology.
	Christian Finn	Christian Finn is a director for product management on the SharePoint team in Redmond. My team is responsible for global product management for SharePoint in the collaboration, portals, social computing, and application development arenas. We manage the Collaboration Capability campaign in BPIO. We also look after interoperability and CPE for SharePoint.
	Nishan DeSilva	Nishan DeSilva is the Director of Information Management & Corporate Records Compliance at Microsoft. Currently leading the LCA’s information management and compliance program using SharePoint 2010 and has accountability for the policies governing Microsoft’s recorded information assets.
Microsoft SharePoint ECM Engineering Team
	Quentin Christensen	Quentin Christensen is a Program Manager on the SharePoint Enterprise Content Management team, specifically working on document and records management. Some of the areas he works on include eDiscovery, policy, document sets, and large scale document repositories. Quentin has authored white papers on large list performance and capacity planning for large document repositories using SharePoint Server 2010.
	Lincoln DeMaris	Lincoln DeMaris is a program manager on the Enterprise Content Management team at Microsoft. He has worked primarily on document management and taxonomy features during his 4 years at the company.
	Ethan Gur-esh	Ethan Gur-esh has been a Program Manager on the SharePoint Enterprise Content Management team since 2004. He worked on Records Management and Compliance during the SharePoint 2007 release, and is currently working on Document Management, Rich Media, and Web Content Management for the SharePoint 2010 release. Additionally, Ethan is the Co-Editor and Secretary of the Content Management and Interoperability Services Specification Technical Committee at OASIS.
	Dan Kogan	Daniel Kogan is a Senior Program Manager in the SharePoint team at Microsoft Corp. He has nearly 20 years’ experience in the IT and software business. Daniel has been in the Web content and Enterprise Content Management space since 1998 and has been at Microsoft since 2001. For the past 4 years Daniel has focused extensively on taxonomies and metadata and how they can be used to enhance productivity and unlock new business potentials and scenarios.
	Kevin Reynolds	Kevin Reynolds is a Program Manager on the SharePoint Enterprise Content Management team and has a passion for customer focused design. He works on a breadth of the Web Content Management features including Master Pages, Page Layouts, Navigation, RTE, and the Large Pages Libraries.
Microsoft SharePoint Product Management Team
	Ryan Duguid	Ryan Duguid is a Senior Product Manager in the IW PMG. Ryan is responsible for Enterprise Content Management and eDiscovery. Ryan has worked in the IT industry in New Zealand, the United States and the United Kingdom for over 15 years. He is passionate about understanding people, identifying their unique problems and helping them to realize their true potential through effective and innovative use of technology.
	Dave Pae	Dave Pae is a technical product manager on the SharePoint team in Redmond, WA. Dave has worked on web and collaboration technologies for over 15 years and started working at Microsoft in 2001. He is focused on the product management of SharePoint specifically for social and collaboration scenarios for 2010 and beyond.
	Pej Javaheri	Pej Javaheri is an industry veteran, having worked in the Business Intelligence (BI) and performance management space for more than 15 years, focusing on helping organizations gain insight, and make better decisions. Part of the SharePoint team, Pej works across Microsoft to bring the bigger BI message to customers and partners, focusing on how the integration of software, data in all its forms, and people can help move organizations forward.
	Erik Schwartz	Erik Schwartz is a Product Manager in the Microsoft Enterprise Search Group. Along with his responsibilities for core product management for connectors and push features for search products, he focuses on customer and field communications, eDiscovery, and key vertical markets, including government globally. Schwartz has managed technical teams of IT Professionals and Software Engineers, and has worked as a Contractor at the Naval Research Laboratory.
	Owen Allen	Owen Allen is a Sr. Product Manager on the SharePoint Partner Marketing Team. His area of focus is SharePoint Partners, and specifically, ISV partners.
Microsoft SharePoint Sales and Evangelism
	Geoffrey Edge	Geoffrey Edge is a Senior SharePoint Technology Specialist working for the Communications Sector North America. His responsibility is to help customers in the Communications Sector learn more about SharePoint Products and Technologies. Geoffrey’s focuses on Enterprise Search and large scale SharePoint deployments.
	Paul Stubbs	Paul Stubbs is a Microsoft Technical Evangelist for SharePoint and Office. He focuses on information worker development community around SharePoint and Office, Silverlight, and Web 2.0 social networking.

This is the largest gathering of Microsoft speakers since our SharePoint Conference in Las Vegas last year and we’re looking forward to meeting you in person next week. We hope you can attend the SharePoint 2010 Summit @ AIIM Expo or join us on the Expo Hall floor. Be sure to bring your burning SharePoint questions and make the most of this opportunity to talk with the experts.

Ryan Duguid
Senior Product Manager
Microsoft Corporation

Variations in SharePoint 2010 – Connecting People with Content

Ryan Duguid — Mon, 12 Apr 2010 20:38:00 GMT

When you provision a new SharePoint publishing site, one of the first options you’ll see on the default welcome page is to use the Variations feature to manage multi-lingual sites and pages. My name is Josh Stickler and I'm the Program Manager responsible for Variations. In this post, I'll provide a brief overview of the Variations feature and highlight main improvements in SharePoint 2010.

If there are additional areas that are of particular interest to you, please post in the comments section and I will try to address as many as I can. I’d really appreciate getting any and all feedback. Thanks!

What is the Variations feature?

Variations is a SharePoint feature that facilitates the management and maintenance of content that can be served to multiple audiences. These audiences can vary in terms of different languages, countries, or regions, but they can also represent different brands or devices.

How does Variations work?

For each channel you wish to serve content, you can specify a Variations label. Labels are instantiated as SharePoint publishing sites and the full set of labels in a site collection is referred to as the Variations Hierarchy. I refer to SharePoint publishing sites created and managed by the Variations feature as “variation sites.”

Using variations, target variation sites reflect one source variation site in terms of pages and site structure. When setting up variations, specify one variation site as the source; all other variation sites are targets. By default, pages published on the source variation site are copied to all target variation sites as draft versions and sites created on the source are created (not copied – this is an important distinction) on all target variation sites. You can only have one source variation site per Variation Hierarchy and you can only have one Variation Hierarchy per site collection.

What’s new in SharePoint 2010?

The concept and core architecture of Variations, in which pages and site structure are replicated across multiple variation sites in a site collection remains the same as in Microsoft Office SharePoint Server 2007; however, we have made significant improvements to better meet the needs of enterprise customers serving content across multiple channels.

These improvements can be divided into four categories:

Server Citizenship
Content Distribution
Editing Experience
Reliability

Server Citizenship

Variations operations now execute in the background via timer jobs. For the end user, this means that you no longer have to wait at a progress screen for operations to complete. For the system administrator, this means that the cost of resource-intensive operations like Create Hierarchies can be better managed.

You can adjust the frequency with which Variations operations run in Central Administration. Next, I’ll explain the difference between the “Create” and “Propagate” timer jobs in the context of improvements we’ve made to the Variations content distribution models.

Site and Page Propagation

MOSS 2007 featured two models for distributing pages across your Variations Hierarchy:

1. Automatic Creation: If “Automatic Creation” is enabled on the Variation settings page (it is enabled by default), then publishing a page on the source variation site will cause that page to be copied to all target variation sites.

2. Manual Creation: If “Automatic Creation” is disabled, then the “Create Variations” Ribbon button is the only way to copy a new page to a specific, individual target variation site.

We’ve received feedback that there are often cases in which changes need to be published locally to the source variation site without being propagated to all targets. For instance, if the source variation site has a typo in English, the correction may not be relevant to a target site in German, so if the correction is published in the source page, it can be unnecessarily confusing to copy this changed English version to all target sites.

In SharePoint 2010, we introduce a third, “hybrid” content distribution model:

3. On-Demand Page Propagation

A setting has been added (configurable through the Object Model) to disable Automatic Page Propagation. When the setting is enabled, publishing or approving a page on the source variation site will not cause that page to be copied to any target variation sites. The "Automatic Creation" setting will be ignored for pages. "Update Variation" and "Create Variation” are the means by which a user can distribute content across the Variation hierarchy on-demand.

I’ll go into more detail on content distribution models in a future post. But so as not to keep you in suspense on how to configure on-demand page propagation, here are the PowerShell commands:

Enable On-Demand Page Propagation:

Disable On-Demand Page Propagation:

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
$site = new-object Microsoft.SharePoint.SPSite("http://yourserver/sites/abc")
$folder = $site.RootWeb.Lists["Relationships List"].RootFolder
$folder.Properties.Remove("DisableAutomaticPropagation")
$folder.Update();

We’ve also made improvements for target variation site content owners to better understand what has changed on the source variation site when new draft versions appear on a target variation site.

Editing Experience

To make efficient use of their time and effort, target variation content editors need an easy and informative way to determine what content is new when pages are propagated from the source variation.

A new “View Changes” button compares the most recent source version propagated to the target with the most recent source version published on the target. Changes are highlighted in a pop-up report to enable content processing directly in the rich-text editor.

Highlighted report

Corresponding location in the Rich Text Editor

This button is available on a target variation page after it has been published once and a new draft version has been copied from the source variation site via one of the Variations timer jobs. I will go into more detail on this new feature in an upcoming blog post dedicated to explaining View Changes with screenshots, a sample workflow, and an example scenario.

Reliability

One of our main goals for Variations in SharePoint 2010 is to make the feature more reliable so enterprise customers can entrust management and distribution of content across multiple channels to Variations.

Now that Create Hierarchies runs in the timer service, we support pausing and resuming this operation during timer service recycles to support long-running operations in large deployments. This also means that the process is not affected by Application Pool recycles. We’ve also made the relationships list, which tracks all target pages linked to a source page, more robust. We now track variations pages using GUIDs for better performance and scale.

Thanks for reading. Check back soon for upcoming blog posts on what’s new in Variations and other exciting developments in Enterprise Content Management.

Regards,

Josh Stickler

Program Manager

Web Authoring in SharePoint 2010

Ryan Duguid — Fri, 09 Apr 2010 21:04:00 GMT

Hi, my name is Kevin Reynolds and I’m a Program Manager on the SharePoint team. Today I will walk you through the process for creating the page above, from creating the page to having it go live on the internet. I will show you the enhanced Web Authoring experience in SharePoint 2010, including editing content, applying styles, using the new UI, changing the layout of the page, and even applying themes to your site. I would highly encourage you to create your own Publishing site and follow along to get a personal feel for the SharePoint 2010 Authoring Experience.

Create New Page

Let’s begin with creating a new page. To create a new page click on the Site Actions menu and choose the New Page option, now in the dialog that comes up type in a name for the page – for this example we will use SharePoint 2010 Communities - feel free to insert your own name. Here is what you should currently see:

Now click Create on the dialog. A new page is created and you can see the Ribbon ()at the top of the page that exposes the most common options that you will use while editing the page. The page should look a lot like this:

Add and Edit Rich Text

Now let’s add some content into the Page and then we’ll get back to exploring more options available in the Ribbon. For this example I’ll add in the following text:

SharePoint 2010 Communities: Work Together Effectively
As part of the 2010 release, SharePoint Communities provides a comprehensive, flexible platform that empowers people to work together in ways that are most effective for them. Allow your people to collaborate in groups, share knowledge and ideas, connect with colleagues, and find information and experts naturally.
Work Together the Way You Want
The global workforce of the twenty-first century is more diverse than ever. Connect and engage all of your employees with a flexible collaboration platform and a diverse set of tools that range from Wikis to Workflows to Workspaces—allowing people to work together the way they want.
Rely on a Secure Collaboration Platform
Let your IT staff rely on an enterprise-ready collaboration platform that is secure and easy to manage and will support your organization’s growing needs. SharePoint 2010 makes social safe with granular security and privacy controls, centralized management and policy setting, and robust reporting and analysis.
Extend the Value of Your Community Solutions
The SharePoint platform seamlessly integrates with the rest of the Microsoft Business Productivity infrastructure, including the Office applications, Exchange Server, Office Communications Server, SQL Server, and Dynamics. In addition, SharePoint provides Business Connectivity Services and adheres to open standards and protocols, making it easy to integrate third-party applications.

Feel free to copy and paste that text into your page as you follow along. We will use the text above to demonstrate the functionality of the editor on the page. Select the first line of text SharePoint 2010 Communities: Work Together Effectively, click on the font color drop down (), put your mouse over the red color, notice how the selected text turns red, now select the orange color, and notice how the text turns orange. We will change the font size now, choose the font size drop down (), choose 18 from the list and notice that the selected text now becomes larger. Choose the text work together the Way You Want, click the Markup Styles menu (), select Heading 1, and now do that for Rely on a Secure Collaboration Platform and Extend the value of Your Community Solutions. Now for those users savvy in HTML if you look at the markup of the page you will notice that the text is wrapped in a <H1> header tags, so you’ve applied a style and have well formed markup. If that last sentence doesn’t mean much to you, no worries, you can just use the menu as a set of styles on your text and leave the HTML markup thoughts to the experts. Now take a moment to play around with the text yourself, apply some fonts, apply some colors, highlights, font size, or adjust your paragraph alignment. No rush, I’ll wait…Really, it’s ok you can come back and continue the blog in a couple of minutes…Welcome back, here is roughly what the current page will look like depending on the formatting you’ve tried out:

Change the Page Layout

We will change the layout of the page, this will allow us to use a standard template that helps us to layout content in a consistent way across the site. Now to change the layout go to the Page () tab, select Page Layout (), and now you can choose a new layout for your page. For this demo I’ll be using a custom page layout – In a later blog I will show you how to create your own page layouts. Click on the Image on right layout and noticed all the new fields that show up and how the page is laid out differently now.

Insert a Picture

The new layout that we have chosen has a Page Image control that allows us to insert a picture onto the page in a specific location. To insert a picture click on the Click here to insert a picture from SharePoint text, then on the dialog that comes up click the first Browse… button, this launches the new Asset Picker, that allows you to choose an image that is already stored on SharePoint, if you haven’t uploaded any pictures don’t worry there are a few that come in the box, you can go to Site Collection Images, choose the Home picture, click OK, and click OK on the next dialog. You’ve inserted your first picture into a page in SharePoint 2010. The page should look like the following:

Insert a Video

In SharePoint 2010 we have enhanced the richness of the media that is natively integrated into pages and now everyone can easily add video and audio files to their page. To add a video put your selection below the text on the page, click the Insert Ribbon tab (), click the Video and Audio button (), now click on the Media Web Part () that is inserted into the page, and you will see a new contextual tab come up that contains commands specific to the Media Web Part:

Click the bottom part of the Change Media button to drop down a menu and choose from computer, this will bring up a new dialog where you can upload a video, click the Browse… button, choose a video on your computer, click OK, change the Upload to: box to be Images, and click OK, and Save on the dialog that comes up after the video is uploaded. You have now inserted your first video in SharePoint 2010, take a moment to use the player, watch the video, and play with the features. The video player will be covered in-depth in an upcoming blog post.

Theme the Site

We have also enhanced the theming capabilities in SharePoint 2010 to make it easy to apply a new set of colors to your site. This will give your site an updated look and feel which can easily be created and updated as your needs change. To update the theme of the site go to Site Actions and choose Site Settings, this will bring up a new page with a bunch of links, click on the Site Theme link, and now you will be in the new theming UI for SharePoint 2010.

We will cover theming of the site and this entire UI in a later blog post, for now let’s update the theme that goes with our content, in the large box with a list of theme choose the Ricasso theme, and click OK at the bottom of the page. Now navigate back to your page and you’ll see that the colors of your page have been updated, according to the new theme that you had chosen:

Apply a New Master Page

The page is really coming together, now we will see how easy it is to change our Master Page. The Master Page is the main component with theming that gives a site it’s look and feel. A master page defines where the company logo goes (or if there is one), where the Ribbon shows up, where the search box is, and all the common elements that should apply to every page. To update the master page go to Site Actions, choose Site Settings, and then on the Site Settings page click on the Master page link.

In the section labeled Site Master Page, click on the drop down box that currently says nightandday.master and change it to v4.master. This tells SharePoint that for this site you want all pages that you author to us the v4.master master page. Now navigate back to the page that you’ve been creating:

Now that you have all the right content and the page looks good, it’s time to get it live.

Go Live

To make the page available to others you will submit it for approval using the Ribbon. This will send the page off to the appropriate approvers for these pages and they will review it and then publish it to customers. To submit this page for approval go to the Publish tab and click the Submit button (), this will bring up a new dialog that will check the spelling and allow you to add comments for the reviewer of the page , add in a comment and click Continue, this will start the approval workflow for the page, a new form will come up, click Start, and now you will be taken back to the page. Now the approver will review the page and Publish it to go live.

You have now created your first page in SharePoint 2010 and you already know how to add pictures, insert videos, change the layout of the page, update the site theme, change the master page, and publish the page to go live. We will go deeper into each of these topics in future blog posts.

Thank you for reading and for following along,

Kevin Reynolds

Program Manager

Enterprise Content Management

It Looks Like You’re Building a Large Library. Would you Like Help?

Ryan Duguid — Mon, 05 Apr 2010 05:37:50 GMT

SharePoint 2010 is more than just SharePoint 2007 plus a bunch of new bullet points on the box. We didn’t just haphazardly build a bunch of new features, look back at the fertile seeds we planted, and muse about how “everything should work pretty well as libraries get large.” We built, and more importantly, tested all the features you’re reading about with scale in mind. We are setting new scale targets for 2010 that go above and beyond what we set in 2007. These numbers are not final yet, but we're shooting for tens of millions of documents in a single library, depending on some specific parameters of your scenario.

When I throw out numbers like that, I’m not talking about just big, static libraries with content that just sits there. We want you to do crazy things with SharePoint 2010 like stuff a million document sets in a single document library with workflows running every which way, a hundred different retention policies firing off actions when you least expect them, and users uploading, tagging, and searching day in and day out. All the goodness of the SharePoint platform will be available to you whether you’re building a team site, a collaborative repository, a knowledge base, or a super large archive.

Like a plump, juicy sausage, much of the good stuff in SharePoint 2010 to give it delicious scalability are things that most people don’t need (or want) to know about. For the most part, scale just works. However, the chef (or information architect) is still a super important player. A well-planned repository is one that will have your users coming back for seconds and writing rave reviews; a poorly-planned one is one that will have them chugging Pepto-Bismol the next morning. Just because you can stuff a bunch of documents in a SharePoint 2010 library without your server igniting in flames the next day at doesn’t mean that you should without first thinking through how to best use the tools available to deliver an excellent experience to your end users.

So, even though scale in SharePoint 2010 just works, you’re not going to install the bits on day 1 and have a massive, searchable, beautiful content storefront on day 2. Guidance still matters, and believe me, we know it; this blog entry is just the beginning of the content we’re planning on delivering to help you on this front. I wouldn��t even call this blog entry guidance; it’s just a primer on the features and capabilities of SharePoint 2010 that you will grow to love if you’re passionate about scale at the library level – if you want to shove a whole bunch of documents in one place and have it be a great experience for both IT and your end users.

So what are these features and capabilities? Here are a few of the most important ones that I’m going to blog about now and in the near future:

We protect your database backend from dangerous queries. If you run a query against any database that requires it to scan through millions of items to find the ones you’re asking for, you’re going to balk at how long it ties up the server’s CPU. Quite frankly, SharePoint is not an exception. Even in SharePoint 2010, there is a class of user operations in certain scenarios that make unreasonable demands on the backend. For these operations, our strategy is to nip them in the bud before they’re executed, which keeps your high-demand servers healthy and responsive. Knowing when this throttling will kick in and planning for it is an important part of large list planning.
We give end-users tools to find content. When you have a sea of documents, the specific one you’re looking for can seem like a needle in a haystack. Structured metadata, easy tagging, metadata navigation, and built-in search refinement make this a less daunting task in SharePoint 2010 out of the box. This is an area we are particularly passionate about; after all, what good is a hugely scalable library if your end users hate it and can’t find what they’re looking for?
We help developers write excellent code. In SharePoint 2007, we didn’t give developers the right tools to write code that scaled well as the amount of content in your site grew. Even worse, it was pretty hard to tell why and when code was bad, and if your site was running slowly, which one out of your ten custom web parts was bringing things down. You had to “build around” SharePoint and do things “just so” to avoid this from happening. In SharePoint 2010, we give you a bunch of tools to make this story better.

Dangerous Queries

One challenge we’ve consistently seen customers run into when building large repositories on SharePoint 2007 is trouble with large containers. As the number of documents in any single container grows – either at the root of a library, or in a folder – bad things start to happen. For one, as your document to container ratio increases, it becomes harder and harder to find exactly what you’re looking for. More serious are the performance implications of large containers. Any of the out of the box ways of retrieving content from containers in SharePoint 2007 – like the All Documents view, the Explorer view, or a Content Query web part – would work, but they don’t scale very well. Loading All Documents in a library with a million items at the root would take a long time to finish. The big problem here is that you wouldn’t be the only one affected; all your friends running SharePoint sites on that same database server would experience things slowing to a crawl as well, as the database server dutifully iterated over those million documents to find the right ones.

Why does this happen? Any time you ask for content from SharePoint, you have to specify how it’s sorted – for example, the All Documents view in SharePoint 2007 asks for the top 100 results, sorted by filename. But items aren’t sorted by filename in the SharePoint content database – so, to bring you this view, SharePoint has to gather up all these million items, sort them, and finally display the 100 ones at the top of the sorted list. Imagine this as being like flipping through the residential section of a phonebook to find the first 100 addresses, sorted in alphabetical order. This would be a miserable task, because the telephone book isn’t sorted in this way – so in order to ensure your sorted list was accurate in the end, you’d have to look through the entire residential section, from start to finish, because after all, the last person listed in the phone book might live at 1000 Aardvark Lane.

Large Lists and Fallback Queries

The laws of physics are the same in SharePoint 2010 as they were in SharePoint 2007; if you run a query that needs to touch a very large number of items, you’re going to have to wait a long time, and so will everybody else. One prominent thing we did in SharePoint 2010 is to nip these queries in the bud before they get executed. To make a long story short (you can read the long story here), a farm administrator can set a threshold which defines the maximum number of items a single SharePoint query can touch. By default this threshold is 5,000. Any library with more items than this threshold is a large list.

Let’s go back to our example of the library with one million items at the root. Say you had that library in SharePoint 2007, and you upgraded to SharePoint Server 2010. First thing you’ll see upon navigating to this library will look something like this:

See the yellow bar above the list view? That’s a sign you have the Metadata Navigation and Filtering site feature turned on and it’s causing something magical to happen! When you load this view, SharePoint 2010 knows that you’re being greedy and asking it to scan through those million items. Since this query exceeds the maximum number of items a single query is allowed to scan (5,000) it doesn’t run the query. But who wants to stare at an empty list view? Instead of running this query as-is, SharePoint finagles it a bit and transforms it into a query that’s almost as good as the one you were asking for, but won’t make the database buckle under the pressure. In this case, we assume that it’s fairly likely that the document you’re looking for is one of the most recently created items in the library – so instead of scanning all one million items, we only scan the top 1,000 or so recently created documents, sort those by filename, and show them to you in the list view. This is what we call a simple fallback query: a query that doesn’t specify an index and asks for too many items in return, so instead of considering the entire list as being eligible for the query, SharePoint considers only the thousand or so most recently added items.

“Wait a second. You’re telling me that SharePoint throttles queries without asking me first? How on earth am I supposed to find anything in this crazy world of fallback queries and partial results?”

Let me assure you; this throttling business is a good thing. It’s a core ingredient in what makes SharePoint 2010 a resource for addressing your scale challenges. Gone are the sleepless nights where you toss and turn and worry about page faults on your database cluster resulting from Mack in Accounting stuffing 6,000 beer pong tournament photos in the root of a library in a forgotten team site in the dusty corners of your SharePoint deployment. The SharePoint 2010 feature set replaces this overarching concern with a set of well-scoped challenges; instead of worrying about every library that might get big, you get to plan for and craft experiences for the set of libraries that need to get big for business reasons.

I should mention really quickly that throttling is about more than just list views. There is a whole class of operations that involve iterating through all the documents in a list, or all the documents in a folder, that will get throttled (in other words, they will not execute) when the list or container is large. These operations include things like:

Adding a column to a library
Creating an index on a library
Deleting a large folder in a library

Metadata Navigation – finding and working on content in large lists

Above is another screenshot from my million item library. This time, we’ve put a couple of SharePoint 2010 features to work. See that I have “demonstration scripts” selected in the left hand side in the tree view, and my list view is rendering without the yellow bar that’s telling me I’m only seeing newest results. That hierarchy of tags you see there represents a taxonomy, Item Type. I am browsing the documents in this library according to their Item Type; in the screenshot, I am filtering to show all documents with the value “demonstration scripts”. Here are the steps that I took to make this happen:

I created a taxonomy that describes my content. You can look forward to some posts from our very own Dan Kogan on this very topic in this very blog in the near future. There’s a lot to learn here. Not just any taxonomy will do here; it needs to be one that broadly divides my content up into evenly-sized buckets. For example, if I had 990,000 demonstration scripts, the query you see above would not get me anywhere. In that case, it wouldn’t make much sense to use Item Type as a piece of metadata and a navigation hierarchy for this library; I would need to find another, more divisive way to pivot the data.
I bound that taxonomy to a field in my library called Item Type. Think of a taxonomy field as a choice field on steroids. Instead of picking values out of a flat list, you pick them out of a tree.
I configured that field as a navigation hierarchy. Every library now has a Metadata Navigation and Filtering settings page where you can configure navigation hierarchies (the filters you see arranged in the tree view) and key filters (the additional filters that show up beneath the tree view)

In these three easy steps, I made “Item Type” a first class navigational pivot over the data. Instead of just staring at a partial list of content at the root, I can now browse with impunity by this virtual folder structure.

Here’s a couple of cool aspects of this feature that aren’t apparent from a single nifty screenshot:

Metadata navigation lets you slice and dice multiple ways. I might have a bunch of taxonomies on my library that classify content in different ways; for example, I might have a Products field, a Region field and a Competitors field, all bound to domain-specific taxonomies that classify the content along those dimensions. Depending on my current task, it might make more sense to filter by the Region field (for example, if I’m looking for the latest sales figures for the North America region). I get more filters than just my virtual folder; I can combine this filter with any number of key filters or list view column filters to drill down to just the content I want (for example, I want to see all demonstration scripts by the ECM team created after 2007).
Metadata navigation thinks about indices and large lists so you don’t have to. Hey, remember just a few minutes ago when we were talking about large lists, indices, and being throttled? Well, metadata navigation thinks a lot about indices and how to run queries the “right way” to make them perform well and prevent throttling from happening. For starters, all the fields you configure as navigation hierarchies and key filters get indexed, and the resulting queries are written in a way that ensures the best index is used to make the query succeed.

You aren’t immune from the laws of physics; if you ask for documents tagged with demonstration scripts and there are 10,000 demonstration scripts, we’re not going to be able to show you all of them. In this case, though, you get something better than a simple fallback; you get an indexed fallback, which means that instead of considering the entire list, the query considers only the items that match the indexed portion of your query.

Wrap-up

This article was just the first in my series of posts about architecting and building large lists filled with discoverable content. Here’s what you can expect over the next few weeks:

A deep dive on metadata navigation, how it works, and some tips to getting the most out of it
A discussion on how other features, like Search and the Content Query Web Part, fit into the equation, and how to configure their metadata filtering capabilities
Some geeky developer tips on writing code that plays nicely with large lists

After that, I’ll be widening my scope a bit to talk about the overall knowledge management story in SharePoint 2010 – which is about more than just browsing for content in a library!

Lincoln DeMaris, Program Manager, ECM

Introducing Web Analytics in SharePoint 2010

Ryan Duguid — Sun, 21 Mar 2010 04:09:55 GMT

As part of SharePoint Server 2010, we have created a new set of features to help you collect, report, and analyze the usage and effectiveness of your SharePoint 2010 deployment – whether it’s used as an internal or external web portal, a collaboration tool or a document and records management repository. These features are part of the Web Analytics capabilities of SharePoint 2010.

This blog post is the first of several that will give you more insight into the enhanced Web Analytics features that we have built into SharePoint 2010. This first post will provide an overview of the new Web Analytics features and we’ll take a deep dive in to specific scenarios in future posts.

Overview

Web Analytics Reports

In SharePoint 2010, we have improved the set of Web Analytics reports that are available out-of-the-box, which will provide insights into the behavior of users of your SharePoint sites. There are three categories of reports that you will find:

Traffic reports: These reports provide metrics such as:

How much traffic your site gets (Number of Page Views);
Who visits your sites (Top Visitors);
How visitors arrive at your site (Top Referrers);
Daily Unique Visitors, Top Destinations, Top Browsers, etc;

Search reports: These reports give you insight into what users are searching for, for example:

How many times users searched (Number of Queries);
What were the most used search terms (Top Queries);
What queries have high failure rates (Failed Queries);
Best Bet Usage, Search keywords, etc;

Inventory reports: These reports display key metrics regarding the inventory of your sites:

What is the total disk drive space user (Storage Usage);
How many sites exist (Number of Sites);
Top Site Product Versions, Top Site Languages, etc;

We aggregate these reports aggregated at the following levels:

Per web application in the farm
Per site collection
Per site
Per search service application

Out-of-the-box, these reports are visible to Administrators at each level. For example, site-level reports are available to Site Administrators of those sites. We have also added a new permission level, “View Web Analytics Data,” that will allow users to access these reports without having to give them Administrator privileges.

You can access Web Analytics reports by going to Site Actions -> Site Settings. Under the Site Actions heading you will see two links, Site Web Analytics Reports and Site Collection Web Analytics Reports.

When you click on either link, you are taken to an overview page that shows you key metrics for your site. You can then drill down to other reports by clicking on them on the left navigation. You can also change the date range for the reports by clicking on the Analyze tab on the Ribbon.

Custom Web Analytics Reports

The out-of-the-box reports are useful to get a general understanding of what is happening on your sites. However, we have made it easy for you to get a deeper level of analysis, or to simply create your own reports. To get started, click on the Customize Report button under the Analyze tab in the Ribbon. Clicking this button will export the data contained in this report to Excel. Excel is a power analytics tools and makes it easy for non technical users to add your own charts, set specific filters, and combine data from multiple reports. In addition, the data within Excel is refreshable, which means that, once you customize the report, it will always be up-to-date with the latest data. To get more details on the great new features in Excel 2010 for building charts, reports and pivot tables, take a look at the Excel Team blog.

Web Analytics Workflows

Web Analytics Workflows is a powerful new feature set that enables you to get reports sent out either on a schedule or when specific conditions are met. For example, you can set them up to receive an email every time the total number of pages views drop by 80% week over week.

To setup a Web Analytics Workflow, go to the Web Analytics report that you are interested in and click on Schedule Alerts or Reports on the Analyze tab in the Ribbon.

Clicking this button will guide you through a series of steps to create your Workflow.

Best Bets Suggestions

Best Bets allow Search Administrators to determine what the most relevant search result is for a given keyword. Up until now, Search Administrators had to look at different reports and data to determine which best bets needed to be added. That process is no longer necessary as SharePoint 2010 periodically sends out suggestions for new Best Bets using all the search metrics it has collected. Now, Search Administrators can simply look through each of the Best Bet suggestions and easily accept or reject them.

To access the Best Bet Suggestions, go to Site Actions, click on Site Collection Web Analytics Reports, and the click on Best Bets Suggestions on the left navigation.

Web Analytics Web Part

We have created a new web part, the Web Analytics Web Part, targeted at Site Managers. This new Web Part is an end-user facing Web Part that can be easily inserted into any page on your site. It can be configured to display the ‘most viewed content’ or the ‘most frequent search queries’ in the site. The data in the Web Part is continuously refreshed as new content or new search queries become more popular.

To use this Web Part, go into the Edit mode of one of your Site Pages and click on any place you can add a Web Part. Then, from the Insert tab on the Ribbon, click on Web Part. Finally, click on the Content Rollup category and select the Web Analytics Web Part.

After you have inserted the Web Analytics Web Part, you can then configure it to display the data you are interested in.

Conclusion

Using the new Web Analytics features in SharePoint 2010, you will be able to get a deeper understanding of what users are doing, what they want from your site and how you can tailor the SharePoint experience to bets meet their needs. Keep an eye out for future posts where we will delve deeper into each of the features mentioned above.

EDiscovery in SharePoint Server 2010

Ryan Duguid — Tue, 16 Mar 2010 04:36:45 GMT

Hi everyone, I am Quentin Christensen and I work on document and records management functionality for SharePoint. Electronic discovery (commonly referred to as eDiscovery) is an area we are supporting with new set of capabilities in SharePoint Server 2010. In case you are not familiar with eDiscovery, it is the process of finding, preserving, analyzing and producing content in electronic formats as required by litigation or investigations. eDiscovery is an important concern for all of our customers and given that SharePoint has grown to be an integral part of collaboration, document, and records management for many organizations, we recognize the need to support the eDiscovery process for SharePoint content.

Microsoft Office SharePoint Server 2007 included a hold feature that could be used for eDiscovery, but it was scoped to the Records Center site template. With SharePoint Server 2010 the eDiscovery capabilities have been greatly expanded to provide more functionality and the power to use these features across your entire SharePoint deployment.

In this post, I want to highlight three major improvements in SharePoint that support eDiscovery. You can:

Manage holds and conduct eDiscovery searches on any site collection
Use SharePoint Server Search or FAST Search for SharePoint out of box to search and process content
Automatically copy eDiscovery search results to a separate repository for further analysis

Read on to learn how SharePoint Server 2010 can support your eDiscovery initiatives and provide you with the tools you need to manage holds, identify, and collect SharePoint content.

The eDiscovery Process

The Electronic Discovery Reference Model from EDRM (edrm.net) provides an overview of the different parts of the eDiscovery process:

SharePoint Sever 2010 addresses the Information Management, Identification, Preservation and Collection stages. While this blog post will focus mostly on the identification, preservation and collection components, SharePoint provides a rich Information Management platform for Collaboration, Social Computing, Document Management and Records Management. This means that you can take a proactive approach to eDiscovery by putting a governance framework in place and using appropriate disposition policies to expire content. Managing content and deleting it when it is no longer needed will reduce the amount of content that must be indexed and searched, and collected for eDiscovery. The result is that eDiscovery costs can be dramatically reduced, changing the problem from finding a needle in a hay stack to finding a needle in a hay bale. Ultimately, the key to achieving legal compliance for eDiscovery obligations is built upon a foundation of robust Information Management.

When an eDiscovery event occurs, such as a receipt of complaint, discovery, or notice of potential legal claim, the identification stage begins. Content that may be subject to eDiscovery must be identified and searches are conducted to find that content. That content needs to be preserved and at some point, the content will be collected.

The eDiscovery Features

Hold and eDiscovery

Hold and eDiscovery is a site level feature that can be activated on any site.

Activating this feature creates a new category in Site Settings that provides links to Holds and Hold Reports lists. There is also a page to discover and hold content that allows you to search for content and add it to a hold. Once the Hold and eDiscovery feature is activated you can create holds and add to hold any content in the site collection. By default only Site Collection administrators have access to the Hold and eDiscovery pages. To give other users permission, add them to the permissions list for the Hold Reports and Holds lists. This will also give access to the Discover and hold content page.

You can manually locate content in SharePoint and add it to a hold, or you can search for content and add the search results to a hold. With the Hold and eDiscovery feature you can create holds in the hold list and then manually add content to the relevant hold by clicking on Compliance Details from the drop down menu for individual items.

Then click on the link to Add/Remove from hold.

And you can select the relevant hold to add to or remove from.

By manually adding an item to hold you will block editing and deletion of that item until it is released from hold. You will notice that the document now has a lock icon showing that it cannot be edited or deleted.

Each night a report for each hold is generated by a timer job. If you need a hold report faster you can manually run the Hold Processing and Reporting timer job in Central Administration.

Search and Process

You can manually add items to hold on any site collection, which is great. But that doesn’t help you find the content you don’t already know about. What if you have a large amount of items you want to find and add to a hold? For that you can use the features on the Discover and hold content page, which is a settings page in Site Settings. From this page you can specify a search query and then preview the results. The configured search service (SharePoint Search Server or FAST Search for SharePoint) will automatically be used. You can then select the option to keep items on hold in place so they cannot be edited or deleted, or if you have configured a Content Organizer Send to location in Central Administration you can have content copied to another site and placed on hold. You may want to create a separate records center site for a particular hold to store all content related to that hold. The Content Organizer is a new SharePoint Server 2010 feature based on the Microsoft Office SharePoint Server 2007 Document Router with richer functionality to automatically classify content based on Content Type or metadata properties. Look for a future blog post covering the Content Organizer.

Holding content in place is recommended if you want to leave content in the location is was created with all the rich context that SharePoint provides, while blocking deletion and editing of content. Be aware that this will prevent users from modifying items. If you prefer users to continue editing documents, then use the copy to another location approach.

When searching and processing, the search will by default be scoped to the entire Site Collection and run with elevated permissions so all content can be discovered. The search can be scoped to specific sites and you can also preview search results before adding the results to a hold. Items can be placed on multiple holds and compliance details will show all of the holds that are applied to an item.

In summary, SharePoint Server 2010 contains key features that make it an essential aspect of your eDiscovery strategy. With the new SharePoint Server 2010 capabilities you can easily apply proper retention policies for all content and make it easier to discover content if an eDiscovery event occurs. eDiscovery often prescribes tight deadlines for production. SharePoint 2010 helps you find the right content and deliver it faster.

Quentin Christensen
Program Manager - Document and Records Management
Microsoft

SharePoint ECM in Force at the AIIM Expo in Philadelphia

Ryan Duguid — Sun, 14 Mar 2010 21:20:40 GMT

As you can see from the last few posts, we are incredibly proud of the evolution of our ECM capabilities in SharePoint 2010 and in April, we are heading to the AIIM Expo in Philadelphia to give attendees the chance to try out SharePoint 2010 and hear directly from the people who built the product. Starting on April 20th, we’ll open the doors on the SharePoint Experience Lab where you can learn about Office and SharePoint 2010, assisted by the ECM team from Redmond and some of our top field specialists. The SharePoint Experience Lab will be in the Expo Hall where we will be joined by a number of our leading partners and best of all, if you register before the event, entry to the Expo Hall is ABSOLUTELY FREE! That’s right, register now and you will get access to a wide range of SharePoint labs, supported by the team from Redmond.

In addition to the SharePoint Experience Lab, we are proud to support the SharePoint 2010 Summit @ AIIM Expo. The SharePoint 2010 Summit @ AIIM Expo consists of almost 30 sessions delivered by the SharePoint ECM Team, customers and leading industry analysts. Entry to the SharePoint 2010 Summit @ AIIM Expo is included with a conference pass that you can pick up for just $599 (UPDATE – Advanced registration has been extended. Enter code A525G to receive a $50 discount). Not a bad price to ask all the questions you ever wanted answered about SharePoint and get the inside scoop from senior product and program managers as well as Eric Swift, the General Manager of the SharePoint Marketing Group.

Here is an overview of the content being delivered by Microsoft speakers at the SharePoint 2010 Summit @ AIIM Expo:

Introducing SharePoint 2010
ECM for the Masses: How SharePoint 2010 Delivers on the Promise
SharePoint and Office: What’s New in 2010
Overview of Social Computing in SharePoint 2010
Web Content Management in SharePoint 2010
Growing SharePoint from Small Libraries to Large Scale Repositories & Massive Archives
Visual Customization Overview: Theming & Branding For Any Site
Using Enterprise Content Types & Managed Taxonomies in SharePoint 2010
Using SharePoint Analytics and End User Feedback to Optimize the Content and Organization of your SharePoint Sites
Document Management in SharePoint 2010
Building Rich, Immersive Sites with Microsoft Tools & Technologies
Enterprise Search Overview
Delivering BI to the Masses at Microsoft
Building an Enterprise Knowledge Management Solution on SharePoint 2010
Records Management Strategies in SharePoint 2010
Better Together Collaboration with SharePoint 2010, Office 2010 & More!
Managing and Sharing Digital Assets in SharePoint 2010
If You Build It, They Will Come: Driving End User Adoption

With the launch of Office and SharePoint 2010 set for May 12th and our intent to RTM (Release to Manufacturing) this April 2010, there has never been a better time to hear from the team that built the product and get the knowledge you need to make SharePoint successful within your business. Spring is coming to Philadelphia and with it comes SharePoint 2010 and the SharePoint ECM team. We look forward to seeing you at the AIIM Expo.

Ryan Duguid
Senior Product Manager – ECM and Compliance
Microsoft

Introducing Web Content Management in SharePoint 2010

Ryan Duguid — Fri, 12 Mar 2010 07:15:00 GMT

Hello everyone! My name is Sangya Singh and I am a Program Manager on the SharePoint engineering team working on Web Content Management (WCM) features. We are very excited about the WCM capabilities that will be shipping in SharePoint 2010 and the possibilities they will open up for our customers to create rich WCM solutions. In this first post, I want to talk about the broad investments we have made in this release around WCM and share with you how we approached it from an engineering perspective.

Enabling different shades of WCM

Taking authoring to the next level

Making it easier to build richer sites

Richer publishing control and greater insight

Scalable platform to power your site

Enabling different shades of WCM

When most people hear WCM, they immediately think dot com, a public facing internet site. A public facing site allows a company to drive brand awareness, deliver marketing campaigns, build community and share information about their products and services. The publishing process behind a public facing site is typically very structured to ensure a consistent look and feel, usage of approved branded assets and a more controlled approval process. Public facing sites are just one use of WCM technology and most companies have far broader needs from a WCM platform. If a public facing site is on one end of the spectrum then a solution like a Wiki is at the end other. Wiki’s are community based and have lots of authors creating content in a very loosely controlled environment. Wiki authors have a lot more freedom on how their content is formatted and organized when compared with a public facing site. There are many shades in-between these two scenarios requiring varying degrees of branding and governance so when we built the WCM features in SharePoint 2010 we set out to empower the business to easily adjust the dial between freedom and control from one site to the next.

Taking authoring to the next level

A modern WCM system has to meet many needs across a business but the number one goal has always be to empower the people who own and create content to easily publish content. With a renewed focus on web analytics, search engine optimization, campaign management and personalization, many businesses and vendors have lost sight of the end user. By empowering content creators, you can rapidly remove the friction between the business and IT ensuring that you can drive content to the right audience in a timely manner. To empower the end user, you need to provide an intuitive user experience that helps employees author and publish content effectively without needing specialized technical skill. Jim Masson discussed the notion of ECM for the masses in his blog ‘SharePoint 2010 Delivering on the promise’, giving insights in to how we think about empowering users and the list below outlines some of the key user experience enhancements that we made in SharePoint 2010:

Quick access to the tools and actions you use most often

The most notable visual change in SharePoint 2010 is the introduction of the “Ribbon” from the Office applications. The Ribbon provides a consistent experience and makes it easy for users to discover the rich features in SharePoint. What’s more, the Ribbon enables quick access to the most common functionality based on the specific task that you are working on. So, let’s say you are authoring a page that requires you to add text, images and videos. When you’re typing, the Ribbon will show you text formatting options like styles, fonts, bold, italics etc. When you click on a video player web part you get options like changing the size of the media player, whether or not the video starts when the page loads or whether the video should loop once it finishes…

Text formatting options available when adding text

Media configuration options are displayed when Media web part is selected

One-click page creation

In SharePoint 2010, one-click page creation allows you to simply enter the page title then you can immediately start authoring the page.

Unlike SharePoint 2007, authors can get into creating their page content by just specifying the page name.

Dynamically changing Page Layout

Page Layouts (templates) provide a way to apply a consistent look and feel to a page. In SharePoint 2010, changing page layout is as easy as picking a layout from a gallery in the Ribbon while the author is editing the page.

The Page Layout ribbon drop-down is available to pick and choose from.

New and improved Rich Text Editor

The new and improved Rich Text Editor (RTE) provides a “Word-like” editing experience that most people take for granted in a non-browser world. The RTE in SharePoint 2010 provides rich formatting of text, live preview of formatting options, easy embedding of images and videos directly into the RTE and drag and drop capability to place them exactly where you want.

Easy to add rich media

SharePoint 2010 makes it easy for authors to select and add rich media content (like images, audio, video and Silverlight controls) to their pages. Authors have quick access to Media, Video and Silverlight Web Parts that they can add to their pages. We’ve also introduced a new experience for selecting rich media content that has features like getting to preview and play the video before you select it.

Support for a wider range of web browsers

With the upcoming release of SharePoint 2010, we will be supporting Internet Explorer 7 & 8.0 as well as the latest versions of Firefox and Safari. This allows users to use their browser of choice when working with SharePoint.

Making it easier to build richer sites

Many people still think of SharePoint as an intranet platform but with customers like Ferrari and AMD betting on our platform for their .com presence, you’re probably asking yourself, how can I use SharePoint to help me build a rich, immersive and accessible web site? The following features would help with that question:

Rich media integration

Earlier I discussed the new web parts in SharePoint 2010 that allow you to add rich media to your pages. To support these web parts, we’ve developed a specialized Asset Library that is optimized for storing, managing and navigating large volumes of rich media including images, audio and video files. We’ve also made investments to ensure that key metadata is promoted from these assets when you upload them to the Asset Library.

The new Asset Library showcasing viewing of assets in thumbnail view and metadata driven navigation

A dialog showing information about an asset as the user’s mouse hovers over it in the Asset Library.

You can preview the video in the hover over dialog before you select the video.

To deliver rich media, we’ve included a customizable Silverlight media player that allows you to customize the ‘skin’ to meet your specific visual needs.

Silverlight based player for playing rich media in SharePoint

Dynamic content

If you want to quickly build landing pages or show dynamic content roll-ups, then you can easily use the Content Query Web Part (CQWP). If you have been developing web sites with SharePoint 2007, then you have no doubt used used this web part. In SharePoint 2010, we’ve made a lot of enhancements to the CQWP. These enhancements support content to content targeting where the query defined in the CQWP can now filter on metadata on the items being queried or a value passed to the page in the URL query string. This rapidly enables scenarios where you need to show related data like services, product sheets, help topics or community content like blogs and wikis. The blog post on Introducing Document Management in 2010 discusses one such scenario with a CQWP. There are other improvements made where data view mapping can now be done via the CQWP tool pane UI.

Managed Metadata tagging

SharePoint 2010 introduces a powerful set of features around defining and managing taxonomies and then leveraging those “terms” to tag content in SharePoint. Leveraging these managed metadata fields in web content enables scenarios around showing dynamic content (discussed above), driving dynamic navigation based on metadata and helps with search engine optimization.

Well-formed mark-up

We made investments in developing and testing against W3C WCAG 2.0 guidelines at the AA level and ensuring that the mark-up within our pages (e.g. page layouts, master pages, content generated in the RTE) is well formed XHTML. This improves accessibility and cross-browser support for sites built on SharePoint. In cases where authors have added content that does not contain well-formed mark-up, we offer a “Convert to XHTML” function in the Ribbon that scrubs the current page mark-up, converting it to well formed XHTML.

Community building tools

The social computing investments in SharePoint 2010 enable scenarios where readers of your site can tag, rate and comment on site content. In addition, you can leverage SharePoint blogs and wikis within your site to foster community and user contributed content so you can easily incorporate social features in your web sites using SharePoint 2010.

Rating control shows the average rating in the form of 1 to 5 stars. And the mouse over tool tip shows the how the user rated the content.

Richer publishing control and greater insight

The publishing platform in SharePoint allows you to control what flexibility is available to authors, how sophisticated the approval process needs to be for content to go live, how the content should be organized in your site, how to orchestrate publishing in different parallel sites and whether to separate the authoring and staging environment from your live site. We’ve also included tools to help you gain insight into what is going on with your site.

Control over what authors can do

Depending on the needs of your site and authors, you can control the functionality available during content creation. You can make all the text formatting options available or only allow the use of predefined markup styles that follow the consistent look and feel of your site while generating well-formed markup. You can give authors the freedom to insert any web part or have the specific, approved web parts available in the page layout.

Orchestrate publishing across different parallel sites

In SharePoint 2007, we introduced the Variations feature. One application of this feature is to support multilingual publishing scenarios where you want to orchestrate publishing between your source site and other global sites that will translate content in to a different language. We have introduced improvements in the translation pipeline to make it easy for someone working in a localized site to understand what has changed in the source site. Users will have 1-click access to a view of what has changed in the latest version of the source page so they can decide what they need to translate or if they need to translate anything at all.

Ribbon action available to the authors on the target sites, to view what changed on the latest version of the page sent by the source site.

We’ve also made improvements around reliability and server citizenship. We’ve moved Variation operations to timer jobs. We support pause and resume during timer service recycles to improve the reliability of long-running operations in large deployments. We give a lot more control to IT on when the expensive process of creating hierarchies should happen. It is also worth noting that the feature set in Variations is complementary to a set of new investments in SharePoint 2010 around Multi-language User Interface (MUI). MUI is the technology that helps SharePoint present all application UI in the preferred language of the user of the site. The combination of Variations and MUI investments provides a great story for managing the translation of your content and managing the display of the SharePoint UI giving a unified experience in multilingual sites.

Deploying content from authoring/staging environment to the live environment

The Content Deployment feature was added in SharePoint 2007 to address requirements for companies hosting their internet sites on SharePoint and wanting a separate environments for authors to modify and review content before it was published to the public facing farm. In SharePoint 2010, we have made significant investments to improve the reliability of the Content Deployment feature. In addition, we’ve made a lot of these reliability improvements available to SharePoint 2007 customers through cumulative updates. Additionally, we’ve made changes to the platform to take advantage of database snapshots to better improve scenarios where authoring on site is going on while the Content Deployment job is running. You can take advantage of this feature if you have SQL Server 2005 / 2008 Enterprise edition. We also provide better logging to get provide insight into Content Deployment jobs.

Publishing workflows

Depending on the type of WCM deployment you have, you can decide how simple or sophisticated your publishing approval process needs to be. You can decide that you don’t need any approval process in place or use simple out-of-box parallel or serial approval workflows or customize the out-of box workflows in SharePoint Designer 2010 to model your business process. We now enable business users to model their workflow in Microsoft Visio 2010 which can then be imported into SharePoint Designer 2010. Another great advantage of building in Visio is that SharePoint uses a new feature, Visio Services to deliver workflow visualization, showing exactly where in the process the workflow is currently executing. We’ve also made improvements in this release where you can reuse the workflow you have created and apply to content types and site templates.

Web Analytics

An important part of any site is understanding what is going on with the content, users and the servers powering the site. SharePoint 2010 provides a range of new Web Analytics capabilities that monitor different aspects of site usage. In addition to the out of box reports, you can subscribe to alerts to monitor changes on key metrics. Beyond traffic insight, there is support for search insight around search queries, popular terms and queries that are succeeding or failing. It also recommends new best bets for the search system by watching what links people are clicking on the search result page so you can promote these to the top of the page.

A report showing information over time on number of page views on the site.

Server Health Monitoring

SharePoint 2010 has made some big investments in logging infrastructure from the analytic side that will help you monitor the performance of your SharePoint deployment. You can now easily find the slowest pages (in terms of rendering) on your site. So in case you have customization where you have one or more Content Query Web Parts making expensive queries and forgot to turn on caching then we’ll help you find that page. Since sites are highly customized with custom web parts and field controls, we’ve introduced the Developer Dashboard feature that allows a developer investigating why a certain page renders slowly to see at a page level which queries went to SQL backend and how long they took. Introduction of Sandboxed solutions allows site administrators to upload custom code that runs in its own sandbox in a way that it can be monitored and throttled so it doesn’t impact the quality of service to other users on the farm.

Scalable platform to power your site

There are a number of investments made in the platform to ensure continued performance and scalability as your site grows.

Large Pages Library and the Content Organizer

We have made improvements in SharePoint 2010 to support thousands of pages in a given pages library but more importantly, we’ve introduced the ability to organize pages in folders with a Pages Library. A new feature called Content Organizer can be leveraged to better organize your web content by setting rules that will decide where page should go. This allows the authors to concentrate on authoring the content and the Content Organizer uses rules to drive the page to the right location. With the investment in large lists, SharePoint 2010 also gives IT the ability to govern how these items are accessed by introducing resource throttling to be able to limit the number of items accessed in a view or a CQWP as an example.

Optimization of the Content Query Web Part

As mentioned earlier, the CQWP can show dynamic content based on a query. In this release we have made query optimizations that leverage indices available on the list that CQWP queries.

Support for streaming rich media

We’ve put a lot plumbing into the product to ensure that the end-user experience of viewing and streaming rich media on your site is smooth and the impact on your network and SQL backend is minimized. The BLOB cache on the web front ends (WFE) has been optimized to read content from SQL in small chunks and start sending the file to the client immediately so the user doesn’t have to wait for the whole file to download. The BLOB cache can also serve requests for parts of the file to the client. So if the user wants to skip to the last chapter in the video and the entire file hasn’t been downloaded yet, the BLOB cache can serve that part of the video immediately.

I hope this post gives you a good introduction to the new and improved WCM capabilities in SharePoint 2010 that allow you to deliver great web sites with rich media, dynamic content and an intuitive user experience for content authors. We look forward to discussing these areas in depth with you in upcoming blog posts and would love to hear your feedback on the investments we’ve made in SharePoint 2010.

Thanks for reading.

Sangya Singh

Lead Program Manager

Introducing Document Management in SharePoint 2010

Ryan Duguid — Mon, 15 Feb 2010 05:35:32 GMT

Hi everyone. It’s Adam here again – this time I want to talk to you today about another key area of the content management world: Document Management (DM). Over the next few months, you’ll be hearing from several members of the engineering team about new DM features that help you get the most value out of your document corpus. We’ll also discuss how key early adopters of SharePoint 2010 used new DM features to solve the toughest information governance challenges.

Today, though, I’d like to spend time talking about what the team has learned about the document management space since SharePoint 2007 and take you on a journey through the key tenets that guided our DM vision this release.

Recap: Document Management in SharePoint 2007

SharePoint 2007 was the first release where SharePoint really broke out of its collaboration role and enabled customers to apply structure and management to their document libraries. A lot of the key DM infrastructure was established in that release: Check in/Check Out, Major/Minor Versioning, Per-Item Permissions, Content Types, Workflows, and the Recycle Bin are just a few examples. Of course, all of these features tightly integrated with the Office client applications such as Word, Excel, and PowerPoint to make it simple for end users to interact with the document repository (a core design tenet that carries through in 2010).

Features like these enabled customers to start creating high-value knowledge repositories on SharePoint 2007.

For 2010, we looked to build off of 2007’s gigantic success, and we rallied our designs around three key ideas:

Tenet #1: Manage the unmanaged

As we looked at how our customers were starting to use the 2007 system’s DM features, we noticed an interesting trend: These features were not just part of managed document repository deployments. Indeed, the traditional DM features were getting heavy usage in average collaborative team sites as well. Customers were using them to apply policy and structure as well as gather insights from what otherwise would have been fairly unmanaged places. SharePoint was being using to pull more and more typically unstructured silos into the ECM world.

This is a key insight that really drove our investments in SharePoint 2010. For instance, one of our key new features in SharePoint 2010 is the notion of a Document Set. You can think of a document set as a “folder on steroids.” It allows you to group related documents together so that they share metadata and have a common homepage, workflows, and archival process:

The Welcome Page of a document set is a customizable page that allows users to discover the content in the set, view and sync metadata between items in the set, and manage the set.

When it came time to design this feature, we knew people would want to use it to manage very structured and rigid official processes (e.g. a pharmaceutical company submitting forms to a regulatory agency). But equally important to us was that the feature can be used in a lightweight team site to manage most processes that requires multiple documents to be bound together (e.g. a team that just needs to put together a pitch book/sales proposal that includes a PowerPoint deck, a spreadsheet of costs, and a document that describes the sales pitch).

Enabling the document set feature to be used informally and easily is one way we are expanding the value of ECM in the minds of SharePoint end users.

Tenet #2: Social computing and enterprise metadata are game changers.

As we started to design out the DM feature set for this release, we quickly realized the power of metadata – both structured taxonomies as well as lightweight folksonomies (keywords) – as transformative forces in the document management space. A SharePoint 2010 document repository would need to take full advantage of both concepts.

There are two key principles that enable SharePoint 2010 users to take advantage of metadata. First is on the tagging side: it’s easy for a site to use enterprise wide content types and taxonomies and it’s also simple for a user to tag with them.

SharePoint 2010 offers consistent management of metadata that any SharePoint site can hook in to with virtually no effort. This allows the entire enterprise to be talking the same language. Tangibly, you can do things such as define the list of products you sell once and have that data available in any SharePoint site.

Note how the type-ahead functionality makes it easy for a user to pick a value from this folksonomy. Also note how the West Coast tag was automatically filled out for the user because it was set as the default value for all documents in this library.

The second key principle is how SharePoint takes advantage of these tags. For instance, a SharePoint 2010 document library can be configured to use metadata as a primary navigation pivot. You can think of metadata based navigation as a virtual folder structure that can be used to filter the items in the library:

Instead of navigating by traditional folders, a user filtered the library to the virtual folder that contains just sales materials about Contoso’s tent products.

It’s a virtuous cycle here: Easy metadata entry allows items to be tagged, which can drive navigation. And because users need the metadata to navigate the repository, this incentivizes them to tag the items!

Tenant #3: The browser as a powerful document management application.

SharePoint has always been used for many scenarios, but perhaps it’s known best for two things:

· A best of breed tool for creating web pages and sites

· A place to store, manage, and collaborate on documents

SharePoint 2010 makes a big bet that creating a knowledge management repository requires the merger of both of these worlds. The browser is increasingly becoming the key technology for information workers – both inside the corporate firewall and on the consumer front. Sure, people will always want to download documents to take with them – but they also want to use the browser to interact with the document and see a wealth of context about the document (e.g. metadata, related documents, wiki pages about the document’s topic).

It’s time for the industry to expect any document management system to also be great at creating pages or wikis that add context to the documents’ content. And any system that doesn’t is going to start looking antiquated.

SharePoint 2010 delivers on this vision in a few different ways. First, if you’ve installed the Office Web Apps (licensed as part of the Office 2010 suite), the default click for a document library can be configured to load Office documents in the browser:

Without ever leaving the browser, users can quickly view Office documents stored in SharePoint.

Second, we spent a lot of time this release thinking about how the web content management features can be used in document repositories. For instance, the ever popular Content Query web part can be used to roll up all the documents related to a particular topic:

A content steward might create a page about a particular topic (e.g. a new product). This page includes text about the product, marketing pictures, as well as roll ups of all the documents tagged with the product.

This vision allows you to combine two very powerful aspects of SharePoint into one solution to your organization’s knowledge discovery problem. It’s a merger of an enterprise wiki and a traditional enterprise document repository.

Wrapping up: A lot more to come!

I hope this post gives some context on where we are going with document management in SharePoint 2010 and beyond. Feature wise, we really only hit a few of the many DM features that make up SharePoint 2010 – stay tuned for future posts as we deep dive into a lot more! And feel free to leave comments about what you’d like us to blog about (especially if you’ve downloaded the Beta and given SharePoint 2010 a test drive already!)

Thanks for reading.

Adam Harmetz

Lead Program Manager, Document and Records Management

Introducing Records Management in SharePoint 2010

Ryan Duguid — Sat, 13 Feb 2010 23:44:00 GMT

Hi everyone. My name is Adam Harmetz and I work on the engineering team responsible for the SharePoint document and records management vision and features. Many of you might remember me from the SharePoint 2007 recman blog. The recman blog was a great way for the team to connect with records managers, IT professionals, and information architects and we'll be continuing that discussion for the SharePoint 2010 compliance features via the Enterprise Content Management (ECM) Team Blog.

I think it makes sense to combine records management with other facets of ECM into one central blog. After all, as Jim discussed, records management is a key component of our ECM strategy. The notion that everyone should participate in ECM processes really served as a guiding principle to help expand the scope of records management in SharePoint 2010. And for all you records managers out there, I think you'll benefit greatly from learning about the other facets of ECM along the way.

To kick off the discussion, here are three key things you need to know about records management in SharePoint 2010.

The Records Center - A Place for Hierarchy, Driven By Metadata

The Records Center was introduced in 2007 as a SharePoint site that served as a conventional records archive. Content from all over the enterprise can be submitted to a Records Center and then routed to the appropriate place where it picks up the right permissions and policies, such as expiration and auditing.
For SharePoint 2010, we know it's important to continue to invest here and add even more "traditional" archive features. When looking at the broad swath of features we had to choose from, our goals here really focused on providing features that allow you to extract the most value out of an archive and find the data you need. For instance, here are a few of the new features in a SharePoint 2010 Records Center:

Document ID: Every document can be assigned a unique identifier, which stays with the document even when it's archived. This allows records to be easily referenced by an ID no matter where the document moves.
Multi-Stage Retention: Retention policies can have multiple stages, allowing you to specify the entire document lifecycle as one policy (e.g. review Contracts every year, and delete after 7 years)
Per-Item Audit Reports: You can generate a customized audit report about an individual record.
Hierarchal File Plans: You can create deep, hierarchal folder structures and manage retention at each folder in the hierarchy (or inherit from parent folders).
File Plan Report: You can generate status reports showing the number of items in each stage of the file plan, along with a rollup of the retention policies on each node in the plan.

Here's the home page of the Records Center in SharePoint 2010 for a fictional government agency, the Joint Task Force. Notice that the home page is a place for records managers to educate the organization on compliance policy, as well as a place to look up a record by its document identifier.

In addition to adding these traditional records management features to our archive, as product designers we made a big bet on the power of metadata to dive 21st century electronic records management. This manifests itself in several ways in the SharePoint archive:

Taxonomy and Centralized Content Types: The archive will be a consumer of enterprise-wide taxonomies and content types, ensuring consistency and context transfer between the collaborative spaces and the archive. We'll be talking a lot more about our 2010 taxonomy investments in future posts.
Content Organizer: The records router can use metadata to route incoming documents to the right place in the hierarchical file plan. For instance, it enables you to automatically enforce rules on content that is submitted, like "If a Purchase Agreement is tagged with Project Alpha, send to the Alpha Contracts subfolder and apply that's folder retention policy to the item."
Virtual Folders: The file plan is a great way to manage a repository but often time isn't what you want to use to navigate and find the content you are looking for. The SharePoint 2010 Records Center makes use of a new feature called metadata based navigation, which allows you to expose key metadata as virtual folders:

Notice that end users discover content in this Records Center by navigating virtual folders based upon metadata properties on the records.

This bet on metadata is all about empowering the end user, thus increasing the chance of successful adoption of the RM system. Instead of choosing a complicated node in a file plan, submitters just fill out a few pieces of useful metadata and they'll use that metadata when they need to find the content again.

In Place Records Management - Injecting Records Management in the Content Creation Experience

With just about every customer engagement my team is involved in, we hear the same message again and again: records management doesn't start (or stop!) in the archive. Content isn't created there and it sure doesn't live there for the most interesting parts of its life.

We've made a huge effort in 2010 to enable you to do effective records management in collaborative spaces. Auditing, Retention, Expiration, Reporting, Records Workflows, eDiscovery, Legal Hold and Recordization are all features you can use in collaborative space as you are striking a balance between SharePoint's value to end users and the need for information governance.

Holding all of this together is a new feature in SharePoint 2010 called In Place Records Management. This allows certain SharePoint documents (or blogs, wikis, web pages, and list items) to be declared records. The system can prevent such records from being deleted or edited, if necessary by your organization's definition of what a record is:

Note that some of the documents have locks, implying to the user that they are dealing with records. When selecting a record, the UI for editing and deleting the item is disabled.

This recordization process can be done either manually, as part of a larger process in a workflow, or as a scheduled part of a document's retention (e.g. after 2 years). The key here is that, when declared a record, the content doesn't move to an archive - it stays where it is so the end users can still find and interact with the content.

Once declared, the system knows about an item's record status, so you can do things such as create different retention policies for records or use record state when defining workflows in SharePoint Designer. We also enable a programmability model so you can perform custom processes and policies upon recordization to meet specialized compliance needs.

Is In Place Records a replacement for a traditional archive? The answer is, of course, sometimes - we'll find some customers who want to use an in place approach exclusively, some who will want the traditional hierarchy and centralization that an archive brings, and many who will want both. It'll be something we'll talk about a lot on this blog, and our documentation has already started discussing the pros and cons of both approaches.

Scale: We're Talking Big

With electronic information growing at a crazy pace and businesses spending billions on eDiscovery every year, records managers have enough to keep them up at night. The scale of their records/content management system shouldn't be another worry.

As the records management engineering team, we take this burden very seriously and a large part of our effort this release has been spent adding features to make it easier to scale to massive archives. Features such as Remote Blob Storage, database query optimizations, internal timer job processing improvements, new database indexing strategies and other engineering initiatives enable us to make a great leap forward this release and allow our customers to have:

Tens of millions of records in a single Records Center
Hundreds of millions of records in a distributed archive: We'll talk in more detail in future posts, but many of the features mentioned above light up to allow many Record Centers to bind together to act as one logical repository.

With our partners on the SharePoint blog, we are looking forward to showing more details on the new scale targets and performance profiles for deployments at this scale over the coming months.

Wrapping Up

It's been a lot of hard work for the team around here to deliver on this vision for 21st century records management. When combined with the integrated e-mail archiving, retention, and discovery capabilities of Exchange 2010, I think you'll see the 2010 wave as a breakout release for Microsoft's records management strategy.

The team here is proud of the work here and eager to talk about it and hear from everyone - feel free to leave suggestions on future blog post ideas in the comments!

Thanks for reading,
Adam Harmetz
Lead Program Manager

P.S. If you are hungry for even for information on SharePoint 2010 records management, check out an interview I did on Don Lueder's blog.

SharePoint 2010 - Delivering on the Promise

Ryan Duguid — Thu, 11 Feb 2010 22:56:00 GMT

My name is Jim Masson, and I’m the Group Program Manager for the Enterprise Content Management team within SharePoint. My team is part of the engineering team, and is responsible for designing the features around content management, including managing documents, web content, rich media assets, records, and a new service for managing shared content types and taxonomy.

With the coming launch of SharePoint 2010, this seemed like a good time to ramp up the ECM team blog, and start a conversation about the SharePoint 2010 release. In the lead up to the offical launch of the product and beyond, various members of the team will be posting details about the major feature areas and features within ECM in SharePoint 2010, including design overviews, walkthroughs, best practices, and eventually interesting case studies. I hope you will subscribe and participate with us in the conversation

ECM For the Masses

When speaking with customers about the content management features in SharePoint 2010, we often refer to the release as being about ECM for the Masses. I wanted to take this first post to outline a little bit about our approach to designing and building SharePoint 2010, and how that has helped us deliver on that vision.

When the team started building 2010, we came up with 3 pillars that drove our investment decisions, and really helped to define the release. These pillars represent design principles that we would apply to each of the feature we built to help us focus in on delivering ECM for the masses. We call them the 3 E’s of ECM, and they are:

Enterprise Ready – This is all about ensuring that SharePoint more easily scales to the amount of content the largest Enterprises deal with and delivers consistently high performance and reliability at any scale. In addition we provide the feature depth, customizability and extensibility that Enterprises need to support the full breadth of business scenarios around content.
Easy to Use – Our focus here is on 2 audiences. First, the features must be Easy for the Information Worker, with best-in-class usability, providing supreme user acceptance and speeding deployment and adoption. Second, the product must also be Easy for IT, providing great functionality OOB that is fast to deploy and easy to manage at the Enterprise, Divisional, Team and Workgroup levels.
Everyone Participates – This is all about ensuring that Everyone in the organization has access to and benefits from the functionality offered by the ECM features – not just a few specialists who have been specially trained, or for whom the organization can justify a high per seat price. This also means that the capabilities can be adjusted to suit the needs of everyone in the organization; from minimal interaction to highly structured and complex workflows – everyone sees exactly as much as they need.

It is my hope that, as we go through the features over the next several months that you will see the impact of those pillars on the product, and how they have helped us to deliver ECM for the masses.