Office Web Service

Office Web Service

The Office Services Web service is used by Office SharePoint Server 2007 as a communication channel between Web servers and application servers. This service uses the following ports:

• TCP         56737
• TCP/SSL     56738

Access to the web service methods is restricted to the farm administrator group, WSS_ADMIN_WPG. None of the web service methods can be called from user code.

Web Services

Depending on features installed, the Office Server Web Services Web application exposes the following internal Web services, which are not available for calls from custom code:

The object model automatically short circuits the web services, i.e. invokes the underlying functionality without invoking the web service, when the target server is also the client primarily for performance reasons. Hence, the web services are not used...

• On a Basic deployment.
• When the administrative action is performed on a WFE that also happens to be the indexer.

Global Web Service (SearchWebService)

Runs in the Office Server Web Services virtual server root application pool, i.e. an application pool that does not belong to any SSP. This GLOBAL application pool runs as NetworkService.

It is used to retrieve low level computer configuration settings before any SSP is created, e.g. system drive info, verify path correctness, the computer's IP Address.

It is also used to create/configure a propagation share. The web method that implements this functionality is special: It impersonates the WindowsIdentity making the request. That identity must be a local admin on the remote server (only local administrators can create/configure shares).

Allowed access: WSS_ADMIN_WPG.

 SSP (Application) Web Service (SearchApplicationWebService)

Primarily used for SSP administration of Search configuration.

A web service associated with a specific SSP on a specific server (indexer and/or query server).

Runs as the SSP web service credentials (the credentials that you enter in the SSP creation/details page).

The SSP web service account can read/write from/to the SSP database and the Search database (only the ones that belong to its SSP).

Allowed access: WSS_ADMIN_WPG and the SSP administration application pool identity.

Security

InterServer Communications

Network traffic can be secured with either SSL on port 56738, or with IPSec on either port.

IPSec is an IP level feature, which means all traffic on the configured ports is protected; whereas, SSL is an application level protection mechanism.

IPSec has the advantage of limiting which pairs of servers can communicate, by configuring the IP addresses. This feature can significantly lock down a server farm.

Service Accounts Used

Search service account

• It is a db_owner in ALL SSP databases.
• It is a db_owner in ALL Search databases.
• It has READ ONLY access to all the content in ALL web applications via a policy.
• It has read/write access to the propagation share on Query servers.
• It has read/write access to the Search registry hive.
• It has read/write access to the Search index location.

SSP administration site application pool identity

• This account is determined by the web application that you select when you create the SSP.
• It has read/write access to the SSP database and the Search database.
• This account has full control over the Search service via its COM interfaces.
• It has read/write access to the Search registry hive.

Global web service account

• This is the GLOBAL application pool account of the Office Server Web Services, i.e. an application pool that does not belong to any SSP.
• It is always set to NetworkService.

SSP (Application) web service

• The application pool account of an SSP web service (the credentials entered in the SSP creation/details page).
• This account has read/write access to the SSP database and to the Search database of an SSP.
• This account has full control over the Search service via its COM interfaces.

It has read/write access to the Search registry hive.

Information Management Policies – Expiration

Information Management Policies – Expiration

The question is, when an information management expiration policy is defined, is the expiration period applied immediately or at some future time? If the answer is, at some future time, exactly what is that time, and can you set it?

Contrived Example

To take a concrete example, create a document library.

Next, modify the default view allow us to see what is happening. Go to Settings, Views, and click on All Documents. Add the columns Created, Exempt from Policy, and Expiration Date to the view. Press the OK button to save the view changes.

Now Go to Settings, Document Library Settings, and click on Information management policy settings.

• Select Define a policy and press the OK button.
• Check Enable Expiration
• Under retention period, select A time period based on the item's properties.
• Select Created + 30 days.
• Under When the item expires, select Perform this action, and Delete.
• Press the OK button.

Return to the library, All Documents view. Upload several documents. The uploaded documents will have an Expiration Date of the current date plus 30 days as expected.

Go back to the Settings, and change the retention period to Created + 60 days. Upload some more documents. The newly uploaded documents correctly show an expiration date of the current date plus 60 days, but the previously uploaded documents still have an expiration date of the current date plus 30 days. It appears there is an inconsistency.

Information Policy Jobs

The key to this inconsistency is the Information Management Policy timer job. This job runs once daily by default. It iterates all the web applications/site collections/sites/lists in the farm, looking for information policy changes. When a policy change is found, all affected item's metadata is updated; consequently, the expiration dates of the library documents in our contrived example are not updated until this job runs. When this job eventually runs, the inconsistency will be corrected.

There is an stsadm command to change the schedule of this job, SetPolicySchedule.

stsadm -o setpolicyschedule   -schedule <recurrence string>

Since this job could affect performance in a large farm, be careful when scheduling it. Daily is probably sufficient. It is also a good idea to have it run about an hour before Expiration Policy job, so the Expiration Policy job will find up-to-date item metadata when it applies the expiration policy to items.

You can find current information on both the Information Management Policy and Expiration Policy jobs by going to Central Administration > Operations > Timer Job Definitions. This will show you the frequency and last run time of each job, but not the complete schedule.

There is currently no stsadm command to change the Expiration Policy job; however, Mattias Lindberg has a code sample to for the job to run using the object model.

Summary

1. Item metadata is not updated for policy changes until the Information Management Policy job executes.
2. The Information Management Policy job schedule can be set with stsadm –o setpolicyschedule.
3. The Information Management Policy job should be scheduled to execute shortly before the Expiration Policy job.
4. The retention policy is not applied to items until the Expiration Policy job executes.
5. The Expiration Policy job currently can only be scheduled through the object model.

My Site Recommendations

My Site Recommendations

The following My Site recommendations are a composite of best practices taken from experiences at Microsoft and other large customers.

Planning

My Sites (even if they are as small as possible and only really used to store a profile picture) complicate backup/recovery, and add complexity and risk to ensuring the availability of the rest of the SSP farm.  Large organizations (100K+ employees) should consider putting My Sites into a separate farm.

Very large organizations might consider multiple My Site farms, perhaps regionally located. This minimizes the number of content databases per farm and places the My Sites geographically closer to the site owners. Having fewer content databases per farm eases administrative burdens. Having My Sites hosted closer to site owners reduces the affects of network latency, thereby enhancing their usage experience. Multiple My Site farms can also provide more flexibility in managing the effort and time required to deploy updates and service packs to any given farm.

Always create a dedicated Web Application to host My Sites. This allows leveraging web application policies to define security, facilitates content database management, and enables creation of zones for external access.

Do not customize the My Site site definition. Besides being unsupported, poorly designed customizations can severely impact server performance and unnecessarily consume valuable CPU and memory resources. Any customization should be done through feature stapling, see http://blogs.msdn.com/sharepoint/archive/2007/03/22/customizing-moss-2007-my-sites-within-the-enterprise.aspx and http://blogs.msdn.com/cliffgreen/archive/2008/03/13/removing-web-parts-from-the-my-site-web-part-gallery.aspx

Indexing

Install the latest service pack (currently SP1). Be sure to get the latest post service pack hotfixes applied, in particular 21243 (Office QFE). There is an issue where incremental crawls will not pick up all changes on My Sites without the post SP1 hotfix.

Use a separate content source for People Profiles rather than allowing it to default to the Local Office SharePoint Server Sites content source. My Sites full crawls can be time consuming due to the large number of site collections. Creating a separate content source enables independent crawl configuration; such as, the type of crawl and crawl frequency for My Sites.

Provisioning

Remember, a user profile page will exist for all employees following a full Active Directory profile import, even if no My Sites have been created yet. Profile pages allow basic employee information to be exposed in search results.

Allow users to create their own My Site on demand. Do not pre-provision My Sites. Generally, pre-provisioning is a time consuming process potentially taking many days or weeks. It gains little, and costs storage and administrative headaches.

Rolling-out

Make My Sites available to everyone on day 1. This allows for "viral" adoption by early adopters. This will eventually encourage others to create My Sites, thereby getting the momentum rolling.

Send out invitations to a small group of "pilot" users who would be interested in trying out My Sites, based on their role in the organization. The pilot group might contain a few hundred users. This gets a critical mass of My Sites in place quickly.

Try regional roll-outs via "soft launches", which include poster campaigns or brown bag lunch training at selected campuses and offices.

About the 3^rd or 4^th month, promote the My Site feature in a story on the Intranet portal home page.

Around the 6^th month, incorporate the concept of "filling out your profile" into new employee orientation as a specific training exercise. Now essentially all new hires will have a My Site (because they need one to store their profile picture).

Encouraging On-Going Adoption

Encourage high profile "executive blogging" to drive awareness and adoption of My Sites. Blogging topics might include annual business planning, corporate strategy, rumor control, etc. This can demystify blogging by encouraging many participants to make daily posts about what they were doing and what they are thinking about. Note: this implies more frequent incremental crawling to incorporate blog entries into the search index.

Also, consider setting up a "Blogs" scope on the search home page to facilitate blog discovery. This can be accomplished by setting up scope based on the Content Type of blog posts. An example follows:

Note that this scope will pick up Blog Posts no matter where the Blog resides, as long as those a SharePoint Blogs crawled by a SharePoint Content Source.

Consider adding a link on the profile page that allows others to send an email to ask the person to fill out their profile ("peer nagging").

Explore holding a contest or raffle – If you fill out (or update) your profile this month you are eligible to win a prize.

Vanity Site Collection URLs

Vanity Site Collection URLs

A customer recently asked for “vanity” URLs for each of the major departments; HR, Finance, Legal, etc.; so for example, HR would be http://hr.bigcorp, and legal would be http://legal.bigcorp.  No problem you say, just create a web application for each department, give the web application default zone the vanity URL, create corresponding DNS entries, and the requirement is fulfilled.

The problem is there are over 30 departments, not to mention foreign subsidiaries, and possibly other “I want my own vanity URL” requests from other corporate groups. Since web applications are heavy resource consumers, each one requiring one IIS web site per zone, basing vanity URLS on web applications would not be feasible.

The traditional out-of-the-box site collection paths did not meet the requirements;

Wildcard inclusion; e.g., http://bigcorp/sites/finance

Explicit inclusion: e.g., http://bigcorp/finance

For both wildcard inclusion and explicit inclusion, the vanity part of the URL is at the end, which is not what was desired.

Host-Named Site Collections

Host-named site collections (what used to be called "scalable hosting mode" in WSS v2) provide exactly the needed capability. Don’t confuse the terms host-named with host headers. They are different concepts. The host-named concept applies to the internal SharePoint virtual path mapping mechanism; whereas host headers apply to IIS web sites independent of SharePoint.

Host-named site collections effectively allow use of an arbitrary URL, which is associated with an existing web application. There can be many host-named sites for a web application.  The net result is freedom to have as many vanity URLs as necessary, while limiting the number of web applications. We can have URLs like http://finance.bigcorp, http://finance, http://my.finance, etc.

Host-named sites cannot be created through the UI. You must use stsadm. This should not deter you, since the syntax is simple. The secret is to add the –hhurl parameter to the stsadm createsite command.

Host-Named Site Collections Limitations

As with all good things, there are some limitations and complications. The following blog is an excellent read: http://blogs.msdn.com/sharepoint/archive/2007/03/06/what-every-sharepoint-administrator-needs-to-know-about-alternate-access-mappings-part-1.aspx. Quoting from this blog:

Host-named site collections short circuit much of the AAM functionality, including the URL remapping functionality.  They're always considered to be in the Default zone, and their URL is always the same URL you supplied when creating the site collection.  You cannot use host-named site collections with off-box SSL termination, port translation, or host header manipulation scenarios.

This whitepaper is also highly recommended reading. It discusses how to enable SSL, and many other configuration issues:

http://office.microsoft.com/search/redir.aspx?AssetID=AM102157711033

Step-by-Step Example

Assume we want to create a site collection with the vanity URL http://finance.litwareinc.com in the http://extranet.litwareinc.com web application.

Step 1: Open a command prompt as a farm administrator, and then enter this command, being sure to include the –hhurl parameter:

>stsadm -o createsite -url http://finance.litwareinc.com -ownerlogin litwareinc\administrator -owneremail administrator@litwareinc.com -hhurl http://extranet.litwareinc.com -sitetemplate STS#1 -title "Finance Department" -quota DepartmentalSite

 

Operation completed successfully.

The new site collection appears in the site collection list of the extranet.litwareinc.com web application.

 

Step 2: Create a DNS entry for this new name, pointing to the http://extranet.litwareinc.com IP address. For local test, make an entry the hosts file.

You can now open a browser and navigate to the site collection.

Step 3: Create a search context for the new site collection, or include it as a starting address within an existing site collection. Start a crawl to include the site contents in the search index.

To test the search, create a simple text document in a shared document library. When the next increment crawl completes, you can then search for the document to verify the search results are using the vanity URL.

The Kerberos Factor

The stsadm site creation command will give a warning if the host web application is using Kerberos authentication (negotiate).

>stsadm -o createsite -url http://finance.litwareinc.com –ownerlogin mossfs\administrator -owneremail administrator

@mossfs.com -hhurl http://dmz.litwareinc.com -sitetemplate STS#1 -title "Finance Department"

 

WARNING: SharePoint no longer customizes Integrated Authentication security settings. This Web application may be using Kerberos, which can require manual configuration. See http://support.microsoft.c

om/?id=832769 for more information.

 

Operation completed successfully.

It is necessary to register the vanity URL with Active Directory using setspn.

>setspn -a http/finance.litwareinc.com litwareinc\administrator

In my testing, I stumbled into another complication. Without realizing it at first, I created a site collection in the Intranet zone of a web application, which was configured for Kerberos.

After creating the site collection, adding the DNS (host file) entry, and executing setspn, I got a “This site is under construction” error page every time I tried to browse to the site. I eventually worked-around this by explicitly adding a host header to the IIS web site. I consider this an unsupported solution, and so I recommend extensive testing before applying it to a production scenario.

“Hidden” SSP Timer Jobs

How are SSP Associations Changed?

Deleting User Profiles

DST and Timer Jobs

Now that daylight savings has arrived in the United States, have you noticed problems with timer jobs not running when expected?

I recently encountered this trying to deploy solution packages using stsadm scripts. In the past, these scripts ran flawlessly. The old solutions were retracted and deleted, the new solution versions added and deployed across all farm servers within a few minutes, and life was good.

Last week I was working with a customer to deploy a new staging farm. We got to the point at which the solution deployment scripts were run. We waited, and waited, and waited. The script was hung at stsadm –o execadmsvcjobs, which was called after executing several stsadm –o deploysolution -name solutionpackage1.wsp -immediate -force –allowGacDeployment statements. What should have happened is the timer jobs to deploy the solution across the farm servers should have executed within about a minute. Instead, over an hour passed. Then suddenly, the jobs ran. What caused this odd behavior, which had never happened before with the same scripts? Was there a farm configuration problem?

Then a thought occurred. The time had changed to daylight savings a week before; moving the clocks ahead one hour. Could there be a connection?

A little research brought the problem to light. WSS 3.0 SP1 includes fixes to timer job DST scheduling problems. See 938663 ( http://support.microsoft.com/kb/938663/ ) One-time timer jobs in Windows SharePoint Services 3.0 are delayed by at least one hour when the jobs are scheduled to occur during daylight saving time (DST).

A quick check in Central Administration > Operations > Servers in Farm, showed the installed version number was 12.0.0.4518. It should have been 12.0.0.6219 if SP1 was installed. We installed WSS 3.0 SP1, and the timer job scheduling problem disappeared!

I strongly encourage installation of WSS/MOSS SP1. It fixes many issues, DST just being one example. Taken time to read and following the installation instructions precisely! Remember, you have to update all servers in the farm at once. There is no rolling update, so if you have a multiple server farm with large content databases, plan on doing this over a weekend to avoid disrupting your user community.

User Profile Change Logging

Profile Change Logging

There is a table in the SSP database named UserProfileEventLog. The table maintains a history of user profile property changes. By default, it retains 7 days of history. This table can cause problems in a couple of ways.

First issue: size. This table contains one row per change of a property in a user profile. The row contains the old and new property values, along with associated metadata like the datetime the property was changed. The implication is that the table can grow large. Assume you just configured the profile import and are ready to start the first Active Directory import. Further, assume your organization has 100,000 user accounts, and each account be populated with 12 AD attributes.  The full import will result in 100,000 X 12 = 1.2 millions rows being inserted into UserProfileEventLog.  To extend the example, assume you also have a BDC import connection populating another 20 properties from your company's HR system. That adds another 2 million rows. There are now 3.2 million rows in the event log table.  If each row is 100 bytes (old value, new value, plus metadata), the table is now approximately 300 MB in size.

Second issue: deleting old entries. The change history is kept for a configurable number of days. The concern is how many rows will have to be deleted on a given day? The number of days of history is 7 days be default, but can be set using stsadm.exe -o profilechangelog -title <SSP Name> -daysofhistory <number of days> -generateanniversaries (http://technet.microsoft.com/en-us/library/cc263013.aspx).  The critical issue is that MOSS has to remove a full day of history every day to honor the daysofhistory setting. Using the numbers in the preceding paragraph, 7 days after the first full import, MOSS is going to delete 3.2 millions rows of data all at once!  What makes this an issue is that this is done with a single SQL statement, something like DELETE FROM  UserProfileEventLog WHERE EventId < @MinEventTime. Think of the implications. As a single statement, this will hold locks until all 3.2 million rows are deleted. These locks might prevent other database transactions from completing; but it also means the transaction log (even with simple recovery mode) cannot be truncated, and will therefore grow until at least this DELETE statement completes. I have seen this delete statement run for 4 hours, with the transaction log quadrupling in size.

This is little you can do to avoid this. Although you can adjust the number of days of history, MOSS will eventually try to delete an entire day's history at some point. The deletion is done by an internal timer job buried within the SSP. The job is hard coded to run at 10:00 PM daily. You cannot change this scheduled time.

What are the take-aways?

• 1. Be prepared for an occasional long running DELETE statement. Ensure you have sufficient transaction log space to accommodate the potential transaction log growth.
  
• 2. Don't schedule any other timer jobs or database maintenance for 10:00 PM to minimize possible deadlocks and transaction timeouts.
  
• 3. Be careful using stsadm -o profilechangelog to reduce the number of days of history, because the next time 10:00 PM comes, MOSS will try to delete multiple days of history all at once.

SharePoint Database Log File Cancer

SharePoint Database Log File Cancer

The database log files can grow like cancer, consuming large amounts of disk storage and impacting performance if not properly managed. This can be a particular problem for smaller deployments without dedicated DBAs or database experienced local administrators.

The underlying issue is that native SharePoint backups do not backup the log files. Content databases are created with FULL recovery model if the System Model database has the FULL recovery model specified. This results in unbounded log file growth. It is not unusual for log files to grow to several times the size of the data files before someone notices the problem or worse yet, the log file consumes all available space such that the database server no longer responds. (view entire article ... )

ULS Log File Data Mining

Problems Importing Profile Data from BDC

By default MOSS imports data from a Master Connection first, which is typically Active Directory but could be another LDAP source. Once a successful master connection import completes, it is possible to import additional data from one or more BDC data sources, also known as supplemental data sources. This is very common, since many organizations keep the bulk of the interesting employee information in HR systems outside of Active Directory. Often these HR systems more accurate and up-to-date than Active Directory, and are therefore considered the official system of record for personnel data. Refer to Importing User Profiles on MSDN for additional information on profile importing.

A relationship between the Master Connection data and the entity methods in the BDC application definition must be established. This is done with an Identifier field. The Identifier is populated by the Master Connection, and then used as the foreign key when executing the BDC methods. In this picture, AccountName is the foreign key; however, most commonly this is a numeric value like PersonnelNumber. (view entire article ...)