Keith Richie : Development

Hey! Where can I get SPSiteManager 2.2??? It's not in the Utility Suite!

krichie — Mon, 11 Sep 2006 15:06:00 GMT

A lot of folks are asking me where they can get SPSiteManager 2.2 since I noted that it was complete. SPSiteManager (along with an update to SPUserUtil, and a couple of other tweaks) were going to be packaged up as the 2.6 release of The SharePoint Utility Suite and updated at the SharePoint Products and Technologies Web Component Directory.

Well, the decision was made to go ahead and move this package of tools over to CodePlex. The CodePlex project is already created at http://www.codeplex.com/Wiki/View.aspx?ProjectName=SPUS , but it's going to take me a few days before I can start getting the individual tools released there. And it will not include all the tools that previously existing there. Also, I'm only sending it out on an very "As Needed" basis right now to ensure we've got all the features fully tested.

In fact, while working with a couple of customers, I found two problems I need to correct before pushing it back out. These two problems exist in the 2.1 version that is in the 2.5 SPUS package. These are not major problems, just strange that no one has reported them yet :).

Problem 1

The first problem is in the MoveSiteToDB() Method as noted here:

MoveSiteToDB()

//====================================================================

// Get the basic information from the source site

// And also setup the restore URL from the target.

//====================================================================

SPSite oldSite = js.GetSite();

string strRestoreURL;

if(js.TargetUrl!=null)

strRestoreURL = js.TargetUrl;

else

strRestoreURL = js.SourceUrl;

bool bSwitched=false;

string strSiteUrl = oldSite.ServerRelativeUrl;

//====================================================================

// We do a couple of really important checks here.

// 1) If the site is the Portals site collection, DO NOT MOVE IT!!!

// You cannot use STS backup/delete/restore on a Portal site collection

// 2) If the site is a portal MySite, DO NOT MOVE IT TO A DIFFERENT URL

// See http://support.microsoft.com/?kbid=889236

//====================================================================

if(oldSite.RootWeb.WebTemplate.ToUpper().CompareTo("SPS")==0)

{

Logging.Warning("You cannot repartition a portals site collection!");

bContinue = false;

}

if(oldSite.RootWeb.WebTemplate.ToUpper().CompareTo("SPSPERS")==0)

{

if(strSiteUrl.ToUpper().CompareTo(strRestoreURL.ToUpper())!= 0)

{

Logging.Warning("MySites cannot be relocated off the virtual server reside on, nor can the URL change during the operation");

bContinue = false;

}

…

//====================================================================

// Restore the site into the target databse on the target virtual server

//====================================================================

targetvs.Sites.Restore(strRestoreURL,strBackupFile,true);

js.ReinitVirtualServers();

From the SDK:, "The Restore method of the SPSiteCollection class restores a site object to the specified URL from the specified backup file."

Parameters:

strSiteUrl A string that contains the URL to which to restore the site object.

The Restore() method requires the full absolute URL of the site, and that's working fine, but when I do the comparison check to ensure you're not moving a MySite between two different URLs, I'm comparing it to the relative version of the source URL. The only problem here, is that this safety check for moving MySites, always prevents you from moving a MySite regardless.

I already have this corrected in version 2.3.

Problem 2

Again, I already have this corrected in version 2.3, but this one needs a full test pass before I release it.

JobSettings.cs GetSite()

Uses absolute URL when it should use Relative URL.

From the SDK: "The Item property of the SPSiteCollection class gets the site object at the specified server-relative URL from the collection."

Parameters

strSiteName A string that contains the server-relative URL of the site object (for example, Site_Name or sites/Site_Name).

Absolute URLs are being passed into the GetSite() method. The fact that it is actually allowing Absolute URLs is a bug IMHO, and I'll work to file this so that it gets corrected. I’ve already corrected to trim the Scheme and Authority from the URL so that the server relative url is used, but since GetSite() is used all over the place, I need to do a full regression. For the most part, this isn't causing a problem except when the Absolute URL of the site as stuffed into the FullUrl column in the Sites tables of your content databases, doesn't match. Again, based on the documentation for this collection, it's expecting a relative URL anyways.

Once I finish doing a full test pass on this version of SPSiteManager, you should see it show up on the CodePlex project noted above. Stay tuned to this channel for updates.

- Keith

Thanks All! - The #1 download from the SharePoint Products and Technologies Web Component Directory is now The SharePoint Utility Suite

krichie — Fri, 25 Aug 2006 14:22:00 GMT

Thanks to everyone for helping make this happen by your suggestions and requests!

Sometime over the last couple of days, the SharePoint Utility Suite moved to the top of the Top Five Downloads (http://www.microsoft.com/sharepoint/downloads/components/results.asp?s=5 )

Thanks John Kozell for your contribution with the .Net Protocol Handler

Thanks Larry Kuhn for your contribution with the SharePoint Portal Server Search Property config

Thanks Dan Winter for your Refresh List contribution.

Thanks to Joel Oleson for recommending it as the #1 Must Have tool set!

Thanks to Chris Gideon and Raymond Hung for their contributions to making SPSiteManager even more powerful and useful than before!!!

SPSiteManager 2.2 is already proving great value with our customers with its new features (http://blogs.msdn.com/krichie/archive/2006/07/28/681634.aspx ), and I could not have done it without you. The updated Analysis and remedial features are helping our customers by identifying growth and usage trends; exceptions to Capacity Planning Guidelines, and improving Search Performance via its site directory and Deep Crawl list cleanup tools.

Not only are customers finding great value out of these tools for their existing Windows SharePoint Services and SharePoint Portal Server 2003 deployments, but they are also

using these tools for preparation for MOSS 2007 Upgrades

http://blogs.msdn.com/krichie/archive/2006/07/12/663278.aspx

Thanks again!

Keith Richie

Try the SharePoint Utility Suite at http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

"After Microsoft, I hope I'm remembered for more than just making one fine cup of coffee..."

SPSiteManager 2.2 complete

krichie — Fri, 28 Jul 2006 19:24:00 GMT

SPSiteManager 2.2 complete

Over the past few evenings, I've finalized changes; tested; regressed; tested; sighed a lot; did the hokie pokie; and performed weird rituals to fight the voices in my head shouting "Implement me! Implement me!" for some new features, yet finally finished this set of updates to SPSiteManager.

I previously posted on this update, but a couple of things noted in that post were cut, while a couple other new things were added. I was hoping to have this and a couple of other updates to other tools complete so that I could get the final SharePoint Utility Suite update complete and live this week, but it's going to take a few more days. This post will cover the "Final" new and enhanced features in this version.

As I noted in my previous post The Next Big Thing - The SharePoint Configuration Analyzer V.Next, we'll be combining the functionality of SPSiteManager with the SharePoint Configuration Analyzer, and incorporating SPUserUtil as well. In the mean time, we've found some immediate additions we need to make to SPSiteManager based on overwhelming customer requests and feature requests. I don't foresee any other immediate changes being needed before moving forward with SCA.Next in full force, so without further ado, here are the new and improved features in SPSiteManager 2.2!

What's New and Improved

Fix for removefromdirectory and removefromcrawl operations
- I found that my comparison was literal, and if you created the site via the Portals Create Site link, then we only stored the relative path to the site in the directory. So I updated the comparison logic to do a literal comparison, and then a relative comparison.
New site directory targeting via -sitedirectoryurl switch
- Not all customers leave their site directory in the same place :) I even commented this in the code and left it with it assuming that your site directory was in the same place. I've corrected this so now you can specify a specific site directory to work on with the -sitedirectoryurl switch
New enumsitecrawl and enumsitedirectory operations
- These give you the capability to export your entire list of sites to crawl, a.k.a. the Deep Crawl List) and site directories to an XML format for later review.
New cleansitecrawl and cleansitedirectory operations
- Unlike the removefromcrawl and removefromdirectory operations, you don’t feed a URL or list of URLs to these operations, rather they enumerate the list of URL references already established in the deep crawl list or site directorys. Using this operation makes it easy to cleanup URL references from sites that are no longer needed.
New adddatabase operation
- This operation can be used to enumerate over the <contentdatabases> section of a Site Distribution Document to create Content Databases. This is extremely helpful if you are mirroring a farm with lots of content databases, and want to match database names, etc on a target system.
- Theoretically speaking, you could run an analysis on one farm, then use SPSiteManager to create the necessary databases, then restore sites. (In the future, I hope to be collecting enough information in the Site Distribution Document, that you could basically replay and duplicate your entire farm settings if needed)
New includefolderinfo and enumversions switches for analysis operations
- Folder, File, and Document Version information can now be analyzed using the –includefolderinfo and –enumversions switches for the analyze operation
New Primary and Secondary contacts for site collection owners are now attributes that are reported in <site> elements for analysis.
New enableversions, disableversions, and purgeversions operations
- Thanks to Raymond Hung, SPSiteManager now has features to help you analysis and gather statistics on version weight in your system, as well as clean them up..
- purgeversions allows you to purge version history across your site collections
- enableversions/disableversions allow you to en-masse adjust version settings across your site collections.
- For example, in the following example, notice that at the <file> element, we sum up the entire size of the file versions via versiontotalsize="59904", then each version itself has a size="" attribute including (Format edited for demonstration purposes):
- <site url="http://www.tailspintoys.com/sites/dev1"
              databaseserver="TAILSPINSQL" databasename="Tailspin1_SITE"
              usercount="1" id="c708258f-bb33-4fb1-8ff9-0ec7e87cb5bf" webcount="1"
              storage="454517" quotaid="0" quotastoragemax="0" quotastoragewarn="0"
              readlocked="False" writelocked="False" primaryowner="TAILSPIN\krichie" secondaryowner="">
      <web url="http://www.tailspintoys.com/sites/dev1"
              subwebs="0" databaseserver="TAILSPINSQL" databasename="Tailspin1_SITE"
              title="Team Web Site" description="" lcid="1033" template="STS" uniqueperms="True"
              acecount="1" id="5cbe8682-89f0-488b-9f6c-456f6449215e" doclibcount="1" listcount="6">
          <doclibs>
              <doclib Title="Shared Documents"
                              Description="Share a document with the team by adding it to this document library."
                              url="Shared Documents" itemcount="7"
                              Author="TAILSPIN\krichie" id="3130c0c8-b768-4b47-a6af-2ea512b113ed">
                  <folder url="Shared Documents" itemcount="1">
                      <file url="Shared Documents/Super Secret Development documentation on SPSiteManager.doc" size="19968"
                                      versioncount="3" versiontotalsize="59904">
                          <version url="_vti_history/1/Shared Documents/Super Secret Development documentation on SPSiteManager.doc"
                                  ID="1" size="19968" Created="7/25/2006 9:43:42 PM" />
                          <version url="_vti_history/2/Shared Documents/Super Secret Development documentation on SPSiteManager.doc"
                                  ID="2" size="19968" Created="7/25/2006 9:44:11 PM" />
                          <version url="_vti_history/3/Shared Documents/Super Secret Development documentation on SPSiteManager.doc"
                                  ID="3" size="19968" Created="7/25/2006 9:44:39 PM" />
                      </file>
                      <folder url="Shared Documents/Forms" itemcount="6">
                              <file url="Shared Documents/Forms/AllItems.aspx" size="7603" versioncount="0" versiontotalsize="0" />
                              <file url="Shared Documents/Forms/DispForm.aspx" size="6138" versioncount="0" versiontotalsize="0" />
                              <file url="Shared Documents/Forms/EditForm.aspx" size="6110" versioncount="0" versiontotalsize="0" />
                              <file url="Shared Documents/Forms/template.doc" size="10752" versioncount="0" versiontotalsize="0" />
                              <file url="Shared Documents/Forms/Upload.aspx" size="9142" versioncount="0" versiontotalsize="0" />
                              <file url="Shared Documents/Forms/WebFldr.aspx" size="6746" versioncount="0" versiontotalsize="0" />
                      </folder>
                  </folder>
              </doclib>
          </doclibs>
          <lists>
                  ....
                  ....    Removed for brevity
                  ....
          </lists>
      </web>
  </site>

Some tweaks need to be made to SPUserUtil over this weekend, and it should be ready to go by the end of next week if not sooner.

I'll keep you posted!

- Keith

The SharePoint Utility Suite recommended as number 1 tool kit of “must haves” !! Thanks Joel!

krichie — Thu, 27 Jul 2006 01:04:00 GMT

Joel Oleson has the following to say in his Top 10 and a few "MUST HAVE" Management Tools for WSS v2 & SPS 2003 posting about the tools I've been writing:

Site, Database & User Management

SharePoint Utility Suite v2.5 – User, Site, & Database Manager & Analyzer

Keith Richie has put in some huge investments to make the life of the IT Pro easier. This is my number 1 tool on the this list of “must haves”. Use it for Sites, Users, and managing databases. Don’t forget to use this to move subsites, My sites, and move users between domains… This stuff is incredible. He’s now working on the next rev of this and other cool stuff. I recommend following his blog.

Thanks Joel!!

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager

krichie — Fri, 21 Jul 2006 01:22:00 GMT

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager

This is the second in a series of posts in regards to "Not being mislead by what your seeing :)"

In the next post in this series, I'll talk about SharePoint index sizes vs the size of your databases, as you can't assume that with a larger database, you have an equally larger index. I'll show you how it can actually be the complete opposite, where your index can actually be smaller for a larger database vs an index of content on a much smaller database

DISCLAIMER: This post shows using Query Analyzer to issue statements against your SharePoint content databases. By no means does this mean that you should change any thing in the database. This is simply for "READING" values. And even this should be done during Off-Peak hours.

Holy SQL Spikes Batman!

A long time ago, in a cube far, far away (Ok, across the hall from where I'm at now, so maybe not so far away)....I worked on an issue with a customer that:

On one hand, literally sucked the life right out of me :)
On the other, give me the opportunity to come up with SPSiteManager (See Codin' till the fingers go numb)

Due to a recent occurrence of another customer hitting this, and the fact that I discovered I really didn't share this before, I felt this posting was warranted.

First off, thanks to Denzil Ribeiro, a SQL Escalation Engineer who originally worked with me internally to help narrow this down when I was an Escalation Engineer for SharePoint.

The Symptoms and Problem

You have a SharePoint site or Portal where you really don't have a lot of users permissions, but you do provide access via possibly a security group.
You don't have a lot of documents or list items, and are well within capacity planning guidelines.
99% of your sites have a relatively low number of users associated to it,

There have never been more than say a few hundred unique users every hit the site, or perhaps they are mainly My Sites, with one or two users (The MySite owner, and a friend or two), or perhaps they are many, many team sites with low activity.

1% of your sites, or rather perhaps just one single Site collection, or Portal Site Collection, has > 10,000 users associated with it.

You've provided access to the site/portal with domain groups, or even a single domain group such as NT Authority\Authenticated Users

Yet, sometimes, your performance or even accessibility to your SharePoint environment is completely blocked.
The Processors on your SQL server spike and sustain a high CPU utilization %, thus bringing everything else to a grinding halt.

In the case where we first hit this, the customer had plenty of web front ends, and was running on an active/active cluster with 16 processors per node, and all 32 procs would spike to 100%, they had one content database with a Portal site collection, and > 9,000 team and My Sites.

The Reason

When any SQL statement is executed in SQL Server, the relational engine first looks through the procedure cache to verify that an existing query execution plan for the same SQL statement exists. SQL Server reuses any existing plan it finds, saving the overhead of recompiling the SQL statement. If no existing execution plan exists, SQL Server generates a new execution plan for the query. (It determines based on the data, the indexes, amount of joins, etc the best approach for retrieving the data to handle the request and presenting the data back to the caller). This execution plan is then cached for use later. This is especially important, if the statement is very parameterized, such as the queries we use to generate result sets for document library and list views.

In this case, the initial statement is compiled with one set of parameters, and for 99% of the time, the plan that the SQL optimizer produced, was a good query plan based on that set of data, and the statement is executed in negligible time. When this same query is executed again with a different set of parameters, since it was a parameterized query, we look up the cached plan and find it, thus use it. The cached plan is good for the first set of parameters, but BAD for the second set, and thus can cause your SQL server to consume the CPU trying to process the results based on the plan it generated previously.

The reason, in this case, why the plan is bad for the second set of parameters was because of the data distribution in the UserInfo/UserData tables. (For more information on how users are cached in the UserInfo and UserData tables, see my posting here: http://blogs.msdn.com/krichie/archive/2006/02/18/534767.aspx ) Because this single site has many users recorded against it via the domain group, it increases the likelihood of this problem. The majority of the sites have very few rows in the UserInfo table for a given site, while the minority has thousands of rows in it for their sites. Hence, the execution plan chosen for a site with very few rows was optimal for any site that had few rows, but when that query plan is used with a site that has a TON of rows it is not optimal for that at all.

I want you to realize, parameterized queries and query execution plans ARE NOT BAD in general. In fact they are good for most cases where there is even data distribution, yet can cause problems if the data distribution is uneven OR if typical parameters are passed initially when forming the plan that is cached.

Oh dear, whaddya gonna do....

I filed a bug with the SharePoint product group team on this, as it wasn't a problem with SQL, it was just how SQL works. The SharePoint product group and sustaining engineering teams both agreed we needed to fix this, it's just the proper fix needed to be thoroughly investigated, regression tested, stamped with approval, etc. We all knew this was going to take a really long time to get the best fix created. (Surprisingly, this didn't take as long as the orphan cleanup fixes I worked with many folks on to push forward :)) Due to timing, we didn't make SP2, so this was a post SP2 hotfix for Windows SharePoint Services, which I HIGHLY encourage and recommend that you deploy. You can get the fix here: http://support.microsoft.com/?id=900929

In the mean time, I had to help this customer. The solution was to move the single site with the large amount of users associated with it to a separate content database, what I call repartitioning or site leveling. Query Execution plans are database specific, so this would prevent the re-use of the plan on large sites.

Ah, that's a snap right? Well, in the case where the site in question may have been a single Windows SharePoint Services team site, I could have used STSADM -o backup/delete/restore to the target database to take care of this one site, and be done with it all..but guess what....It wasn't a team site that was the problem, but rather the Portals site collection. That was the one with the large user dataset. Guess what, you can't use STSADM for that :) Also, you can't use SPSBackup for this either, because SPSBackup backs up the entire content database, not just the portals site collection.

What's the only other way to do this? Repartition the other 9,999 team sites out of this content database, and into another.

So just how would you do that? Ah shucks that's easy...Here's the steps:

In order to repartition Site Collections in and out of content databases you have to use the following steps:

If you have not done so already:
1. Create a new content database to place the backed up site into.
Read/Write lock the site so that it cannot be accessed during the backup operation
1. Navigate to Windows SharePoint Services Central Admin page
2. Select Manage Quotas and Locks
3. Select Manage Site Collection Quotas and Locks
4. Type in the URL of the site you need to lock and click the “View Data” button.
5. Select the “No Access” radio button in the “Site Lock Information” section and choose OK.
STSADM –o backup (to backup the site collection to disk)
STSADM –o deletesite (To delete the site from the current content database)
Visit the Content Database management page from the Windows SharePoint Services Central Admin pages.
1. Select the First content database and either:

i. Set it’s status to Offline or

ii. Set it’s max-sites and warning-sites counts to that equal to the number of current sites in the content database (Be sure to have a notebook handy to write down all of the original values so that you can re-set them in a later step.)

Repeat step 4.a for each content database EXCEPT the one you are restoring to (This prevents the WSS Logic from load balancing the existing content databases, to determine which content database to restore the site in)

STSADM –o restore (restore the site into the new targeted database)
Read/Write unlock the site to allow access to the site again
1. Navigate to Windows SharePoint Services Central Admin page
2. Select Manage Quotas and Locks
3. Select Manage Site Collection Quotas and Locks
4. Type in the URL of the site you need to lock and click the “View Data” button.
5. Select the “Not Locked” or “Adding content prevented” radio button in the “Site Lock Information” section depending on the sites previous state, and choose OK
Visit the Content Database management page from the Windows SharePoint Services Central Admin pages.
1. Select the First content database and either:

i. Set it’s status back to Online (If it was online to begin with) or

ii. Set it’s max-sites and warning-sites counts to their previous values (refer to the notes taken in Step 4.a.ii)

Repeat step 6.a for each content database

Repeat steps 2-8 (unless you need to create more content databases, in which case you restart at step 1).

In other words, do these steps ~10,000 times. Let's see, would YOU want to do those steps manually? Ah shucks that's easy...NOT!

As you can see this is a very labor intensive and time consuming process. If you had to do this for many sites it becomes impractical to do all of this manually.

The Solution, and birth of SPSiteManager

I had to come up with a solution for my customer. Otherwise, I would not be able to sleep....It was my issue, my customer, and they were in some serious pain, and besides...my job at the time required me to work the issue to closure. They were experiencing the issue all the time, and the outage was a sustaining 3 minutes for every user who hit this one tiny little document library, on this VERY popular portal area :). So what would happen, is users would notice the problem, and then keep trying to click on this document library...As soon as it would render, another user would come along and hit it, thus...down again. They were quite literally down, completely down, until they prevented access to this document library. I knew that we could perform the same steps needed as in the manual steps above, via the SharePoint Object Model, so off I went, Codin' till the fingers go numb.

While I was writing the repartitioning logic, I knew we would have other customers eventually hit this before the fix, and it would also be a great tool to have around, in case you needed to move sites around regardless of the bug or not. I also wrote a simple detection algorithm, to analyze and detect where content databases may be in this state, thus the reason you see the lowusercount= and highusercount= attributes in the <database> element in SPSiteManagers Site Distribution Document. I worked with MSIT to get it run on our own Microsoft internal SharePoint servers, to look for the occurrence of the problem. The product group even gave me a mirror of their internal server (A really big heavy portal server) to work against to ensure we got the logic flawless for solving this type of problem. (And if I haven't told you enough PG, MSIT, etc.....Thanks!!!!)

Heck, since I had to go examine the sites, webs, etc..I thought what the heck, I'll even go add in the checks against the capacity planning guidelines we have for SharePoint. The result, was yet another little tool that went "Boom"

In the end, when the customer ran SPSiteManager (in batches) to repartition their sites out, they completed it in about 10 man hours. That's ~10,000 site collections moved in 10 man hours...Now, imagine the amount of time it would have taken to do the manual steps above.

The ultimate fix

Now before you go off running, and telling everyone that they must repartition their large user associate sites immediately, STOP NOW. You do NOT have to do this. As I noted above, a post SP2 hotfix is available to remedy this problem. See http://support.microsoft.com/?id=900929 .

I will tell you right now, though, that:

The KB does not necessarily call this out, and in fact is worded improperly

It's currently worded as:

"When you run an ad hoc query on a Windows SharePoint Services site, the CPU usage on the server that is running Microsoft SQL Server consumes 100 percent of the CPU time. Therefore, users cannot access the server."

Which is VERY misleading. This in by NO means whatsoever implies that you should go run ad hoc queries on your database.
We are working to get the official KB updated with the proper wording.

It IS RECOMMENDED however, that you keep a portals site collection in it's own database, as with a portal, it is very common that you would have a large set of users associated with it, and it can get quite large, and if you need to move it later, guess what you can't. You'll have to repartition all those team sites out of it, rather than re-locate the Portal site collection to a new database. What I would recommend, is that when you create a new Portal, go set it's "Max Number of Sites" to 1 in the Manage Content Databases page, then create a separate content database for your team sites.

In Conclusion

If you feel you may be experiencing this problem, and you do not have the Post SP2 hotfix installed as noted above, a couple of simple tests can be executed

While experiencing the problem, issue the following statement against the content database in question.

DBCC FREEPROCCACHE

If your hang is cleared up by this, chances are you are being impacted by this issue.

Use SPSiteManager to perform a simply analysis (Without the need to do a deep level analysis) and look at your user associates on your content databases:

SPSiteManager -o analyze -allvs
- If you see a lowusercount <= 1000 users and a highusercount >= 9,000 to 10,000 users, chances are you are being impacted by this issue.

To put it bluntly, install the fix now (or the most recent rollup hotfix which will include this.) :)

Hope this helps!!

- Keith

Previous Posts on this series:

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Additional References:

Description of the Windows SharePoint Services post-Service Pack 2 hotfix package: November 15, 2005
http://support.microsoft.com/?id=900929

SQL Server Architecture: Execution Plan Caching and Reuse
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/architec/8_ar_sa_4azp.asp

SQL Server Books Online: DBCC FREEPROCCACHE
http://msdn2.microsoft.com/en-us/library/ms174283.aspx

Welcome John Kozell to the blogging world!

krichie — Thu, 20 Jul 2006 16:34:00 GMT

John Kozell was co-author for the Portal additions to SPUserUtil. If it were not for John, those changes would probably never made it in. John is also the author of the .Net Protocol Handler example in the SharePoint Utility Suite.

John has now joined the blogging world!!

He's already made a few posts, and you can check them out here:

http://blogs.msdn.com/johnkoz

- Keith

SPSiteManager update coming soon

krichie — Tue, 27 Jun 2006 15:25:00 GMT

In the mean time, we've found some immediate additions we need to make to SPSiteManager based on overwhelming customer requests and feature requests. Thus, I'm working on an update to SPSiteManager to include the following before moving forward.

First I'll note what is complete, and then what is in progress to go into this version before I release it. This of course, is not an all encompassing feature request list :).

What's Complete

Fix for removefromdirectory and removefromcrawl operations

I found that my comparison was literal, and if you created the site via the Portals Create Site link, then we only stored the relative path to the site in the directory. So I updated the comparison logic to do a literal comparison, and then a relative comparison.

New site directory targeting via -sitedirectoryurl switch

Not all customers leave their site directory in the same place :) I even commented this in the code and left it with it assuming that your site directory was in the same place. I've corrected this so now you can specify a specific site directory to work on with the -sitedirectoryurl switch

New enumsitecrawl operation

This exports your entire list of URLs from your site crawl list.

New enumsitedirectory operation

This exports your enitire list of URLs from your Site Directories

New adddatabase operation

This operation can be used to enumerate over the <contentdatabases> section of a Site Distribution Document to create Content Databases. This is extremely helpful if you are mirroring a farm with lots of content databases, and want to match database names, etc on a target system.
Theoretically speaking, you could run an analysis on one farm, then use SPSiteManager to create the necessary databases, then restore sites. (In the future, I hope to be collecting enough information in the Site Distribution Document, that you could basically replay and duplicate your entire farm settings if needed)

New site owner attributes reported in <site> elements for analysis.
New Version Purging

Thanks to Raymond Hung, SPSiteManager now has features to help you analysis and gather statistics on version weight in your system, as well as clean them up..

What's In Progress

Content Source Auditing

The SharePoint Utility Suite has a tool called SPAddCS, and I made some changes to it to "Read" and "Write" those content sources. I'll be incorporating those changes into SPSiteManager so that you can utilize it to "Capture" your content source settings, and "Replay" them onto a server if needed

Various Alert help operations

Alert Purging
Alert "URL" fixup. I.e., if you server name changes, your alerts may still fire with the link to the old URL.
Email changes for Alerts

I hope to have these changes complete and tested by the end of the week. Once they're done, I'll post again with an update.

- Keith

Dispose of those Objects! SPSites, SPWebs, and SPBears oh my!

krichie — Thu, 15 Jun 2006 21:42:00 GMT

For the past two years or so, I've constantly worked with customers on issues related to not properly disposing of SPSite/SPWeb and other objects when necassary in long running applications, web parts etc. We've pushed for SDK updates etc where necassary, but we didn't have a complete "Developers Guide" on this subject alone.

Scott Harris took the initiative and took all the hard work that I, Chris Gideon, himself and others have struggled with over the past two years and put it into a great document along with Mike Ammerlaan (A PM for the SharePoint Product Group)

After many weeks of composing, reviewing, editing, fact checking :) We've completed it and it is now available:

Best Practices: Using Disposable Windows SharePoint Services Objects

http://msdn2.microsoft.com/en-us/ms778813(office.12).aspx

Hope this helps, as it's part of my daily mantra with folks when I discuss this. (Now I need to go an make sure I'm practicing what I preach in my tools :))

- Keith

The Maestoso Interstellar Suite - Part II

krichie — Mon, 12 Jun 2006 16:33:00 GMT

First I want to thank those who placed comments on and the emails I received with comments on Part I of The Maestoso Interstellar Suite. I was quite surprised at the many positive comments. I was hoping "someone" would like it, but didn't realize it would generate so many kudos back.

Over this past weekend, I started working on Part II. Nowhere near complete, but it's a start. The versions available there will not be the final versions. Once I have them wrapped up, I'll reduce the links to just a couple minute samples etc as I prepare it for release.

I'm in Redmond this week to work on the Desired Configuration Monitoring project for SharePoint. I have no estimate of when the DCM for SharePoint will be complete, but I'm here to help get it going.

During my down time this week, I also have a few changes and enhancements I've made to SPSiteManager (More on those in a later post) that I'll need to finish up this week and get posted. One of the bigger items, is a Document Version Purging option to allow you to purge document versions that are no longer needed within a date range or version count option.

Troubleshooting WSS Event 1000 DB Connectivity Problems - Part 2

krichie — Wed, 24 May 2006 14:37:00 GMT

Chris Gideon has posted a great article with a list of the most common reasons why you'll see the dreaded WSS Event 1000 DB Connectivity even as noted in my previous article Troubleshooting WSS Event 1000 DB Connectivity Problems. In my article I noted that this can be caused by numerous things.

Chris provides a great 20 item check list of things to ensure are correct that can cause this problem, and we both know there are more. The fact is, it's a "General" error message that bubbles up because of various database connectivity problems, and SharePoint is simply a victim. As he notes though, V3 makes the bubbling up of this warning alot clearer to help you identify the problem.

I also love how he notes that we use SPSiteManger frequently to help identify problems like these. We're diggin this tool :) I can't wait to go full steam on SCA.Next in which I'll be incorporating all of SPSiteManagers features.

Read the full post on his blog.

- Keith

SharePoint Account Management using SPUserUtil - Part 3 - Auditing Accounts

krichie — Mon, 22 May 2006 22:15:00 GMT

SharePoint Account Management using SPUserUtil - Part 3 - Auditing Accounts

SPUserUtil will mean either WSSUserUtil or SPSUserUtil respectively (WSSUserUtil is used to administer Windows SharePoint Sites on a standalone WSS Farm/Virtual server OR Windows SharePoint sites in the same virtual server of a SharePoint Portal Server 2003 site.) SPSUserUtil is a superset of WSSUserUtil, designed for working on SharePoint Portal Server Areas.

I went back and renamed some of the previous posts I had written about SharePoint Account Management to reflect that these are all really within the realm of SharePoitn Account Management, and each post covers a specific aspect of such.

In fact, the oldest post is now called "Part 0" :)

For a recap (And to unravel any confusion this may have caused :)), here is the series so far.

In this post, I'll show you the many different ways you can use SPUserUtil to audit accounts in your SharePoint environment

Out of Box methods for auditing accounts

The OOB (Out of Box) way to currently audit security principals on your sites and webs is to visit each respective site collection or web and look at their corresponding users list. But nothing (outside of purchasing a third party tool) gives you the ability to quickly audit for users and groups across site collections and webs.

Out of Box methods for auditing accounts

Auditing accounts for a web through the UI

If you just want to view the current account rights on a web, you simply need to access the "Manage Users" page for the web in question. The navigation path to this page (for Windows SharePoint Services) is:

From the home page of the site, click "Site Settings" on the top navigation bar
Click "Manage Users"

If you have removed the navigation bar on your site, or you want to just jump right to the page, you can access it by navigating to the users.aspx layouts page directly in your browser. For example:

http://server/sites/asite/asubweb/_layouts/1033/user.aspx

This will take you to the "Manage Users" page for the subweb named "asubweb" underneath the top level site "asite" within the managed path "sites" on the "server" in question.

From this page, you can view the account resources on the web. If you have many accounts, you will need to click on the previous and next links on this page to paginate through the list of accounts.

Note: Keep in mind, that the Windows SharePoint Services Capacity Planing Guide states that the The size of the access control list is limited to a few thousand security principals, in other words users and groups in the Web site. The Guideline for optimum performance is 2000. If you need to utilize more than 2000 individual security principals, consider using NT Security Groups, as I have seen many customer experience problems when they exceed these guidelines. SPSiteManager (Also located in the SharePoint Utility Suite) can assist you in detecting and reporting where you have exceeded this capacity planning guideline.

Auditing accounts for a site collection through the UI

If you want to audit accounts that have access from a site collection perspective you simply need to access the "Manage Site Collection Users" page for the site collection in question. The navigation path to this page (for Windows SharePoint Services) is:

From the home page of the site, click "Site Settings" on the top navigation bar
Click "Go to Site Administration"
Click "Go to Top-level Site Administration"
Click "View site collection user information"

If you have removed the navigation bar on your site, or you want to just jump right to the page, you can access it by navigating to the siteusrs.aspx layouts page directly in your browser. For example:

http://server/sites/asite/_layouts/1033/siteusrs.aspx

This will take you to the "Manage Site Collection Users" page for the site collection whose top level site is "asite" within the managed path "sites" on the "server" in question.

From this page, you can view the account resources at the site collection scope which shows you all users who have at least Guest rights on a web within the site collection. This shows you a concise list of all users whom have hit a web within the site collection. If you have many accounts, you will need to click on the previous and next links on this page to paginate through the list of accounts.

Cumbersome Problem Number 1

This becomes an administrative headache if you have 100's or 1000's of account resources on your webs or sites, in which case you have to paginate through all these resources on either the users.aspx page or the siteusrs.aspx pages, to "find" an account.

Cumbersome Problem Number 2

If you have 100's or 1000's of site collections, it makes it nearly impossible to find what sites and webs a given account has, or ever had, access to.

SPUserUtil to the rescue.

You can use the analyze operation of SPUserUtil to analyze a Windows SharePoint Services web or SharePoint Portal Server Portal for auditing purposes. This operation will scan your targeted URL and produce two files (the user map and the webs manifest file) which can be used for security audit reviews.

You can adjust the scope of this analysis to a single web, a series of webs, a complete site collection, or multiple site collections on a specified virtual server.

The Webs Manifest (depending on your analysis scope) can give you a holistic view of user security across your SharePoint farm.

Note: SPUserUtil and SPSiteManager are being merged into one all encompassing tool. This will give you the added benefit of taking advantage of the SPSiteManager SDD (Site Distribution Document) as well as the ability to scan multiple virtual servers at once for auditing purposes.

Note: Pay special attention to the new -asuonly and -usermask switches noted below

Auditing accounts for a web using SPUserUtil

By Using the analyze operation in SPUserUtil, it's extremely easy to analyze a single web, or multiple webs and produce a file in XML that can be used with any tool capable of displaying XML to audit user rights.

For example:

wssuserutil -o analyze -url http://server/sites/asite -r -usermap c:\all-users.xml -webfile c:\allusers-webs.xml

This example starts its analysis at the web/site designated by –url, and scans all subwebs recursively underneath which results in producing two files as noted below:

UserMap file

The UserMap file will contain a distinct list of unique users found within the scope of the analysis operation

Webs Manifest File

This file gives you a complete hierarchical view of the webs and their individual rights on each.

For example:

<?xml version="1.0" standalone="no"?>
<!DOCTYPE SPUserUtilWebFile>

<webs>
<web url="http://myserver/sites/artists" title="Artists Site"
description="Cool Artists Web" lcid="1033" template="STS" uniqueperms="True">
    <user loginname="MUSIC\edgarf" displayname="Edgar Frose" email="edgarf@tangerinedream.org" notes="" sid="S-x-x-xx-xxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxx" >
      <crosssitegroups />
      <sitegroups>
        <group name="Contributor" />
      </sitegroups>
      <listpermissions />
    </user>
    <user loginname="MUSIC\cfranke" displayname="Christopher Franke" email="cfranke@sonicimages.com" notes="" sid="S-x-x-xx-xxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxx" >
      <crosssitegroups />
      <sitegroups>
        <group name="Contributor" />
      </sitegroups>
      <listpermissions />
    </user>
    <web url="http://myserver/sites/artists/movies" title="Movies Subweb" description="This site contains information related to movies" lcid="1033" template="STS" uniqueperms="True">
      <user loginname="HOLLYWOOD\bwillis" displayname="Bruce Willis" email="bwillis@hollywood.com" notes="" sid="S-x-x-xx-xxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxx" >
        <crosssitegroups />
        <sitegroups>
          <group name="Contributor" />
        </sitegroups>
        <listpermissions />
      </user>
      <user loginname="HOLLYWOOD\glucas" displayname="George Lucas" email="glucas@hollywood.com" notes="" sid="S-x-x-xx-xxxxxxxxx-xxxxxxxxx-xxxxxxxxxx-xxxxxx" >
        <crosssitegroups />
        <sitegroups>
          <group name="Contributor" />
        </sitegroups>
        <listpermissions />
      </user>
    </web>
</web>
</webs>

Notice that for each web found within the scan, an individual <user> element is written for each security principal found. Each <user> element will contain a <crosssitegroups> and <sitegroups> container that may contain a list of Cross Site Groups or Site Groups the security principal is in. These individual groups will be noted by a <group> element with each respectively.

Also, for each document library or list where you have explicit permissions set, you'll see a <list> entry within the <listpermissions> container with the title of the list/document library and the permission mask.

The web manifest file only shows the accounts which currently have permissions on sites/webs detected in the scan, but does not have the complete list of users that may exist in the site collection, and thus the usermap will only reflect those users. If you want to see a complete unique list of accounts that have ever hit the site collection (and were not removed from the site collection) use the -asu or -asuonly switch.

The -asu switch causes the usermap to be populated with the complete list of Site Collection users regardless if they currently have permissions on a web or not. When using this switch, we write out all the site collection users, then proceed to do the normal web analysis.

The -asuonly switch performs the same function as the -asu switch, but bypasses the web scanning to create the web manifests file.

If you don't specify a -webfile switch, a webs manifest file is still created for you using the name of the usermap appended with "-webs" in the filename.

Auditing accounts for a site collection using SPUserUtil

In the example in the previous section, asite is the top level site in the site collection, thus the complete site collection was analyzed with the above example. If you wanted to analyze every single site collection on a virtual server, add the -ac switch

For example:

wssuserutil -o analyze -url http://server/sites/asite -r -usermap c:\all-users.xml -webfile c:\allusers-webs.xml -ac

Even though the above example still specifies a direct site collection in the -url, by specifying the -ac (All Collections) switch we simply connect to the virtual server for the URL, and analyze every Site Collection found for the virtual server.

In this case, the usermap will contain a distinct list of users detected in the scan by distinct across the entire virtual server for all site collections.

The Webs Manifest file will also have a top level <web> element for each site collection found within the <webs> container.

Auditing a single account across the farm using SPUserUtil

In this example, we'll use the -usermask switch to filter the analysis down to a single account, or find accounts using a given mask

For example to scan an entire virtual server to determine where a specific user has rights:

wssuserutil -o analyze -url http://server -r -usermap c:\all-users.xml -webfile c:\allusers-webs.xml -ac -usermask "TAILSPINTOYS\krichie"

This will produce a webs manifest file, but only the account TAILSPINTOYS\krichie will be reported on.

If I wanted to filter for accounts only matching a portion of the login name, say perhaps all users from the WINGTIPTOYS domain

wssuserutil -o analyze -url http://server -r -usermap c:\all-users.xml -webfile c:\allusers-webs.xml -ac -usermask "*WINGTIPTOYS*"

This will produce a webs manifest file, but only report on accounts which have WINGTIPTOYS within the login name

You would repeat this process for each virtual server in your farm to get a complete list of account right locations through your farm.

Note: Once the merge of functionality of SPUserUtil and SPSiteManager is complete, the resulting output will not include the nested webs when filtering on specific account masks as it does now. The way SPUserUtil works in it's current iteration, you could have many <web> elements and nested <web> elements in the resulting file for your entire farm, even though the account only exists on one web.

I hope you see the benefit in the auditing capabilities of SPUserUtil. Our goal is to improve this experience in the next release of this tool set. If you have any questions or comments, please let me know!

- Keith

For more information in regards to the Schema of the Various SharePoint Tables, see the Databases section in the SharePoint Products and Technologies SDK at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/spptsdk/html/SPPTWSSDatabases_SV01072208.asp

For more information in regards to Managing Users and Cross Site Groups in SharePoint
http://office.microsoft.com/en-us/assistance/HA011608091033.aspx

SPUserUtil is contained in the The SharePoint Utility Suite at:
http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

For More information on the Windows SharePoint Services MigrateUserAccount() API:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/spptsdk/html/tsamSPGlobalAdminMigrateUserAccount_SV01234066.asp

For More information on the SharePoint Portal Server MigrateAccount() API:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/spptsdk/html/mPortalAccountMigManagerMigrateAccount2_SV01187841.asp

For more information on Windows SharePoint Services and SharePoint Portal Server 2003:
http://www.microsoft.com/sharepoint

SharePoint Account Management using SPUserUtil - Part 2 - Adding Accounts

krichie — Fri, 19 May 2006 22:10:00 GMT

SharePoint Account Management using SPUserUtil - Part 2 - Adding Accounts

You don't need a separate tool to just add an account resource to SharePoint sites and webs, but when it comes to scenarios where you need to "Replicate" someone's rights to another account, it can be an extremely difficult and time consuming process to do this manually.

This is the second of a few posts in regards to Adding accounts via SPUserUtil.

See the first post in this series SharePoint Account Management using SPUserUtil - Part 1 - Cloning Accounts

Adding Users

Someone has just joined your team, and you want to give them access using a mix of rights to the same sites as someone else. Cloning doesn’t work here, because you may not want them having the exact same rights as the source account

In this scenario, We’ll add a user and security group to sites/webs that the source account is already an Administrator of. The Security group will be given Contributor rights, and the user will be given Web Designer rights

If you needed to do this for tons of sites and webs, it can be very time consuming if you have to manually visit every single site and web to perform this change.

SPUserUtil to the rescue.

With SPUserUtil, you can automate the above process easily to help relieve the headache of managing users in SharePoint.

Adding Users using SPUserUtil

Using the add operation in SPUserUtil, it's extremely simple to add account resource rights to a series of sites and webs, whether it be a user account or NT Security Group using the following steps:

Perform an quick single user analysis to get a proper UserMap file and a Webs Manifest file showing the location of the single user across your virtual server, or create one from scratch (See SharePoint Account Management using SPUserUtil - Part 0 - Removing Accounts for more information)

WSSUserUtil –o analyze –url http://server -usermap singleuser.xml –r –ac -usermask "*krichie*"

Note: Unlike the cloning example in my previous post, we don’t want to use the –asuonly switch. We want to use the –r <recursive> and –ac <all site collections> switch so that the webs manifest file will have the location of the user.

This will generate the file singleuser.xml. An additional file, singleuser-webs.xml, will be generated that contains the webs manifest file which shows you the location and rights of all users found in the scan. You could of course, just create this file from scratch in notepad if you wanted to.

Here is a version showing this scan on my test environment from a scan on a single user account.

<?xml version="1.0" encoding="utf-8" standalone="no"?>

<!DOCTYPE SPUserUtilWebFile>

<webs>

</sitegroups>

</user>

</sitegroups>

</user>

</web>

</sitegroups>

</user>

</web>

</webs>

Following the analyze operation, I performed the following steps to prepare for my add operation.

Copied the singleuser-webs.xml file to addusers.xml
Edited the addusers.xml file to change the loginname attribute to the loginname of the target user
1. I also changed the group name attribute from “Administrator” to “Web Designer”
Added a <user> element at each web to add a reference to the new Security Group, and give it a <sitesgroups> and <group> element to add this group to the “Contributor” site group in each web.

<?xml version="1.0" encoding="utf-8" standalone="no"?>

<!DOCTYPE SPUserUtilWebFile>

<webs>

</sitegroups>

</user>

</sitegroups>

</user>

</sitegroups>

</user>

</sitegroups>

</user>

</web>

</sitegroups>

</user>

</sitegroups>

</user>

</web>

</webs>

Notice that on subweb2 “NT Authority\Authenticated users” is NOT given the “Contributor” right. The purpose here is to show you that you can change the rights at any point. You can use a mix and match of additional <group> elements within the <sitegroups> and <crosssitegroups> containers if you wanted to add a user to multiple site groups. This minimal example is just to show you that it can be done.

Note: I would NOT recommend using NT Authority\Authenticated Users as a NT Security Group to use. It’s purpose of use in this post is for example purposes ONLY.

5. Use the add operation to add these users with their prescribed rights to SharePoint.

WSSUserUtil –o add –usermap singleuser.xml –webfile c:\addusers.xml -verbose

Note: The usermap operation is a required parameter even though is NOT used in the add operation. This is a small inconsistency that will be corrected at a later date.

In a future post I'll cover Auditing Rights with SPUserUtil to cover more on the analysis features.

I hope this helps, and let me know if you have any questions!

- Keith

For more information in regards to Managing Users and Cross Site Groups in SharePoint
http://office.microsoft.com/en-us/assistance/HA011608091033.aspx

SPUserUtil is contained in the The SharePoint Utility Suite at:
http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

For more information on Windows SharePoint Services and SharePoint Portal Server 2003:
http://www.microsoft.com/sharepoint

SharePoint Account Management using SPUserUtil - Part 1 - Cloning Accounts

krichie — Wed, 17 May 2006 22:15:00 GMT

SharePoint Account Management using SPUserUtil - Part 1 - Cloning Accounts

This will be the first (Really the second, as we're now going 0 based :)) of a few posts in regards to Adding accounts via SPUserUtil.

Replicating Rights/Cloning.

Someone has just joined your team, and you want to replicate, or more precisely clone, your rights on webs/sites to his account. If you didn't use an NT Security Group to manage common rights, you would have to do the following to note all your rights so that you can then add the new team member to sites/webs with the same rights:

Figure out exactly what you have access to across the farms you have access to :)
Note your individual user Role/Rights where you have an individual ACE entry on a a site/web/
Note the groups you are a member of and that groups Role/Rights on every single web, etc.
Visit each site/web through the UI and and add the user.

If you have permissions spread across many webs and site collections (And across farms too) it can be a daunting task.

SPUserUtil to the rescue.

With SPUserUtil, you can automate all the above processes easily to help relieve the headache of managing users in SharePoint.

Note: Pay special attention to the new -asuonly and -usermask switches noted below

Replicating Rights using SPUserUtil

Using the clone operation in SPUserUtil, it's extremely simple to replicate account resource rights from one account to another, whether it be a user account or NT Security Group using the following steps:

Perform an quick single user analysis to get a proper UserMap file, or create one from scratch (See SharePoint Account Management using SPUserUtil - Part 0 - Removing Accounts for more information)

WSSUserUtil –o analyze –url http://server -usermap singleuser.xml -asuonly -usermask "*margie.richie*"

Note: The –asuonly switch means Give me All Site Users associated on the site collection, regardless if they have any direct ACE entries on any webs, and prevents normal web scanning for the user. It implicitly includes the default -asu logic as noted above,

This will generate the file singleuser.xml. An additional file, singleuser-webs.xml, will be generated but it will be completely empty, because we're bypassing the normal web scanning that occurs in an analyze operation. It's a crude but effective way to generate a proper user map file for a single user (or multiple users based on the -usermask). All we're concerned about is getting one <user> element created for the user. You could of course, just create this file from scratch in notepad if you wanted to.

Note: A future version of this tool will eliminate the need to generate a UserMap file for single user operations. In other words, you'll be able to specify -userlogin to target a specific user without the need to generate the UserMap.

Here is a version prepared to work on a single user account.

<?xml version="1.0" standalone="no"?>

<!DOCTYPE SPUserUtilUserMapFile>



<users>

<user loginname="MYDOMAIN\margie.richie" newloginname="" />

</users>

Edit the singleuser.xml file to add the newloginname attribute. For example:

<?xml version="1.0" standalone="no"?>

<!DOCTYPE SPUserUtilUserMapFile>

<users>

</users>

Pass this file back into SPUserUtil to replicate the rights for MYDOMAIN\margie.richie to the security group MYDOMAIN\SomeSecurityGroup using the clone operation
WSSUserUtil –o clone –url http://server -usermap singleuser.xml -r -ac

This will enumerate over every single web (via the -r <recursive> switch) on every single site collection (via the -ac <all collections> switch) and replicate/clone the rights for MYDOMAIN\margie.richie to the security group MYDOMAIN\SomeSecurityGroup.

You could of course use a user account as the source or target, as well as a Security Group, or mix and match between them. Also, by using the UserMap, it allows you to batch up a series of cloning operations.

Note: A future version of this tool will eliminate the need to generate a UserMap file for single user operations. In other words, you'll be able to specify -userlogin to target a specific user without the need to generate the UserMap.

If you were to then do an analysis of the site collection now, you'll see where MYDOMAIN\SomeSecurityGroup matches the rights of MYDOMAIN\margie.richie.

Of course, it's always best to use NT Security groups for common rights so you don't have to perform this process all the time.

In a future post I'll cover Altering Rights using SPUserUtil as well as Auditing Rights.

I hope this helps, and let me know if you have any questions!

- Keith

For more information in regards to the Schema of the Various SharePoint Tables, see the Databases section in the SharePoint Products and Technologies SDK at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/spptsdk/html/SPPTWSSDatabases_SV01072208.asp

For more information in regards to Managing Users and Cross Site Groups in SharePoint
http://office.microsoft.com/en-us/assistance/HA011608091033.aspx

SPUserUtil is contained in the The SharePoint Utility Suite at:
http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

For more information on Windows SharePoint Services and SharePoint Portal Server 2003:
http://www.microsoft.com/sharepoint

SharePoint Account Management using SPUserUtil - Part 0 - Removing Accounts

krichie — Wed, 17 May 2006 19:26:00 GMT

SharePoint Account Management using SPUserUtil - Part 0 - Removing Accounts

There are many ways to remove an NT account resource (User or Security Group) from a SharePoint resource. You remove an account resource at the web level, or the site collection level.

Considerations

Consider an account resource which only permissions on one web within a site collection. In this scenario if the account resource only had permissions on one single web in the site collection, it's easy to just navigate to that one web and remove it.

If the account resource had permissions on a lot of webs in the site collection, you can just visit the siteusrs.aspx page for the site collection and remove it from there. When you do this, it completely removes the account resource from the site collection as a whole, thus removing all references to the resource on any web they exist on.

If you don't want to just mass remove the resource from the site collection in whole, but only want to remove a resource from multiple webs in a site collection, you have to visit each web and remove the resource manually for each one.

I'll detail each of these OOB (Out of Box) methods further down, then detail how to do the same process (which I believe is easier) using SPUserUtil

It's not a bug :)

When an account resource is deleted from Active Directory or NT, SharePoint does not automatically remove the account resource information from it's sites and webs. This is not a SharePoint problem, as the same issue exists on any other NTFS resource. For example, a folder on your PC, or a file share on the network.

For the most part, most administrators don't worry too much about cleaning up dead account resources from NTFS resources, as those accounts ARE deleted from AD, and can never compromise security...They're just (for the lack of better words) orphaned ACE's on the resource.

When you do decide to remove the account resource, it's pretty easy to just bring up the resources properties dialog....Scroll down to the orphaned ACE, and remove it.

For SharePoint on the other hand, it is quite cumbersome because you have to paginate over, and over, and over, and over until you get to the account resource in question to remove the ACE.

Basic scenario

A user in your enterprise is terminated, or departs on his own free will :) You need to either

Remove the user from SharePoint Sites.
Transfer Ownership of SharePoint sites/webs/etc to another individual or

This post will deal with simply removing the user completely. We'll cover ownership transfer in a future post.

Leaving the user information on the sites doesn't necessarily present a problem, but it can be an administrative headache. I have seen a few reports that when trying to remove users from Portal areas, that it balks because it still tries to do a domain lookup on the user when you're deleting them. If you have a repro of that, you should call Customer Support Services if it is preventing you performing your administrative task.

Out of Box methods for removing users

Removing a single resource from a web through the UI

If you just want to remove an account resource from a web (whether the account resource is deleted or not), you simply need to access the "Manage Users" page for the web in question. The navigation path to this page (for Windows SharePoint Services) is:

From the home page of the site, click "Site Settings" on the top navigation bar
Click "Manage Users"

If you have removed the navigation bar on your site, or you want to just jump right to the page, you can access it by navigating to the users.aspx layouts page directly in your browser. For example:

http://server/sites/asite/asubweb/_layouts/1033/user.aspx

This will take you to the "Manage Users" page for the subweb named "asubweb" underneath the top level site "asite" within the managed path "sites" on the "server" in question.

From this page, you simply select the users/groups you want to remove via the check box next to each resource, then select the "Remove Selected Users" link.

This removes an account resource from this web only. If you need to remove this user from multiple webs, follow the same process noted above for each web.

Removing a single resource from a web using STSADM

You can simplify the process by using STSADM if you have sufficient rights using the deleteuser operation.

stsadm.exe -o deleteuser url http://server/asite/asubweb userlogin DOMAIN1\AUser

This removes an account resource from this web only. If you need to remove this user from multiple webs, follow the same process noted above for each web.

For more information on STSADM operations, see: http://office.microsoft.com/en-us/assistance/HA011608091033.aspx

Removing a single resource from a site collection

Removing a single resource from a web, DOES NOT remove them entirely from the site collection, even if the account resource only had permissions on the single web in the site collection from which you just removed them. For some of the reasons why, see my previous blog post about Using SPUserUtil to synchronize SharePoint user Display Names and Email addresses with the information in Active Directory.

If you want to remove a single resource from an entire site collection (whether the account resource is deleted or not) you simply need to access the "Manage Site Collection Users" page for the site collection in question. The navigation path to this page (for Windows SharePoint Services) is:

From the home page of the site, click "Site Settings" on the top navigation bar
Click "Go to Site Administration"
Click "Go to Top-level Site Administration"
Click "View site collection user information"

http://server/sites/asite/_layouts/1033/siteusrs.aspx

This will take you to the "Manage Site Collection Users" page for the site collection whose top level site is "asite" within the managed path "sites" on the "server" in question.

From this page, you simply select the users/groups you want to remove via the check box next to each resource, then select the "Remove Selected Users" link.

This removes an account resource from this site collection only. If you need to remove this user from multiple site collections, follow the same process noted above for each site collection.

There is no equivalent STSADM operation (such as seleteuser noted above) to remove a user from the entire site collection

Cumbersome Problem Number 1

This becomes an administrative headache if you have 100's of account resources on your webs or sites, in which case you have to paginate through all these resources on either the users.aspx page or the siteusrs.aspx pages, to get to the resource in question so you can mark them for deletion. You could certainly script it using the deleteuser operation for STSADM, but this does not remove them from the site collection.

Cumbersome Problem Number 2

As noted above, scripting calls to STSADM can still be cumbersome, as you would have to run it on every single web in your entire farm to remove the user from the webs. Since there is no STSADM equivelant for removing users from the entire site collection, you have to manually visit each site collection in your farm (There could be literally 10's of thousands of site collections) to remove the user.

SPUserUtil to the rescue.

With SPUserUtil, you can automate all the above processes easily to help relieve the headache of managing users in SharePoint.

Note: Pay special attention to the new -asuonly and -usermask switches noted below

Removing a single resource from a web or series of webs using SPUserUtil.

If you just want to remove an account resource from a web (whether the account resource is deleted or not), using SPUserUtil, following these steps:

Prepare a SPUserUtil usermap file with a user element which has the loginname= attribute set accordingly.
loginname= should be the NT Login Name as reflected in SharePoint.
I actually recommend running a analyze of the site to get a complete list of all users. You can then use this file as a template.

For example:

WSSUserUtil –o analyze –url http://server/sites/site -usermap users.xml –asu

Note: The –asu switch means Give me All Site Users associated on the site collection, regardless if they have any direct ACE entries on any webs.

This will produce a listing of all users and generate the file users.xml. An additional file, users-webs.xml, will be generated that contains the web hierarchies and the location of user permissions on each web.

Here is an example of the output (Of course, your results will be different :)).

<?xml version="1.0" standalone="no"?>

<!DOCTYPE SPUserUtilUserMapFile>

<users>

<user loginname="MYDOMAIN\margie.murphy" newloginname=""

displayname="Margie Murphy" email="margie.murphy@MYDOMAIN.com"

sid="S-1-5-21-5555555555-555555555-5555555555-1111" />

<user loginname="MYDOMAIN\Administrator" newloginname=""

displayname="" email=""

sid="S-1-5-21-4444444444-555555555-5555555555-500" />

<user loginname="MYPORTALSVR\administrator" newloginname=""

displayname="MYPORTALSVR\administrator" email="administrator@MYDOMAIN.com"

sid="S-1-5-21-3333333333-555555555-5555555555-500" />

</users>

Here is a version prepared to work on a single user account. (Note: the account listed below would NEVER be deleted from my world :))

<?xml version="1.0" standalone="no"?>

<!DOCTYPE SPUserUtilUserMapFile>

<users>

</users>

It's actually much easier than that in the current release of SPUserUtil. With this updated version there is a new usermask switch that will filter down to a specific user. For example:

WSSUserUtil –o analyze –url http://server/sites/site -usermap singleuser.xml -asuonly -usermask "*margie.richie*"

Note: The –asuonly switch means Give me All Site Users associated on the site collection, regardless if they have any direct ACE entries on any webs, and prevents normal web scanning for the user. It implicitly includes the default -asu logic as noted above,

This will generate the file singleuser.xml. An additional file, singleuser-webs.xml, will be generated but it will be completely empty, because we're bypassing the normal web scanning that occurs in an analyze operation. It's a crude but effective way to generate a proper user map file for a single user (or multiple users based on the -usermask).

Note: A future version of this tool will eliminate the need to generate a UserMap file for single user operations. In other words, you'll be able to specify -userlogin to target a specific user without the need to generate the UserMap.

With this file prepared, we can utilize the -o delete operation of SPUserUtil to delete a user from a web, or series of webs

When running the -o delete operation using WSSUserUtil, it will read in and process all the user elements found in the xml file.

For Example:

WSSUserUtil –o delete –url http://server/sites/asite/asubweb -usermap users.xml

Every single user found in the users.xml file will be removed from the web pointed at by the -url switch.
If you want to recursively remove the user from the web and all of it's subwebs, use the -r switch.

For Example:

WSSUserUtil –o delete –url http://server/sites/asite/asubweb -usermap users.xml -r

Every single user found in the users.xml file will be removed from the web pointed at by the -url switch and all of it's sub webs.

Much easier than using the UI or STSADM -o deleteuser for every web.

Removing a single resource from a site collection using SPUserUtil

If you just want to remove an account resource from a site collection (whether the account resource is deleted or not), using SPUserUtil, following these steps:

Prepare you list of users as noted in the previous section
Run SPUserUtil with the -o deletecollection operation using WSSUserUtil, it will read in and process all the user elements found in the xml file.

WSSUserUtil –o deletecollection –url http://server/sites/asite -usermap users.xml

Every single user found in the users.xml file will be removed from the site collection pointed at by the -url switch.
To remove a user from every single site collection on a virtual server:
WSSUserUtil –o deletecollection –url http://server -usermap users.xml -ac

Every single user found in the users.xml file will be removed from every site collection on the virtual server specified using the -url switch.

Purging all those deleted accounts

So you've discovered that a deleted account resource (Either a User or Security Group) currently has permissions on a web or site in your environment and you want to remove them completely. With the above information in hand, the following short list of steps using SPUserUtil should to the trick.

Generate a valid usermap for a single resource or multiple resources
a) Generate a usermap for a single resource

    WSSUserUtil –o analyze –url http://server/sites/site -usermap singleuser.xml -asuonly -usermask "*krichie*"

b) Generate a usermap for alot of resrouces

    Either run an analyze operation:

    WSSUserUtil –o analyze –url http://server/sites/site -asuonly -usermap users.xml

    And trim out the users you don't want to operate on.

c) Or Create and additional <user> element in the UserMap for each account in question.
Run WSSUserUtil with the deletecollection operation and the -ac switch to completely purge the user from ALL site collections on the virtual server.

WSSUserUtil –o deletecollection –url http://server -usermap users.xml -ac
Repeat step 2 for each virtual server

Note: Like SPSiteManager, there WILL be an -allvs switch added in later that will allow you to perform all operations across all virtual servers, etc, in a future version. That would make the process even simpler by executing a command such as the following:

WSSUserUtil –o deletecollection –allvs -usermap users.xml -ac

My next postings will be about Ownership transfers and Adding accounts en-masse, Brahahahahahha (I'm feeling like a mad scientist right now :))

I hope this helps, and let me know if you have any questions!

- Keith

For more information in regards to Managing Users and Cross Site Groups in SharePoint
http://office.microsoft.com/en-us/assistance/HA011608091033.aspx

SPUserUtil is contained in the The SharePoint Utility Suite at:
http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

For more information on Windows SharePoint Services and SharePoint Portal Server 2003:
http://www.microsoft.com/sharepoint

Keeping the dream alive

krichie — Wed, 03 May 2006 01:21:00 GMT

I get to actually flag multiple categories on this post :)

Over the past few months, I've waited patiently for payments to arrive from the various digital download services that host my music. I knew that those sales, along with parsing my logs on my music site were going to help me getter a better handle on which specific songs, styles, etc folks find more enjoyable than others.

Keeping the dream alive is a song from my Sidetracks - Volume 3 CD and the data shows this is by far the most popular song I've released by a LONG shot which is really cool for the following reasons:

It helps me determine what specific styles I should focus more on.
It helps me gauge where my energy should be spent when drafting up new musical works.
But most importantly, this one song has a lot of personal meaning for me.

The song reflects one key attribute I hold high in myself and anyone I meet. That no matter how tough life can be; what struggles you may face; barriers to success; just all things that can interrupt your path towards achieving your goals, you have to just "Keep the dream alive" and move forward, and with lots of patience and hard work...In the end, you'll reach your full potential and look back and go "Wow! What a great life I've had".

I had a dream when I started doing software development

I wanted to use the right language for the best situation and write multiple components utilizing C/C++ where appropriate, as well as .NET related components where most appropriate, Language X where appropriate, etc yet have them all speaking with and consuming one another. The work I’ll be doing with the SharePoint Configuration Analyzer will help to achieve this dream

I had a dream when I started doing SharePoint work

SharePoint has this really rich OM. Is it documented all that well these days? No J Not really, but I saw the potential in it, and it has allowed me to create some really cool stuff that I think folks are really enjoying these days. The work I’ve done with the SharePoint Utility Suite has helped to achieve this dream.

I had a dream when I started with my musical endeavors

I just wish I had more time to put into it J But the results from my work happening at my music site are helping me to achieve that dream.

With patience, and my mind set on Keeping the dream alive, I know I’ll achieve all my goals and reach my dreams. Or as my pal Joe Dirt might say.."You just got a keep on keepin' it on. Lifes a garden man, dig it."

Just some thoughts I wanted to share.

- Keith