Imagine Think Create Share : Search

PauseCrawl vs PauseBackgroundActivity

carloshm — Mon, 07 Dec 2009 23:37:00 GMT

Hi,

Just a comment about my pausecrawl powershell script. Although it seems clear the pausecrawl command pauses an active crawl for the content source; I get last week some questions about it.

On the other hand, the PauseBackgroundActivity pauses all content-related activity for a Shared Services Provider's search service.

local vs global.

adios!

Follow SEO recommendations in your public SharePoint deployment

carloshm — Wed, 25 Nov 2009 16:47:00 GMT

Hi,

Today I received a request about SEO recommendations in your SharePoint deployment. Well you should care about it always, no matter if it is SharePoint or not (although it would be better :)) In case you came from other dimension, search is the technology behind everything.

Just to the spec, I wanted to share that information:

"I would start caring about the content (SEO is content quality), and resolve features SharePoint doesn't have. You have a good pointer at http://blog.mastykarz.nl/tag/seo/ , http://blogs.technet.com/vedant/archive/2009/06/24/search-engine-optimization-in-sharepoint.aspx or http://www.thesug.org/Blogs/sharepointseo/default.aspx

On the other hand I would work with tools as IIS SEO Toolkit to monitor the site."

Finally it is recommended to check third partner solutions and any generic SEO recommendations

Nice to Read
Search Engine Optimization Starter Guide
http://www.google.com/webmasters/docs/search-engine-optimization-starter-guide.pdf
How to Optimize SharePoint Server 2007 Web Content Management Sites for Search Engines
http://msdn.microsoft.com/en-us/library/cc721591.aspx

Bye!

How to: Programmatically read the ChangeLog in PowerShell

carloshm — Fri, 05 Jun 2009 15:33:00 GMT

Hi,

When I was working to get powershell scripts from MSDN samples to get/set information, I created too one for reading the changelog. It was a bit tricky to build, but I manage to port Querying for specific changes to PowerShell.

You can review the changelog API in the Microsoft SharePoint Developer Documentation Team Blog and MSDN (previous link):

We wanted to monitor which changes were been read by the crawl process and been able to filter specific changes.

## SharePoint Reference [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint") [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Administration") [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.Search.Administration") [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.Search") [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server")

function global:Get-ChangeLog($url, $minutes, $changetype)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$startTime = [System.DateTime]::Now.Subtract([System.TimeSpan]::FromMinutes($minutes));

$db_id = [Microsoft.SharePoint.Administration.SPContentDatabase].getmethod("get_Id");
$result_id = $db_id.Invoke($s.ContentDatabase, "instance,public", $null, $null, $null);

$db_CurrentChangeToken = [Microsoft.SharePoint.Administration.SPContentDatabase].getmethod("get_CurrentChangeToken");
$result_CurrentChangeToken = $db_CurrentChangeToken.Invoke($s.ContentDatabase, "instance,public", $null, $null, $null);

$startToken = New-Object Microsoft.SharePoint.SPChangeToken([Microsoft.SharePoint.SPChangeCollection+CollectionScope]::ContentDB, $result_id.ToString(), $startTime);

$changeQuery = New-Object Microsoft.SharePoint.SPChangeQuery($False, $False);
$changeQuery.User = $True;
$changeQuery.ContentType = $True;
$changeQuery.Add = $True;
$changeQuery.Delete = $True;
$changeQuery.Field = $True;
$changeQuery.File = $True;
$changeQuery.Folder = $True;
$changeQuery.Group = $True;
$changeQuery.GroupMembershipAdd = $True;
$changeQuery.GroupMembershipDelete = $True;
$changeQuery.Item = $True;
$changeQuery.List = $True;
$changeQuery.Move = $True;
$changeQuery.Rename = $True;
$changeQuery.Site = $True;
$changeQuery.SystemUpdate = $True;
$changeQuery.Update = $True;

$changeQuery.ChangeTokenStart = $startToken;
$changeQuery.ChangeTokenEnd = $result_CurrentChangeToken;

$db_GetChanges = [Microsoft.SharePoint.Administration.SPContentDatabase].getmethod("GetChanges", [Microsoft.SharePoint.SPChangeQuery]);
$result_GetChanges = $db_GetChanges.Invoke($s.ContentDatabase, "instance,public", $null, $changeQuery, $null);

write-Output $result_GetChanges | Where-Object { $_.ChangeType -eq $changetype };

$s.Dispose();
}

Get-ChangeLog -url http://your_site_url -minutes 120 -changetype Add

Take into account that the time will get changes based on UTC

Bye!

Sharepoint: supportability scenarios

carloshm — Wed, 01 Apr 2009 00:41:00 GMT

Today I posted some samples about Search and Powershell integration. I mention about the supportability of querying sharepoint tables. I thought about it and I wanted to clarify my point of view and what is public & published.

I have listed the main areas (I'm primarly focusing on Office12) where I have seen supportability questions. If you have any question, please call your support contact or access http://support.microsoft.com/

On the other hand, there are articles as SharePoint Database Access where it is recommended to not query database but instead use the object model.

Looking back to the scenario of my original comment if you are debugging with the SQL Profiler your SSP Database and at the same time running the powershell script or getting the crawl history, you will see something like:

SP:Starting Event

exec dbo.proc_MSS_GetCrawlHistory @ContentSourceID=NULL,@MaxRecords=NULL,@BeginTime=NULL,@EndTime=NULL,@CrawlStatus=NULL

SP:StmtCompleted

SELECT A.CrawlID,A.CrawlType,B.ContentSourceID,A.Status,A.StartTime,A.EndTime,CAST(A.SuccessCount as int) as SuccessCount,A.ErrorCount,A.WarningCount from MSSCrawlHistory as A inner join (select distinct CrawlID,ContentSourceID from MSSCrawlContent) as B on A.CrawlID = B.CrawlID WHERE A.ProjectID = 1 AND DATEDIFF([Day], A.StartTime, GETDATE()) <= 7 ORDER BY A.EndTime DESC

So you may think: "as these tables are not heavy used, a directly querying with some changes may have less locking, we should follow that", something that you may test only in you lab/dev environment.

SELECT A.CrawlID,A.CrawlType,B.ContentSourceID,A.Status,A.StartTime,A.EndTime,CAST(A.SuccessCount as int) as SuccessCount,A.ErrorCount,A.WarningCount from MSSCrawlHistory as A with(nolock) inner join (select distinct CrawlID,ContentSourceID from MSSCrawlContent with(nolock)) as B on A.CrawlID = B.CrawlID WHERE A.ProjectID = 1 AND DATEDIFF([Day], A.StartTime, GETDATE()) <= 7 ORDER BY A.EndTime DESC

Just thinking in the task, this may seems better, but think about the possibility that committed data may be a requirement.

Finally, should you TSQL your sharepoint database? I would say no, as:

You may affect overall response of your environment
Unless you have enough information through our open specification program you may conflict with you EULA acceptance.

What would be better, querying or using the object model?

Working with the object model gives you supportability for your code as you won't have breaking changes (backward compatibility) or you will be advised about deferred/obsolete code and have migration paths. something that using TSQL you won't have.

Namaste!

How to: Programmatically Manage the Crawl of a Content Source in PowerShell

carloshm — Tue, 31 Mar 2009 19:53:00 GMT

Hi,

If you have read my previous post you may think, why did you stop there? Well, that is what I thought too :), and started with this article from MSDN.

We wanted to start managing the crawling of our content source more programmatically, as we have seen that running several at the same time affects the overall process.

So running these powershell scripts as scheduled task and monitoring the status can improve in crawling.

You may improve them reusing the context and content sources and doing some pipeline

to be included in the overall file:

## SharePoint Reference
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Administration")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.Search.Administration")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.Search")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server")

To start an incremental crawl of the content source

function global:StartIncremental-Crawl($url, $csname)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$sc = new-Object Microsoft.Office.Server.Search.Administration.Content($c);

$cs = $sc.ContentSources[$csname];

$cs.StartIncrementalCrawl();

$s.Dispose();
}

StartIncremental-Crawl -url http://your_site_url -csname "your content source name"

To start a full crawl of the content source

function global:StartFull-Crawl($url, $csname)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$sc = new-Object Microsoft.Office.Server.Search.Administration.Content($c);

$cs = $sc.ContentSources[$csname];

$cs.StartFullCrawl();

$s.Dispose();
}

StartFull-Crawl -url http://your_site_url -csname "your content source name"

To pause a crawl in process

function global:Pause-Crawl($url, $csname)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$sc = new-Object Microsoft.Office.Server.Search.Administration.Content($c);

$cs = $sc.ContentSources[$csname];

$cs.PauseCrawl();

$s.Dispose();
}

Pause-Crawl -url http://your_site_url -csname "your content source name"

To resume a paused crawl

function global:Resume-Crawl($url, $csname)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$sc = new-Object Microsoft.Office.Server.Search.Administration.Content($c);

$cs = $sc.ContentSources[$csname];

$cs.ResumeCrawl();

$s.Dispose();
}

Resume-Crawl -url http://your_site_url -csname "your content source name"

To stop a crawl of the content source

function global:Stop-Crawl($url, $csname)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$sc = new-Object Microsoft.Office.Server.Search.Administration.Content($c);

$cs = $sc.ContentSources[$csname];

$cs.StopCrawl();

$s.Dispose();
}

Stop-Crawl -url http://your_site_url -csname "your content source name"

To check the crawl status values for a content source

function global:Get-CrawlStatus($url)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$sc = new-Object Microsoft.Office.Server.Search.Administration.Content($c);

Write-Output $sc.ContentSources;

$s.Dispose();
}

Get-CrawlStatus -url http://your_site_url | Format-Table -property CrawlStatus, CrawlStarted, CrawlCompleted

Enjoy!

How to: Programmatically Export the Crawl History to a CSV File in PowerShell

carloshm — Tue, 31 Mar 2009 18:51:00 GMT

Hi,

When I came across the article at MSDN How to: Programmatically Export the Crawl History to a CSV File I thought I would never create such a tool just for that specific feature, as you end up with additional requirements in order to create an admin tool.

But today I needed to get data from crawl history, and I didn't want to get them from SQL (remember it is not supported ;)), so I started to write down a simple powershell script to do it. And then I realized that for this atomic actions, indeed it is a great options!: you give admin people multiple commands that they can use/combine to monitor/get information about the environment (and yes many, many things more)

## SharePoint Reference
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Administration")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.Search.Administration")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.Search")
[void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server")

function global:Get-CrawlHistory($url)
{
trap [Exception] {
  write-error $("ERROR: " + $_.Exception.GetType().FullName);
  write-error $("ERROR: " + $_.Exception.Message);

  continue;
}

$s = new-Object Microsoft.SharePoint.SPSite($url);
$c = [Microsoft.Office.Server.Search.Administration.SearchContext]::GetContext($s);
$h = new-Object Microsoft.Office.Server.Search.Administration.CrawlHistory($c);

Write-OutPut $h.GetCrawlHistory();

$s.Dispose();
}

Then you can just execute: Get-CrawlHistory -url http://your_site_url/ | Export-Csv your_path_and_file_name

Then you can import to excel and make some charts.

In order to filter the information some useful columns should be denormalized: CrawlType, ContentSourceID, Status.

Cheers!

Information Design for MOSS

carloshm — Thu, 23 Oct 2008 05:02:00 GMT

Hi,

I was reading today Stephen P. Anderson's book and then I realized that Information Architecture is to Findability what Information Design is to Understandability.

We focus on the type of information we present to the user, in order to be completely relevant, but I do not really see focus on innovative/automatic ways to "communicate ideas, illustrate information or express relationships".

Microsoft, Yahoo or Google, as many start-ups work to improve this areas. There were shortcuts, alternate views, or browsing experiences and many more; but I would like to see Search Land to include this concept in depth.

Imagine a future search product that based on the user question shows, depending on the type of information, knowledge of the user, and relevant information; different type of representation in order to maximize the interaction with him/her.

Would it be possible to build a PoC based on (MOSS/FAST) to manage that?

How would it look like?

Federation webparts that shows information in a map
RSS Media exposed on video queries.
DeepZoom Image with size data blocks resizing
Audio clustering

Cheers!

Global Asset Picker - SharePoint

carloshm — Thu, 02 Oct 2008 02:16:00 GMT

Hi,

Did you get in trouble finding elements through OOB features with hundreds (or thousands) of hundreds of documents/pages? I think this is something everybody meet sooner or later in a SharePoint environment.

I have several opportunities to improve Asset Picker, but the way it is built (based on a Browse & Find instead of Search concept) it was really difficult to hack it and make it globally available.

At the end we got to a scenario with too many custom development to be able to accomplish this solution.

http://msdn.microsoft.com/en-us/library/ms543215.aspx
http://sharepointbergen.blogspot.com/2007/09/modifying-asset-picker-dialog-in.html
http://sharepointkb.wordpress.com/2008/09/10/asset-picker-browse-field-control/
http://blog.mastykarz.nl/adding-new-locations-to-the-look-in-dialog/

So, did I find any other suggestion to improve authoring/editing experience?

Imagine

Been able to Search for content inside MOSS/WSS without moving through several windows, and using search built-in capabilities, just right where you are editing.

Think

A Feature deployed through a Solution package, that contains a delegate control begin included in the Additional Page Header. This control manage a modal css window, that allows the user to search and filter results from a form control.

Create

Options for the "client engine" would be:

Personally, I would choose JQuery (even before it was going to be included in VS :))

Adios!

SearchMonkey, OpenCalais and MOSS

carloshm — Thu, 21 Aug 2008 01:15:00 GMT

Hi all,

This is something I really wanted to start including in my public projects (and I will do), as certainly Search will need to improve in coming years.

just check above examples, and you will see.

Well this post was just to show you the example of OpenCalais integration with MOSS: http://www.opencalais.com/MOSS

Open Calais allows you to add semantic metadata to content already in your environment. A very nice way to improve editors usability, as you can see in tagaroo wordpress plugin.

How does it work?

It installs (through solutions+features):

Custom List
Custom WebPart
Events Receivers to Documents and Page Library (ItemDeleting, ItemCheckedIn, ItemAdded and ItemUpdated events)

and when an item is created/updated and so on, the OpenCalais Webservice is called, with the information from the file (via WebRequest) and related metadata is stored in the Custom List.The custom webpart gets the current item, and it searchs for the metadata into the Custom List, so calls to the webservice are made only in editing state.

Finally as the rendering of the custom webpart matches UserAgent of Search Engine Bots, it will only display that rich information to bots matching those rules.

By default this rule has Yahoo bot, in order to be relevant to SearchMonkey initiative with microformats.

Adios!