Jie Li's GeekWorld : SharePoint

October 2009 Cumulative Update Packages for SharePoint Server 2007 and Windows SharePoint Services 3.0 are published

Jie Li — Thu, 29 Oct 2009 03:07:00 GMT

The server-packages of October 2009 Cumulative Update for Microsoft Office SharePoint Server 2007 and Windows SharePoint Services 3.0 are ready for download. October 2009 Cumulative Updates introduce more rules on Pre-Upgrade Checker, which can help customers to prepare the upgrade of their SharePoint farm to SharePoint 2010.

Download Information

Windows SharePoint Services 3.0 October 2009 cumulative update package
http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=974989

Office SharePoint Server 2007 October 2009 cumulative update package
http://support.microsoft.com/hotfix/KBHotfix.aspx?kbnum=974988

Detail Description

Description of the Windows SharePoint Services 3.0 October 2009 cumulative update package
http://support.microsoft.com/kb/974989 (link may not be live yet)

Description of the Office SharePoint Server 2007 October 2009 cumulative update package
http://support.microsoft.com/kb/974988 (link may not be live yet)

Installation Recommendation for a fresh SharePoint Server

To keep all files in a SharePoint installation up-to-date, the following sequence is recommended.

Please note: Start from April 2009 Cumulative Update, the packages will no longer install on a farm without a service pack installed. You must have installed either Service Pack 1 (SP1) or SP2 prior to the installation of the cumulative updates.

After applying the preceding updates, run the SharePoint Products and Technologies Configuration Wizard or “psconfig -cmd upgrade -inplace b2b -wait” in command line. This needs to be done on every server in the farm with SharePoint installed.

The version of content databases should be 12.0.6520.5000 after successfully applying these updates.

You can also refer to April Cumulative Update post for deployment guides, slipstream how-to links and FAQs.

Jie Li

Technical Product Manager, SharePoint

Index and Search PDF Files in SharePoint Server 2010

Jie Li — Tue, 20 Oct 2009 16:38:00 GMT

Like Office SharePoint Server 2007, there’s no OOTB PDF iFilter in SharePoint Server 2010. If you add PDF as a file type for SharePoint Search, you will get the following result:

You can see that only the file attributes are indexed.

You need to install a x64 PDF iFilter for this. There’re three PDF iFilter on market, Adobe, Foxit, and TET. You can refer to my earlier post for comparison. Since the registry name is changed in 2010, you may need to manually modify it to make the iFilters registered. Foxit recently updated their installer to reflect this change.

http://www.foxitsoftware.com/pdf/ifilter/

Quote from Foxit PDF iFilter change log:

Version Number: 1.0.0.3213

* Fixes a crash issue that is caused by embedded fonts.

* Adds the following registry settings in the installation program:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Setup\Filters\.pdf]

"Extension"=".pdf"

"FileTypeBucket"=dword:00000001

"MimeTypes"="application/pdf"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Setup\ContentIndexCommon\Filters\Extension\.pdf]

@="{987f8d1a-26e6-4554-b007-6b20e2680632}"

So run the installer, and then restart SharePoint Server Search 14 service. This service name is subject to change when RTM, but you can easily get the idea.

Recrawl the files.

It worked. Please note the installer will not get you PDF icon file, you need to follow the steps here http://www.foxitsoftware.com/pdf/ifilter/installation.html to download icon file and modify DOCICON.XML.

This also applies to Search Server 2010. FAST Search index PDF files OOTB, so you don’t need to go with these steps.

Jie Li

Technical Product Manager, SharePoint

SharePoint 2010 - What does that mean for IT Pros?

Jie Li — Mon, 19 Oct 2009 16:00:00 GMT

So, it’s time to lift the curtain. For people at SharePoint Conference 2009, feel free to meet me on the site. For those IT administrators who can’t make it to Vegas, here’s something you should know about SharePoint 2010. Of course, you should check out http://go.microsoft.com/fwlink/?LinkId=165421 (link may not be activated yet) for IT Pro Evaluation Guide. Shane Young did a good job on this guide and he definitely should be given a credit. :)

If you are a new Administrator for SharePoint, you can try our free screencast series Getting Started with SharePoint 2010 on TechNet. It is done by 5 SharePoint MVPs, covers from level 50~300, 9 modules.

Installation and Upgrade Requirements

Requirements! Keep in mind that SharePoint 2010 is 64bit only, so it’s time to upgrade your server operating systems to x64. 64 bit architecture brings more memory support, not only the total amount of memory can be supported, but also how much memory that a single process can address. You can refer to the following two articles for 64bit benefit details and plan for upgrade.

Advantages of 64-bit hardware and software (Office SharePoint Server 2007)

Migrate an existing server farm to a 64-bit environment (Office SharePoint Server 2007)

Note: Nearly any server hardware you can find on the market after year 2005 is 64 bit capable.

Hardware requirement

I don’t want to go through the whole piece of hardware requirement, but just to point out some important things.

For production environment, the memory is recommended to be 8GB or above. SharePoint 2010 introduces a lot of new service applications, things like Visio service, Access service all consume memory. You can still run SharePoint with 4GB memory, or maybe even less, but for performance consideration you should give it more. After all, 16GB DDR2 FB-DIMM only costs ~400 dollars today (10/19/2009) on Newegg.

Always use fast disk arrays for your SQL Server. Don’t install SQL Server on the same machine with SharePoint unless it’s not for production.

Software requirement

Operating System: Windows Server 2008 or Windows Server 2008 R2 (Patches will be installed with pre-req installer)

Developers can install SharePoint 2010 on Windows 7, but this should not be used in production.

Database: SQL Server 2005 SP3 with CU3 or SQL Server 2008 SP1 with CU2 or SQL Server 2008 R2 (still in beta but will be supported when RTM)

The following figure shows what will happen if you don’t have the correct CU applied on SQL Server.

.Net Framework version: 3.51 (.Net Framework 4 is not supported currently)

If you want to upgrade from SharePoint Server 2007, then you need to make sure at least service pack 2 is applied on the farm. Otherwise it will be blocked. The following figure is to show the logs when Upgrade-SPContentDatabase cmdlet is run on a database older than SP2 (12.0.0.6421) attached to the 2010 server farm. The upgrade is blocked right away, because database schema did not meet the minimum requirement.

Client Browser Support

We made a great progress to replace ActiveX controls with AJAX and Silverlight, and made the pages compatible with the web standards, to support a broad selection of the browsers cross platforms.

Support Level	Browser	Jie’s Comments
Level 1	Internet Explorer 7 & 8, 32 bit	Even you are running a 64 bit operating system, your Internet Explorer is 32 bit by default.
	Firefox 3.x on Windows, 32bit	Yeah, Firefox jumps their versions quickly, we will have new support claims when they have new version 4 after testing work is done.
Level 2	Internet Explorer 7 & 8, 64 bit	Most of the time you won’t have this scenario.
	Firefox 3.x on non-Windows	Linux, Mac…
	Safari	Good for Mac and iPhone users
Level 3	Internet Explorer 6, Chrome, Opera…	Some of the browsers might work, but since we did not test them, they are not supported.

Note: Mobile Browsers are supported with Mobile View.

Details of system requirements can be found here:

http://technet.microsoft.com/en-us/library/cc262485(office.14).aspx (Link may not activated yet)

Improvements for IT Pros

Scale with Governance

Investment on SharePoint 2010 focused on better scalability, with less chaos. With the new architecture of Service Applications and Protocols, SharePoint 2010 can scale from a single server for small business to multiple server farm or even multi tenancy hosting deployment in large enterprise. With the benefit of x64 hardware, we tested up to 50 million items in a single document library, 100 million items in a single search index. Unlike SharePoint 2007, now you can have multiple indexer to crawl your content at the same time. With FAST for SharePoint and FAST ESP, your enterprise search solution can also scale up to billions of items easily. Resource Throttling can help when end users make some “bad” moves on the server. For example, if a user want to sort on a big document library with more than 5000 items in a view, he will be stopped and informed that he can use other ways to achieve his goal, without slow down the whole server farm.

Deployment Flexibility

If you have installation experience from SharePoint 2003 and 2007, you will be surprised on how many steps you need to go to get a 2010 server farm up and running. Prerequisites can be downloaded and installed automatically, Configuration Wizard reduces the pain to setup every service applications. We also support scriptable deployment with Windows Powershell, automation guys would love it. Virtualization is supported everywhere, it helps on scalability, management, high availability and disaster recovery.

Intranet, Internet, Extranet…SharePoint 2007 has been proven to handle all these deployment scenarios. Now SharePoint Server 2010 also support multi-tenancy. You can even setup a SharePoint hosting solution in your own organization.

Keep in mind, no matter when, we always recommend you to use complete server farm installation mode. And it would be better to avoid installation on a Domain Controller in production unless you are running SBS. SQL Server should always be separated on a different box too to get much better performance.

IT Productivity

Windows PowerShell, Ribbon UI, Health Monitoring… There’re a lot of tools and features to make SharePoint Administrators exciting.

People already started to use Windows PowerShell for 2003 and 2007 management. However, it is more like to deal with object models. For me, it’s more like a developer job, not that easy for IT Professionals. In 2010 we introduced more than 500 cmdlets for SharePoint, from site creation to service application management. Although sometimes you still need to use STSADM, most of the time, Windows PowerShell is the way to go. With the powerful script environment, some large operations used to take hours to finish now can be done in minutes.

For example, the following script can get the sizes of all sites owned by a user quickly.

Get-SPSite -Limit ALL -Filter {$_.Owner -eq "contoso\sp_admin"} | Select URL, @{Name="Storage"; Expression={"{0:N2} GB" -f ($_.Usage.Storage/1000000)}

A health and monitoring center is offered directly in Central Administration. It is also called SharePoint Best Practice Analyzer (SBPA). SBPA is rule based, it use timer jobs to run the rules and detect problems in the server farm. It can also has Actions implemented, to fix the problems just by several clicks. When a problem is found, it is displayed in a health report page, administrators can also set alerts with emails or SMS to get informed. This rules system can also be extended with your own custom rules.

Ribbon UI is introduced in SharePoint 2010 everywhere. With a completely redesigned Central Administration site, It greatly reduced the time and clicks to manage services, web applications and site collections.

Visual Upgrade can help Administrators to switch between 2007 look and 2010 look. If the new one does not work, switch it back to fix the problem.

Conclusion

That’s already a lot of things, right? Especially Windows PowerShell support, that is a huge change in IT Pro’s life. At first you may find it’s a little bit tough, but after you know the basic concept of it, you will benefit from it and never want to turn back.

There’ll be a lot of blog posts coming after, so please stay tuned. Meanwhile, remember to download the bits when it is public on Nov. Check out IT Pro Evaluation Guide on TechNet for complete explanation of all new features for IT Pro.

Jie Li

Technical Product Manager, SharePoint

“Fix” SharePoint 302 redirect problem by IIS7 and URL Rewrite

Jie Li — Tue, 30 Jun 2009 23:00:00 GMT

Recently in my spare time I’m helping my friends to get their internet facing sharepoint site up and running. Since this is for the internet, the first thing they need to consider is SEO. So we have a well known problem now: SharePoint use 302 temp redirect instead of 301. If you do a search for “sharepoint 302” you will see a lot of articles talking about the problem.

Here’s the default SharePoint Site. The request was temporarily redirected (302) to Pages/default.aspx. Search engine bots don’t like it, they like 301. So this is BAD.

How to solve it? Oh, I’m a IT Pro person, I don’t want to deal with a custom redirect HttpModule – god knows what will happen if those custom code mess up my sites! So any other options?

I’m lucky because I installed SharePoint on Windows Server 2008, so I can use IIS7 features. I downloaded URL Rewrite module from http://www.iis.net/extensions/URLRewrite, installed it, and started to configure the redirect.

Choose your site in IIS Manager, click URL Rewrite, and create a new blank rule.

Use Regular Expressions to match ^$ (which means “empty”). Set Action Type to Redirect, and add the redirect URL (by default should be Pages/default.aspx), set redirect type to Permanent (301). You are all set!

Now, clear browser cache and revisit the site:

It is 301 now! Pretty easy, isn’t it?

URL Rewrite module is great. If you are a regex guru you can also create more complex rules to make everything fit for your site.

PDF iFilter Battle! FoxIT vs.. Adobe, 64bit version

Jie Li — Wed, 10 Dec 2008 07:46:28 GMT

After so long a time Adobe finally released its 64bit version of PDF iFilter!

http://www.adobe.com/support/downloads/detail.jsp?ftpID=4025

“In response to customer requests, Adobe is releasing Adobe PDF iFilter 9 for 64-bit platforms, which will allow searching PDF files on Microsoft® Windows® 64-bit platforms for applications such as Microsoft Office SharePoint Server 2007, Microsoft Exchange Server 2007, and Microsoft SQL Server 2005.”

But, what about performance? How does it compare with FoxIT 64bit PDF iFilter?

My friend Deb Haldar did a performance test last year for their 32bit iFilters. You can find the result here: FOXIT vs.. Adobe PDF IFilter [ 32-bit only ]. Let’s say, FoxIT 32bit PDF iFilter is more than 4 times faster than the Adobe one.

Will the story change in 64bit age?

I picked about two sets of PDF files. Set I contains ~1000 PDF files, 1.7 GB in total. Set 2 contians ~2600 files, 2.4G in total. Language is mixed by 30% Chinese, 70% US English. The hardware spec is a two-way dual core XEON at 3.4GHz, 4G Ram. SharePoint was patched with October CU. Here’s the result.

	File Set	File Number	Total File Size(MB)	Avg File Size(MB)	Crawl Time(m:s)	Crawl Time(s)	File Per Second	Success	Error
FoxIT	I	1041	1751	1.68	6:02	362	2.88	1064	0
Adobe	I	1041	1751	1.68	30:03	1803	0.58	1063	1
FoxIT	II	2676	2406	0.90	7:46	466	5.74	2759	0
Adobe	II	2676	2406	0.90	40:58	2458	1.09	2757	2

On average, FoxIT x64 PDF ifilter is still ~5 times faster than the Adobe one. But FoxIT charges 330 USD for a 2 core machine, while Adobe PDF iFilter is free. So if PDF indexing is the key to your business, go with FoxIT to get much better performance. If not, you may play with Adobe PDF iFilter to furfill some simple and basic request.

Search Suggestions in IE8 with SharePoint/Search Server

Jie Li — Fri, 05 Sep 2008 03:23:13 GMT

IE8 Beta 2 has been there for a while. Although it is not supported to use together with SharePoint products yet (there’s no chance to support a beta product), you can still try it out. There’re couple of new features introduced like WebSlices, Accelerators and Search Suggestions.

Search Suggestions! Isn’t it cool to make your intranet SharePoint portal to be a Search Provider and have this lovely suggestion feature?

However, by default you can make SharePoint a Search Provider, but no way to add a suggestion feature.

So here it comes – an update to “Search As Your Type(SAYT)” codeplex project, with Search Suggestion working in IE8! And I also included a small green “S” logo icon file for that, all free:)

Since it needs time to update codeplex project, I’ll put something here.

1. Install SAYT on your SharePoint Server/Search Server as instructed. Do some test searches, to make sure it works.

2. Download new update from here:

http://cid-8007edf5c56fc334.skydrive.live.com/self.aspx/Public/SharePointSearchIE8.zip

3. Extract this zip file, copy all files to the directory you put GetInfo.aspx in first step, and overwrite it.

4. Modify ssprovider.xml as needed. Replace SharePointSearchCenter and SAYTUrl with your own ones.

5. Use IE8 to navigate to add.html, and add search provider.

6. Choose the green “S” provider and try it out!

You can add a sample provider on http://www.mssearch.cn:8099/add.aspx, and try type in “search server” to see the result.

Disclaimer: This code is not support by Microsoft, if you have problems, leave your comment here. SAYT Codeplex project will be updated later to include this feature.

IE 8.0 Beta 2 with SharePoint, what’s the story?

Jie Li — Thu, 28 Aug 2008 09:47:50 GMT

Today Internet Explorer 8.0 Beta 2 was released. Well, if you tried Beta 1 with Office SharePoint Server 2007 or Windows SharePoint Services 3.0, you may notice that some of the features like content editor webpart does not function properly, even in IE7 mode.

But IE7 mode should be the same as original IE7, not to cause new problems. After several months’ bugfix and feature improvement, that problem now is gone.

There’re also some changes between IE8 Beta 1 and Beta 2. For example, there’s no IE7 mode button anymore, it is changed to a “Compatibility View” button next to refresh button. And that is not turned on by default.

So what will happen if I use IE8 Beta 2 to access a SharePoint site?

That depends on your master page of the site. IE8 Beta 2 will check DOCTYPE and meta tag to determine wether to use Compatibility View or not. If there is no DOCTYPE indicated in the page, IE8 Beta 2 will use this mode by default, and you will not see the button at all. This is what happened for default master pages come with SharePoint Server. All things should work by default.

It is quite common that you are using your own modified themes in master pages. So what if DOCTYPE is there in the pages? Will it cause any problem?

Maybe. But if you want to ensure everything works, I would recommend to add a meta tag in your master pages:

When IE8 Beta 2 reads this line, it will automatically switch to Compatibility View and make everything right. This can be seen as a temporary solution. IE8 native mode will be supported in future SharePoint v3 service pack.

SharePoint Search Admin 0.80b – Now with Keywords and Bestbets Backup and Restore Feature

Jie Li — Mon, 02 Jun 2008 03:17:00 GMT

It has been a long time since last update. In this version of SharePoint Search Admin, I added a feature by very popular request: Keywords and Bestbets.

I mark it a “Beta” because there’re still something I didn’t finish due to time limitation, so I disabled “Add Keyword” and “Add Bestbet” button. They are not the focus because using SSA to add them cannot provide a better experience than default SharePoint UI.

Let’s take a look at the interface.

Still ugly :) I promise I’ll improve it.

So you can remove individual Keyword, remove all keywords by a single click, backup and restore whole keywords/bestbets collection in a few seconds with this new feature.

A sample XML generated by SSA Keywords backup.

What may happen when I crawl MILLIONS of files in MOSS/MSS? Part II - Why I need X64 instead of X86?

Jie Li — Sun, 13 Apr 2008 20:44:35 GMT

In last post of this series we talked about the crawl time and and CPU usage. This time we will talk about process memory usage and x86/x64 issue.

Many customers asked me questions about x86/x64 comparison. Most of them consider x64 would be a benefit, but they don't know what kind of benefit it really brings.

So now I tell you why it's needed.

As we know, SharePoint/search server has three layers. Each layer can only be single architecture.

Layer 1: Web Front Server (WFE)

WFE is needed to host the web site, process different user events. This layer, of course can be and should be x64, unless you have some pretty old webpart which still need x86. At this layer, the most memory consummation comes from IIS(w3wp.exe). For IIS in X86, it can only use at about 1.1-1.2G memory. If you hit this barrier, the process may just hang there. This situation happens when very big number of request lasted for a long time(several hours or days, it depends on how many users are accessing the site at the same time). You can have multiple WFE in one server farm.

It's quite important to have IIS recycle automatically, and sometimes you even need to manually recycle it.

Layer 2: Query Server.

Query Server hosts query engine. It continually receive index propaganda from index server. When users make a query, it will be sent to query engine. Query engine will check SQL Server for document properties, and check index for content chunk. So the disk performance is important for query server when query load is high. Other workload, like query time security trimming by custom security trimmer, is also done by Query Servers.

You can have multiple Query Server per farm. And Query Servers, should be x64 if you have the hardware.

Layer 3: Index Server.

Index Server is the spider. Because you can only have one Index Server currently in SharePoint 2007/Search Server 2008, you need to take great care of it. DO NOT put ANY other applications on it, DO NOT share the box with SQL. If you do, you will quickly be hit by the bad performance.

Yes, Index Server should be x64. The problem is, sometimes we cannot make it x64. For example, we have a 32bit ifilter which does not have any x64 implementation but is very important to customer business, or we only have 32bit protocol handler like Lotus Notes PH... In these situations, you can only have a 32bit Indexer.

You may suffer from the same limitation when using 32bit Indexer, especially in Lotus Notes. When you continues crawl too many Notes docs into SharePoint, index engine may hang there for several hours because of memory limit. I suggest, if you happen to come across such problem, make a simple application to monitor and automatically restart search service after one DB is finished. This can manually recycle the memory used by the engine.

In the future, nearly everything can be made x64. So stay tuned.

Database: SQL Server

No matter querying or indexing, the backend database is always a main contributor to the whole performance. So make it independent, make it faster, and even make it cluster. This will help with overall performance. x64 is the best choice for DB.

Index FTP content with SharePoint 2007/Search Server 2008

Jie Li — Thu, 10 Apr 2008 10:03:00 GMT

Question: Can SharePoint 2007/Search Server 2008 index and search ftp?

Answer: OOTB you cannot. But we have a FTP protocol handler in SharePoint 2001 ResKit, and that one can be used on SharePoint 2007 or Search Server 2008.

Important Notice: This approach is NOT supported by Microsoft. It is just for your test purpose only, and should not be used in an important production envoriment.

Some restrictions may apply: x86 and anonymous ftp only.

Here's the steps.

1. Get a copy of ftpph.dll. It is in SharePoint 2001 ResKit.

2. Copy it to c:\Program Files\Microsoft Office Servers\12.0\Bin

3. Register this dll. Open command prompt, navigate to c:\Program Files\Microsoft Office Servers\12.0\Bin, run "regsvr32 ftpph.dll".

4. Open Regedit, navigate to HKLM\SOFTWARE\Microsoft\Office Server\12.0\Search\Setup\Protocol Handlers

5. Insert a new string item: "ftp". The value is "FtpPH.SearchProtocol.1".

6. In command prompt, restart search service by "net stop osearch" and "net start osearch"

7. Download SharePoint Search Admin at http://www.codeplex.com/SearchAdmin , and then add a new custom content source for your ftp site.

8. Start full crawl, and you will see the crawl log. Job done!

Smart Search for SharePoint v1.2

Jie Li — Fri, 07 Mar 2008 21:00:06 GMT

Many many thanks to Shawn Feldman!

Quote from him:

"I was having a few issues getting smart search installed and granting it the right permissions to access the db (I’m running a multi account environment) so I put it in a feature, Gac’d it, and added SPSecurity.RunWithElevated for whenever it makes a database connection. Here’s the code if you’re interested."

It's my fault. I know this problem when I was working on a Internet(anonymous) facing SharePoint site. The problem of SmartSearch v1.1 is it will use the current user to access db, of course, cannot be successful when using IUSR_XXXX accounts. RunWithElevated is the definitely the solution but I don't have time to do as I'm alwasys travelling around.

So here you are:

Smart Search for SharePoint ver 1.2 - Mar 8 2008

I think this is the reason why we made these open source tools - everybody can contribute to it, and make our work better.

Build Custom Federated Search Connector in Microsoft Search Server (and SharePoint) - Solve Problems and Extend Your Ideas

Jie Li — Fri, 29 Feb 2008 14:17:23 GMT

I assume the read of this article understand what is federated search. So we already know that in order to use Federated Search webpart in Search Server, you need to provide a RSS feed to it, which can also be called "OpenSearch" stuff.

But, not every application you search will return this kind of RSS/ATOM feed. For example, Google, Baidu and many other web sites. So how can you federate search results from this kind of web sites?

http://msdn2.microsoft.com/en-us/library/bb931083.aspx

Scenario 2: Connecting to an External Search Site That Returns Results in HTML Format

Scenario background: The site is configured to use Anonymous access.

Possible solution: Use a Web application outside of the context of a SharePoint site, which contains a lightweight ASPX page that does the following:

Submits a search request to the site by using the search terms passed in the initial request URL.

Converts the results in the HTML response received from the external search site to RSS format.

Returns the RSS XML in the response to the search server.

In this scenario, the federated connector’s Web application could be located on a remote server; however, a simpler solution is to create the Web application within the _layouts folder for the SharePoint site. For more information about creating this type of Web application, see How to: Modify Configuration Settings for an Application to Coexist with Windows SharePoint Services.

In a variation for this federated connector solution, you can add support for multiple external search sites by modifying the ASPX page to include details for more than one site within a case statement. The query template specified for these locations could then include a custom parameter that specifies which site in the case statement receives the federated query. Another variation is to combine the results for multiple external search providers, incorporating logic to order the results based on relevance.

Well, there're already some people who did a nice job, for example Andrew Woodward:

http://www.21apps.com/2008/01/search-server-2008-federated-sites-that.html

I would go a little further on this. Here I take Baidu as an example. Baidu is the biggest Internet search engine in China. (Google China? God knows where them are. Baidu introduced many interesting applications that Chinese users love to use. But Google China, is only famous for stealing the input method dictionary of another major Internet company SOHU, and then made its own Pinyin input method. After this was exposed to the public, they did a not so honest "apologize" and said that were two interns who did it. Perfect, later this became a popular phase in China, if anyone did evil things but was discovered by the public, he would say it's intern's or temporary employees' fault. Well, what a shame on this "not to be evil" company. - little off topic) .

Baidu.com does not return any RSS feed. What's more, it is using GB2312 encoding method to show the results. So if you directly use regex to capture something in Baidu, you will get some squares which do not make sense.

And there're some limitations in asp.net Request.QueryString method. It cannot correctly process Gb2312 encoding. So the Page Load Method must be changed to the following code:

protected void Page_Load(object sender, EventArgs e)
    {
        if (Request.QueryString["q"]!= null)
        {
            query = Request.Url.Query.ToString();
            query = query.Remove(0,3);
        }
    }

In this way, a query string will be kept so you can process it with Encode and Decode. If you use QueryString, you will get a stupid behavior that it incorrectly use Decode method in a wrong encoding charset...The result is a disater. Stupid, stupid, stupid. I want to slap the guy who wrote this method. Does he know there're not only English in this world?

For example, my nickname opal, in Chinese is 猫眼石. If queried from IE, it will be encoded using UTF-8. But Baidu can only consume GB-2312.

In UTF-8, 猫眼石 is %E7%8C%AB%E7%9C%BC%E7%9F%B3.

In GB2312, 猫眼石 is %C3%A8%D1%DB%CA%AF.

It's quite different. If you want do a search for %E7%8C%AB%E7%9C%BC%E7%9F%B3, and it is treaten as a GB2312 string, it will become 4.5 Chinese charactors. and none of them will make sense.

Okay, compain less, do more. So then we need to decode query string.

 private string getRssItemXml(string query)
    {
        //first you must decode it as UTF8. Because when IE access a utf-8 based website, it will pass the corresponding encoded strings.
        //Of course, you can modify web.config to make this application using Gb2312, but that doesn't make sense.
        query = HttpUtility.UrlDecode(query, Encoding.UTF8);
        //Then we need do encode it to gb2312. Baidu can only consume that.
        query = HttpUtility.UrlEncode(query, Encoding.GetEncoding("gb2312"));
        string url = string.Format("http://www.baidu.com/s?wd={0}", query);

        WebClient client = new WebClient();
        byte[] byteData = client.DownloadData(url);
        //Returned results are also in GB2312, so you have to rebuild it.
        string strData = Encoding.GetEncoding("gb2312").GetString(byteData);
        Regex searchPattern = new Regex("\\)\" href=\"(?<link>.*?)\" target=\"_blank\"><font size=\"3\">(?<title>.*?)</font></a><br>(?<desc>.*?)<br>");
        StringBuilder sb = new StringBuilder();

        foreach (Match m in searchPattern.Matches(strData))
        {
            sb.AppendFormat("<item><title><![CDATA[{0}]]></title><link><![CDATA[{1}]]></link><description><![CDATA[{2}]]></description></item>",m.Groups["title"].Value,m.Groups["link"].Value, m.Groups["desc"].Value);
        }

        return sb.ToString();
    }

So then put this aspx file to a website, have your federated search webpart point to it, like http://www.abc.com/Baidu.aspx?q={searchTerms}, and then you can get Baidu federated search in Microsoft Search Server 2008.

I put part of my work here:

http://cid-8007edf5c56fc334.skydrive.live.com/self.aspx/Microsoft%20Search%20Server/CaptureWeb.rar

It contains:

Baidu Federated Search Web Service
Baidu News Federated Search Web Service
iCiba (English-Chinese Dictionary) Federated Search Web Service
Dictionary.com Federated Search Web Service

Yes! You can put dictionaries on your federated search web page so if anybody want to search a word, he will get the meaning immediately! You can also have some triggers to make this happen only with numbers or charactors, etc.

Okay...Here comes Smart Search bug fix and installation guide...

Jie Li — Tue, 26 Feb 2008 19:53:37 GMT

It's my fault to ignore it for so long a time. It is not written by me originally, but by a colleague of mine, Gang Chen. But he asked me to do a favor to create a project on codeplex, so now it is maintained by me.

http://www.codeplex.com/smartsearch

And finally, I fixed the foolish bug and replaced the Chinese charactors to English words. I spent a whole afternoon to get it installed on my WSS+Search Server Express box. Everybody can try it here.

http://www.mssearch.cn:5000/Search/results.aspx?k=lotus

A installation guide is also there. I think a experienced sharepoint user can install it within 10~30 minutes.

It is quite interesting that the installation guide is also the 11111th release on Codeplex. :)

The reason I spent so much time is, WSS is using Windows Internal Database. In this code, we need to create a seperate sql table in content db, so it failed of course. To workaround this problem, you need to manually modify the code and point it to a Sql Server(Express) instance.

Real World Lotus Notes Index Result in SharePoint 2007

Jie Li — Wed, 20 Feb 2008 17:47:51 GMT

Crawled Notes data: 1240 Gbytes

Crawled Notes Items: 2.7 Million

SharePoint Index: 70 Gbytes

SharePoint Search DB: 120 Gbytes

Data vs Index = 124:7 = 18:1, which is a ratio of 6%.

Index vs Search DB = 7: 12. Search DB is 160% of Index.

In total, (Search DB+Index)/Data=16%

Remember, this ratio varies in different environment, depends on how you use Notes Applications, how many attachments are there, and many other factors. I put it here only as a reference. Maybe update later.

Next steps of SharePoint Search Enhancement, and...

Jie Li — Sun, 17 Feb 2008 10:53:00 GMT

I was collecting feedbacks from many people these days about the enhancement we did for SharePoint Search in China. Good feedback, bad feedback, that's okay. I just put the whole list here:

Search As You Type, SAYT: The live search feel for your SharePoint search box.

TODO: Check if there's any chance we can make use of AJAX.NET.

http://www.codeplex.com/SearchAsYouType

Predefined Search: Some kind of Saved Search. Just by a single click, and your query will be remembered and can be share to the public.

TODO: Currently none. If you have ideas, contribute to the open source project!

http://www.codeplex.com/MOSSPredefinedSearch

Smart Search: 1. Display top ten hot search keyword. 2. Display relevant search keywords.

TODO: Bugfix, and a pure English version.

http://www.codeplex.com/smartsearch

SharePoint Search Admin: A GUI based tool which can do SharePoint search administration, much better and easy to use than SharePoint Search settings page itself.

TODO: Bugfix, add more tricks so that can help service people to deliver things on time.

http://www.codeplex.com/searchadmin

Chinese "did you mean" webpart: Deliver a did you mean feature for Simplified Chinese.

TODO: None

http://www.codeplex.com/cndidyoumean

Some people have a concern that these codeplex tools are not supported by Microsoft. That's true. But I must emphasize here, that codeplex thing is OPEN SOURCE. Yes, we are talking about Open Source in Microsoft. Personally I am some kind of open source lover, I think if you really want to do some thing like SOA or SaaS, open source development model would be a pretty good alternative to a complex API set. The pain of API set is the blackbox development. Product team people may not understand the business needs as deeply as we field people do. So, it is quite possible that they do something unreasonable with the product. The gain of API set is also the blackbox model. It simplified the process of development, lowered the entry barrier of newbie developers. Of course, open source may lead to bad documentation and unreadable code, also complexity of the code you write to extend it. So, a balance between opensource and api would be better for service development. So far, Firefox has been proven to be a good example.

A little off topic. But things really are different when you talk to IT pro. It depends on what is the model: a product only solution or a product+service solution? For platforms like SharePoint, a product+service solution is definitely the wiser choice. Because in a more successful implementation, nearly everything needed to be customized for the organization. In this way, the codeplex thing would be a great help.

It may be hard to change people's thought immediately, but time would prove this.