Jie Li's GeekWorld : OpenSearch

Build Custom Federated Search Connector in Microsoft Search Server (and SharePoint) - Solve Problems and Extend Your Ideas

Jie Li — Fri, 29 Feb 2008 14:17:23 GMT

I assume the read of this article understand what is federated search. So we already know that in order to use Federated Search webpart in Search Server, you need to provide a RSS feed to it, which can also be called "OpenSearch" stuff.

But, not every application you search will return this kind of RSS/ATOM feed. For example, Google, Baidu and many other web sites. So how can you federate search results from this kind of web sites?

http://msdn2.microsoft.com/en-us/library/bb931083.aspx

Scenario 2: Connecting to an External Search Site That Returns Results in HTML Format

Scenario background: The site is configured to use Anonymous access.

Possible solution: Use a Web application outside of the context of a SharePoint site, which contains a lightweight ASPX page that does the following:

Submits a search request to the site by using the search terms passed in the initial request URL.

Converts the results in the HTML response received from the external search site to RSS format.

Returns the RSS XML in the response to the search server.

In this scenario, the federated connector’s Web application could be located on a remote server; however, a simpler solution is to create the Web application within the _layouts folder for the SharePoint site. For more information about creating this type of Web application, see How to: Modify Configuration Settings for an Application to Coexist with Windows SharePoint Services.

In a variation for this federated connector solution, you can add support for multiple external search sites by modifying the ASPX page to include details for more than one site within a case statement. The query template specified for these locations could then include a custom parameter that specifies which site in the case statement receives the federated query. Another variation is to combine the results for multiple external search providers, incorporating logic to order the results based on relevance.

Well, there're already some people who did a nice job, for example Andrew Woodward:

http://www.21apps.com/2008/01/search-server-2008-federated-sites-that.html

I would go a little further on this. Here I take Baidu as an example. Baidu is the biggest Internet search engine in China. (Google China? God knows where them are. Baidu introduced many interesting applications that Chinese users love to use. But Google China, is only famous for stealing the input method dictionary of another major Internet company SOHU, and then made its own Pinyin input method. After this was exposed to the public, they did a not so honest "apologize" and said that were two interns who did it. Perfect, later this became a popular phase in China, if anyone did evil things but was discovered by the public, he would say it's intern's or temporary employees' fault. Well, what a shame on this "not to be evil" company. - little off topic) .

Baidu.com does not return any RSS feed. What's more, it is using GB2312 encoding method to show the results. So if you directly use regex to capture something in Baidu, you will get some squares which do not make sense.

And there're some limitations in asp.net Request.QueryString method. It cannot correctly process Gb2312 encoding. So the Page Load Method must be changed to the following code:

protected void Page_Load(object sender, EventArgs e)
    {
        if (Request.QueryString["q"]!= null)
        {
            query = Request.Url.Query.ToString();
            query = query.Remove(0,3);
        }
    }

In this way, a query string will be kept so you can process it with Encode and Decode. If you use QueryString, you will get a stupid behavior that it incorrectly use Decode method in a wrong encoding charset...The result is a disater. Stupid, stupid, stupid. I want to slap the guy who wrote this method. Does he know there're not only English in this world?

For example, my nickname opal, in Chinese is 猫眼石. If queried from IE, it will be encoded using UTF-8. But Baidu can only consume GB-2312.

In UTF-8, 猫眼石 is %E7%8C%AB%E7%9C%BC%E7%9F%B3.

In GB2312, 猫眼石 is %C3%A8%D1%DB%CA%AF.

It's quite different. If you want do a search for %E7%8C%AB%E7%9C%BC%E7%9F%B3, and it is treaten as a GB2312 string, it will become 4.5 Chinese charactors. and none of them will make sense.

Okay, compain less, do more. So then we need to decode query string.

 private string getRssItemXml(string query)
    {
        //first you must decode it as UTF8. Because when IE access a utf-8 based website, it will pass the corresponding encoded strings.
        //Of course, you can modify web.config to make this application using Gb2312, but that doesn't make sense.
        query = HttpUtility.UrlDecode(query, Encoding.UTF8);
        //Then we need do encode it to gb2312. Baidu can only consume that.
        query = HttpUtility.UrlEncode(query, Encoding.GetEncoding("gb2312"));
        string url = string.Format("http://www.baidu.com/s?wd={0}", query);

        WebClient client = new WebClient();
        byte[] byteData = client.DownloadData(url);
        //Returned results are also in GB2312, so you have to rebuild it.
        string strData = Encoding.GetEncoding("gb2312").GetString(byteData);
        Regex searchPattern = new Regex("\\)\" href=\"(?<link>.*?)\" target=\"_blank\"><font size=\"3\">(?<title>.*?)</font></a><br>(?<desc>.*?)<br>");
        StringBuilder sb = new StringBuilder();

        foreach (Match m in searchPattern.Matches(strData))
        {
            sb.AppendFormat("<item><title><![CDATA[{0}]]></title><link><![CDATA[{1}]]></link><description><![CDATA[{2}]]></description></item>",m.Groups["title"].Value,m.Groups["link"].Value, m.Groups["desc"].Value);
        }

        return sb.ToString();
    }

So then put this aspx file to a website, have your federated search webpart point to it, like http://www.abc.com/Baidu.aspx?q={searchTerms}, and then you can get Baidu federated search in Microsoft Search Server 2008.

I put part of my work here:

http://cid-8007edf5c56fc334.skydrive.live.com/self.aspx/Microsoft%20Search%20Server/CaptureWeb.rar

It contains:

Baidu Federated Search Web Service
Baidu News Federated Search Web Service
iCiba (English-Chinese Dictionary) Federated Search Web Service
Dictionary.com Federated Search Web Service

Yes! You can put dictionaries on your federated search web page so if anybody want to search a word, he will get the meaning immediately! You can also have some triggers to make this happen only with numbers or charactors, etc.

Federated Location Definition Files Collection for MSS

Jie Li — Wed, 21 Nov 2007 07:16:00 GMT

I just put some of my work here:

http://cid-8007edf5c56fc334.skydrive.live.com/browse.aspx/Microsoft%20Search%20Server

You can download and import these FLDs into your Microsoft Search Server installation. Some of them are not polished, and others maybe used to show special effects. Later I will explain how they work.

Locations are the same with my opensearch post:)

WikiPedia (en-us), WikiPedia (zh), Live Search, Live News

Google Blog, Google Blog (zh-cn), Google News, Google News (zh-CN)

Flickr, LinkedIn, MSDN, MSDN China, TechNet, TechNet China

Technorati, Technorati (zh), Wired, Yahoo, Yahoo Images, Yahoo News

Britannica, del.icio.us, Youtube, Amazon

Some OpenSearch sites for reference

Jie Li — Sun, 04 Nov 2007 09:52:44 GMT

Do you know www.opensearch.org? This is a open standard for search engines, which is created by A9.com from Amazon.

Why does it matter? Don't look at all those boring documents on opensearch.org, generally speaking, this is a standard that requires search engines return a RSS/ATOM feed. So you can use your favorite feed reader, get the best answers just in time.

Meanwhile, a standard RSS feed allows you intergrate different search engine results into your own application. For example, you have a enterprise search engine inside your company, but you want to intergrate some search results from outside, an iframe is okay but not very good. With XML+XSLT, you can intergeate them very easily. This is not limited to search engines, you can try different ideas, Yahoo Image Search, Google News Search, Live Spaces Search, LinkedIn expert search... Anything.

Well, what if a site does not return RSS feed? Wikipedia, Linkedin, they don't have anything. Oh, Wikipedia said they support opensearch, but they only support a part of the standard. What they returned can only be used in IE7/Firefox live search.

Live.com has a very useful feature to solve this problem. It can return RSS feed for any results. To give wikipedia a RSS feed, just use

http://search.live.com/results.aspx?q={searchTerms}+site%3aen.wikipedia.org&format=rss

And that's all.

Here I list some of the sites to be used with RSS feed:

WikiPedia (en-us)	http://search.live.com/results.aspx?q={searchTerms}+site%3aen.wikipedia.org&format=rss
WikiPedia (zh)	http://search.live.com/results.aspx?q={searchTerms}+site%3azh.wikipedia.org&format=rss
Live Search	http://search.live.com/results.aspx?q={searchTerms}&format=rss
Live News	http://search.live.com/news/results.aspx?q={searchTerms}&format=rss
Google Blog	http://www.google.com/blogsearch_feeds?hl=en-us&q={searchTerms}&lr=&ie=utf-8&num=10&output=rss
Google Blog (zh-cn)	http://www.google.cn/blogsearch_feeds?hl=zh-cn&q={searchTerms}&lr=&ie=utf-8&num=10&output=rss
Google News	http://news.google.com/news?hl=en-us&ned=us&ie=UTF-8&q={searchTerms}&output=rss
Google News (zh-CN)	http://news.google.com/news?hl=zh-CN&ned=cn&q={searchTerms}&ie=UTF-8&output=rss
Flickr	http://api.flickr.com/services/feeds/photos_public.gne?tags={searchTerms}&lang=en-us&format=rss_200
LinkedIn	http://search.live.com/results.aspx?q={searchTerms}+%28site%3Awww.linkedin.com%2Fpub%29&format=rss
MSDN	http://search.msdn.microsoft.com/search/Feed.aspx?brand=msdn&query={searchTerms}&lang=en-us&feed=rss
MSDN China	http://search.msdn.microsoft.com/search/Feed.aspx?brand=msdn&query={searchTerms}&lang=zh-cn&feed=rss
TechNet	http://search.technet.microsoft.com/search/Feed.aspx?brand=technet&query={searchTerms}&lang=en-us&feed=rss
TechNet China	http://search.technet.microsoft.com/search/Feed.aspx?brand=technet&query={searchTerms}&lang=zh-cn&feed=rss
Technorati	http://feeds.technorati.com/search/{searchTerms}?language=en
Technorati (zh)	http://feeds.technorati.com/search/{searchTerms}?language=zh
Wired	http://www.wired.com/search/rss?query={searchTerms}
Yahoo	http://api.search.yahoo.com/WebSearchService/rss/webSearch.xml?appid=yahoosearchwebrss&query={searchTerms}&adult_ok=0
Yahoo Images	http://api.search.yahoo.com/ImageSearchService/rss/imageSearch.xml?appid=yahoosearchimagerss&query={searchTerms}&adult_ok=0
Yahoo News	http://news.search.yahoo.com/news/rss?p={searchTerms}&ei=UTF-8
Britannica	http://www.britannica.com/opensearch?query={searchTerms}&ct=eb&start=0&count=20
del.icio.us	http://del.icio.us/rss/tag/{searchTerms}?language=en
Youtube	http://www.youtube.com/rss/tag/{searchTerms}.rss
Amazon	http://amazon.com/rss/tag/{searchTerms}/new?length=100