Subject Search Result Evaluation Tool

Jie Li — Mon, 17 Dec 2007 08:08:20 GMT

http://www.codeplex.com/SubjectSearchEva

This is another result for Hylanda wordbreaker testing...

The story is very interesting. After we got 18000+ search results for each wordbreaker, we need to evaluate the results to decide when, where and how the new wordbreaker is better than the original one. Such thing cannot be decided by myself, or by any of the physical measure. When people say "Hey, you know sometimes Yahoo is better than Google", they are measuring it with their own eyes.

This is quite similar with my research for my Master's degree. I studied Psychological Acoustics for two years in that time, try to use some way to measure sound quality. This is called subject quality evaluation. So I used the same method, to measure search result quality.

This method is called Paired Comparison. The tester will be given two sets of result, and choose which is better. For example, there're 50 results in each set(A, B). So first, A1 and B1 show up and the user choose A, then A2/B2 show up and user choose B... In the end, count the number of A and B, you will get tester's preference.

To make sure there's no psychological interfere during the test, the order and set can be randomized. Tester will not know which is A and which is B, they just need to choose the better one.

Here's the interface of the program.

Testing. If tester cannot choose which is better(sometimes the two sets are the same good or bad), he can choose not to vote or vote for none.

Final result will displayed in a message box.

Of course, this is just a simple proof of concept. I shared the source and a sample in the release package so you can do your own research work.

Improve User Experience in Enterprise Search Step By Step - Part IV - Relevancy Tuning by WordBreaker

Jie Li — Fri, 23 Nov 2007 16:18:04 GMT

We have been talking about XSL/XML for so long a time. Now we want to give relevancy a shot.

Hey, relevancy?

Yes, relevancy.

Relevancy is the most important thing for a search engine, more important than the page numbers it crawled, more important than the result update interval in most of the case - because users always look at the top results.

Relevancy is a very complex problem. It is affected by many factors, it is quite different in different languages. In this article, we will take English and Chinese for example. (I know a little German and Japanese as well but ...)

In this series we will go through wordbreaker, weighting, and other useful stuff. Because I'm now in a Karaoke party, I cannot describe everything in detail. I assume you already know how to deal with Bestbets and Did you mean feature. If you want to have some other information, please read Luca Bandinelli's multilingual whitepaper.

Wordbreaker

Wordbreaker is the first issue if you hate a search engine. although word breaking in Latin languages(English, Spanish, French, German, Dutch...) is much easier than that in other languages(Chinese, Japanese, Korean, Arabic...), it's still a boring thing to deal with.

MOSS/MSS comes with many wordbreakers, but sometimes you may not be very satisfy with them. Is there any 3rd party word breaker I can use? Yes, some Microsoft partners have quite a long history in delivering word breaking technology to production use. For example, Hylanda is a leading Chinese word breaking technology company. They did a pretty good Chinese wordbreaker for SQL Server 2000. Since we didn't change the interface of wordbreaker even in MOSS/MSS, it can be used directly here.

To change a wordbreaker, you need to do the following things.

a. Register the wordbreaker.

This depends on the installation manual of the wordbreaker:). But generally speaking, it should be something like this:

regsvr32 YourWordBreaker.dll

b. Get GUID string of your new wordbreaker. We will need it in the next step.

Search for your wordbreaker dll's name in registry, and you will find something located in CLSID branch. For example:

HKEY_LOCAL_MACHINE\SOFTWARE\Classes\CLSID\{4474fffd-da87-4116-9be9-874939d2bd04}

Copy this guid string for further usage.

c. Navigate to the branch of your language. Replace the values with your wordbreaker.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\12.0\Search\Setup\ContentIndexCommon\LanguageResources\Default\YourLanguage

WBDLLPathOverride is the path of your wordbreaker dll. In my case, my wordbreaker is located at C:\Hylanda\HlChsBrKr.dll

WBreakerClass is the GUID string you just got.

Don't forget to restart your search service by net stop osearch, net start osearch. Then do a FULL crawl for all the content source. If the wordbreaker did the crawl job mismatch with the new installed wordbreaker, it will result a bad search result because of query time word breaking.

Sorry I can't post the images of search results, they are still in testing process. But I can tell you the improvement is HUGE.

Jie Li's GeekWorld : WordBreaker

Subject Search Result Evaluation Tool

Improve User Experience in Enterprise Search Step By Step - Part IV - Relevancy Tuning by WordBreaker