Windows Live Agents Team Blog : data sources

Parsing, manipulating and examining data in Buddyscript

imiluk — Tue, 23 Sep 2008 20:46:00 GMT

As an Agents developer, you need to be able to handle and present data accurately and efficiently. There are many string parsing, manipulation, and examination functions in the Buddyscript arsenal. Many are of the standard variety that are found in most programming languages. The more common and obvious ones that follow the same concepts as most other high-level computer programming languages are the following (for more details on these, you can look at the online help under Procedures and Functions in the String Handling section):

Common String Parsing Functions:

StringSub - the most common substring function that gets a part of a string. Don't confuse this with the StringSubstitute function, which does string replacing.

StringGrabPart - this function splits out a string into different parts based on a given separator. A number is provided as one of the arguments in the function, indicating which part to return. Compare this with StringSplit.

StringSplit - given a string, splits the string into parts based on a separator string. These parts are put into an object list.

Common String Manipulation Functions:

StringUppercase - uppercases a string

StringUppercaseFirst - uppercases only the the first letter in the string

String UppercaseInitials - uppercases the first letter in each word of the string

StringLowercase - lowercases a string

StringConcat - concatenates two or more strings. Each string to be concatenated is a parameter.

StringTrim - trims off leading and trailing spaces from a string

StringLTrim - trims off leading spaces from a string

StringRTrim - trims off trailing spaces from a string

StringSubstitute - given a target string, replaces all occurences of one string with another string. Don't confuse this with StringReplace (see below)

StringReplace - given a target string, replaces a string with another string at a particular position in the string.

StringFormat - similar to the printf function in standard C programming language library, where a format is given to define the string.

Common String Examination functions:

StringLength - returns the length of a string

StringSearch - finds the position number (Buddyscript uses an offset starting at 0 in its string) in a string, given a target string.

StringStartsWith - used as a conditional (true or false, comparing a string with a desired start-with string to see if the string starts with the start-with string

StringEndsWith - used as a conditional (true or false, comparing a string with a desired end-with string to see if the string ends with the end-with string

StringAscii - returns the ascii value of a character

StringChar - the opposite of StringAscii - returns the character value of an ASCII code

Buddyscript does provide a few more string parsing and manipulation functions that are not as common to programming languages, but are very powerful. Some of these functions are quite useful. I'm going to go into a little more depth on some of these functions.

The concept of a thawed string - Using StringClean

Let's say you need to capture data that can come in in different formats. For example, a telephone number. A telephone number could come in looking like 650-555-1212, (650) 555-1212, 650.555.1212, and so on. One pretty solid approach you could take is to use the StringClean function to normalize the data. What the StringClean function does is to convert a string into its thawed state. The term 'thawed' is something germaine to Buddyscript. A thawed string is a string that has been stripped of all special characters except for letters and numbers.

Here is a little test code snippet:

+ _testme TESTSTRING=Anything

STRINGTHAWED = StringClean(TESTSTRING)

STRINGNORMALIZED = StringSubstitute(STRINGTHAWED, " ", "")

- STRINGNORMALIZED

Let's run this: _testme (650) 555-1212

The STRINGTHAWED variable will look like: 650 555 1212

By then replacing all whitespace characters with nothing using the StringSubstitute function, the STRINGNORMALIZED variable will then contain 6505551212.

Hashing in Buddyscript - Using StringHash

A hash is usually used as a numerical representation of a longer string. In many programming languages, It can be used as a way to better index a set of data, i.e. indexing on the hash rather than on a long string. A hash can also be used to check that a string or computer file has not been altered. We can do this by comparing an expected hash value with the hash value of the string or file. The form of the syntax for StringHash is as follows

HASH = StringHash(URL, 2147483647)

The 2nd number is the maximum number allowed for hash values. You can specify any number up to 2147483647. This number is 2^31 -1, which is the absolute maximum you can specify. Any number greater will return zeroes. For example, the hash generated from the statement StringHash("http://www.microsoft.com", 2147483647) would be 251890749.

Also, note that there are also two other hashing functions that are undocumented, GetMD5 and GetSHA1, which are also built-in. These use the MD5 and the SHA-1 encryption algorithms. Both return a hexadecimal string.

HEXSTRING = GetMD5(String)

HEXSTRING = GetSHA1(String)

Using the regex functions

If you're not familar with regular expressions, hang on to your seat belt. Regular expressions are commands that also do string processing. But think of regular expressions as string processing on steroids. With a small set of commands, one can do a lot of very flexible and powerful processing of text data.

BuddyScript has a slew of string functions that accept regular expressions as a means for searching and manipulate data. Let's look at a real world example.

Currently you are reading my blog post right now. So let's say, just as a crude example, that you wanted to find the last posting date in this blog and report it in your agent. So how would we do this?

This is known as (web)site scraping. In general, this is not a good practice, since HTML can change all the time at the whim of a web page designer. It is far better to use a web service that has an RSS feed or an XML feed that contains this data. (Also, hitting against a website too often can lead to administrators thinking that there is a Denial of Service attack going on, so don't try this often). Having said this, let's pretend that no feeds exist, and you really have to grab the date from the website.

First of all, you'd set up a datasource to access the data, and then put the data into a variable. Here's an example:

datasource dsBlog() => PAGE

http

http://blogs.msdn.com/windowsliveagents/

DATA = dsBlog()

The variable DATA now contains all the HTML, most of it which is fairly unreadable unless you are an HTML guru. We know that the most recent posting date is between the word "Posted" and the word "by" (as I said before this is a very unstable technique but bear with me). We can now use the StringMatch function to grab this specific data. The StringMatch function has 3 parts to it, the string, the regular expression, and the object hash where the splitting of the data will go into for the regular expression.

The actual HTML snippet might look something like this:

"Posted <a id="postlist" href="http://blogs.msdn.com/feedback.aspx">Saturday, September 13, 2008 12:32 AM</a> by"

The statement to extract this would look something like this:

EXTRACT = ()

MATCHES = StringMatch(TESTSTRING, "Posted(.*?)\.aspx\">(.*?)</a> by", EXTRACT)

- EXTRACT[1]

We allocate a list called EXTRACT, which will contain the contents as specified in the regular expression inside the parenthesis, (.*?). So what does .*? mean? The '.' dot represents matching any one character. The '*' is a quantifiers. The '*' means to grab all characters. The ? means to look for the first occurence. There is a lot to regular expressions which this blog post won't be able to cover.

EXTRACT[0] will thus contain <a id="postlist" href="http://blogs.msdn.com/feedback

EXTRACT[1] will contain Saturday, September 13, 2008 12:32 AM

MATCHES will contain the number 2, since there are 2 matches found.

There are other string functions that use regular expressions as input as well. Once again, look at the online documentation for more details.

StringSplitRegex - is like StringSplit, but instead of a fixed separator value, a regular expression is used instead.

StringSubstituteRegex - is like StringSubstitute, but instead of a fixed string to be looked for for replacing, a regular expression can be used instead.

An advanced look at Web Services and DataSources - Part II

imiluk — Mon, 11 Aug 2008 22:00:00 GMT

This is a continuation on An Advanced Look at Web Services and DataSources. The original entry is located here: http://windowsliveagents.spaces.live.com/blog/cns!5BCD45E519E07634!711.entry

Now let’s take a look at the datasource itself. The datasource essentially is a function itself, with arguments to pass, and variables to return. In the preprocess section, the POST_DATA variable where the XML SOAP request string was built is put into here. In addition, the actual web service URL is stated here as well.

In the preprocess section, there are also two built-in variables that can be used, LIMIT and OFFSET. These two variables are used to ‘page’ results in a cursor. In the example above, we look at LIMIT to populate a variable called MAXRESULTS. The MAXRESULTS variable is then used in the COUNT element (in this case 10) to bring back 10 results per request. If the user needs more, then the datasource then starts at the next row and retrieves 10 more results.

The simple xml section is a hierarchical representation of the XML response from the SOAP API, to be flattened out into a 2-dimension look when the data is retrieved. Indentation is used to signify a parent-child relationship. The {loop=content} statement acts as a loop within the XML, iterating through the XML. The end nodes (highlighted in BOLD) are the fields that is used to capture information and passed back to the calling routine. Note that fields can be skipped in the simple xml section if the user does not need it.

It should be noted here that by using simple xml to represent the XML response, there is no provision for providing a “dynamic” representation of the XML using simple xml. So in essence, you would have to potentially write a different datasource function for each different search type in this case. For generate an advanced datasource that could output differently depending on the search type would require outputting a datasource in Buddyscript. We’ll cover this in a different blog.

datasource LiveSearchAPI(SEARCH, CULTURE_INFO) => Title, Description, Url, Source, NewsYear, NewsMonth, NewsDay, NewsHour, NewsMinute, NewsSecond {expire="in 1 hour" continue_on_error="true" timeout="15" }

preprocess

if LIMIT>10 || LIMIT<=0

MAXRESULTS = 10

else

MAXRESULTS = LIMIT

FIELDLIST = "Title Description Url Source DateTime"

POST_DATA = BuildSearchAPIPostData(SEARCH, "News", OFFSET, MAXRESULTS, CULTURE_INFO, FIELDLIST)

http

http://soap.search.msn.com:80/webservices.asmx

header

Accept: application/soap+xml

postdata {encode=no}

POST_DATA

simple xml

Envelope

Body

SearchResponse

Response

Responses

SourceResponse

Offset => RESULTOFFSET // Where we're starting from.

Total => TOTAL // Total number of results.

Results

Result {loop=content}

Title

Description

Url

Source

DateTime

Year

Month

Day

Hour

Minute

Second

postprocess

INFO.Offset = RESULTOFFSET

INFO.MaxCount = TOTAL

return INFO

There are other datasource properties that should be considered to either increase performance and or deal with potential errors in accessing/retrieving information from the datasource. The first one is the Timeout property. You can specify this time in order to lengthen or shorten the time it takes before the datasource quits accessing the web service. The default value is 10 seconds. In our case, we have it at 15 seconds. The next property is the continue_on_error property. By changing this property to ‘yes’, execution will still continue and the datasource caller can retrieve the error message in the SYS.Data.Error variable. This is only on those sources that call the ABErrorProc. The final property is very important. It is the Expire property. This determines how long retrieved data should be valid, i.e. kept in cahsed memory. The ability to cache retrieved data in memory will improve performance on retrieving information in the datasource. You should consider these factors:

1) how often does the data change?

2) how often will the same retrieved data be asked again?

3) how large is the retrieved data set?

4) server memory cache size (N/A on hosted applications)

5) how fast does the web service perform?

All of these are considerations. In our case, since news items change frequently, we’ll set it for a relatively short time period, say 1 hour.

Examples of the Expire property:

Expire=”never” /* this is the default expiration for most non-Buddyscript datasources */

Expire=”in 1 hour”

Expire =”now” /* no caching at all, same as “never” */

Expire=”tomorrow at 5am” /* Note that this time is the server time, not the client time. In hosted applications, this is in GMT time */

The postprocess section is important for returning a range of information. For datasources that do not handle the processing of data using offsets and limits (i.e. simple xml), if the postprocess section is missing, the processing QueryServer will process the data coming back from the datasource in its entirety. In cases where the output coming back is one entity or one row, or if the amount of data needed to be processed is small, the postprocess section is not needed.

postprocess

INFO.Offset = RESULTOFFSET

INFO.MaxCount = TOTAL

return INFO

Looking at the postprocess section, this section is used to set the offset and total count of rows in a variable. This variable is then used by Buddyscript to control the display of output.

In this case, INFO is the name of an object variable. The names of the variables inside the object is Offset and MaxCount, and these values are populated from the datasource:

Offset => RESULTOFFSET // Where we're starting from.

Total => TOTAL // Total number of results.

Finally, here is a crude routine to pass a request to the Live Search API, access the web service and display the contents of the data, using Buddyscript code to control the amount of data coming in.

? Tell me some news about STRING=Anything

LOCALE="en-us"

TITLE, DESCRIPTION, LINK, SOURCE, YEAR, MONTH, DAY, HOUR, MINUTE, SECOND = LiveSearchAPI(STRING, LOCALE) show 10

* Here are the results:

- TITLE, SOURCE

* <blank/>

else

- Sorry, no news sites were found for your input.

In the input, you could ask a question such as “Tell me some news about Baron Davis” for example, and get back results that looks like this (notice that the output only contains 2 out of the 10 arguments returned, Title and Source):

Here are the results:

Baron Davis Going South, San Francisco Gate

The Baron Davis, Gilbert Arenas Switch-a-roo?, San Francisco Gate

Clippers set sights on Baron Davis, Los Angeles Times

Baron Davis on verge of signing with Clippers, Washington Post

NBA: Warriors trying to woo Brand, Newsday

Baron Davis becomes free agent, Chicago Sun-Times

Report: Davis to ditch Warriors for Clippers, FOXSports.com

Logo? Colors? History? Don't mean a thing if you ain't got that team, CBS Sportsline

Davis on verge of joining Clippers, CNN Sports Illustrated

Davis on verge of signing with Clippers, Salon

Type “more” for more news.

Notice that in the pattern routine, there is an option called SHOW 10. This means to output 10 rows at a time. Buddyscript will go to the output datasource to retrieve the information, which in this case happens to be exactly 10 rows, since the request was to buffer 10 rows per datasource request. If the user were to type in “more”, another 10 rows will be retrieved from the datasource and 10 more rows will be displayed, and so on. (Note that there is a Buddyscript variable named SYS.Presentation.Maxlength that also controls the number of characters that can be displayed on an IM client. Depending on what this is set to, this number will also control the number of rows displayed back.)

With the SHOW command, this allows the user a quick and equivalent way of emulating a forward read-only cursor, i.e. displaying x number of rows of output at one time. The other option would be to put the data into an object and loop through the object, displaying each row, which involved more coding. It’s very possible that for control purposes, the latter method is the right way to go, but for quick coding and display, SHOW is very powerful.

Hopefully you have gotten a chance to absorb the intricacies of using datasources by accessing a really powerful web service. Thanks for your attention!

An advanced look at Web Services and Datasources - Part I

imiluk — Mon, 11 Aug 2008 21:59:00 GMT

An advanced look at Web Services and Datasources

Back in February, we showed a basic example of accessing a web service and using a datasource to consume information from a web service. (Refer to: http://windowsliveagents.spaces.live.com/blog/cns!5BCD45E519E07634!443.entry) Today, we’ll take a further look at a real-world web service, and some further things you can do with a datasource to make your application perform better, and to potentially reduce coding effort.

Many agents rely on rich content in order to create a solid user experience. Many times, this involves accessing an external data store, such as a web service. With some web services, you can parameterize your request in the URL itself, for example http:/URL?<input>. More often, web services involve using request and response model via a SOAP API.

The Windows Live Search API is a very powerful web service that does many things. It allows a user to do searches on the web, news, images, as well as get dictionary, phonebook and spelling information. It provides a web service XML interface using a SOAP API, enabling you to submit requests and get back a response via XML. For more information on the XML, refer to this link: http://msdn.microsoft.com/en-us/library/bb251794.aspx

Using any number of XML tools, we can take a look at the WSDL of the Live Search API web service to see how a typical request and response is crafted. The WSDL endpoint is this:

http://soap.search.msn.com/webservices.asmx?wsdl

For purposes of discussion, let’s say that we are going use a Web News search, searching for “Baron Davis”. The query of “Baron Davis” is put into the Query element. There are numerous other elements that are part of the response. They are highlighted in red. The XML response would look like this:

<?xml version="1.0" encoding="utf-16"?>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<soap:Body>

<Query>Baron Davis</Query>

<SafeSearch>Moderate</SafeSearch>

</Location>

<ResultFields>All DateTime</ResultFields>

</SearchTagFilters>

</SourceRequest>

</Requests>

</Request>

(Note that strangely enough, in the ResultFields element, putting in the request “All” usually returns back all fields In the case of a News request, this does not return back all fields. “All DateTime” will return all fields, plus the datetime in this case.)

The XML response looks like this, with output fields in bold:

<?xml version="1.0" encoding="utf-16"?>

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<soapenv:Body>

<Title>NBA free agent roundup: Baron Davis to L.A.?</Title>

<Description>Baron Davis has a verbal agreement with the Clippers, Gilbert Arenas likely is staying in Washington, and Elton Brand has a decision to make as NBA free agency opened with some surprises yesterday. The dominoes began to tumble when Davis surprised ... </Description>

<Url>http://www.newsday.com/sports/basketball/knicks/ny-spnba025749266jul02,0,2266274.story </Url>

<DisplayUrl>http://www.newsday.com/sports/basketball/knicks/ny-spnba025749266jul02,0,2266274.story</DisplayUrl>

<Source>Newsday</Source>

</DateTime>

</Result>

<Title>Later Daze, Baron: Davis leaves Warriors for Clippers</Title>

<Description>del.icio.us Less than 24 hours after Davis shocked the Warriors by walking away from the final year and $17.8 million left on his contract, the franchise guard delivered a second strike Tuesday afternoon. Davis agreed in principle with the Clippers ... </Description>

<Url>http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2008/07/02/SPCF11ICKB.DTL</Url>

<DisplayUrl>http://www.sfgate.com/cgi-bin/article.cgi?f=/c/a/2008/07/02/SPCF11ICKB.DTL</DisplayUrl>

<Source>San Francisco Gate</Source>

</DateTime>

</Result>

</Results>

</SourceResponse>

</Responses>

</Response>

</SearchResponse>

</soapenv:Body>

</soapenv:Envelope>

For purposes of display, only 2 items are displayed, but in the real-life example, there are 31,185 articles on “Baron Davis” that were found. The 31,185 is indicated in the <TOTAL> element in the response. Note that not all 31,185 rows are retrieved. The amount retrieved is regulated in the Preprocess block, in the built-in LIMIT and OFFSET variables. More on this later, when we describe the datasource.

Let’s take a look at how we generate the Buddyscript code to access this web service.

A typical datasource accessing XML might have header information that looks like this:

datasource dsSample(ARG1, ARG2, ARG3) => A1, B1, C1

http

http://someurl/sample.xml

simple xml

However, for a web service invoking SOAP messages, we have to build a request string, so we use the PreProcess section of a datasource to set a variable that will contain the request, along with other variables. Thus, the beginning of the datasource would look like this instead:

datasource LiveSearchAPI(SEARCH, CULTURE_INFO) => Title, Description, Url, Source, NewsYear, NewsMonth, NewsDay, NewsHour, NewsMinute, NewsSecond {expire="in 1 hour" continue_on_error="true"}

preprocess

if LIMIT>10 || LIMIT<=0

MAXRESULTS = 10

else

MAXRESULTS = LIMIT

FIELDLIST = "Title Description Url Source DateTime"

POST_DATA = BuildSearchAPIPostData(SEARCH, "News", OFFSET, MAXRESULTS, CULTURE_INFO, FIELDLIST)

http

http://soap.search.msn.com:80/webservices.asmx

header

Accept: application/soap+xml

postdata {encode=no}

POST_DATA

simple xml

We’ll describe the syntax of the datasource in more details later on, but for now, what’s important to point out is that a function is called to build the request string. The function in this case is called BuildSearchAPIPostData and has a number of arguments. Let’s take a look at the function:

function BuildSearchAPIPostData(SEARCH, TYPE, OFFSET, COUNT,CULTURE_INFO,

FIELDLIST, RADIUS, LATITUDE, LONGITUDE)

if (LATITUDE eq "")

LATITUDE = 0

if (LONGITUDE eq "")

LONGITUDE = 0

if (RADIUS eq "")

RADIUS = 5

POST_DATA = '<?xml version="1.0" encoding="UTF-8" standalone="no"?>\n'

POST_DATA = StringConcat(POST_DATA, '<SOAP-ENV:Envelope xmlns:SOAP-ENV="http:/\/schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:tns="http://schemas.microsoft.com/MSNSearch/2005/09/fex" > <SOAP-ENV:Body>')

POST_DATA = StringConcat(POST_DATA, ' <tns:Search xmlns:tns="http:/\/schemas.microsoft.com/MSNSearch/2005/09/fex"> <tns:Request>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:AppID>', GetLiveSearchAPIAppID(), '</tns:AppID>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Query>', SEARCH , '</tns:Query>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:CultureInfo>', CULTURE_INFO, '</tns:CultureInfo> <tns:SafeSearch>Moderate</tns:SafeSearch> <tns:Flags>None</tns:Flags>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Location>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Latitude>', LATITUDE, '</tns:Latitude>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Longitude>', LONGITUDE, '</tns:Longitude>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Radius>', RADIUS, '</tns:Radius>\n')

POST_DATA = StringConcat(POST_DATA, ' </tns:Location>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Requests> <tns:SourceRequest>\n')

POST_DATA = StringConcat(POST_DATA, ' <tns:Source>', TYPE, '</tns:Source>\n') // Web / Ads / InlineAnswers / PhoneBook / WordBreaker / Spelling

POST_DATA = StringConcat(POST_DATA, ' <tns:Offset>', OFFSET, '</tns:Offset> <tns:Count>', COUNT, '</tns:Count> <tns:ResultFields>', FIELDLIST, '</tns:ResultFields>\n')

POST_DATA = StringConcat(POST_DATA, ' </tns:SourceRequest> </tns:Requests>\n')

POST_DATA = StringConcat(POST_DATA, ' </tns:Request> </tns:Search>\n')

POST_DATA = StringConcat(POST_DATA, '</SOAP-ENV:Body> </SOAP-ENV:Envelope>\n')

return POST_DATA

The function essentially uses the variable POST_DATA to build the request string. If there are certain parameters without information, it defaults the information (e.g. LATITUDE). The function also calls another function to get the Windows Live APP ID. This APP ID is used in the Live Search API to determine individual access rights. (See the Windows Live API for more information on obtaining this APP ID.) The function itself simply looks like this:

function GetLiveSearchAPIAppID()

return "ABCDE12345"

where “ABCDE12345” is the App ID.

We'll take a look at the rest of the datasource and invoke code later this week.