Windows Live Agents Team Blog : Debugging

Debugging Your Agent: step-by-step debugger

imiluk — Tue, 16 Sep 2008 20:52:00 GMT

Today we're going to talk again about debugging your agent's inner workings, but this time through the use of debug output instead of the step-by-step debugger. This is generally where you want to go to check up on what's happening behind the scene if your agent's output is not helping enough. The following contains scary exhaustive lists of obscure words, but fear not, the good news is that you won't need to remember any of them!

In the SDK, the queryserver's output can be seen in the Output tab when the agent is running, next to Conversation.
When running the Agent in the console, you can find the same data under Monitoring and Control > View Component Logs, then checking the latest Queryserver logfile.

The level of debug is highly customizable; you can pick and choose the outputs important to you, whether you want to know what's happening to your datasources, how the scoring is done exactly, why the agent is taking so long to startup, the order in which the domains and packages are being compiled, how your supercomplex routine ends up in a loop, what's happening with the notifications....

There are various ways of setting the debug level:
- In the SDK, your agent's properties have a field called Debug, which takes the argument "tag1 tag2 tag3...."
- In the SDK when the agent is running, you can change the level at any time using the command "!d tag1 tag2 tag3..."
- For a console agent, you can change it in the config file under Query Server:
<debug>tag1 tag2 tag3...</debug>

- For a running console agent, there is also a way - see at the end of this post for more details!

So what exactly would you put in this TAGS list?
Well the easiest approach is to use the preset debug levels. They are numbered from 0 to 9, 0 outputing nothing and 9 outputting per minute the equivalent of the whole contents of the Library of Alexandria.
By default, the SDK uses level 5 which is a good compromise but still quite verbose, while console Agents are usually set to 1 or 2 due to the amount of traffic they can get.

Here are, for future reference, the tags associated with each level:

1 = platforminfo domainerror matchingtestupdate extrainfo qddpevents qdmrevents qdmisc externalcomp dispatcherrunlevel miscloadingerrors preloadingagentfailed idsapiaccess

2 = 1 + platforminfoex plugins miscloadinginfo queryprocessor bestmatchoverall predicatereloading qduserevents qdmessages qddns sourceblocker1

3 = 2 + datasourcebasic queryconstruction querytimings gatewaynotfound gatewayfailure gatewaydbnotfound gatewayblocked externalext executionrephrase scriptexe1 threadid sourceblocker2 matcherlow agentpreloaded miscloadingdetails

4 = 3 + userprofile domainallstats match moreenum

5 = 4 + datasourceaverage conditiontreebasic userprofileex keepallmaybeyoumeant scriptexe2 timedactions matcheraverage

6 = 5 + conditiontreeaverage matchpossible contextbasic presentationfeed

7 = 6 + datasourcedetailed conditiontreedetailed patternexplosionbasic tokengrouping compilationrephrase scriptexe3 xslttrransform1

8 = 7 + patternexplosiondetails matcherhigh sourceblocker3

9 = 8 + displayallunify currentcommand patternexplosionfull xslttrransform2

For the global setting, using such a number can be quite enough to define how much general output you want to see, but let's say you have a problem with your datasources and want to see what happens with them without getting too distracted with the rest. Well you could call the command : "!d 1 datasourcebasic" which resets the main debug level to 1, then adds "datasourcebasic" on top of it. Note that the tags are cumulative, so if you were to call it again with: "!d datasourceaverage", your debug would then contain all of 1 + datasourcebasic + datasourceaverage. On the other hand, calling a numbered level resets to that level, so "!d currentcommand 1" is the same as "!d 1".

Now, you've noticed that if you're only interested in datasources, you still have to list a number of tags to achieve the level of debug you want. Here comes another helper: the tag groups! Much like the numbered levels, these help you by setting a bunch of tags at once:

platform = platforminfo platforminfoex

miscloading = miscloadinginfo miscloadingerrors miscloadingdetails

init = plugins conditiontreebasic conditiontreeaverage conditiontreedetailed dispatcherrunlevel

up = userprofile userprofileex

domain = domainerror domainallstats

context = contextbasic

pattern = patternexplosionbasic patternexplosiondetails patternexplosionfull

matching = bestmatchoverall match keepallmaybeyoumeant matchpossible displayallunify matchingtestupdate matcherlow matcheraverage matcherhigh

conditions = conditiontreeaverage conditionsbasic conditionsdetailed patternsbasic patternsdetailed

script = extrainfo moreenum currentcommand

gateway = gatewaynotfound gatewayfailure gatewaydbnotfound gatewayblocked

sourceblocker = sourceblocker1 sourceblocker2 sourceblocker3

xslttrransform = xslttrransform1 xslttrransform2

rephrase = executionrephrase compilationrephrase

qdbase = qddpevents qdmrevents

qdsmart = qdbase qduserevents qdmessages qddns up querytimings

qdall = qdsmart qdthread

datasource = datasourcebasic datasourceaverage datasourcedetailed

external = externalcomp externalext

thread_id = threadid

Another practical example: let's say you don't want too much information in your log window, and decide to set the debug level to 2. It just so happens that this poses a problem with the Comprehension Info tab, as that tab requires special debug outputs (notably matcheraverage, found in debug level 5) to be able to present you with the rephrasing rules details and scores reliably. Normally, you would keep the debug level 5 and feel sorry for all the virtual paper being wasted and the virual forests becoming extinct, but now you know better! All you need to do is to change the setting to "1 matching rephrase", or even "1 executionrephrase matcheraverage" to be super conservative.

A few words about performance.

As you can imagine, outputting so much data to the screen or to a file takes up resources, so whenever performance is a concern, you want to be conservative about the level of debug. Here are two ways to help without hurting:

- if your agent has a good-size codebase, starting it up can take a bit of time depending what it loads. So in the SDK where you require numerous restarts, you don't want to add on top of that the cost of outputting huge quantities of debug output you don't need at start up.

Lo and behold! the 'debug-startup' option is your new friend. It tells the queryserver what debug level to adopt until it's fully started. In the SDK you use it by adding "--debug-startup FLAGS" to the Extra Options field in the agent's properties. **

- For a console agent already running and live, as noted before you want a pretty terse debug level due to the traffic, but you may need to change it on the fly to investigate a particular issue. For this, Buddyscript has a built-in procedure called SetDebugLevel()

+ setdebuglevel TAGS=AnythingStrong {MACRO_PROTECTED_MATCHING}

- Changing the debug level to TAGS...

call SetDebugLevel(TAGS)

However make sure you turn it back once you're done investigation! One safe way of implement in it is with the following code:

variable G_CHANGED_DEBUG_LEVEL = false

+ setdebuglevel TAGS=AnythingStrong {MACRO_PROTECTED_MATCHING}

- Changing the debug level to TAGS...

G_CHANGED_DEBUG_LEVEL = true

call SetDebugLevel(TAGS)

procedure ABEndSessionProc()

if G_CHANGED_DEBUG_LEVEL

call SetDebugLevel("2")

This is assuming that your 'config' debug level is 2.

And that's it! Next time you have an issue for us to look at and we ask you to send us the log with debug level set to 'currentcommand' (this one outputs the whole execution of the code), you'll know why and how.

Thanks for reading the blog and come back soon!

** an even better friend in that regard is the command "!ddlreload" in the conversation window, that doesn't restart the agent but only reloads the domains and packages that changed since the last reload. Be advised though that it won't always reload everything correctly, so a restart once in a while is recommanded.

Type "?" to get a list of all those useful and lesser-known commands.

Debugging your Agent

imiluk — Thu, 28 Aug 2008 00:53:00 GMT

Depending on the complexity of your Agent, you might want to add debug code right of the bat. That makes it easier (and less painful) to find issues you will come across. Look for examples further down the post.

How to debug your Agent:

Syntax: break [file:]line

Example: In the SDK, type “!break Numbers.pkg:291” (this will set a break point in the package “Numbers” at line 291)

Once you set a break point, enter a query that you want to investigate further. Eventually you will hit the break point you set and you can now dig in by using the debugging commands listed below.

While in debug mode, type “help” to see a list of possible commands:

Debugger static commands:

break [file:]line: sets a breakpoint

show breakpoints: lists the breakpoint

show files: lists the different files that are loaded

delete number: delete a breakpoint

delete all: delete all breakpoints

show global vars: lists all the global vars

print or p expr: evaluates the expression

start: the next execution will stop at the first instruction

debugger on|off: turns on or off the debugger

Debugger execution commands:

continue or c: continue the execution

next or n: continue to the next line

step or s: continue to the next command, entering procedure or function

step out or so: continue until outside the routine, procedure or function

backtrace or bt: lists the call stack

up: goes up in the call stack

down or do: goes down in the call stack

frame or f number: goes to a given frame in the call stack

show vars: lists all the local vars at the current frame

quit or q: terminates the execution

Note: !d [debug_level] // Displays/changes the debug level

How to add debug code to your Agent:

// Debug code to place in a .pkg file.

variable G_PROJECT_NAME_DEBUG_MODE

procedure TurnDebugMyProjectOnOff(ON_OFF)

G_PROJECT_NAME_DEBUG_MODE = ON_OFF

if ON_OFF

- Debug mode turned on!

else

- Debug mode turned off!

procedure DisplayIfDebugMode(DISPLAY)

call IncreasePresentationMaxLengthIfDebugMode()

if G_PROJECT_NAME_DEBUG_MODE

- DISPLAY

// Debug code to place in a .ddl file. This is to turn debug mode on/off in your project.

+ _debug_my_project ON_OFF=Anything {MACRO_PROTECTED_MATCHING} // i.e. type “_debug_my_project true” to activate debug mode.

call TurnDebugMyProjectOnOff(ON_OFF)

If you have a lot of different domains in your Agent that interact with each other (i.e. weather, location, etc.), then it might make sense to split up the debug code accordingly. Create some per feature, i.e. procedure DisplayIfWeatherDebugMode(DISPLAY) and procedure DisplayIfLocationDebugMode(DISPLAY), etc.

Now, place your new procedure(s) wherever you want to display debug information throughout your code. For example:

function GetWeatherChangeForInitiative(WEATHER_OBJ, PRECIP, TEMP)

call DisplayIfWeatherDebugMode(GuessWeatherLocationObject())

CHANGE_MUCH_HIGHER_DATE = ()

CHANGE_MUCH_LESS_DATE = ()

WEATHER_OBJ_KEYS = Keys(WEATHER_OBJ)

DAYS_W_NO_CHANGE_AHEAD = 0

for value DATE in WEATHER_OBJ_KEYS

if PRECIP

CHANGE_TO_CHECK = GetWeatherObject_ForecastChangePrecipitation(WEATHER_OBJ, DATE)

if TEMP

CHANGE_TO_CHECK = GetWeatherObject_ForecastChangeTemperature(WEATHER_OBJ, DATE)

call DisplayIfWeatherDebugMode(StringConcat("Condition to check: ", CHANGE_TO_CHECK))

//much higher

if CHANGE_TO_CHECK eq "MACRO_WEATHER_PATTERN_CHANGE_MUCH_HIGHER"

insert last in CHANGE_MUCH_HIGHER_DATE DATE

call DisplayIfWeatherDebugMode(StringConcat("Much higher on: ", DATE))

break

//much less

if CHANGE_TO_CHECK eq "MACRO_WEATHER_PATTERN_CHANGE_MUCH_LESS"

insert last in CHANGE_MUCH_LESS_DATE DATE

call DisplayIfWeatherDebugMode(StringConcat("Much less on: ", DATE))

break

//no change

if CHANGE_TO_CHECK eq "MACRO_WEATHER_PATTERN_CHANGE_NO_CHANGE"

DAYS_W_NO_CHANGE_AHEAD++

call DisplayIfWeatherDebugMode(StringConcat("Days without change: ", DAYS_W_NO_CHANGE_AHEAD))

// ...

Happy debugging! ;)

------------------------------------------

Blog post contributor(s): Mirco

Visual Studio hints

imiluk — Mon, 11 Aug 2008 22:01:00 GMT

If you are moving from the old Colloquis IDE to the Visual Studio-based Windows Live Agents SDK, and you've never used Visual Studio, you may be a little lost. This post aims to collect simple hints and tricks for agent developers new to working with Visual Studio. This is just a start -- if you have any hints, please do comment, and later on we'll post a sequel.

Also, this is by no means much of a guide to Visual Studio as a whole, and is really meant to get you started and oriented for agent-development tasks. Also, it's not a guide for creating agents. The first resource for using the new SDK is the Windows Live Agents SDK documentation. You should start there before doing anything.

Agents menus

Here are some common menu locations for agent-development tasks:

Edit -> Find and Replace ... Lots of options for searching through single or multiple files. Regular expressions work!

View -> Solution Explorer ... This is rougly equivalent to the Explorer feature in the old IDE, and lets you see the project files in a tree view.

View -> Class View ... Lets you see all the project's domains and packages as a flat list of classes.

View -> Error list ... Displays compile errors and warnings.

View -> Output ... Similar to the "Misc Debug" pane of the old IDE, displays compiler messages.

View -> Other Windows -> Conversation Window ... Where you compile and talk to the agent.

View -> Other Windows -> Comprehension Info Window ... Displays match scoring information.

Project -> Add DLS Item ... Add datasources and other DLS items.

Project -> ProjectName Properties ... Specify compile parameters like buddy id, filter, and command-line options.

Tools -> Windows Live Agents Tools -> Code Management ... The SDK's interface to the Partner Hosting Infrastructure.

Tools -> Windows Live Agents Tools -> Request License Certificate ... Get a new license cert.

Tools -> Windows Live Agents Tools -> Management Console ... Launches the web management console within Visual Studio.

Tools -> Windows Live Agents Tools -> Update connections file ... Attempts to update the connections file based on the content of your project.

Tools -> Windows Live Agents Tools -> Performance ... Various tools for measuring performance.

Tools -> Options ... Set things like tabs and syntax highlighting.

Show All Files

By default, the SDK will mount a filesystem in the Solution Explorer, which may affect performance if there are a lot of files, causing the editor to lag. If this is the case, you can click "Show All Files" to remove the All Files view, and navigate to files through the Class View. Editing will be a lot faster. Set Show All Files to false by default at Tools -> Options -> Windows Live Agents SDK -> SDK Settings -> General -> "Show All Files on project open."

Keyboard Shorcuts

As of right now, there isn't a keyboard shortcut for starting/stopping the agent in the Conversation Window. However, here are some shorcuts you may find useful:

CTRL-K CTRL-C ... comment

CTRL-K CTRL-U ... uncomment

CTRL-ALT-L ... Solution Explorer

CTRL-TAB ... toggle through open windows

CTRL-Shift-<arrow key> ... move forward/backward by one word

Many other shortcuts are listed next to the menu items they correspond to.

Window Arrangement

One of the best things about Visual Studio is the flexibility it provides for laying out your environment. Any window can be tabbed, docked, or floating, and you can drag everything around the way you like. Right click a window title, or click the small arrow in the top of the pane, to see your options

Guidelines for Testing your Agent

imiluk — Mon, 11 Aug 2008 21:57:00 GMT

As compared to Web sites and traditional software applications, conversational agents are subject to some unique policy compliance risks. These risks arise because:

· End users’ interactions with agents are freeform and unpredictable.

· Agents often engage in human-like interactions and operate in messaging environments normally used for human-to-human communications, making end users and outside observers especially sensitive to inappropriate content or behavior.

Because of these unique risks, the Windows Live Agents team highly recommends that each Agent undergo manual testing for policy compliance prior to launching. Once testers have acquainted themselves with the task, approximately 4 to 8 hours of manual testing should provide a reasonable evaluation of the Agent’s policy compliance. Testers should:

· Be native speakers

· Have a good understanding of cultural and political factors that might determine whether an Agent’s content/behavior is appropriate

· Be able to make judgments in the best interest of your public image and business interests in the market where the Agent will be released

· Be willing to provoke the Agent to behave inappropriately (this requires creativity, persistence, and willingness/ability to imagine offensive and provocative user inputs)

· Understand the Agent’s feature set

· Ideally not have been directly involved in the Agent’s development

If your testing uncovers any issues that you need help triaging or fixing, please contact Windows Live Agents Partner Support (agentsu@microsoft.com). Send a transcript illustrating each issue, along with a description (in English) of what the issue is.

Overview for Testers

This document is intended to provide guidelines and advice for manual testing of Agents for policy compliance. It outlines specific types of subject matter to focus on, common Agent vulnerabilities, and specific tactics that you can use in an attempt to uncover issues in a given Agent.

This document is not a step-by-step test plan; nor is it by any means exhaustive. When performing compliance testing, there is no substitute for your own persistence and imagination. Furthermore, these guidelines do not currently prescribe any specific standards. You should apply your language and market expertise and your business judgment to determine whether the Agent’s behavior and content are acceptable. We strongly advise erring on the side of caution.

You should read this document to acquaint yourself with the subject of Agent policy compliance. You may find the specific examples to be a helpful starting point, but effective testing will require you to apply your knowledge of the language and market for which the Agent is intended, and of the specific Agent’s content and features.

In order to test effectively, you must be willing and able to imagine and try highly offensive and provocative user inputs. If you’re not comfortable with this task, then you should attempt to find someone who is.

Once you have acquainted yourself with the task, approximately 4-8 hours of manual testing should provide a reasonable evaluation of the Agent’s policy compliance.

Sensitive/Inappropriate subject matter to test

· Profanity

· Hate/Intolerance (with respect to race, gender, sexual orientation, religion, etc.)

· Violence and criminal behavior

· Drug use

· Sexual content

· Suicide

· Culturally/Politically sensitive subjects in your market

Scenarios to test

· Imagine you are one of the Agent’s target users

· Imagine you are a child

· Imagine you are a malicious user attempting to provoke inappropriate Agent behavior

Common Agent vulnerabilities

· Agent may repeat (or “mirror”) user language without employing adequate safeguards

· Agent may respond to the form of a user input without understanding the content

· Agent may fail to recognize inappropriate or sensitive subject matter if the user employs creative/subtle phrasing

· Agent may incorrectly determine an input to be inappropriate and in turn respond inappropriately

· Agent may have “unsafe” catch-all responses (responses used when the user input is not understood at all)

Some specific tactics to try

· See how the Agent responds to blatant abuse and provocation

· Try to trick the agent into repeating an inappropriate word or phrase

· Try to elicit an inappropriate opinion from the Agent

· Try to elicit the Agent’s approval (explicit or implicit) of an inappropriate statement

· Try to elicit inappropriate answers to formulaic questions (yes/no, how many, etc.)

· Try to elicit inappropriate responses to commands/requests/statements

· Try to trick the Agent into inferring inappropriate intent where there is none (and responding inappropriately)