Windows Live Agents Team Blog : Best Practices

Guidelines for Testing your Agent

imiluk — Mon, 11 Aug 2008 21:57:00 GMT

As compared to Web sites and traditional software applications, conversational agents are subject to some unique policy compliance risks. These risks arise because:

· End users’ interactions with agents are freeform and unpredictable.

· Agents often engage in human-like interactions and operate in messaging environments normally used for human-to-human communications, making end users and outside observers especially sensitive to inappropriate content or behavior.

Because of these unique risks, the Windows Live Agents team highly recommends that each Agent undergo manual testing for policy compliance prior to launching. Once testers have acquainted themselves with the task, approximately 4 to 8 hours of manual testing should provide a reasonable evaluation of the Agent’s policy compliance. Testers should:

· Be native speakers

· Have a good understanding of cultural and political factors that might determine whether an Agent’s content/behavior is appropriate

· Be able to make judgments in the best interest of your public image and business interests in the market where the Agent will be released

· Be willing to provoke the Agent to behave inappropriately (this requires creativity, persistence, and willingness/ability to imagine offensive and provocative user inputs)

· Understand the Agent’s feature set

· Ideally not have been directly involved in the Agent’s development

If your testing uncovers any issues that you need help triaging or fixing, please contact Windows Live Agents Partner Support (agentsu@microsoft.com). Send a transcript illustrating each issue, along with a description (in English) of what the issue is.

Overview for Testers

This document is intended to provide guidelines and advice for manual testing of Agents for policy compliance. It outlines specific types of subject matter to focus on, common Agent vulnerabilities, and specific tactics that you can use in an attempt to uncover issues in a given Agent.

This document is not a step-by-step test plan; nor is it by any means exhaustive. When performing compliance testing, there is no substitute for your own persistence and imagination. Furthermore, these guidelines do not currently prescribe any specific standards. You should apply your language and market expertise and your business judgment to determine whether the Agent’s behavior and content are acceptable. We strongly advise erring on the side of caution.

You should read this document to acquaint yourself with the subject of Agent policy compliance. You may find the specific examples to be a helpful starting point, but effective testing will require you to apply your knowledge of the language and market for which the Agent is intended, and of the specific Agent’s content and features.

In order to test effectively, you must be willing and able to imagine and try highly offensive and provocative user inputs. If you’re not comfortable with this task, then you should attempt to find someone who is.

Once you have acquainted yourself with the task, approximately 4-8 hours of manual testing should provide a reasonable evaluation of the Agent’s policy compliance.

Sensitive/Inappropriate subject matter to test

· Profanity

· Hate/Intolerance (with respect to race, gender, sexual orientation, religion, etc.)

· Violence and criminal behavior

· Drug use

· Sexual content

· Suicide

· Culturally/Politically sensitive subjects in your market

Scenarios to test

· Imagine you are one of the Agent’s target users

· Imagine you are a child

· Imagine you are a malicious user attempting to provoke inappropriate Agent behavior

Common Agent vulnerabilities

· Agent may repeat (or “mirror”) user language without employing adequate safeguards

· Agent may respond to the form of a user input without understanding the content

· Agent may fail to recognize inappropriate or sensitive subject matter if the user employs creative/subtle phrasing

· Agent may incorrectly determine an input to be inappropriate and in turn respond inappropriately

· Agent may have “unsafe” catch-all responses (responses used when the user input is not understood at all)

Some specific tactics to try

· See how the Agent responds to blatant abuse and provocation

· Try to trick the agent into repeating an inappropriate word or phrase

· Try to elicit an inappropriate opinion from the Agent

· Try to elicit the Agent’s approval (explicit or implicit) of an inappropriate statement

· Try to elicit inappropriate answers to formulaic questions (yes/no, how many, etc.)

· Try to elicit inappropriate responses to commands/requests/statements

· Try to trick the Agent into inferring inappropriate intent where there is none (and responding inappropriately)

Best Practices for Developing a Windows Live Agent: part 3

imiluk — Fri, 08 Aug 2008 22:14:00 GMT

Use Dialogs

Dialogs create a more interactive experience and help the user to feel like he/she is driving the conversation.

Example:

- Hey, I have really cool games. You wanna see them?

? Yes.

- Here are my games:

Hangman {*}

Word Scramble {*}

Reversi {*}

? No.

- OK! Next time you want to see my games, just type “games.”

Best Practices for Developing a Windows Live Agent: part 4

imiluk — Fri, 08 Aug 2008 22:14:00 GMT

Use Canonical Questions

A canonical question establishes the most concise, “representative” question for a topic. If a user asks an ambiguous question, the agent responds with a clarification request using the canonicals for each matched topic. Note that only one question should be marked as a canonical in a topic.

Example:

+- What is my horoscope?

? What is my horoscope?

? Zodiac

+- What is my Zodiac sign?

? What is my Zodiac sign?

? Zodiac

User: zodiac

Agent: What did you mean?

1 What is my horoscope?

2 What is my Zodiac sign?

Best Practices for Developing a Windows Live Agent: part 2

imiluk — Fri, 08 Aug 2008 21:51:00 GMT

Create Good Subpatterns

Subpatterns are one of the most powerful NL recognition features in the BuddyScript language. They allow you to write efficient, reusable code. Many subpatterns are defined in various packages in the Vocabulary directories for each language in BuddyScriptLib. Have a look at the subpatterns created there before you create your own. Many general parts-of-speech and “chat” subpatterns, as well as subpatterns for industry-specific terminology, are defined there. Create subpatterns for anything else you foresee your end users chatting about – product names, peoples’ names, place names, etc.

NL-brick Subpatterns

When you define a subpattern as an NL-brick, every element of the subpattern that is recognized at maximum strength is a candidate to replace sections of your NL handles. This is a very useful property, but use it carefully. If you want only certain elements to replace the natural language in an NL-brick subpattern, make the other elements score lower.

Define subpatterns as NL-bricks by putting them within NL-brick zones. If your project is derived from the WLATemplate, add NL-bricks in “YourAgent/YourLanguage/CustomSubpatterns.pkg,” which already contains an NL-brick zone.

Example:

begin properties SubpatternsAreNLBricks // Start of NL brick zone

…

### “n” prefix: Noun Stems
subpattern nBike + (bike|bikes)

subpattern nMotorcycle
+ (motorcycle|motorcycles)
+ motor (cycle|cycles)

subpattern nMotorbike
+ (motorbike|motorbikes)
+ motor =nBike

### “ns” prefix: Noun Senses

subpattern nsMotorcycle

+ (=nMotorcycle|=nMotorbike)

+ =nBike {score=MACRO_ MEDIUM_SCORE}

…

end properties SubpatternsAreNLBricks // End of NL brick zone

In the above example, the word “motorcycles” in the natural language handle “? How much do motorcycles cost?” will be replaced by = nsMotorcycle, but the word “bike” in the natural language handle “? Can I ride my bike to work?” will not be replaced by =nsMotorcycle as “bike” does not match at maximum strength in the subpattern. Therefore, if the user types “How much do bikes cost?” it will match on “? How much do motorcycles cost?” (albeit with a lower score), but if the user types “Can I ride my motorcycle to work?” it will not match on “? Can I ride my bike to work?”

Example:

begin properties SubpatternsAreNLBricks

subpattern nsNewYorkCity

+ (new york [city]|manhattan)

+ nyc {spellcorrect=no}

return “New York City”

end properties SubpatternsAreNLBricks

subpattern ACity

+ CITY=NewYorkCity

+ CITY=SanFrancisco

+ CITY=Chicago

return CITY

In the above example, the =ACity subpattern should not be an NL-brick because you would not want “New York City” in a natural language handle such as “? What are some good restaurants in New York City?” to be replaced by =ACity (if it were, the user query “What are some good restaurants in Chicago?” would match on “? What are some good restaurants in New York City?”)

In general, specifying some subpatterns as NL-bricks and leaving others as non-NL-bricks (the default behavior) allows you great control over how subpatterns are used in your matching.

Binding Subpatterns to Variables

Bind subpatterns to variables whenever you need to use the value returned by a subpattern, either as a parameter in a data request, to store interesting user data, or simply to make the agent’s response “smarter.”

Example:

- Where do you live?

? I live in CITY=ACity

? CITY=ACity

STATE = GetStateForCity(CITY)

USER_LOCATION = CITY

- Wow! I love CITY! That’s in the state of STATE, right?

User: I live in nyc

Agent: Wow! I love New York City! That’s in the state of New York, right?

Best Practices for Developing a Windows Live Agent: Part 1

imiluk — Fri, 08 Aug 2008 21:49:00 GMT

Today we will start a series on best practices for developing a Windows Live Agent (WLA). Here we go:

Use the WLATemplate Project as a Starting Point

The WLATemplate Project (“C:\Program Files\Colloquis\Colloquis SDK\Projects\WLATemplate“) is a generic project you can quickly customize into your own Agent. Simply launch the instantiation script “StartInstantiation.bat” and follow the instructions to start a new Agent in the right directory.

Use Natural Language Handles

The fastest and most efficient way to get your Agent to recognize User questions is to add Natural Language Handles. NL Handles are analyzed against the libraries of rephrase rules and synonym subpatterns defined in your project and in the BuddyScriptLib packages (included in the SDK) and are transformed into regular “+” style matching patterns.

Example:

This natural language handle:

? What can you do?

is rephrased as:

? What do you do?

and “patternized” into these subpatterns:

+ =qsWhat =vDo =You =vDo

This user query:

Tell me what you do

is rephrased as:

What do you do?

Case and punctuation are stripped from queries, so:

what do you do

matches perfectly on the patternized natural language in the agent.

Natural language handles should be:

Concise—ideally 15 words or less and one sentence
Unambiguous
Grammatically correct and properly punctuated (including question marks or periods)
Free of spelling mistakes