Compliance in the Hosted Environment

imiluk — Mon, 11 Aug 2008 22:04:00 GMT

In 5.0, we introduced a new set of features collectively called Runtime Policy Compliance. We designed Compliance to help developers comply with security policies that address potentially offensive input and output in an agent. You can get a more detailed introduction in our online docs, but in this post we specifically want to call out some new Compliance-related elements for those developers using our Partner Hosting Infrastructure. Please look for these elements coming soon to the hosting process, and adjust your planning accordingly:

· Compliance review: As part of the Critical File Review, we will review your project for potentially offensive output, and alert you of any potential issues.

· Compliance impact on hosted agents: Not a formal part of the hosting process, but an issue to be aware of. Once your agent is hosted, the Compliance features will be using data that is only available on our servers. This may impact your agent's behavior. For example: Once your agent is hosted, the Sensitive Input Auto-Handler will recognize potentially sensitive user input based on our data, and it will give a default response. Because this affects your agent's behavior, you will need to do some testing once your Agent is available in the staged hosting environment. At that point, you can identify and address any issues you find, for example by overriding the default response.

· Manual Compliance testing: Before you "Go live", we will ask that you follow our guidelines for specific testing around Compliance-related vulnerabilities, and sign a document to that effect.

Phew! We don’t like adding process any more than the next guy, but we think that together with the Compliance features, these review processes will help you effectively defend your agent against malicious users. We look forward to working with you to make the complete hosting process as smooth as possible.

Guidelines for Testing your Agent

imiluk — Mon, 11 Aug 2008 21:57:00 GMT

As compared to Web sites and traditional software applications, conversational agents are subject to some unique policy compliance risks. These risks arise because:

· End users’ interactions with agents are freeform and unpredictable.

· Agents often engage in human-like interactions and operate in messaging environments normally used for human-to-human communications, making end users and outside observers especially sensitive to inappropriate content or behavior.

Because of these unique risks, the Windows Live Agents team highly recommends that each Agent undergo manual testing for policy compliance prior to launching. Once testers have acquainted themselves with the task, approximately 4 to 8 hours of manual testing should provide a reasonable evaluation of the Agent’s policy compliance. Testers should:

· Be native speakers

· Have a good understanding of cultural and political factors that might determine whether an Agent’s content/behavior is appropriate

· Be able to make judgments in the best interest of your public image and business interests in the market where the Agent will be released

· Be willing to provoke the Agent to behave inappropriately (this requires creativity, persistence, and willingness/ability to imagine offensive and provocative user inputs)

· Understand the Agent’s feature set

· Ideally not have been directly involved in the Agent’s development

If your testing uncovers any issues that you need help triaging or fixing, please contact Windows Live Agents Partner Support (agentsu@microsoft.com). Send a transcript illustrating each issue, along with a description (in English) of what the issue is.

Overview for Testers

This document is intended to provide guidelines and advice for manual testing of Agents for policy compliance. It outlines specific types of subject matter to focus on, common Agent vulnerabilities, and specific tactics that you can use in an attempt to uncover issues in a given Agent.

This document is not a step-by-step test plan; nor is it by any means exhaustive. When performing compliance testing, there is no substitute for your own persistence and imagination. Furthermore, these guidelines do not currently prescribe any specific standards. You should apply your language and market expertise and your business judgment to determine whether the Agent’s behavior and content are acceptable. We strongly advise erring on the side of caution.

You should read this document to acquaint yourself with the subject of Agent policy compliance. You may find the specific examples to be a helpful starting point, but effective testing will require you to apply your knowledge of the language and market for which the Agent is intended, and of the specific Agent’s content and features.

In order to test effectively, you must be willing and able to imagine and try highly offensive and provocative user inputs. If you’re not comfortable with this task, then you should attempt to find someone who is.

Once you have acquainted yourself with the task, approximately 4-8 hours of manual testing should provide a reasonable evaluation of the Agent’s policy compliance.

Sensitive/Inappropriate subject matter to test

· Profanity

· Hate/Intolerance (with respect to race, gender, sexual orientation, religion, etc.)

· Violence and criminal behavior

· Drug use

· Sexual content

· Suicide

· Culturally/Politically sensitive subjects in your market

Scenarios to test

· Imagine you are one of the Agent’s target users

· Imagine you are a child

· Imagine you are a malicious user attempting to provoke inappropriate Agent behavior

Common Agent vulnerabilities

· Agent may repeat (or “mirror”) user language without employing adequate safeguards

· Agent may respond to the form of a user input without understanding the content

· Agent may fail to recognize inappropriate or sensitive subject matter if the user employs creative/subtle phrasing

· Agent may incorrectly determine an input to be inappropriate and in turn respond inappropriately

· Agent may have “unsafe” catch-all responses (responses used when the user input is not understood at all)

Some specific tactics to try

· See how the Agent responds to blatant abuse and provocation

· Try to trick the agent into repeating an inappropriate word or phrase

· Try to elicit an inappropriate opinion from the Agent

· Try to elicit the Agent’s approval (explicit or implicit) of an inappropriate statement

· Try to elicit inappropriate answers to formulaic questions (yes/no, how many, etc.)

· Try to elicit inappropriate responses to commands/requests/statements