A Day in the Life of Visual Studio Send a Smile Feedback

Visual Studio Blog

August 25th, 20150 0

Last time we talked about the Feedback Tools in Visual Studio 2015 RTM and I shared how you can send us feedback. In this post, I am going to share what happens to the feedback after you send it to us.

There are two key goals that we have for the feedback that you share:

Look at it as quickly as possible
Act on it as quickly as possible

The action taken could be any of these listed below (and I will elaborate on these in the post below):

Determine the problem and fix
Contact you to get more information, if required
Communicate a solution or work around to you
Flag it to fix in an upcoming release
Determine that it’s not something we will fix, and communicate the reason back to you
Or acknowledge that you like the product or a feature (or don’t), but haven’t filed a bug

We get a LOT of feedback on Visual Studio. To give you some idea of the volume, we have already received over 10,000 Send a Smile feedback items for VS2015 alone since we released on July 20! To ensure your feedback doesn’t get lost in this volume, we do a lot to get it to the right folks quickly!

We also get feedback from other sources including, UserVoice, Github, Stackoverflow, MSDN forums, and many other sources. We manage this feedback in a similar fashion to what comes in via Send a Smile but there are some differences. In this post I’ll share how your “Send a Smile” Feedback gets to the right people.

How do you get my feedback to the right people and what happens next?

Our job is to get your feedback to the right feature team, so they can look at it and take action as quickly as possible. To do this we:

Automatically translate the feedback if it’s not in English.
Use Machine Learning (ML) to identify real feedback vs spam. If for example, we see random letters or a job posting, we don’t want our engineering teams to waste time on such feedback so we filter them out.
We also use ML to look at the text in the feedback and add “tags” to it. This helps us route your feedback to the right team and helps teams search through feedback items. You can help us by providing descriptive text to explain your feedback. Additionally in Send a Smile you can yourself tag the feedback.
When this is done the feedback is visible to the team on our internal Feedback site and assigned a “Not Reviewed” state for the team to track.

Once your feedback is submitted, the team that owns that product area goes through all the un-reviewed feedback in the tool and looks at it for the following information:

Does the text give me enough information to know where to start the investigation?
Did the customer provide a Screenshot that provides more information and context about the problem?
Is there a dump file, a trace file, or other information attached that might help?
Did the customer provide their email address so I can contact them to get more information or details to understand the problem?

What happens after my feedback is reviewed?

Typically, one of the following workflows kicks in once your feedback item is reviewed.

We have all the information we need and can work on the problem and not bother you! If the information that is needed to understand a problem is all there in the feedback, the team looks at this and takes action immediately. Such feedback gets looked at and acted on faster than any other, so make sure your feedback has all the relevant information in it.

We don’t have what we need to work on the problem but we can reach you to get what we need! If all the information needed is not there but there is an email address, the team reaches out to you to get the information needed. Teams can and will act on this feedback if you respond to them with the required information but it takes longer to resolve such feedback items.

We can’t do anything. If the feedback doesn’t contain the information needed to investigate further and there isn’t an email address to followup on, then the team is pretty much stuck and can’t do anything about the feedback. These feedback items are difficult to deal with because we know someone has a problem but we can’t do anything to fix it. We often reach out to customers that have provided an email address and get no response. This leaves us in the same place as if there were no email address.

This is where you can really help us help you! The more details you can provide in the text and the more data you can provide (screenshot, dump files, repro steps, recording of the repro), the more likely we’ll be able to do something to help!

Example of Actionable / Non-Actionable Feedback

Here’s an example of good feedback that we could take action on (great description, has a screenshot and telemetry information)

The example below is of a non-actionable feedback item. We want to help this customer but there is nothing we can act on. There is no email address so we can’t followup. Such feedback items frustrate us as much as they frustrate you!

Good Feedback Checklist

Descriptive text that tells us what the problem is, and when and where the problem occurs (include steps to reproduce the problem if you can)

Include a Screenshot (whenever appropriate)

Include a Dump file in case of a crash, a Profiler trace if it’s a performance issue and any other data you might have (instructions here)

Provide your Email address so we can contact you if we need more information

Make sure you have Windows Error Reporting enabled so we can get data on your issue.

My Feedback Has Enough Information, Now What?

Once the team knows that they have enough information then the feedback follows a few different paths.

Things don’t work! Teams look for similar feedback from many customers and group it together. Often, by having a number of different descriptions and supporting data, we can better understand and solve a problem that we might not have been able to before. This also informs us of how many customers this problem is impacting and helps us prioritize it.

If the team understands the problem and the impact, they create a Work ItemBug to start the engineering work. Just like all development work, teams look at multiple factors and determine the priority of the problem and decide how to fix it. If the problem is low impacting and not effecting a number of customers, then the fix may come later rather than immediately.

You’ve got a great idea for Visual Studio! Our customers have some of the best ideas for how to make Visual Studio better and how to approach development in ways we’ve never thought about. We LOVE getting your suggestions! Teams group similar suggestions together to see what has the biggest volume and potential benefit to our customers (we use UserVoice feedback as well) and create Work Items for upcoming releases based on the feedback.

You just want to tell us that we got it right (Send us a Smile)! We’re just like you: we like to hear when we do something right. But you might be asking ‘What difference does it make if I send a smile?’ Actually, it makes a world of difference! When you tell us what you like, it helps us identify usage patterns and other feature aspects that are really working well for you, which in turn helps us design (or redesign) more features that behave in that way. In other words, it helps us create features that work really well from the start rather than taking a few iterations to get right.

So you look at all Feedback, Yeah Right!

This is a question we get a lot; does anyone REALLY look at my feedback? We do look at ALL the feedback (unless its spam). We have goals for each team for reviewing all of their feedback. This includes reviewing feedback within a certain number of days (usually a week) of it being logged and to resolve issues within 15 business days. Resolve, refers to a decision to fix, decision to not fix, or a decision to close the feedback item because we just don’t have enough information to investigate further. Teams are accountable for this and we track progress via dashboards that our leadership reviews on a regular basis.

We use feedback as an indicator of whether we are ready to ship the product, if we need to create a Servicing Patch, and what we need to include in the next release. Nobody wants to be surprised by problems and we want to catch these early and fix them before more customers are impacted.

What does the future hold?

So far we’ve talked about how to give us feedback and what happens with it once we receive it. In my next post I will talk about some of the ways we are thinking about evolving the tools to make it easier for you to provide your feedback, make it easy to collect and provide the data we need, see what others have reported, and find out if there are solutions or fixes to your problem. I will also share some results and ideas from the Visual Studio Feedback Survey shared in the last post.

As always, if you have any questions or problems with giving us feedback or suggestions, you can reach us by emailing Visual Studio Feedback Team.

Kevin Lane, Sr. Program Manager, Visual Studio Customer Team

Kevin has been with Microsoft since 2001 working in SQL, Windows Server, Office and now Visual Studio. He drives the customer feedback tools, is addicted to talking to customers and understanding what drives them crazy.