We > Me: Joe Morel's Blog : Reputation

Traveling to Online Community Unconference 2007

joemorel — Tue, 05 Jun 2007 20:36:48 GMT

Tomorrow I will be attending the Online Community Unconference in Mountain View, CA. It's been a little bit over a year since the last time I attended this conference, and I'm looking forward to seeing whether or not people's focus has changed over the past year. Last year most of the topics revolved around two themes:

How do I measure my ROI (return on investment) in my online community so I can justify my existence to my management?
How do I design a reputation system in my online community that makes my community more "sticky" but not necessarily ultra-competitive?

I was very interested in reputation a year ago, and it remains something that I think is of paramount important. As a corollary to that, I'm also very interested how to give the community the power to solve their own disputes and quarrels. I've recently been in the middle of a few online mud-slinging sessions. They are ugly and take away from the focus of the community. The way they are currently "resolved" is through emails directly to me. Ugh--that breaks down pretty quickly...I can't respond to all of the mails, it's not really my business, and frankly, I'm not going to be the point contact for the MSDN Forums forever.

From this conference, I'm going to try to get some "best practices" from other community managers on how they deal with this all-too-common occurrence. If anybody else reading is attending and wants to chat, get a beer, dinner, or whatever, let me know!

And, now, off to SeaTac!

The Reputation System We're Actually Working On...

joemorel — Wed, 23 May 2007 02:37:00 GMT

Over the past year, I've done quite a bit of blogging on reputation and got quite a bit of feedback back from you as a community. Well...I've got good news and I've got bad news. The good news is that we're actually working on a reputation system. The bad news--it's "step one", and fairly basic.

Below is a version of the spec that I'm working on implementing, minus the boiled down requirements list. Any feedback? Comment away...

Microsoft Forums 2.3: User Point System

Summary

The current Microsoft Forums use an answer count-based system that only rewards high volume answer contributors, but doesn’t reward users that contribute high quality, helpful answers, or other important actions. This specification describes a basic point-based reputation system that achieves the above goals in a more comprehensive manner than is currently implemented. The basic reputation system described here uses an event-driven model to award or deduct points in particular situations.

These point values will be displayed in a user’s profile, along with a basic visual representation (star rating) that makes the user’s current rank apparent to novice users. These points will also be used in place of answer count in the current “Top Answerer” lists on the forum site homepages, individual forum pages, and the “Hall of Fame” page.

User Events and Point Values

The basic reputation enhancements require the events below to be tracked and awarded with a configurable number of points. The values below are suggested default values. The events will be tracked in a table in the forums database, with each record containing the following information:

·EventType

·DateTime

·UserID

·SiteID

·ForumID

·Points Awarded

Events (Points it's worth by default)

User replies to a question/bug thread that they did not start (1)
A user reply to a question thread that they did not start is marked as an answer (5)
User receives a helpful vote for a reply that they posted (5 x (# of votes))
User marks a reply as an answer (1)
User has a reply that they marked as an answer unmarked as an answer (-5)

The point values should be:

1. Calculated per user,

2. Scoped to forum site and,

3. Scoped to only events that have occurred in the past year.

For example, a user that answered a question on the MSDN forum site would not get points for their TechNet reputation. Furthermore, a user that answered a question 18 months ago on the MSDN site would no longer get points for that answer. For the edge case where a forum is displayed on multiple sites, the user should get points on both sites. This should be handled by creating multiple separate records for each event that occurs on a forum that is displayed on multiple forum sites.

The point values should be cached whenever possible and only need to be refreshed every 4/8 hours to reduce the load on the live forum site.

Display of Point Values on the Forums

The point values will be displayed and utilized in multiple forums on the forums, but will only be used in areas where post counts or answer counts are currently used.

Hall of Fame Page

The Hall of Fame page should be updated from number of answers to number of points.

User Avatar Areas

The current “user avatar areas” contain the user’s name, whether or not they are a moderator and/or an MVP, and the raw number of posts that they have made in the Microsoft Forums. The only changes will be that the raw number of posts made in the forums should be replaced with the number of points they have based on the current forum site that is being viewed, and the addition of a simple star rating icon that will contain zero to five stars, directly based upon star rating.

Figure 1 - Current user avatar area features only number of Posts

Figure 2 - Enhanced user avatar area includes star rating and the number of points.

The star rating thresholds should be configurable per site, but the suggested point values are:

Star Rating (number of points)

0: (0 – 9)
1: (10 - 99)
2: (100 – 499)
3: (500 - 999)
4: (1,000 – 4,999)
5: (5,000+)

Some of the top moderators will pass well past these thresholds on certain high-volume sites (such as MSDN) and will need to be adjusted upwards. This should be done immediately upon rollout of the reputation system. Changing the thresholds after the reputation system has been rolled out will cause problems in the forums community.

Who's Rated the "Most Helpful" in the forums?

joemorel — Thu, 12 Apr 2007 02:44:00 GMT

A little bit ago, I started talking about ways that we could measure the success of the community without using pure "answer count" as the only barometer for community health. I decided to do a little bit of data mining today and looked for the people who have received the highest number of "Helpful Votes" over the past three months. Below that is the chart of the top answerers over the same time period.

Top Helpful Rated List:

User Name	Total Votes	Who is it?
nobugz	404	MVP
Ilya Tumanov	202	Microsoft
DMan1	192	MVP
Shawn Hargreaves - MSFT	184	Microsoft
Figo Fei - MSFT	168	Microsoft Support
Cindy Meister	164	MVP
Zhi-Xin Ye - MSFT	147	Microsoft Support
einaros	139	Community
Jay_Vora_b4843e	135	Community
Jens K. Suessmeyer	132	MVP
Bruno Yu - MSFT	129	Microsoft Support
Jim Perry	125	MVP
Arnie Rowland	124	Community
Phil Brammer	122	MVP
TilakGopi	122	Community
Mike Danes	114	MVP
Dave299	113	Community
ahmedilyas	111	Community
Bite Qiu - MSFT	106	Microsoft Support
Andreas Johansson	106	MVP
Dick Donny	101	Community
Peter Ritchie	90	MVP
ReneeC	90	Community
Richard Berg MSFT	88	Microsoft
nogChoco	85	Community

Top Answerers List:

User Name	Total Answers	Who is It?
nobugz	886	MVP
Bruno Yu - MSFT	682	Microsoft Support
Figo Fei - MSFT	672	Microsoft Support
Zhi-Xin Ye - MSFT	501	Microsoft Support
DMan1	444	MVP
Ilya Tumanov	432	Microsoft
Cindy Meister	405	MVP
Arnie Rowland	382	Community
Bite Qiu - MSFT	361	Microsoft Support
Andreas Johansson	339	MVP
ahmedilyas	307	Community
James Manning - MSFT	281	Microsoft
LesterLobo - MSFT	272	Microsoft
Tom Lake - MSFT	268	Microsoft
ReneeC	264	Community
Jens K. Suessmeyer	261	MVP
Richard Berg MSFT	259	Microsoft
Damien Watkins - MSFT	254	Microsoft
einaros	246	Community
TilakGopi	238	Community
Dick Donny	234	Community
Gavin Jin - MSFT	232	Microsoft Support
Simple Samples	212	Community
Shawn Hargreaves - MSFT	198	Microsoft
Bob zhu - MSFT	187	Microsoft Support

Sounding Off: Feedback from the Forum Moderators

joemorel — Fri, 06 Apr 2007 01:02:00 GMT

Last Edited: 4/6/2007 [I'll be amending the lists in this post based on the comments and continual feedback that comes in. Thanks for hanging with me here.]

We've recently released a service pack for the forums that have changed some functionality in the forums. It's the first batch of noticeable, user-facing changes in the forums in over a year, so it's not surprising that some of the changes have caused some problems. These problems, however annoying, have also spurred some very interesting threads in the MSDN Moderators Forum, a private forum that's the "coffee lounge" for our community moderators to hang out and discuss issues in the forums.

I thought it might be interesting to summarize some of the discussion here and see if we can spur some more constructive input. I'm going to summarize what I've been hearing into three sections. For the "bad" and "ideas" categories, I'm going to try and break them out into individual blog entries.

The Good

Code Snippet Blocks - I'm glad that there's something in the forums now that will protect my code from being "emoticon-ized"...woohoo!
Graphs/Stats of Forums Progress - Josh shared some internal information about forums on his blog recently...it's information that I see everyday, but the community doesn't get to. I got the feedback that we should share it more often.
???

The Bad

Transparency about Support - We've recently started having a new product support team answering questions on the forums, with the goal of getting the answer rates up to "healthy" levels. We really haven't been good at communicating this to any of the moderators in the forums. As a result, there's a huge amount of confusion as to the role of the support people compared to the role of the moderators in the forums. It's a problem, especially because the support technicians aren't aware of the culture and process that the moderators have painstakingly created over the past two years.
Can't Find the Answer Box - This one has actually been brought up both internally and by the community. The "Can't Find the Answer?" box that's next to a forum isn't time scoped. Because these questions are essentially the most viewed questions in the forum, they are old and outdated. The easy resolution--only display questions asked in the past three months here. Of course, there are posts that are older than three months that could still be very useful to people. What's the best solution for the most people?
Top Answerer List - The "Top Answerer" list on the forums doesn't measure quality. (I blogged a bit about this yesterday.) It drives people to answer too quickly, and people can inflate their stats easily by just doing housekeeping stuff--not by answering difficult questions. If reputation is important to a community, then this is not the right solution.
Code Snippet Quirkiness - The code snippet handling just isn't where it should be in a technical support forum. Emoticons render still in one-liner snippets that aren't in a code snippet boxes. The general quote here would be: "C'mon Microsoft! These are developer forums...get it together!"
No Permalinking to Individual Posts in a Thread - There's no way to link to an individual post in a thread--it makes it hard to link to a particular answer. (Note: there are indexes available on the page that can be used.)
RSS Feeds Keep Sticky Posts at the Top of the Feed - And they don't currently include replies. These should both be fixed in "SP4", which should be released at some point in April.

The Ideas

Mark as Resolved with a Reason - Instead of the "Mark as Answer" button, why not make it "Mark as Resolved" and then offer a list of reasons that a user could pick from. "Mark as Answer" is being used even when the question isn't truly answered...just to drive up answer rates or get the "Top Answerer" credit. (Personally, I love this idea.)
Label the Moderators - Who is a moderator on the forums? Right now, I'm not sure. Almost all forum platforms have moderator labeling, but ours doesn't. Let's label the moderators.
Rewards? Recognition? - We really haven't done much beyond the "Top Answerers" list in the forums for recognition or reputation. We need to start moving in that direction...today. (And yes, I've been pushing for this for over a year now. Want proof? Check out my blog category about reputation.)
Are the Metrics Right? - Once again...I blogged about this yesterday. The answer rate metrics are great...they are measurable, and in general, a high answer rate really does mean that people are getting answers to their questions in that forum. But is that really the *best* way to measure things?
Volume *Might* Cause Problems - I was recently very happy to see the 1,000,000th post in the forums. Of course, I also have the theory that more eyeballs = more posts = more answers, and everything scales very well. This isn't necessarily so. The moderators are starting to get overwhelmed with the number of questions coming in...we should be careful on how quickly we drive more and more traffic to the forums.

Let's talk and make a list. What's good/bad/or just a good idea?

Democracy for Online Communities

joemorel — Sat, 13 Jan 2007 01:47:01 GMT

When I first started with the DevDiv Customer Connection team, Josh gave me the thirty-second elevator speech about his goals for the forums. The phrase "self-sustaining community" was used more than once. The idea was that just creating the forums or getting Microsofties into the forums would make them a true success. The forums would be only be successful when Josh and I could both get hit by a Microsoft shuttle and the forums would still not miss a beat. The answer rate would remain the same, and the forums could continue to grow, evolve, and operate with our intervention.

The forums aren't there yet. I've been acting as benevolent dictator in the forums for over a year now—creating new forums, monitoring answer rates, sending "Community Mail" updates to the division, and moderating disputes between moderators. The community has grown and evolved, but I'm still not confident that we have the proper mechanisms in place to allow the forums to be self-sustaining. That's because for the forums to be self-sustaining, the forums need to be democratic.

The top community members involved in the forums are the moderators, and they do a great job bringing issues to my attention and ensuring that I'm staying on top of issues in the forums, but they themselves are unable to make any changes. I'd love to see a system where the moderators are able to make decisions autonomously, especially in the following scenarios. What do you think? Would this work, or would it turn into a bad episode of Survivor? Can true democracy work for an online community?

Creating a New Forum – The only way to make a new forum right now is through a little applet that sits on my desktop. Typically, a product team has to send me an email and ask for a new forum to be created. This is bad—there are dozens of opportunities for new forums that would help developers right now that the moderators are likely aware of, although I'm not. Why not let moderators propose a new forum? If 25% of the active moderators confirmed that they could help moderate the new forum, the new forum request would automatically go through and be made on the site.

Monitoring Answer Rates – I send a biweekly mail out to Developer Division that contains how every team is doing in their community efforts, including the 2 and 7 day answer rates on each team's forums. The purpose is to be a not-so-subtle reminder—please remember to pay attention to your forums and help out your customers. The community should have access to the same information. What forums are unhealthy and need more attention? Which are doing great? Which ones need more Microsoft help? The moderators should be able to ping Microsoft and request more participation in certain forums were the answer rates are below the ideal.

Adding New Moderators – We are also looking for good community members who are answering a decent volume of questions with thoughtful answers—they are usually our next moderators. Currently, we rely on other moderators to send mail and nominate other users to be moderators. Once again, let's take out the middleman. Any moderator could nominate a user to be a moderator. If 10 moderators voted positively (and less than 2 voted against), the user would automatically join the ranks of moderators.

Removing Moderation Rights – Unfortunately, even moderators sometimes need checks and balances, and it might be necessary to remove moderator rights from another user. Once again, a moderator could nominate somebody to lose their rights, and if ten people vote in the affirmative (and less than 2 in the negative), then that user would be demoted.

Banning a Spamming User – Spam users should be dealt with as quickly as possible. Unfortunately, that sometimes involves sending me email asking me to ban somebody. The moderators should be able to nominate and vote to ban a user from the forums for spamming.

Nominate a User for a Commendation – We don't really have any great awards in the forums now, but wouldn't it be great if the moderators could nominate exceptional users for "Most Helpful Answerer" awards? It could appear as a small avatar next to their names. Once again, the moderators could nominate and vote.

Any other ideas?

Finding the Good Stuff on the Forums

joemorel — Thu, 01 Jun 2006 02:13:00 GMT

I recently had the opportunity to attend the Online Community Camp run by Forum One Communications in San Francisco this past week, a gathering of people throughout the industry that were involved in some way connecting people using the power the Internet. Some of the conference attendees were “big” companies like Microsoft, Apple, Cisco, SAP, who were thinking about community in the same way that I am: basically using the collaborative power of an online community to help people get fast and accurate answers to their questions. Other people’s organizations were based primarily around online community interactions—it was absolutely fascinating to listen to the stories that some of these people had to share about the way they were connecting people around the world with similar interests.

There were three breakout sessions that I was able to attend, and thankfully, the first breakout session of the day was about reputation in online communities. Right up my alley, right? There were a few key points that people shared, such as to remember that reputation systems are not one-size-fits-all (some communities do not lend themselves well to competition between users), and to avoid reputation systems based purely on volume (something I touched on a bit ago.)

The most interesting point brought up was that reputation systems actually have two main focuses: they can judge a user’s reputation (which is what I’ve been focusing on) or they can judge the reputation of content. Wow...completely obvious, but it’s something that I’ve nearly entirely looked over, with the exception of one lowly post were I talked about post quality.) So, how can we judge the reputation of posts, namely answers, on the MSDN Forums?

One core concept about rating post content is that people simply don’t “vote” for content without an incentive. We currently have “Helpful/Not Helpful” buttons on every post on the forums—and they are clicked at about one-third of the rate of the “Mark as Answer” buttons. Why? There’s no reason to mark something as helpful. Nothing appears to happen. Yes, we take those votes and use them to help with search on the backend, but an average user has no reason to click. So, any post rating system actually asking a user to separately “rate” the quality of a post is likely to not get a ton of involvement.

The idea that Josh proposed today during a thread with another team at Microsoft thinking about the same problem was to build the post rating directly into the answer marking. Instead of a “Mark as Answer” button, there could be a drop-down for Mark as Answer with the following options:

Not an Answer
Maybe an Answer
Answer
Good Answer
Excellent Answer

The polled and averaged results of all of the votes from every registered user on the site could be compiled for an overall “answer score” for the post, that would be exposed on the post itself. Every thread containing a reply post with an average above 3 would be considered an “answered” question. Each answer score would roll up directly to a user’s reputation—the higher rated their answers, the higher rated the contributor.

This answer score could be combined with the number of raw page views on an individual thread to help determine what posts are the most popular and good. A post with a high answer rating and a high number of page views is likely going to be something that people are going to want to read…what do you think?

New Forums Features!

joemorel — Sat, 20 May 2006 01:37:00 GMT

Josh scooped me to the story, but yesterday the Microsoft.com team released a service pack for the forums that deploys a few new features and fixes. Check out his blog post for the full run down.

My favorite new feature, at least as far as taking baby steps towards a real reputation system, is the Top Answerers page. It's a quick way to find out who the top people have been on the forums in the past 30 days. I had no idea our new forums rock star is a user named "spotty". 274 answers in 30 days--rock on!

The Top Answerers stuff has also been expanded to include a "Top Ten Answers in the Past 30 days" box on every question list page for a forum. Now, if you in a particular forum, you can scroll to the bottom of the page and see who the top answerers are for that forum.

Thanks to the team over at Microsoft.com for making the MSDN Forums experience even better!

Forums Reputation Phase 3 – Profiles

joemorel — Tue, 25 Apr 2006 03:04:00 GMT

I used to think that rich profiles were dumb—that they were really only something that were only used by teenagers hell-bent on getting the latest and greatest AIM virus. Then I saw the Xbox.com Forums. Wow. Now those were great profiles. Why? The profiles mattered. It was more than just a bunch of stuff somebody else typed—profiles were the home base of reputation. There are two very powerful elements that make these profiles so interesting:

1.) Recent Activity – In Xbox Live land, this is really more about your recent game activity. Little glyphs are next to my profile showing the past games I’ve played (Hexic HD, Marble Blast Ultra, and Bankshot Billards 2, if you must know.) Clicking on those glyphs leads you to a richer view that actually shows a user’s achievements in that individual game. Expand this to Microsoft communities—a community profile has glyphs indicating the past few communities you’ve participated in. Click on any one of the glyphs and it automatically brings up a page with your personal reputation and roles in that community, along with your recent activity (posts, bug submissions, wiki changes, etc.)

2.) Community Score – In every way, shape, and form, the community score is visible in both the mini-profile that shows up next to somebody’s post, and their richer full-page profile view. Whenever you read a post, you know a user’s gamer score (how many achievements they’ve made in games), community forum ranking (how “good” they are on the forums), and gamer “reputation” (how well other people liked playing with them.) Not only does this let other people know quite a bit about who they are talking to on the forums, but it also gives more for a user to care about all of those numbers. For example, if reputation exists for other community channels at Microsoft (say, product bug submission websites, wikis, chat, and blog), there could be a rolled up “Microsoft Community Score” displayed, alongside their MSDN Forums reputation.

3.) Comparison – It’s great to know that you have 1,200 points on the forums, but what does that mean if you don’t know where you stand against others on the forums? Clicking on another user’s profile should bring up a page that shows a side-by-side comparison of your reputation with theirs. In addition, you should be able to quickly see your ranking on the forums, along with a list of the people who are above and below you. For example, you want a “Top 100” icon next to your name, and find out that you’re currently in 103^rd place…but if you answer just 14 more questions this week, you could move into 98^th place. It’s probably enough incentive to answer those extra questions!

4.) Avatars – I know. Avatars can be very…well…lame. In fact, the avatars are my least favorite part of the Xbox profile system, but mainly because I think the pictures are fairly lame and I’m not going to pay any extra money to change the little picture next to my name. Of course, if those avatars actually helped you find experts in a technology area, they’d probably make more sense, right? You know those questions you answer when you register a new product—the ones that ask what technologies you are interested in and what year you were born in? Why not tie the results of those questions into a custom avatar on the forums? Enjoy C#? You get a C# icon next to your name. Feel like you’re an expert on the Visual Studio IDE? You get the cool little Visual Studio infinity icon. You get the picture.

5.) Make it and Take it – People spend quite a bit of time in the community trying to build this reputation, but it really only is visible when you’re on the forums site itself. Why not take it with you? The “mini profile” displayed next to your posts should be available as a web part that you can plug into your personal or professional webpage. What better way of showing potential customers that you are passionate about technology than displaying your contributions to the Microsoft community?

PageRank for Answers?

joemorel — Tue, 18 Apr 2006 03:25:00 GMT

A few of the comments I've gotten about my reputation system are telling me that I'm on the right track, but I'm really not doing enough to measure answer quality, not just quantity. Recently, the community lead for the C# team told me that she had been finding some of their newer moderators (who are excellent) by using the "Most Helpful Rated" column--not the "Top Answerers" column that I evangelize so much.

It's really too easy to just pick off low-hanging fruit and marking simple answers as answers. If somebody posts something that's truly great in the forums, shouldn't they get more credit than a random Microsoftie that closes a thread with a link to the newsgroups and marks that as an answer?

Of course, the logical answer is "Of course", but it's just not that simple. It adds a layer of moderation on top of nearly everything in a system that's already fairly taxing for moderators (for the answer marking to work well, you really need moderators that care about answer marking.) So, how do you judge answer quality without asking people to do extra moderation?

One idea I've toyed around with was using a combination of page view counts and "trackbacks" on a given post. The former we already measure--every time somebody looks at a post on the forums it increments a little counter. Sure enough, the best, most-helpful posts have the highest view counts by far. The latter--"trackbacks" to posts, might not be entirely feasible with our current forum implementation, but the idea is simple (and the key to Google's billions)--the more people that link to your post, the more likely it is that your post was useful and of high-quality.

This is some fairly half-baked brainstorming, and I'm very aware of some of the holes in the system--it's easily gamed, there's no real assurance that a given post isn't just being read quite a bit because it appears helpful until you read it, and so on, but I'd really love to brainstorm with everyone...what do you think?

How can you gage answer quality without asking a moderator to do it?

Forums Reputation Phase 3 - Community Ownership

joemorel — Thu, 13 Apr 2006 03:27:00 GMT

On for the next part of phase 3 of the forums reputation system...

It's not Microsoft's community...

…it’s yours. So, as a community, it should be possible to do all of the same things I’m able to do in my site administration tool. Through a voting process available only to users with a certain level of reputation or higher, it should be possible to ban users, approve users for new levels of moderation rights, nominate MVPs, and even recommend new forums for creation. If the forums and other community channels are built for the customers, why be afraid to give up control to the customers? If five or more people recommend that a user should be banned for spamming within 12 hours—shouldn’t they be banned? If a group of MVPs think that there’s a member of the forums that should really be an MVP—why shouldn’t they be able to nominate users themselves?

Forums Reputation Phase 3 – Expansion

joemorel — Wed, 12 Apr 2006 01:47:00 GMT

Over the past month I’ve blogged a ton about my “crawl, walk, run” plan for a reputation system in the Microsoft Forums. Honestly, I’ve been putting off writing this “final” phase, because it’s pretty tough to write a vision for the do-all and end-all of online reputation systems. J So, let’s not call this “final”…let’s call these “guideposts to the future.” In addition, this post is much, much longer than I originally thought it would be, so I’m going to divide it up into subject areas, each with their own posts. I’m also going to create a “reputation system” tag on my blog, and tag each reputation focused post. Today I'll start with "Expanding Your Microsoft Reputation."

The more I’ve thought and wrote about reputation systems, the more I’ve decided that reputation is 100% linked with broad community moderation and making the Forums something that the community is able to take ownership of—a true self-sustaining community. (To read a great post about broad community moderation, check out Josh’s recent post.)

Expanding Your Microsoft Reputation

Why should your reputation only be visible in the forums? Your reputation should always be scoped to your current context (if you are participating in the English MSDN Forums, your reputation score that is displayed should only be your reputation from the English MSDN site), but that doesn’t mean that your accomplishments on other forums sitees—and other Microsoft communities—shouldn’t be shown. Are you a top bug submitter on the MSDN Product Feedback Center? Are you also an Excel expert? Involved in a beta program? Your community profile should allow anyone to find out more about you and what technologies and communities you are passionate about, no matter what Microsoft community they run into you in.

Answer Marking: It's Not Just For Moderators Anymore

joemorel — Sat, 08 Apr 2006 00:49:00 GMT

I've been writing quite a bit about reputation and auto-moderation on the MSDN Forums on this blog. It's been a cool way for me to organize my thoughts and collaborate a bit with other people on what we be great for the forums in the future--but it's basically just been a bunch of writing.

Until now. I'm happy to announce one of the very, very first ideas we had about auto-moderation has finally seen the light of day. Every single forums user that has contributed 5 or more answers in the MSDN Forums, as of the beginning of April, now has rights to mark replies as answers.

This is an experiment to see if more replies actually get marked if more people are able to mark them. Will this help our answer rate? I hope so. I'll check back in next month and see if this has had a noticeable effect.

Today's 2 day answer rate: 41%

Today's 1 month answer rate: 58%

Microsoft Forums Reputation System - Phase 2

joemorel — Tue, 28 Feb 2006 04:09:00 GMT

A couple of posts ago, I covered my vision for the first phase of a reputation system for the Microsoft forums. Now it’s time to build a case for Phase 2 (I feel a bit like I’m building a subdivision here…) Phase 1 of the reputation was focused on incenting and rewarding top answerers. Phase 2 of the reputation system aims at helping another group of key users of the forums--the question-askers.

A few months back I did an informal survey of some of the top answerers on the MSDN Forums site. One of the questions I asked was "Why do you answer questions on the forums for free?" I was expecting answers such as "respect", "advertising my consulting business", etc. What I got was completely different. People said that they learned by answering interesting questions on the forums. The best way to get more answers was really to get more interesting questions posted.

Of course, some of the questions on the MSDN Forums are, admittedly, not very interesting. Many newbie posters will post something like "Help! I am using Visual Studio and can't run my program." Listen, I don't care what your third-grade teacher said--there are such things as stupid questions. :) Questions like that clog some of the forums, keeping our top answerers from wanting to answer questions there. In addition, new questioners are the ones most likely to post off-topic. It's not their fault--they aren't familiar with the structure of our site yet, but you'll often see databinding questions in the "Visual Studio General" forum, or questions about GridView in the "Visual Basic Express" forum. This makes it very, very difficult to find a good question to answer, and it keeps our more experienced questioners with the good questions from getting as many eyes looking at their on-topic question.

What do we do? We allow questioners to build reputation. If you're known for writing really good questions, or even known for good answers, you should have more flexibility than a newcomer to the site. So, let's give people some points for question-asking activities. Here's an idea:

Question Asking Points

1 Point for every question asked that was answered

1 Point if you marked a reply to your question as an answer (and it was never unmarked)

1 Additional Point for every question asked that was marked as helpful
-1 Point for every post that was marked "off-topic" and moved to another forum
-3 Points for every post that was so off-topic that it was deleted
-10 Points for every reported post that was approved as abusive

Answering Points

5 Points for every answer given
2 Additional points every time your given answer was marked as helpful

So, "answer count" would be replaced by a synthetic "community score" in phase 2 of the reputation system that took into account more of the behaviors on the forums that we wanted to encourage--good question asking, helpful posting, answering...and discouraged the bad behavior--off-topic posting, abusive/spam language, etc. So, what would we use this new score to do? Remember--any reputation system needs to be linked back to something "tangible"--a number on a screen doesn't mean much to me, one way or another.

Points Based Rights

10 - able to post in a forum without going through forum selection steps

Newbie posters would need to answer a few questions that helped select an appropriate forum before they posted, and possibly would force new users to search before posting. For example, I would be asked what version of Visual Studio I'm using, what language (C#, C++, VB), and what I'm trying to do (writing code, designing UI, debugging, deploying, etc.) The forums system would then suggest a forum that is on topic for me to post in.

25 - Answer Marking Rights (for others)
50 - Answer Marking Rights (for everybody)
100 - Can Move Posts
etc...based upon the levels in Phase 1
And, of course...
-20 - Nomination for banning from the system, which would go to the site administrator

I toyed around a bit with the idea of having the top users questions actually appear higher on a forum thread list than others, but I really don't like the idea of the forums becoming an exclusive club. A new user should have just as much chance of having their question looked at as a veteran user--but I think it's okay to make them jump through a few hoops while they learn the site, ensuring quality for the older users of the forums.

This whole "rewarding question askers" idea is relatively new--it really stemmed from a conversation I had with Josh last week. As a result, I'd really welcome some feedback. What do you think?

Microsoft Forums Reputation System - Phase 1

joemorel — Fri, 17 Feb 2006 03:57:00 GMT

In my last post about Trust and Reputation, I outlined what I believed were the key elements of reputation building in an online community:

Activity Based
Community Rewarded
Visible
Linked to Additional Privileges
Obvious on How Reputation is Determined
Reputation Takes Time

Since then, I've had a chance to sit down with Jana Carter (blog), the product manager for the Microsoft Forums, discuss my vision for a reputation system on the forums. I see the reputation as having three "phases" of implementation--one that happens in the next few months, the second an ideal view for a year from now, and the third, a final, far-reaching view of what reputation could look like in Microsoft communities in a few years.

Today I'll describe "Phase 1" of the reputation system...the framework which the "cool" and "fun" stuff can be built on in the future.

Most of the first phase of the forums reputation system is focused around "incenting and rewarding top answerers"--encouraging those key members of our forum community to contribute as much as possible. Without further ado--Phase 1:

Special User Glyphs--If you have a special role with Microsoft or on the forums, you should have a glyph next to your name in a thread. These glyphs would identify Microsoft Employees, MVPs (Microsoft Most Valuable Professionals), and Forum Moderators.
Number of Answers Displayed--In the "thread profile" area, we currently display the number of posts you've made. Who cares? If we are about answering questions on the forums, this number should be changed to the number of answers.

Special "Top 50 Answerers" Glyph--If you are one of the top 50 answerers in the past month on the forums site, you should get special recognition. How about a glyph next to your name?

Top Answerer Lists--Up at the top of the page, easy to find, and scoped to whatever context you are currently viewing the forums in. Are you looking at the entire MSDN site? Great! The top answerers list is the top answerers for the entire MSDN site. Just looking at the Visual Basic Express Edition forum? No problem--we've got your list of top answerers for that forum right at the top of the page.

Hall of Fame Page--We've got a member list right now in the forums...but how interesting is a list of 60,000 user names? The Hall of Fame page will allow users to view the list of users--sorted by number of questions answered. If you're number one all time, you get to sit on the top of that list.

Automatic Moderation Rights--This one's a bit more sensitive that the rest. One of the big problems we've had on the forums has been getting more community people to moderate and mark replies as answers. Of course, there is going to be some human-based "checking up" to make sure the system isn't gamed, but here's the plan. If, over one year, you have:

5 Answers--You get the ability to mark other people's replies as answers.
10 Answers--You can mark your own replies as answers as well.
25 Answers--You get the ability to move off-topic threads around on the forums.
50 Answers--You can edit a post.
100+ Answers--Congratulations--you're a community moderator!

Those are the main pillars for the first phase of a reputation system on the forums. Notice that instead of "post rating", which many communities use, I'm choosing to just use the answer marking, which is already part of the culture on the forums.

Any comments or suggestions? Let me know...

Trust and Reputation on the Microsoft Forums - Part 1

joemorel — Fri, 10 Feb 2006 03:56:00 GMT

In my last post, I talked about how difficult it was to try and decide what groups of people you can trust in an online community. Outside of my simple and naive "Why don't we just trust anybody?", most online communities use a "reputation system" to establish trust and order on their sites. Some common examples include:

Ebay's Seller Rating System which uses feedback from every online transaction to create a reputation for each seller.
Amazon.com's Top Reviewer's System
Slashdot's Karma System

Even technical support sites, such as Experts-exchange.com and Apple.com Discussions use reputation systems to give users on their systems some sense of personality and let the community know how "trustworthy" that particular user is. Even Microsoft's own Xbox.com Forums have a rich reputation system. Simply put, the most successful online communities--the ones that people want to come back to time and time again--have reputation systems.

The reputation systems are based around a few common principles:

Activity Based--This one is obvious. People get better reputation by doing something that is considered "good". Selling an iPod on Ebay and actually sending it in good condition. Reviewing a book with thought on Amazon instead of writing a mindless mess. Posting something intelligent on Slashdot. Alright...maybe the last one is less likely... :)
Community Rewarded--The best reputation systems allow other people in the community to reward "reputation points". I get to choose whether or not the seller I bought from on Ebay gets a point. Users on Amazon can choose whether or not a particular review was helpful to them.
Visible--The results of a reputation system must be visible. Ebay has its stars, Amazon has a "Top Reviewers" list, Xbox.com has awesome user profiles/avatars. If a user is acheiving reputation, and nobody can see it, it's not really worth anything.
Linked to Additional Privileges--Once a user acheives a certain level, there should be some sort of reward that makes a user care that they have acheived that level of reputation. Typically, this is more moderation privileges--think of it as becoming a "deputy" of that particular online community.
Obvious on How Reputation is Determined--Some sites do this better than others. Ebay and Amazon have clear, great pages on how you get a better reputation--sell more stuff, or write more reviews. The more difficult it is to determine exactly how you get a better reputation, the less likely it is that you'll behave in that way.
Time--None of the online communities I mentioned above are brand-new. Most have been around for years. It takes years to establish reliable reputations and build a community around them.

So, what am I getting at? Well, I'm priming people for my next few posts, in which I'm going to propose a reputation system that should be used for all of the Microsoft Forums, starting with the MSDN Forums. I'm currently working with the Microsoft.com Forums team on this, and I'd love to get broader feedback from everyone on this. Stay tuned!