 |
Rep systemsFebruary 2nd, 2006 |
Gamasutra -Soapbox – “Designing an MMORPG Feedback Rating System” is an article by David Edery on how we should make use of reputations systems in MMOs.
Dude, I’ve tried…! It’s a wonderful excuse to dump some accumulated knowledge on you.
Here’s some of the things that people don’t tend to realize in working in this space.
Negative feedback reputation systems have been shown mathematically to always spiral into chaos. The reason? People with bad reps just start up a new identity. Effectively, this means a free pass on bad behavior. And let’s not kid ourselves — there’s something called the “fundamental attribution error” by the psych types, which says that we mistakenly think that nobility and good behavior is inherent in people’s character. But it’s not. Given the opportunity, most people will do something that can be classified as “bad.” I know this is a kinda cynical thing to say, but it’s what the research shows.
Positive feedback reputation systems do work, as long as there’s an expectation of repeated interaction (in other words, if you think you might see the other person again). In a large enough population that isn’t segmented into cohorts or groups of appropriate size, this repeated interaction may not be a reasonable expectation. In other words, too mobile and big a group of players on your server, such as interacting with a different group of folks every night, and the incentives to get a good rep go away. (In fact, they might actually reverse: farm good rep in order to blow it on a big bad action).
Single-stage reputation systems are vulnerable to farming. This can be summarized by pointing to eBay. In a single-stage rep system, your “star” rating is based purely on how many folks give you a positive rating. The quality of their rep doesn’t matter. The result is that you can farm positive reputation pretty easily, because any new account can give positive transaction ratings.
In general, in a large enough system, farming becomes statistical noise. Of course, another way to look at it is that a guild that all rates its leader highly every day is technically not farming as far as the algorithm is concerned — they actually are showing their support.
In addition, in any environment where there is a low transaction cost, what starts to happen is the system norms to maximum. Any rating below the max is seen as negative feedback relative to the norm. This means that soon the ratings become meaningless — everyone is perfect. Over time, there’s an inflationary effect, because more and more ratings enter the system.
Of course, the core challenge here is finding a transaction to hook your reputation rating to. Many of the things that we would like to rate in MMOs, such as a random obnoxious chat remark, aren’t really peer-to-peer transactions. But giving reputation without a transaction (as in Cory Doctorow’s concept of “whuffie”) just exacerbates the problems listed above. So you have to hook reputation into specific actions you can hang a code trigger off of — rating a grouping session, or a purchase, or a murder.
More successful rating systems have either had a multi-stage system (such as Slashdot’s Meta-Moderation system) to act as a corrective, or like Advogato and other such systems, rely on authentication of reputation from trusted sources. This solves that guild farming issue.
This latter approach is very useful under certain conditions. The way it works is that when glancing at a rep rating, it displays to you the rep based only on sources that you yourself have rated. A high rating by someone you consider a jerk won’t show as a high rating to you. People your friends like will show as higher. Effectively, it’s rating mediated by social network. Unsurprising, because it actually derives from security models for untrusted peer-to-peer networks.
The downside, of course, is that if you aren’t hooked into a social network, you’ll effectively have no ratings. Since one of the primary values of a rating system to is provide cues to newbies as to the social landscape (“is this guy trying to scam me?”) this method fails that general usability test; instead, it works best for intermittent interactions in a large population of veteran users. Uh, something like maybe Eve Online might benefit from it.
This method can also get computationally expensive, of course, particularly depending on how far out through the network you search the data… does your friend’s friend’s friend’s friend’s rating count?
Ironically, a reputation system is fundamentally just about conveying data in a rapid abstracted manner. If your group of players is tight-knit enough — like say, a cohort of players levelling at the same rate through the same zones through their career — real world word of mouth is more efficient. This is a large part of why I believe that we have not seen the same sorts of misbehavior in the level-based systems that we have seen in the classless game systems; the latter do not tend to naturally create groups of a size smaller than Dunbar’s Number that are guaranteed repeated interaction, and thus peer pressure serves to police misbehavior. In other words, if you can get your players into clumps of 150 or less who tend to play mostly with each other, an abstracted rep system is effectively overkill at that point — and so is much of your customer service concern.
This argues that a coded reputation system is really intended for specific uses that your design may not call for. You really need it when you have a large population without repeated expectation of interaction, and you need it then for newbies. But the most secure methods won’t help, and if baddies can create newbies to do their dastardly deeds with, even those methods won’t help.
FWIW, UO went through most of the items described in the article on Gamasutra at one point or another, including fading of ratings, double-stage systems, recidivist tracking (causing permanent changes based on multiple movements between positive and negative) — and they all fell prey to the fact that it was a negative-feedback system. On SWG, we discussed trying a system like Advogato, and ended up concluding that word of mouth was what worked best anyhow.
One of the biggest problems, of course, is that what’s really needed is a lengthy background check, but we want a summary with just a five-star granularity. Enriching player profiles with past activities would go a very long way towards clarifying the meaning of the five-star rating, but even showing “rated 5/5 by 1 person” versus “4/5 by 127 people” is a bit too much data to parse in many circumstances.
I’m still a believer in reputation systems, but I think we’re going to need to look towards more robust mathematical models in order to really crack this particular nut.
You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.
I thought your article was quite nice, and I wasn’t trying to attack the methods you described, but rather just provide an infodump of stuff I’ve learned in trying to tackle similar problems.
Who Are You? What Do You Want? I was going to approach this slightly differently, but Raph made a post about social feedback
Blogroll Joel on Software Raph Koster Sunny Walker Thoughts for Now Sex, Lies and Advertising
[...] Farming bezeichnet das endlose Sammeln von bestimmten Ressourcen an den daf�r optimalen Stellen des Spiels. Oft um es dann bei Auktionsh�usern zu verkaufen.. [...]
[...] He also responds to a Gamastura article entitled Designing an MMORPG Feedback Rating System, which was also discussed on Slashdot. [...]
[...] Raph on Rep systems [...]
[...] Long story short, I’d like to know more before accepting any pronouncements. That said, I’d bet a few developers would be pleased to learn that social interaction is not so important, since accommodating social interaction tends to engender the thorniest design problems. [...]
[...] Rep systems [...]
[...] Source: http://www.rhsmith.umd.edu/faculty/cdell/reputation.html http://www.raphkoster.com/?p=303 [...]
[...] Your page is on StumbleUpon [...]