AJ Repost #2
by Tasteless | 22/01/2009 19:26:35 Source: http://www.arenajunkies.com/showthread.php?t=59023 "Background: A persistent player rating system was first hinted at back in TBC as something that might be introduced in WotLK. When I first read about it I was shocked because (as with anyone who played Wc3 competitively) I was aware of how easily the system can be abused, and how it kills competitive PvP. I sincerely hoped that this was merely an inexperienced member of their team getting a little over-excited with a naïve idea for improving match-making. I sincerely hoped that their development team had the competence to stop this ever going live. The primary motivation for a persistent rating system was to avoid pairing poorly skilled players against highly skilled players because bad players don't enjoy being stomped. We will later see that this is impossible to achieve since a player's "skill" level is something that that may be projected to anything a player wishes that is below his actual skill level, but we will go into this in detail later. Anyway, 3.0.8 came along and out-of-the-blue the new and untested system was dumped upon us. What made it even more unsavory to swallow was that it came with major bugs leading to widespread chaos with skyrocketing ratings, prestigious achievements being awarded for mediocre play, and general inconsistency. However this post is not concerned with these bugs, and in fact these bugs are currently clouding a far more serious issue. The issues I'm trying to highlight exist with any persistent rating system, and they would still exist even if Blizzard had smoothly implemented the system. Even if they had kept the elegant zero-sum scoring system for team ratings and used the persistent (or hidden) rating solely for match-making, the problem would still exist! The fundamental flaw with a persistent rating system --------------------------------------------------------------- Essentially a persistent rating attaches a sense of "state" to a player that persists between arena teams. This "state" affects how the player is to be paired, and therefore affords the player a means by which to manipulate how he is paired. A player will therefore aim to be in the optimal "state" before beginning a competitive team. Laughably Blizzard have hidden this persistent rating (and gone so far as to call it a "hidden" rating) in the naïve hope that players will not attempt to manipulate that which they cannot see. A simple way to abuse this system would be to deliberately play badly for many games: It need not require losing every game, even winning 50-50 at a low rating will be enough to establish a low persistent rating. When this low rating is established, the player can then simply begin a new team, start playing well, and reap the benefits of beating players who are now judged (by a flawed system) to be more skillful. Flawed regardless of the rate of evolution ------------------------------------------------------------ A persistent rating attempts to judge a players skill level by looking at his current team or personal rating. This means that the persistent rating must evolve according to actual ratings. For a persistent rating to be meaningful and persistent it must evolve slowly and remain stable. However this makes establishing a false persistent rating more potent, and so the system may be abused to a greater degree. A fast evolving persistent rating undermines the whole desired purpose of a persistent rating system since a player can easily "reset" his persistent rating by playing poorly: Suppose a gladiator has been playing at 2300+, and wishes to boost a friend to 1800. The gladiator need only quit his team, begin a new team and play badly for however long it takes to decrease his persistent rating before helping his friend out. Either way the system does not work - whether it evolves quickly or slowly or anywhere in between it will be susceptible to establishing a false persistent rating. Even if the rating were non-decreasing, you would see players deliberately maintaining low ratings (or even rerolling characters!) to establish a convincing low persistent rating so that they could capitalize towards the end of the season. In fact it's impossible for a persistent rating system to succeed: if it's potent enough to pair low skilled players consistently and award them for occasionally beating more skillful players, then it will be potent enough to be heavily abused. If it is not potent enough to achieve this, then it servers no purpose other than to require a "reset time" for players wishing to help out lower-rated players. ---------------------------------------------------------- Stop this folly! ------------------------------------------------------------ I [ Post edited by Tasteless ] |
by Kalgan | 22/01/2009 23:20:06 Although I can't go into great detail at the moment, I'd like to make a few comments. - The team/personal ratings and the hidden ratings are related to each other. Your team/personal ratings essentially drift toward your hidden rating, so if you deliberately lower your hidden rating and remake a team, your team/personal ratings simply won't get up to high ratings until your hidden rating does. His exploit as proposed simply doesn't work because his assumptions about the relationship between the visible and hidden rating are incorrect. - Yes, a group of good players could still deliberately stomp nubs by intentionally losing lots of games to lower their hidden rating. A rating system won't stop that, but it still solves most cases of high rated players beating up on lower rated players. First, many of the cases where high rated players were being matched against low rated players wasn't because the high rated players were deliberately trying to stomp people, it was often accidental as high rated players had normal/natural reasons to change teams on occasion (helping friends, trying out new players, new comps, new strategies, etc). Second, it takes far more effort to deliberately deflate your rating down to low levels than it did to simply create a new team. - Something that many of you have mentioned is that the rating adjustments are now not always "zero sum". However, a rating system doesn't need to be zero sum in order to prevent inflation. The new system uses a bayesian prior distribution in order to prevent inflation, which in simpler terms means there's math that enforces that players won't have ratings outside of some range (when working without bugs, of course). For the population size we have for arenas, this means we should see ratings range from around 600 to 2400 (again, assuming no bugs). - There were two bugs with the system that we're in the process of fixing before we reactivate arenas (should be soon) that were causing serious problems (and distorting everyone's perspective on the system). One was in the functionality that seeded everyone's hidden rating based on their pre-3.0.8 patch performance, causing players to "drift" toward wildly inaccurate ratings, the other being in the way that personal ratings were adjusted. [ Post edited by Kalgan ] |
by Kalgan | 22/01/2009 23:34:30
PvP definitely needs to become less bursty, and we will be addressing that. However, I still think the rating system matters. |
by Kalgan | 22/01/2009 23:40:44
I think you say that with the assumption that if it had been the start of the season, the bugs wouldn't have been there (not a good assumption in this case imo). The new system (when working) doesn't significantly change how hard it is to get the gear or who you have to beat to get the title/mount/etc (that's still entirely relative). It also tends to be mid/late in the season when the problems with the old system were the most exposed (with highly rated players starting new teams, helping friends, etc). |
by Kalgan | 22/01/2009 23:50:52
It was on the PTR. I blame you for not catching the bugs. jk ;] As you may have noticed though, players don't participate in the arena ladder on one PTR server the way they do on a live battlegroup. |
by Kalgan | 22/01/2009 23:56:14
I'm avoiding this question until we're ready to re-activate arenas (in case something we want to do turns out to not be do-able). |
by Kalgan | 23/01/2009 01:08:16
Correct. And yes, re-acquiring your personal rating for that team is under consideration (although some of the considerations are technical, not purely design). |
by Kalgan | 23/01/2009 02:14:59
I don't expect them to keep their rating. |
by Kalgan | 23/01/2009 02:16:26
The SGR says it likes that name, not sure whether we do anything to stop it at this point. |
by Kalgan | 23/01/2009 02:56:12
It is not like personal rating. The hidden rating uses a different approach to determining player ratings. It's more complex, but it tends to figure out a player's skill level far more quickly/accurately than an Elo system does. However, changing player's actual team/personal ratings extremely rapidly isn't exactly what we want. So, we're using an approach where we use the hidden rating (internally we refer to it as the GDF rating since it uses gaussian density filtering) for matchmaking since it creates better matchups than Elo and we use it as an anchor to figure out where the team/personal ratings should be moving toward. |
by Kalgan | 23/01/2009 03:02:00
Yes, I'm implying that we hope to roll back team/personal ratings back to what they were right before 3.0.8 went live. |
Recent Blizzard Announcements
