Reputation System Revisited - Need Feedback

[Royalite]

Also can we get a raid canceled button?

-XX

"Delete raid" is by the save button

I personally feel that reputation is fine for what it is and that people are just trying to make it out to be something bigger than that
....
REPUTATION IS REPUTATION WITH THE SITE ONLY AND WILL NEVER BE A GOOD INDICATOR OF A PLAYER'S SKILL!

You may have come to this conclusion on your own for your raids; I did as well a long time ago. However the system is currently setup to evaluate skill. I think that is the disconnect that should be adjusted.

[Fuggo]

Again, it seems very obvious we will NOT be wiping reputation, as was stated in the orignal post. There is a very simple way for us to convert people' current raid statistics over into the new system. We can also, as people have suggested, add a note/achievement/Feat of Strength on their account stating "Earned an Honored reputation in the classic rep system". So, please avoid keeping your feedback solely focused around rep wipes, it isn't happening.

As for rep decay, yes, we were also considering adding in reputation decay in some capacity after a few weeks.

A note from me to you all; please keep in mind this proposed system splits reputation and the idea of rating. Reputation would evolve into "what is this person's worth to OpenRaid?", whereas rating would be "what is this person's worth as a player?".

To the people claiming this concept is pointless, we have received a lot of feedback over the months stating the opposite, you guys want to rate performance. The current system mixes the two, and makes the whole concept confusing. Is reputation currently performance based or personality based or OpenRaid participation based, etc? While OF COURSE my philosophy of how to give a rating to a healer and your's could be different, the rating is not meant to be a literal translation of one's ability. That is the point of it being an average. Averages also help people recover from a bad performance one time. To the people claiming that everyone will hand out 3's...yeah, I think that is very true. How is that bad though? Wouldn't a 3 represent someone who was good enough to not be noticed but not great enough to stand out? Is getting a 3 a bad thing?

On to ratings based on which instance they are running. I think you need to look at the whole picture to better understand how this rating system can help you. First, all of this new information would be displayed on the signup page. All of this information would be easily accesible for raiders and raid leaders alike. Also, for those who have said, "just use armory", what if we could bring armory to you? Our goal is to provide as much information about a raider as possible based on what they have done on OpenRaid. Now tying this back into what I started this paragraph about. We want to track which raids people run, as that is very easy on our end, and display it for people to see. It was suggested that we tie ratings to an instance, and honestly, I think that could be a great idea if it is practical. So, you can see that someone has a 4/5 star rating with 76 total votes as a tank, but you look at their OR raid history and they have run 10 ICC's and 1 Tier 11. That is information that some raid leaders will not care about, and others will. The point is that the information is available.

We are also trying to incorporate a person's raid history on armory better. Much like Blizzard's system at the bottom of the armory page, we are trying to use the API to tell you how many times someone has killed a boss, on top of telling you already how many bosses they have killed in the instance.

Another piece of feedback was, do not make it required to leave a comment in order to rate someone. Fair enough, we just want to make sure people take rating seriously, not so much from the "just give a 3" point of view, but more so from a "do I actually remember this person?". Asking for a comment makes you think of whether or not you can come up with one, however I think you make a good point, how does "good dps" help? So, what if you only have to leave a comment if you give out a 5 or a 1? Obviously, you can leave a comment for any rating, but we make those mandatory?

We also considered having the rating that you have displayed, your average, display 2-3 comments that were given to you based on the rating tied to it. So if your average was a 4.2, your "quick" comments would be 2-3 that are 4's.

Trust me, we love comments more than ratings, but reading through 10 pages of comments is impractical. (we are fixing that with the new website in a pretty nifty way, but there is still sifting to do) A rating is a quick means to judge someone. Comments are a more "complete" way of judging someone. It gives you an idea for more progressive content. Who cares about measuring dps in ICC? Few. Who cares about measuring dps in Dragon Soul? Everyone. We are moving forward under the assumption that more progressive content will be available for you all to raid using our service, and we want to have a system in place that offers you help while building a raid. I don't see how just because instances are easier than others, or my rating philosophy is harsher than another's, makes the whole system pointless. This, again, is not meant to be a literal definition of one's talent. Think of armory, you can see someone got 10 heroic Deathwing kills right now. What does that mean? What if they were the lowest dps in a 25 man guild and were essentially a warm body now that nerfs are so harsh? What if they were a warm body even if their first kill was in the 2nd month? Every judgement given or read, outside of actually playing with a person, is an approximation and/or assumption.

For the people who like the current system, but don't like the idea of rating people in the new concept...it is exactly the same thing. People use the current system to rate others, the only difference is that people rate each other based on their own thoughts. My +5 rep vote could be because I liked your personality as a raider, but you were ok dps. Another +5 could be because in his mind, your 25k dps in ICC was good enough. The point is...any system that involves more than a simple 0 or +1 always brings in relative values. I do not see how that is bad if the whole point of the rating system is to simply provide other people a quick look at what you have earned based on what you have done. With all of the other information we will be providing, one can simply pull people in based on a 3+ rating and decent rep, or they can dig deep to find out they have killed Heroic Rag only once, they gem incorrectly, and based on a comment 3 pages deep they actually have a pretty rough time with world in flames. Options!

As for attendance, yes, I love the idea of adding more options in. Options for people who were kicked, or disconnected, or left, etc. I also think that there needs to be a bit more functionality available for raiders, if the raid leader bails right before or during the raid. We will certainly look into implementing that. Great ideas. Also, yeah, we need to figure out how the reserve works. Sadly, most people take reserve to mean "do not give a crap". Adding consequence should help mop up a bunch of the current "no show" issues.

Someone asked "if the rating system is based on attendence...". No, it won't be. Attendance would just be something we track and display separately. So I sign up for your raid as a tank. You see me listed in signed up and want to either approve me or leave me out of the raid. You would now see my ilevel, the rating for the role I signed up as, as well as the total votes i have received for the role, you would see my attendance, you would see my reputation via the new system, you COULD see (this is just an example of more we can show) how many times I have ran the instance your raid is for on OpenRaid, how many times I ahve killed the end wing boss, etc, etc.

The addon...yeah sadly addons have a rough time communicating with the website. Is it possible to sync up the rating system with the addon? Maybe. However, it would involve a lot of generating and pasting text strings, like the current one uses. How we would get the text strings to properly generate in WoW to paste into OR could be a pain. We can certainly look into it.

Why have rep? Because people love it! The concept of grinding something out that shows you are skilled/dedicated/reliable/sexy/a good person is what we all play these type of games for, right? We just want to take that core concept, and really flesh it out into something that has a value to other users as well as simply being a badge of honor.

Will people take the new rating system seriously? I am not sure really. Will the people who won't take the new system seriously take any system seriously? I think if this new idea has some weight behind it, and more people admit to looking into other players before raiding with them, I think any new system would get more traction.

For people claiming this concept is complex, sure maybe on paper without an example to play with it might be, but I assure you it is not. A 5 star rating system with comments is simple. Rep being a measure of what you do for OpenRaid and what people think of you as a person is simple. Tracking and displaying more information is simple. I think you need to have some faith that we can make this system function properly, and not simply cast it aside when obviously no one outside of the staff have had a chance to play around with the concepts.

For people who think just because it is participation based, everyone will be exalted in a week. No. We control the thresholds and monitor rep gains very closely. Also, another reason rep decay is nice.

For people who think rep doesn't matter, and ratings don't matter, and the only thing that will matter to you is a full scan of their armory. Outside of us bringing more of armory to you, I don't know...just use armory then? Why is this bad for other people because you personally do not want to use it as a gauge? We are trying to make these things have more meaning, and give you more information, and you are responding as if that is a bad thing JUST BECAUSE it is not a 1 to 1 copy of Armory.

To the person who thinks this is just too much for a "glorified pug system", I would say this to you. We have plans to make it a lot more than that, and incorporating new things like a more valuable rep system, pushes us forward...at least I think. If OpenRaid is simply LFR with a litmus test for you, then you can continue to use it as such. Mark no response to all the ratings, give no one rep, ignore attendance and enjoy raiding. We will not make any changes that would completely overhaul the system, for exactly the reason of still being a simple LFR tool with a litmus.

Sorry, I think the covers a lot of the feedback, but if I missed you I apologize.

[Nery]

Reputation is still easily cheesable, I'd very much like to see an entire removal of the reputation system.

Regarding some people here, I used to be on top 100 reputation here till I decided to take a break off WoW a month ago.

At least on Alliance side, a couple of very specific groups cheesed people's reputation through the roof - even my own, admittedly. I'm sure you know what I'm talking about.

I say, leave comments, leave the voting system up - but remove reputation as a whole. Votes can't be as easily cheesed as reputation can be.

QuickEdit, sorry Fuggo but your post is gigantic:
Why have rep? Because people love it! The concept of grinding something out that shows you are skilled/dedicated/reliable/sexy/a good person is what we all play these type of games for, right?
- Except as of right now rep shows who has been here for the longest, or who has been helping lead the bigger raids. I believe some sort of Rep Decay is in order (even if this affects me negatively).

[Disq]

[color=#]lots of talking (nicely said), but what about performance tracking, what you mentioned in your first post? Not a word has been spoken of that, only in the lines of difference between ICC and DS HC.[/color]

[Darrie]

Skill/Performance tracking....Someone in iLvL 333 joins a Ruby Sanctum run. They do 12k, which ends up being less than the i400 geared tank. Most people will glance at their recount to do these evaluations, right? So poor mister 333 ends up with a abysmal score/stars when really he did just fine. Won't this scenario be presenting itself fairly often?
Doesn't seem fair. The number of extreme raids on OpenRaid will always be the exception rather than the norm, unless they suddenly allow progression xrealm, so why put an unneeded focus on dps? This shouldn't be an elitist numbers based community, unless those numbers are attendance, effort, or attitude.

[Xyriin]

I would love to have a viable skill ranking but here are the problems I see...

The Human Flaw...
There is already a 'skill' ranking system. Sure it's combined with participation but at the same time the real problem is that people don't use it correctly. An interesting data point would be how many negative points have been received from Not Good and Awful ratings. This should be able to be pulled from the database (assuming of course the ranking values are uniquely stored per per raid). Now compare the number of times those negative ratings have been given versus the number of times No Opinion/Good/Great/Exceptional have been handed out. Considering the skill range of players this should be a fairly linear distribution, I'd bet a couple thousand dollars it isn't even close.

If those detailed values don't exist then at least you could get a distribution of the final ranked score of a person in a raid divided by the number of people who ranked the raid. Once again in a perfect world you would have a linear distribution from worst score to best score and once again I'd bet anything it is skewed high with a high majority of people well above the average possible score. So what does all this actually say? That people aren't rating honestly, and accurately. I'm sure some just get through it as quick as possible, others may not care at all and just rank everyone as high as possible (I can confirm in some raids I hear people say "rank everyone great!" at the end). Others probably rate their friends highest no matter what or may favor a specific role.

So in the end the problem sits mainly with the people doing the ranking.

How to correct the human factor:
Weight an individual user's ratings. This would require more database tracking but devise a system to baseline each user. Say you have potential ratings from 1-5. A completely unbiased person over the course of time doing ranking purely by skill would have an average of 3 (in other words he would rate just as many people exceptional as he would awful).
Sample set of 1000...
100 people ranked a 1
200 people ranked a 2
400 people ranked a 3
200 people ranked a 4
100 people ranked a 5
Now let's look at something more similar to your normal OpenRaider at the moment...
10 people ranked a 1
40 people ranked a 2
300 people ranked a 3
450 people ranked a 4
200 people ranked a 5
The 'perfect' ranker has an average of 3 compared to your normal ranker who has an average of 3.79. Since this normal ranker is 0.79 above the optimal average you simply weight ALL of his ratings. So in this case if he gives someone a 4, they only get an effective 3.21.

If the math is too complicated then you can enforce this type of even distribution by limiting ratings similar to you have a cap on 'exceptional' ratings now. Have an unlimited number of average ratings, limited best/worst (1 in 10 raid spots), and limited for good/bad (2-3 out of 10 raid spots). This of course results in problems when you have raids that are mostly good or mostly bad.

The other item mentioned was role favoritism. Split out the rankings by role for even more accuracy as some people tend to favor specific roles for their rankings. A good analysis would be to check out people's average score per raid based on role. If healers, tanks, or dps are significantly off from the other the easy solution would be a site wide variable to essentially buff or nerf points gained from those roles. In my opinion this is a bit of a fringe case since most people tend to stick to the same role so you're 'competing' inside your own role for raids most of the time anyway. That said, this would be a good site statistic to monitor if using a serious skill system.



The Difficulty Question...
So obviously an average person could look like a superstar on that BC rep/transmog run but they can't seem to avoid standing in fire on current expansion content. Old content only enhances the problems with the human factor mentioned above because who really does horrible on old content? Yet all these raids are currently lumped in with things like heroic Firelands. At the same time there is a certain amount of skill needed to successfully complete things like old achievements, a few rare boss mechanics, etc. Personally, I run a weekly Uld25 farming run to get people legendary items, get transmog gear, and knock out a good amount of achievements. I have no problem rating someone awful because they failed to listen on vent with regards to stopping dps, strategy for an achievement, etc. So lesser raids can sometimes serve as a valuable analysis tool for raiders before they head into more current content.

Using math to compensate for time and gear:
The easy solution for older content is to simply scale all ratings given for an older raid. This way you do this will of course vary by the exact rating system you implement. Assuming we're using a rating of 1-5 you don't want to let doing good on an easy run skew someone high, and if anything you want to let someone doing bad on an easy run skew them low. So for people who do excellent you want this to factor in lightly...they still got a 5 but it's a 'weak' 5. Let's use Uld25 as an example again. OpenRaid admin/moderators decide that Uld25 is only 50% the difficulty of a current raid so all ratings are adjusted by the difficulty factor. So the score of 5 you receive in Uld25 would only be worth half what it would if you got a 5 in Firelands. As that scales down a 3 may be weighted 75%, a 1 weighted 100% (so that you get just as negative marks from poor performance on an easy run as you would on a hard run).



Complementing the Skill System with Outside Data...
The goal is to obviously create the most accurate and valid skill system as possible. But in any system with human interaction you can have errors, flaws, and bias. The number one thing I'd like to see added to character information in addition to the overall progression is the number of kills and date of the first kill. Wowprogress generates a value like this called PvE score and it's simply a point value based on when you killed bosses compared to everyone else. Date of kill lets me know the very important aspect of if they killed the boss during progression or way after during times of lol-DS 25% zone nerf for example. Or maybe they went back and cleared FL only after they had full DS gear. Similarly the number of kills lets me know if they were carried or had a fluke kill. Probably over the top for most raids on here but WoL parse ranks would be very useful for evaluating DPS as well. Maybe don't make it a mandatory listing but toss a star by the zone they have parses. All of these things would provide a quick one glance assurance that their OpenRaid skill rating is as described.

[Bluriel]

Ok - one silly idea... but here we go:

Could we have a different category for "rated raids"?

I for one would prefer to keep "fun/socializing" (or some category to be invented) without the "performance" option. In my raids I care more about attitude than DPS (or even more than staying out of fire!).

Whoever does dunno - ... normal? hardcore raids? Sure - have your "performance meter". I'd rather keep the raids I lead/join without it.

Or... just a check from the raid leader to enable/disable it, also visible to the participants?

Thank you,
Blu

[Xyriin]

That could probably be tied into the existing raid difficulty system. Right now it's just there to give people an idea of maybe the type of raid they're signing up for. Slightly re-tooled it could be used to almost set the raid as 'rep only' or 'skill rating'. This might also help that raid option have more meaning and get people funneled to the appropriate raid. Maybe even more importantly place a bit more emphasis on someone trying to run a 'skill' raid so there are less of the 7/7H FL raids each with a raid leader and 4 other signups expecting to be carried.

[Bluriel]

Thank you X!

May I say it is equally frustrating when the leader posts a "fun" raid and people start posting dps meters and/or start whispering raid leader about performance and/or rage quit with/without a bratty fit

[Singularity]

Comments mean more to me than rep. I dont really care what happens to the rep bar but i think a network strength meter or something would be cool. How big is your OpenRaid network and of what quality are the people that are on it kind of thing. I think it may support raid leaders too who are the lifeblood of a system like this imo. Getting on a raid leader's friends list would increase your network strength alot and give the raid leader a small bump. The amount of raids you have attended/led increases the amount of bump for both sides. If you come and do work like it is a religion people want you on their list. If you come and troll and act a like a fool the consequences will reflect your behavior pretty quickly. If your network strength is strong people could at least see that if you are invited you probably know alot of other raiders or at least some very skilled/dedicated ones. Just a quick thought between queues