Problems with ratings - official thread

Micah
Micah
edited February 2008 in General Discussion
«1

Comments

  • Micah
    Micah
    Posts: 894
    It seems that making ratings public and not anonymous has not fixed the problems. We never thought that people would be so aggressive about low ratings, especially once anonymity had been removed. In addition, the current system doesn't take into account the fact that campaigns change over time. A 1-star today (no content, basic description, "coming soon") may be a 5-star tomorrow. Unless people come back and update their ratings to reflect the current standings, they fall out of sync pretty fast.

    Ryan and I will try to figure out a solution, and we invite feedback from everyone. Keep the comments in this thread (and keep things civil). Together we'll come up with something that works.
  • Noshmek
    Noshmek
    Posts: 9
    Thank you for this thread.

    If a user who is not involved in a campaign wishes to give a rating, my belief is that they should offer constructive criticism to justify that rating. Are they impressed with a particular facet of my campaign? Do they believe that a particular element is not to their liking? Or are they granting a rating based simply on a desire to skew a campaign in or out of the top 5, in essence, cooking the books - with no justification what-so-ever? As I said in my other post, the system appears to be arbitrary. If another member of this awesome community wants to give me a rating, I'd like some feedback, so that I may possibly improve my own game, or even just food-for-thought. But a low rating based simply on the fact that my campaign is rated higher than yours is, to me, offensive and dishonorable.
  • RobJustice
    RobJustice
    Posts: 178
    I'm a proponent for making votes optional. I think if you decide to have ratings, then you take that can of worms as is.

    Some other potential solutions:
    Requiring ratings to have comments. Ala E-Bay's Positive/Negative feedback system.
    Ratings expire after X amount of time, perhaps set by the user perhaps not. Example: Ratings are valid for 30 days and then they expire.
    Requiring so many posts/creations before allowing ratings.
    Restricting ratings to Friends lists, although I've heard some people like strangers giving them ratings its still an option.
    Doing away with the ratings system entirely and allowing comments to fill their void.

    Just a few ideas
  • mcoorlim
    Posts: 39
    I like the idea of allowing constructive criticism in any form. I welcome it. Positive or negative.

    I just don't like seeing accounts created, marking my campaign a 1, and then never doing anything else - not creating any campaigns, characters, objects, or even rating anything else. It gives the impression that someone went through all that effort just to tell me "YOU SUCK!" and then vanishing into the void once more.
  • IceBob
    IceBob
    Posts: 98
    I've been trying to think of a decent solution to this conundrum, but none leapt immediately to mind. I like the idea of ratings 'expiring' after so long, or at least having notification sent to the rater when the rated campaign/character/item changes so they can go back and take a look if/when it improves.

    One other possible, though more complex, solution would be to create an algorithm whereby older ratings lose their weight in comparison to newer ones (they decay over time, until effectively expiring). They would remain at their set value (a rating of 2 would still be a rating of two even after three months), but they would be given less weight when determining the overall average rating. That way, a low rating that the campaign no longer deserves will simply fade away over time.
  • Noshmek
    Noshmek
    Posts: 9
    But isn't there an issue with the fact that the campaigns, as shown on here, are not indicative of how that campaign is being run? This lends to the arbitrary essence of the rating system as we all know that what is written is in no way indicative of the total package of a campaign. How could it be judged by someone who spends less than three minutes looking over what you have managed to put on the site? It's like judging an apple pie without ever tasting it; sure it may look tasty, but you would never guess that the recipe included sauerkraut and licorice.

    To be truthful, I would much prefer that the ratings be turned off completely - particularly when the individuals who have no connection to your campaign give you a rating and then fail to provide any feedback. A time elapsed rating, or a degenerating rating will not change this, and some individuals will continue to create dummy accounts to boost their ratings, and destroy the ratings of others out of spite. It's not enough to limit ratings to the creation of posts or items, as anyone can put jibberish in until they reach that goal. And limiting it to friends is simply going to skew results to a higher level for everyone - Would you be friends with someone who would rate your game as 1 star?

    My opinion: turn the ratings off as this site can only provide an incomplete view of the entire gaming experience for a particular campaign. The service this site provides is invaluable in the context of individual games. It is also interesting to see what other people are doing, and perhaps mine that for inspiration or ideas. In as far as total package, however, the site can not hope to compete in its current form, and the rating system cannot take into account the wide breadth and diversity of games on this site.
  • outrider
    outrider
    Posts: 46
    the ratings could go but I believe there needs to be a mechanism for campaigns turning up to allow people to check out the ways that people run their games. I enjoy looking at the different formats that say legacy of brutality and sins of the father use. Their creative use of pictures and formats enhance my enjoyment of reading their sites.
  • RobJustice
    RobJustice
    Posts: 178
    bq. But isn’t there an issue with the fact that the campaigns, as shown on here, are not indicative of how that campaign is being run?

    Except people aren't voting on how the campaign is run, their voting on how it looks. To continue your analogy, perhaps this apple pie is made with sauerkraut but it could still win the prize for best presentation.

    bq. To be truthful, I would much prefer that the ratings be turned off completely

    I'm in agreement, I'm not a fan of the ratings but I'll continue to play devil's advocate anyway. :D

    bq. A time elapsed rating, or a degenerating rating will not change this, and some individuals will continue to create dummy accounts to boost their ratings, and destroy the ratings of others out of spite.

    True, but thats a lot of work just for spite, I don't know most people are THAT a-holeish.

    bq. It’s not enough to limit ratings to the creation of posts or items, as anyone can put jibberish in until they reach that goal.

    A reporting system could help combat this, if someone is posting gibberish they should be removed from the site plain and simple.

    bq. And limiting it to friends is simply going to skew results to a higher level for everyone – Would you be friends with someone who would rate your game as 1 star?

    Actually, i like to think that my friends give me an honest appraisal. If I'm running a two-star game, I want my friends to tell me that I'm running a two-star game.

    I think my big argument against a rating system is that its inaccurate and unneeded. Someone who doesn't like Fantasy games could vote low on a game simply because its Fantasy. Now, thats a valid opinion but its inaccurate to how good the actual campaign is. Feedback and conversation is the best method of judging if your game is a good game or not.
  • Noshmek
    Noshmek
    Posts: 9
    I think my big argument against a rating system is that its inaccurate and unneeded. Someone who doesn’t like Fantasy games could vote low on a game simply because its Fantasy. Now, thats a valid opinion but its inaccurate to how good the actual campaign is. Feedback and conversation is the best method of judging if your game is a good game or not.

    Couldn't have said it better myself. Actually, I tried, but failed miserably, and you rocked it. So, everything else I wrote - just pretend it was you saying this. LOL
  • DarthKrzysztof
    DarthKrzysztof
    Posts: 132
    The comments and the favorites/fans system are enough feedback for me; I wouldn't mind losing the ratings system altogether. Mind you, _Cold Blood_ been fortunate enough to escape negative ratings since they went public, but we accumulated a few before that, presumably to knock us out of the Top 5.

    I've mentioned this somewhere before, but my only real interest is in keeping my players happy, since that also makes me happy. It _shouldn't_ bother me when some troll one-stars something I've created, but it certainly does. That sort of thing is inevitable when you're gaming in a public place, and what's more public than the internet?

    Taodon's right; I don't know how accurately everyone else's campaign pages reflects the _experience_ of those games. My adventure summaries, which Jennifer writes, are pretty representative, though they do include some "enhancing." (Conversations get jumbled in chat-based games, and Lady Margone's particular flavor of nasty is hard to extemporize, among other things.) Rating based on presentation is all we really _can_ do.

    And RobJustice is also on the money - just because something isn't my cup of tea is no reason to give it a low vote. If I can't think of anything nice to say, I'd just as soon not say anything at all...

    P.S. I would still like to see a "Top 5" (or however many) on the front page for the campaigns with the most fans!
  • IceBob
    IceBob
    Posts: 98
    The only reason I don't want the ratings dumped is that I look for highly-rated games to raid for ideas. Having the top 5 decided by favorites would enable me to continue that practice, but limit me somewhat to those games that have already attracted a lot of attention.

    I like seeing my page rated, and I like getting the ratings from my players and friends, because, as RobJustice said, I want them to give me an honest evaluation of how much fun they're having - and I think they do.
  • Noshmek
    Noshmek
    Posts: 9
    I like seeing my page rated, and I like getting the ratings from my players and friends, because, as RobJustice said, I want them to give me an honest evaluation of how much fun they’re having – and I think they do.
    But wouldn't that sort of rating come to you in ways other than this site, particularly if they are playing with you? I appreciate when my players can give me feedback, and often allows me to use that to make my games better. But there are times on here where the person who is giving you an "honest evaluation" of a single click on a star is not giving you an evaluation, feedback, or anything else - even when you later contact them for that specific reason. So, one must surmise that the only reason you were given a rating was for singular purpose known only to the evaluator. What can you take from that to improve your page or your game? Nothing.
  • IceBob
    IceBob
    Posts: 98
    bq. But there are times on here where the person who is giving you an “honest evaluation” of a single click on a star is not giving you an evaluation, feedback, or anything else – even when you later contact them for that specific reason. So, one must surmise that the only reason you were given a rating was for singular purpose known only to the evaluator. What can you take from that to improve your page or your game? Nothing.

    You are correct in that. As I previously mentioned, the major reason I don't want all forms of rating abolished is because I use the ratings to find games with good material to help inspire me, and to learn from the storytelling tricks of other GMs. If ratings were entirely removed and the favorites served that purpose, it would suit me almost as well. The other reason to have ratings is the little juvenile rush I get when I see that some total stranger likes my material. It soothes my ego.

    I haven't gotten bent out of shape over the one random negative rating my campaign has, because I checked that individual's profile and looked at his voting record. The other ratings from non-players I take as high accolades from my peers in the gaming community (see also the gratuitous compliment thread).
  • redstar
    redstar
    Posts: 119
    (My proposal to fix rating system: Do you like my campaign, circle y/n: Y N ... just like grade school)
  • RobJustice
    RobJustice
    Posts: 178
    You forgot Maybe. haha
  • mcoorlim
    Posts: 39
    Actually, I think that a simple "Like/Dislike" plus mandatory comment field would be pretty cool. Popular campaigns would just have the most "likes-dislikes" - as it is, if a game with three people marking '5' is more popular than one with eight '5's and 2 '4s'.

    At least, I think so?
  • Charsen
    Charsen
    Posts: 85
    It is really hard to figure something out that works, and I'm constantly surprised to see top games get booted down... any time my game gets on the top 5, I feel slightly panicked because I think someone will rush over and give it a 1 just to boot it off the island, you know? So, I kind of dread the top 5 as much as I like it...

    The biggest problem is... what determines a 5 or a 4? How do I know they're done writing? What if they've got 2 pages for 5 months, and I give it a 1 star... and then BAM tomorrow, they have 500 pages written by Neil Gaiman with original music by David Bowie and Danny Elfman and personalized art by Julie Bell and Boris Vallejo. I mean, then I look like a horrible person for rating them before all that content came out, and it's likely I won't even realize it's been updated so I can fix my score. I tend to just take a "wait before you rate" approach, typically, rather than give out a low rating. My 5 might be different than [the poster below me]'s 5... and some people think a 3 is a good score while others might rip their hair out!

    Though I have only rated a few games, it's because I really try to read a lot of the game before I rate. Now I'm planning on going back through and leaving some comments... and oy! At least spring break is coming up right? :)

    I don't know how to solve the problem with retaliatory votes. I suppose you could IP log, but that might be a hassle for people who share PCs with their gaming families. Or people who would go to the public library or get their chat buddy across the pond to do it, if they were so determined!

    One thing that might help.. you could have top 5 lists for genres, like High Fantasy, Low Fantasy, Sci-Fi, Modern, Comedy... But then you'll probably have a High Fantasy glut as well.. I don't know.

    But yeah, the Like / Dislike is nice, however if you said "Dislike" and said "I don't like fantasy games!" then it's still not a great rating, you know? I guess this is the trouble with any rating systems..

    You could rate specifics of the games, like :
    Plot 1-5
    NPCs 1-5
  • photoneater
    photoneater
    Posts: 182
    bq. Popular campaigns would just have the most “likes-dislikes” – as it is, if a game with three people marking ‘5’ is more popular than one with eight ‘5’s and 2 ‘4s’.

    I don't think that is exactly how it works. There is an average involved, but it isn't quite that basic. After all, there are plenty of campaigns with three or four fives and nothing else, and yet most of the regularly top rated campaigns have less than perfect averages.

    I honestly don't know what I think of the issue. Is the negative voting really happening that much right now? It seems to me like it is happening, but not nearly as much as possible. That doesn't really excuse it when it does happen, though. That said, if everyone wants a change, we really should come up with something.

    bq. Feedback and conversation is the best method of judging if your game is a good game or not.

    Yeah, but is that happening? People are still preferring to cast a vote over other forms of feedback, in my experience. My campaigns are among the most voted on in the site, but I've received maybe one comment or other form of feedback. And from my observations on the site, most campaigns are not getting a lot of comments- one or two here and there from a friendly passerby, and occasionally one from a player, but it is hardly the feedback avenue I personally had hoped it would be, personally.

    bq. I don’t know how to solve the problem with retaliatory votes. I suppose you could IP log, but that might be a hassle for people who share PCs with their gaming families. Or people who would go to the public library or get their chat buddy across the pond to do it, if they were so determined!

    Right, I agree here. For example, I share my PCs with both three of my players, across my different campaigns. Furthermore, people wanting to use proxy accounts to be duplicitous and make negative votes can still find ways to do it without ever needing to worry about their IP betraying them.

    bq. Plus a comment. Or you could replace the numbers and say “One in a million” “Exceptional” “Great” “Not Bad” “Needs Work”... each associated with a number value (great = 5) and then you do an average of the scores to get a value at the end, like 4.75 or what-have-you.

    I like this a lot.

    bq. Other possible ways to tilt the hand to games is by having other categories… Most Active, Most Viewed, Most Commented, Most Loved (amount of fans), Most Pages… I dunno, there are lots of options.

    This too.

    bq. It is really hard to figure something out that works, and I’m constantly surprised to see top games get booted down… any time my game gets on the top 5, I feel slightly panicked because I think someone will rush over and give it a 1 just to boot it off the island, you know? So, I kind of dread the top 5 as much as I like it…

    Does this still happen as often as before? I have been sitting comfortably in the Top 5 ever since the vote reboot, and I don't think I've ever received a negative vote because of it. By negative vote, of course, I mean a notably critical vote, like a 1 or 2 out of nowhere as opposed to a 3 or 4 that might cause me to drop off the page for a day or so.

    bq. Though I have only rated a few games, it’s because I really try to read a lot of the game before I rate. Now I’m planning on going back through and leaving some comments… and oy! At least spring break is coming up right? :)

    That's a good policy.

    bq. True, but thats a lot of work just for spite, I don’t know most people are THAT a-holeish.

    Oh, they are. Maybe not here, and not now, but the potential is definitely there- and pre-vote reboot, we were seeing pretty clear evidence of it.

    bq. Requiring ratings to have comments. Ala E-Bay’s Positive/Negative feedback system.

    This idea has been mentioned many times, and I honestly think it is the only way to preserve the current system and resolve the issue.

    bq. Restricting ratings to Friends lists, although I’ve heard some people like strangers giving them ratings its still an option.

    Yeah, I don't like this one. As far as I am concerned, the votes I get on my campaign are how I get feedback from the community. If I want feedback from my players and other friends, I generally just ask them.
  • DarthKrzysztof
    DarthKrzysztof
    Posts: 132
    bq. You could rate specifics of the games, like : (snip) Originality 1-5

    Ooo, I'd draw some negative votes there. Almost everything I've done has been based on published Planescape material, even if it's only obvious to the people who've read or played it before. Mind you, finding ways to link it all together into the semblance of a cohesive story requires _creativity,_ but that isn't the same thing. :P
  • Charsen
    Charsen
    Posts: 85
    bq. Does this still happen as often as before? I have been sitting comfortably in the Top 5 ever since the vote reboot, and I don’t think I’ve ever received a negative vote because of it. By negative vote, of course, I mean a notably critical vote, like a 1 or 2 out of nowhere as opposed to a 3 or 4 that might cause me to drop off the page for a day or so.

    Well, not that I have seen since the vote reboot (knock on wood), but I guess we'll see! Yeah I mean a 1 or 2 though. Before the vote reboot, yeah, that was harsh.

    bq. Ooo, I’d draw some negative votes there. Almost everything I’ve done has been based on published Planescape material, even if it’s only obvious to the people who’ve read or played it before. Mind you, finding ways to link it all together into the semblance of a cohesive story requires creativity, but that isn’t the same thing. :P

    Just because you use published materials doesn't mean your stuff isn't original in how you handle and publish it - but you're right, _creativity_ is probably a better word choice! :)
  • Micah
    Micah
    Posts: 894
    I've been pretty busy lately, so I haven't had time to keep up on this. I'll take some time in the next day or so to read through all these and discuss with Ryan. He's mainly in charge of the rating stuff, as he did all the work. He wrote an "article on star ratings":http://blog.aisleten.com/2007/05/03/ajax-css-star-rating-with-acts_as_rateable/ for our developer blog, and it's by far our most popular read.
  • MythicParty
    MythicParty
    Posts: 78
    Midnight Oil is a great band. Anywho, my first thought when reading Ryan's tutorial was if the stars could be replaced by something more 'Obsidian Portalish.' Say, the Obsidian Portal symbol itself?

    There's an answer to the problems somewhere between getting rid of rankings entirely and keeping the current system as is.

    I'm admittedly not a big fan of the 1-5 numerical ratings. At the least there should be a rubric to help create a more universal standard of what a '1' is, what makes a campaign a "5," the differences between a '2' and a '3' etc.

    But I think what would be a fairer way to rank campaigns would be by the number of people who have made it one of their favorites. You'd still have each person's comments. Lastly, have a way to easily see what other campaigns said fans have also favorited, with the idea being that if someone likes the same campaign that you like, you may also like a campaign that they like and which is new to you.
  • IceBob
    IceBob
    Posts: 98
    bq. You could rate specifics of the games, like :
    Plot 1-5
    NPCs 1-5
  • viz
    viz
    Posts: 19
    I would prefer the elimination of ratings. If it's a vote, that's mine.
  • MythicParty
    MythicParty
    Posts: 78
    I think a more likely possibility is what photoneater advocated; "I personally would like to see a way to make ratings optional."
  • commonsd
    commonsd
    Posts: 5
    I would agree with the separating these like somebody said:
    "Other possible ways to tilt the hand to games is by having other categories… Most Active, Most Viewed, Most Commented, Most Loved (amount of fans), Most Pages… I dunno, there are lots of options."

    as an aside, I think part of the problem with the ratings is that there isn't really enough voting traffic and so rating remain pretty static.
  • ryan
    ryan
    Posts: 126
    Well, it's definitely not a perfect system. The initial idea was that when we start having campaigns we can use user ratings to help us determine who has the best campaign that month. Also, it helps us keep an eye one different campaigns each month when looking for our new featured campaign and character/items. But definitely, I tend to either rate 5s, 4s, or nothing at all.

    bq. Ratings expire after X amount of time, perhaps set by the user perhaps not. Example: Ratings are valid for 30 days and then they expire.

    I'm not really a big fan of this, but would consider for displaying the top 5 rated campaigns for this month on the homepage.

    bq. Doing away with the ratings system entirely and allowing comments to fill their void.

    We are working on comments and will probably have a +/- rating similar to digg, et al.

    bq. One other possible, though more complex, solution would be to create an algorithm whereby older ratings lose their weight in comparison to newer ones (they decay over time, until effectively expiring). They would remain at their set value (a rating of 2 would still be a rating of two even after three months), but they would be given less weight when determining the overall average rating. That way, a low rating that the campaign no longer deserves will simply fade away over time.

    I tried working through a algorithm similar to Netflix recommendations. Unfortunately it's extremely complex (as expected).

    bq. Except people aren’t voting on how the campaign is run, their voting on how it looks. To continue your analogy, perhaps this apple pie is made with sauerkraut but it could still win the prize for best presentation.

    I admit I am guilty of this... :(

    bq. I think a more likely possibility is what photoneater advocated; “I personally would like to see a way to make ratings optional.

    Interesting idea...

    bq. I would prefer the elimination of ratings. If it’s a vote, that’s mine.

    If it's causing this much of a headache, we may go down that road.

    We really want to revamp the profile page to allow better overview of each persons campaigns, characters, and items and also to follow other people, campaigns, characters, and items that they are interested in. We're always open to suggestions. I'm considering putting up a poll to see what options people would like to see done with the rating system.....











    Ryan
  • dm_punks
    dm_punks
    Posts: 18
    So, uhm...

    I guess this would be a bad time to ask you guys how much you like my campaigns?

    _hastily ducks_
  • bevinflannery
    bevinflannery
    Posts: 114
    Bumping this thread as a topic for general discussion -- We've had some one-star ratings being given to some of the site's top-rated campaigns by folks in their first day of membership, for no apparent reason other than to knock the recipients down a peg and to boost the "reviewer's" own new Wiki a higher average. (Yes, I got one of the one-star ratings, and I'm sobbing in my beer at this moment ... well, not really, I don't drink at work.)

    Since we've had a lot of people become more active since the thread was last active whose thoughts I've appreciated on a number of different topics, I figured I would bump it and try to start a conversation on the ratings system: What does the community want the ratings to accomplish, if anything? Can the current 1-5 star system meet those goals? Can the current system be changed to minimize the impact of drive-by "teh suxors!" ratings, without giving the GM's the ability to artificially inflate their own ratings by erasing adverse ratings?

    Insert musings of choice here...
  • sandman
    sandman
    Posts: 155 edited May 2010
    I think the hole star rating system could shift to a *"Like/Unlike"* rating, just like in Google Reader (to mention one, but it's the same principle in many different tools or social networks). Basically, if you like what you see, you just hit "I Like", but if you change your mind later, you can take it back and "Unlike" it. This way, you only give a *positive rating* (without degrees) to the campaign/s you think deserve it, and you don't to the ones that don't appeal to you (instead of giving a negative rating).

    I think it's simple and intuitive too.

    Do you guys agree?
    Post edited by sandman on
Sign In or Register to comment.

March 2024
Wrath of the Highborn

Read the feature post on the blog
Return to Obsidian Portal

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Discussions