Problems with ratings - official thread

2»

Comments

  • ChainsawXIV
    ChainsawXIV
    Posts: 530
    I'd tend to agree, and for what it's worth, that's essentially what the existing 'favorite' system is.
  • vstraydogstrutv
    vstraydogstrutv
    Posts: 209 edited May 2010
    I also agree that the rating system is pretty unnecessary. A simple like or dislike should really fit the bill (which, as Chainsaw pointed out, is what the favorite system is). My campaign has been up for months now, and the only rating I have is the one I gave myself.

    In OP's infancy the rating system was probably a great idea, but with the sheer quantity of different systems represented and all of the different tastes it seems to lose its credibility over time. There are already two solid ways to offer feedback to a person's campaign: Favoriting and commenting. You could even go as far as to tie these two things together, having the favorite on the sidebar and under it is the ability to leave a comment.

    THAT BEING SAID, however, the ability to rate ITEMS should still stay in place. We gamers love to make our own little gadgets and love to know what people think of said shwag. I think knowing an item is a 2-star creates more drive for the next item to be better put together. Also, if the item IS a 2-star, there should be an option while editing to clear the current rating as a way of saying "Hey, I've improved this item in some way, please check over it again and let me know what you think."
    Post edited by vstraydogstrutv on
  • sandman
    sandman
    Posts: 155 edited May 2010
    bq). I
    Post edited by sandman on
  • arsheesh
    arsheesh
    Posts: 850 edited May 2010
    h4. Problems with the Current Rating System

    Hey all, thought I'd throw my two cents in here. Regarding ratings, what I've seen to be the biggest problems are as follows:

    1. *Relativity*: There does not seem to be any established criteria for what ought to count as a 1 star vs a 5 star site. As a result, people are left to their own tastes and preferences to decide how to rate a site, and these can vary considerably. For instance, if I like fantasy, but not sci-fi, I may tend to give higher ratings to the fantasy sites I visit than I do to the sci-fi sites. These ratings are not reflective of the actual quality of the sites however, but of my own idiosyncratic tastes and preferences. In addition to this problem, I have noticed that by and large, most OP members tend to only give out 5 star ratings to the sites that they like. Now as I see it, if 3 stars is supposed to be representative of the average site, then rating a site at 4 stars ought to signify that "this site is a cut above the rest", whereas rating a site at 5 stars ought to indicate that "this is a truly exceptional site." However, if most users only rate sites at 5 stars, then 5 becomes the new 3, and in this inflated rating system, rating a site at anything less than 5 stars seems to indicate that "this site is less than average" since the average rated site is a 5 star site (as a result of this ratings inflation, I myself tend to only give out 5 star ratings, though I would prefer that on the whole, the ratings system was more accurate)

    p. 2. *Ambiguity*: Let's assume that you are a fair minded individual who genuinely tries to rate the sites you visit based upon perceived merit, as opposed to your own idiosyncratic tastes. Well, when visiting a site, how do you determine how to rate the site? Even if you are working from a set of normative criteria to make such a judgment, at what point is it appropriate to rate a campaign site? It seems that sites that are still in the early stages of construction ought to be given time to mature before an evaluation of their merit is given, but it's not always clear when a campaign has reached maturity.

    p. 3. *Viciousness*: Of course not all members of the community _are_ fair minded. Some quite frankly are tools who get their jollies from tearing other people down to build themselves up. As Bevinflannery pointed out, recently one such member decided that it would be fun to scroll through the most popular sites on the Portal and hand out 1 star ratings to those sites. Currently, there do not seem to be any brakes on this kind of behavior built into the rating system.

    As a result of these and other issues, some contributors to this thread are in favor of getting rid of the point-based rating system altogether (perhaps substituting a "like/dislike" system in its place). I myself like the idea of a point based rating system,since _ideally_ it would give you a bit more objectivity as to a site's merit than a "like/dislike" rating system. As things stand now however, the current rating system is far to vulnerable to misuse and abuse, and so lacks the objectivity that it was meant to embody. I doubt that there is any perfect solution to the problems with this rating system (leastwise not where imperfect people are concerned). However, here is my suggestion for modifying the system that might help to decrease some of the problems mentioned (see next post).
    Post edited by arsheesh on
  • arsheesh
    arsheesh
    Posts: 850 edited May 2010
    h4. Proposal: Modify the Rating System by Member "Reputation"

    The current rating system could be complimented and modified by a user "reputation" system. In addition to giving out ratings to sites, individual members of the OP could be awarded "rep" points for excellence related both to their contributions to a campaign site they are involved in (hence, players could receive rep points as well), and/or for contributing to the OP community in some other way, such as by offering new users HTML help in the forums etc. Reputation would thus be an indicator of how well respected a particular member of the community is. Currently, we have a democratic, egalitarian rating system: everyone get's exactly one vote, and all votes are accounted equal in value. However, it seems like the opinions of established pillars of the community such as Micah, Ryan, Grunn, Chainsaw (and several other names come to mind) ought to be reckoned as more weighty than that of a new user who has just created an account (or for that matter, a user that created an account a year ago and has done next to nothing with their site ever since). These are members who have invested a substantial amount of time and energy not only into their own campaign sites, but also into helping other members of the community build their own sites. Their evaluations ought to thus be more important to that community than the opinions of fringe users.

    This said, I propose both that a reputation system be implemented, and that this system would also modify the existing star rating system in the following way. In order to rate a site at all, you need to have received at least 1 point of reputation within the community. Moreover, the more reputation one possesses, the more weighty their ratings will be. So for instance, if I possess 1 point of reputation, my rating will count as only 1 vote. However, if I have achieved 10 points of rep, that same rating will count as 2 votes, and at 20 points of rep, 3 votes etc. I think that modifying the weight of a member's vote by the weight of their reputation would really help to mitigate some of the problems of the existing system. Consider the problem of relativity. It seems that someone who has invested allot of time and effort into their campaign, and has been involved in the community enough to have seen many other campaigns on the site, ought to have developed a good intuitive sense of what a poor, average and excellent campaign look like. Hence, their opinions will likely be more reliable than most (though of course, some relativity will inevitably remain). I'm not sure that this proposal would do anything to resolve the problem of ambiguity, but it seems to me that it would greatly mitigate the problem of viciousness. For one thing, it would prevent individuals who had just created an account from using the rating system to smear other people's campaigns, since they do not have enough rep in the community to vote. For another thing, say that these individuals do gain a point of rep and thereby earn the right to vote on the ratings of other people's sites. Now suppose that out of spite they go and give several popular sites a 1 star rating. Well their 1 star rating isn't likely to affect those other sites all that much since it only counts as "1" vote. However, the perpetrator of the 1 star vote now faces the threat of heavy retaliation. Members of the popular sites that just got the 1 star rating could always return the favor, and in doing so, severely lessen the rating of perpetrators site, since their ratings are likely going to count for significantly more votes than that of the perp. Hence, fear of retaliation might prevent members with a vested interest in the ratings of their own campaigns from maliciously rating more popular sites.

    Anyway, I apologize for the length of these posts, but I felt I needed some space to be able to articulate both my proposal, and to give reasons favoring it. Curious to here what other think of this idea.

    Cheers,
    -Arsheesh
    Post edited by arsheesh on
  • gnunn
    gnunn
    Posts: 423
    Arsheesh, I definitely like your rep idea. I think the Cartographer's Guild has a good model to follow, if OP decides to go that route.

    As to 1-star trollers, the requirement to have rep before giving ratings could very well help solve the issue, but I'm also not super concerned if some faceless internet entity disapproves of my stuff. I know my players are happy, and all y'all lovely people appreciate the stuff I've done. I hope you feel the same.

    Is there an option to flag users for this sort of behavior? Since OP is not a paid site, it might be tough to ban folks, but it might also hilight who is trolling and who is legit.
  • arsheesh
    arsheesh
    Posts: 850
    Grunnn, yes, the Cartographers Guild reputation system was exactly the model I had in mind. I was also thinking that the admins could have the power to assign negative rep to those who abuse the system, as a way of discouraging abuse, and highlighting who such abusers are.
  • Micah
    Micah
    Posts: 894
    I'll jump in here...First warning: I haven't read all the arguments. I've been super-busy with the dynamic character sheets and have pushed everything else aside.

    Right now I'm leaning toward removing the ratings altogether. I don't see it as adding all that much value overall, and it's obvious that it can be abused. On my list of fun things to do, battling trolls and devising anti-abuse systems ranks down around stabbing needles in my eyes. To put it another way: I have limited development time. Which would you rather have: the cool dynamic sheets we're rolling out, or a system to deal with trolls? When it comes to areas like this, I have to carefully consider how much value it adds to the system, versus how much it's going to tax our development resources. If it doesn't pass the test, it gets the axe.

    I also much prefer the upvote/favorite system we have for campaigns. If you like it, favorite it. Over time, the best will rise to the top. Obviously, this can be abused as well, but only to promote something, never to demote it.

    The team is going to be discussing the ratings system soon. Hopefully we can come up with a good solution.
  • JimTriche
    JimTriche
    Posts: 483
    With the current ratings system, I tend to avoid rating something without content.

    If there's a potential there, I may favorite the campaign to keep tabs on it, occasionally I'll talk with the person running it and ask what they're going to do with it, etc. Especially if the campaign is new.

    If a campaign hasn't been updated in more than 6 months, I tend to forget about it.

    If I see a really good campaign marred by stagnation I might give it a 4 star rating and watch it. I'm not sure if I can adjust that up later, but the opportunity hasn't arisen yet for me to re-evaluate those.

    And if a campaign has planning, effort and a good amount of activity, I tend to 5 star rate, because it's not the art and fancy formatting that does it for me, it's the effort.
  • gaaran
    gaaran
    Posts: 740
    Honestly, I think we should leave it the way it is. So far, with all those campaigns out there, I've only seen one guy go and rate the top campaigns a one just to spite them. The vast majority of the people are here for the same reason, and i trust them to use their judgment. As far as everyone rating either 4, 5, or nothing, not a whole lot we can do about that. Sure it might skew the rating upwards, but I've noticed that some of the most active people in the community (who would probably have a decent amount of rep) are guily of that too (I am myself as well). It's not a perfect system, but it gives us a general idea that people are enjoying our campaigns, and that's really all we need. If we were to change to a thumbs-up, thumbs-down system, that would be acceptable to, as that'd sort of what it is now anyway, however, that would be even easier for a troll to abuse. That's my two cents, I love the fact that I'm getting people looking at my page, and getting followers, but if one guy comes and rates me a one? I'll get over it.
  • jdcengel
    jdcengel
    Posts: 7
    I also like the ratings system the way it is. Once campaigns hit 25-30+ ratings, a single "one star" vote doesn't change the value all that much. Don't get me wrong - I complain as much as the next guy when it happens, and am guilty of wasting the developer's time with this issue - but the current ranking system provides a good way to easily find the best campaigns. I particularly value the non-anonymous aspect of the ranking system. I have contacted each of the three people who have given my campaign a 1 star rating, and each one was willing to change his rating after interacting with me on a personal level. Ironically, it's turned out to be a great way for me to meet people.

    Failing that, the upvote/favorite system seems like a good way let decent campaigns rise to top. I like that it still preserves non-anonymous aspect.

    jdcengel
    DM of Imperium Illithidus
    http://www.obsidianportal.com/campaign/imperium-illithidus
  • kenurion
    kenurion
    Posts: 80
    In my opinion, the benefit of a rating system is to help users find well-done campaigns to look at. Perhaps some kind of objective "completeness" metric could accomplish that. Well-thought-out and highly-detailed campaigns will have a lot of wiki pages, many npcs, adventure log posts, etc.

    This "completeness" metric could be a number determined solely by OP. It could be as simple as total MB of material uploaded, or it could include number of images, number of npcs, average kB of data per npc (proxy for depth of description of npc's), total updates during the last 30 day period, etc.

    Of course this could still be faked, but it would be a lot harder to fake a high "completeness" score than creating free accounts and doing nothing but handing out "1-star" ratings to campaigns you're envious of. This way, users can find campaigns that are highly likely to be well-done. If some of them are complete, but boring, well, users can still appreciate the thoroughness.

    This could be in addition to the star system. In fact, it could make the star system more robust. Most campaigns don't get that many ratings, so a few spiteful 1's can really drag them down. It's a bit of a chore for users to dredge through all the 'well-intentioned' efforts out there and find the campaigns that have real momentum. Knowing which campaigns are data-rich and therefore likely to be worth looking at could make that search more rewarding and encourage more well-behaved users to hand out stars thus diluting the influence of the spiteful users. Maybe once a campaign achieves a certain level of completeness, OP could list it on a 'rate-me' box on the home page to drive users to rate it.
  • gaaran
    gaaran
    Posts: 740 edited June 2010
    Personally I think the rating system is fine the way it is. It's not perfect, but it works as well as can be expected. In any online community that uses things like this, there are people who use the system to "troll" if you will. Because this is a community in the thousands rather than the millions, it's just a little more pronounced when it happens. Also, correct me if I'm wrong, but it seems to me that when searching for campaigns, they are organized according to number of fans, not according to star rating, so people giving a good campaign a one-star out of spite wont hurt their visibility. You can only favorite a campaign, you can't un-favorite it, so no one can affect other sites' visibility negatively.
    Post edited by gaaran on
  • Duskreign
    Duskreign
    Posts: 1,085
    I think I would like the rating system a lot more if you had to leave a text comment describing the reason why you gave the campaign such a rating. I recently got a 1-star rating for Wyrmshadow, a site I work very, very hard on making as nice as I possibly can, and seeing that 1-star rating actually does affect my resolve and confidence.

    I love Wyrmshadow. When someone insults her, I want to defend her honor, but I can't do anything about this.
  • arsheesh
    arsheesh
    Posts: 850
    You know the more I think about it, the more Sandman's idea (several posts back) of keeping the "Favorites" option but switching out star ratings for a "Like" system seems to make allot of sense. In that proposed system, it is not possible to provided undue negative criticism. You only have the options of "Liking" and "Favoriting" the site. You can always choose to remove a campaigns from either list, but this does nothing to damage the reputation of the site. Also, this would allow you to support all the sites you are fond of by "Liking" them without spamming your "Favorites" update log.
  • sandman
    sandman
    Posts: 155 edited July 2010
    Thanks for the support Arsheesh. That's exactly why I think it would be a good solution to this problem, because it doesn't relly on "negative" and "positive" ratings (or even different degrees of ratings).

    To recap, this was the idea I proposed (the theory):

    bq). I think the hole star rating system could shift to a
    Post edited by sandman on
  • johnkretzer
    Posts: 1
    I am new here...and the very first thing was to give a bunch of games 1 star...why to see what happens(I have since gone and read though the campaigns and have changed my ratings). The rating things is not a good idea in my opinion. Or atleast should be taken with a huge grain of salt.

    I think if you want to keep any sort of rating sysytem you should be forced to leave a comment...this is not a perfect solution as you will get the '1-star with the "You Suck" comment..but the moderators can weed those people out and remove their rating and/or slap the offending account with a ban. So hopefuly that would be...a self-correcting problem.

    There is one rule I think should be added...I don't think you should be able to rate your own work. GMs and players can't rate their own campaigns and players can't rate their own characters...I was surprised that I could do so...

    That is my two cents from new member.

    Anyway I think this is a great site and do hope to be a more active member in this community and have recommended it to all the GMs I play with.
  • FrankSirmarco
    FrankSirmarco
    Posts: 250
    The rating system is completely arbitrary, totally open to opinion, and useless for anything except sorting by "most popular". And as much as I like Obsidian Portal, using the rating systems as a means of ranking campaigns is a MAJOR misstep.

    I realize that there are a lot of users (myself included) that strive to have the coolest campaign in their given gaming system. Other users acknowledge your hard work by adding your campaign to their "favorites" list, or by leaving positive feedback on your comment page. Obsidian Portal features the best campaigns on the site every month with their "Featured Campaign".

    How many different ways do you need to have your hard work acknowledged?

    If you want my two cents, you eliminate the rating system and the "Most Popular" sort, make the Featured Campaign a bi-weekly feature instead of a monthly feature (14 extra campaigns feature a year), and go from there. You could even have a yearly award (perhaps a free one-year Ascendant membership) that is given to the best campaign in each gaming system (call it the "Opie"). That way, the trolls can't affect your rating, and serious users get the acknowledgment they crave.

    Thoughts?
  • arsheesh
    arsheesh
    Posts: 850
    Well Frank, I agree that there would be some gains by doing away with a "Most Popular" sorting filter, but I think overall this would represent a net loss to the OP community, here's why. Getting rid of such a feature might give newer, less "visible' campaigns an equal opportunity at being viewed (on the assumption that there are some dedicated OP members that still browse through the thousands of sites on the system). However the downside is that it makes it extremely difficult for members who wish to check out quality sites to find them. They would have to waist hours browsing through allot of "dud" sites to find the ones of higher quality. This seems like a definite regress to me, since one of the best ways to improve your own site is by gleaning ideas from other quality sites.

    Now, I realize that there is some arbitrariness of quality built into the "most popular" filter. For one thing, there are plenty of campaigns whose high quality is _not_ represented by their popularity. On the other hand, I think there are a small handful of popular campaigns whose quality does not match their popularity. But overall, most of the popular campaigns are of pretty high quality, and those that are not are just anomalies. Likewise, usually the creme does rise to the top, so while it may take time, most high quality campaigns do eventually make it to the top 3-4 pages on the popular list. Hence, I don't think we should get rid of the "Most Popular" sorting filter.
  • gaaran
    gaaran
    Posts: 740
    I agree Sheesh, it may not be perfect, but for the most part, high quality campaigns are the ones to receive favorites from people, and number of fans is how the popular campaigns are sorted. Unless they were to add different sorting criteria, I think they should leave that as is, not so that those of us with the popular pages can sit on top, but because we've put in a lot of time and effort to make our pages look good, and if someone is just starting out, there is a great way to look at some of the more popular sites for ideas. For example, most of the featured campaigns get a lot of favorites (rightly so) so those tend to be at the top of the lists for people to find them easily.
  • FrankSirmarco
    FrankSirmarco
    Posts: 250
    Arsheesh & Gaaran, wouldn't it make sense to eliminate the rating systems and make the "most popular" campaign pages the ones with the most followers? I might be wrong, but it seems that as of right now, the "most popular" campaigns are based on their average rating.

    Eliminate the rating system, and you eliminate the means by which the trolls are sabotaging other campaigns.
  • arsheesh
    arsheesh
    Posts: 850
    Oh I see, I must have misunderstood your earlier point. I thought you meant that the "sort by most popular" filter ought to be removed altogether. If you are just saying that the star-rating system ought to be removed that's another matter entirely. Actually though, I think you are mistaken about how the "most popular" campaigns are sorted. Popularity is not determined by ratings, but rather by fans/followers. Negative ratings do not affect popularity status of a campaign. Hence, I'd be fine with removing the current rating system and simply keeping the "favorites" option, I just like Sandman's proposal of supplementing it with a "Like" option, that's all.
  • FrankSirmarco
    FrankSirmarco
    Posts: 250
    My apologies, arsheesh. You're right. I assumed the "Most Popular" campaigns were being sorted by this info:

    4.8/5 Stars (137 votes)

    Instead, I see that they're being sorted according to the number of fans.

    So, what the heck is the point of the star system? By all accounts, it serves no real purpose. And with all due respect, I'm not sure that a "Like" option would serve much of a purpose - unless you believe that by clicking "like", a guest to your site is acknowledging your hard work while not necessarily needing to add it as a "favorite" campaign.
  • sandman
    sandman
    Posts: 155 edited July 2010
    bq). So, what the heck is the point of the star system? By all accounts, it serves no real purpose. And with all due respect, I
    Post edited by sandman on
  • gaaran
    gaaran
    Posts: 740
    Yeah, frank, I agree with Sandman, sometimes I will click on the random campaign button on the front page, and I'll see something that i think someone has done a good job on, but I don't care to get updates on it, because it's a game/genre I'm not interested in. But I want to acknowledge that I think they've done a good job, so I rate the site. Is it really so wrong to like receiving positive feedback on things that we'd spent tens to hundreds of hours on? I know I like it :).
  • FrankSirmarco
    FrankSirmarco
    Posts: 250
    I don't have a problem with having your hard work acknowledged. And positive feedback is wonderful. But having an arbitrary rating system that is based on opinion, has no established set of criteria, and can be used to sabotage someone's campaign isn't necessarily a good thing.

    If a "like" system can do the same thing the rating system can do, without allowing other users to sabotage a campaign, then I'm all for it.
  • Duskreign
    Duskreign
    Posts: 1,085
    The star ratings imply a competition that doesn't, or at least shouldn't, exist here. My campaign isn't better than anyone elses. I am not out to prove that. I work hard to ensure that my site is as nice as I can make it. I don't do it to keep up with the Jones'. For example, I love Gabe's campaign, but I don't think it is better than any other featured campaign. I'm biased, but I like my campaign at least as much. I'm not competing against Gabe, or Arsheesh, or anyone else, so a points-based system seems a little inappropriate to me.

    Think of this in D&D terms. Let's say that a person's Charisma score is being determined by a vote from both friends and total strangers in their community. After all, who better to judge someone's likability and general charm than those that have to deal with that person. So, the people assign this person a Charisma of 16. Suddenly, someone shows up in the community, knows nothing of this person or what they do, and arbitrarily decides to sabotage their Charisma score, just for the f#%^ of it. His Charisma drops to a 15. Only one point, but that point drops his Cha bonu.... you know what? I hate analogies. Nevermind.

    Another point is that with the 5-star rating system, I think anything less than 5 stars is a little bit of an embarrassment. I don't know about you guys, but I either give a campaign I like 5 stars or I give one I don't like as much no stars at all. I might, then, be in the like/dislike category. Even then, I think when liking or disliking a campaign, you should be forced to select a reason from a drop-down menu, with an "other" option that requires a text input. If you don't like Wyrmshadow, I would like to know why. I'm not averse to criticism, but I prefer that it be constructive.

    I heard the argument that "this is the Internet. Douchebags aplenty on the Internet. You should expect some of that no matter what." I refute that. This isn't the Internet. This is Obsidian Portal. It is a community. We should be allowed to police our citizens, as is the right of all communities.

    I think that when adding a campaign to favorites, there should be an opt-out for updates from that campaign, which could be later opted-in if the person should later decide they want to be more in-touch with the goings-on in a particular campaign. Do that, and the star-rating system becomes quite moot.
  • arsheesh
    arsheesh
    Posts: 850 edited July 2010
    Can I get an "Amen"?!
    Post edited by arsheesh on
  • Duskreign
    Duskreign
    Posts: 1,085
    I think you get one free with a large sandwich at McD...

    oh, Amen? I thought you typed small soft drink. I need new glasses.

    Anyway, I'm glad someone agrees with me.
  • gaaran
    gaaran
    Posts: 740
    My apologies, i didn't mean that we needed to keep the 5-star system, merely that to have a system (such as the "likes" method that has been tossed around) that allows us to acknowledge the campaigns we like gives people warm fuzzies. It need not be a rating system however.

    Also, the idea of simply having a favorites system, but being able to opt out of the updates would be a decent way to do it too.

    Oh, and Amen, too! (It's more fun if everyone shouts it together :)
Sign In or Register to comment.

August 2022
In Over Their Heads

Read the feature post on the blog
Return to Obsidian Portal

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Discussions