Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Fuzzy Avatars Solved! Please re-upload your avatar if it was fuzzy!

General: Forget the Scores

24

Comments

  • NBlitzNBlitz ZwollePosts: 1,904Member

    Originally posted by SBFord

    @nblitz

    Welcome to the American grading system employed at EVERY school in the country.

    Oh, so everybody's a winner. 

  • BillMurphyBillMurphy Managing Editor Berea, OHPosts: 2,377MMORPG.COM Staff Uncommon

    The thing is, we do have a wide range of scores late on the site.  And I feel that every score we've given in recent months has been justified by the reviewer.  The problem lies in the fact that scores ultimately remain subjective, just as the opinions contained within the written review do.

    We are neither "right" nor "wrong".  But we are always honest.

     

  • KyleranKyleran Tampa, FLPosts: 20,008Member Uncommon

    Originally posted by BillMurphy

    The thing is, we do have a wide range of scores late on the site.  And I feel that every score we've given in recent months has been justified by the reviewer.  The problem lies in the fact that scores ultimately remain subjective, just as the opinions contained within the written review do.

    We are neither "right" nor "wrong".  But we are always honest.

     

    No, the problem most folks had with your SWTOR reviews is that in reading your text we did see the positives and negatives you wrote about.

    What surprised us is that after mentioning the obvious shortcomings most of you went ahead with scores like 8.7 which in your own words can be considered "almost perfect".

    Call it grade inflation or whatever, but SWTOR likely deserves no more than a 7.5 to 8 in any stretch of the imagination.

    Which once again brings up the issue of the failure of your grading system.  Perhaps mimicing the American grading system isn't really good for MMORPGs?  There really doesn't need to be 6 levels of failure (ok 7, because in my house, Mom always said C stood for Crummy, not satisfactory) so that the difference between success and failure is a matter of millimeters.

    The 5 point system would be much better IMO, that way a title like SWTOR could have gotten a 4/5 (and been rated more consistently. (though you'd have to not fall into the temptation to rate them as 4.5 at launch trying to get them to 90%.

    So we do read your text, and we do think you're trying hard to be honest, however you also tend to be a bit too forgving for some folks tastes, but like you said, that's just your opinion.

     

    In my day MMORPG's were so hard we fought our way through dungeons in the snow, uphill both ways.
    "I don't have one life, I have many lives" - Grunty
    Still currently "subscribed" to EVE, and only EVE!!!
    "This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon

  • ArthedainArthedain NaplesPosts: 3Member

    Originally posted by BillMurphy

    The thing is, we do have a wide range of scores late on the site.  And I feel that every score we've given in recent months has been justified by the reviewer.  The problem lies in the fact that scores ultimately remain subjective, just as the opinions contained within the written review do.

    We are neither "right" nor "wrong".  But we are always honest.

     


     

    But if the scores are subjective, what's the point in using them?

  • solarinesolarine IstanbulPosts: 1,203Member

    Originally posted by SBFord

    @nblitz

    Welcome to the American grading system employed at EVERY school in the country.

     

    And *that* right there is the reason the US has no top tier universities - o wait. :P

    Honestly, though, it feels as unwieldy as the imperial measurement system. Maybe it's just because I haven't grown up with it. Or maybe it's because you guys have grown up with it that you don't realize how odd it looks.

  • teakboisteakbois Parlin, NJPosts: 2,154Member

    Originally posted by Loke666

    Lol, reviews in themselves are usually good but then they mess up the score.

    95% of all computer games get the score between 6/10 and 10/10, if not more. Average should logically be 5.5 which for some reason is no average score but something that completely and utterly sucks.

    I usually just subtract 5 and see it as out of 5 instead, it always give a pretty realistic score.

     

    This is actually pretty good advice.

     

    3.7/5....Sounds about right to me

  • itgrowlsitgrowls newport news, VAPosts: 2,951Member

    Soooo did they write this after the backlash they received in the forums about how corrupt their scoring system is (along with all of the other sites out there) that gave the poor excuse of an mmo SWTOR an almost perfect rating or what? Is this before the review or after the review of that game? (goes and checks) yep, it has to be because of the SWTOR review response. We know it can be hard to come up with a good review but to blatently fanboi your review on such a game tells all of us that you weren't being honest. Despite all of the problems of this game NOT a single website magazine spoke about those problems or included them in their scoring, those problems were swept under the rug in EVERY review which means that either someone got paid lotsa LA money or reviewers weren't paying attention. The damage to the game reviewer community's credibility has been done, apologizing for it won't change it.

  • ZX81SpectrumZX81Spectrum BordingPosts: 39Member

      I do not pay much attention to the score, but actually read what is written in the review.

      When I have read the pros and cons I can make up my own mind. The final score made by the reviewer may include "fluffyness" in terms of fun, aesthetics and/or love for a franchise. The eternal battle between objectivety and subjectivety (it's not bad to be subjective!) will always present itself as a problem when dealing with issues like putting a number to something that cannot quite be measured.

     

     

    40 Year old couples and gaming. Toons borrowed from Cyanide and Happiness.Text is Yours truly. Paint ftw :)


    image

  • TealaTeala SomewherePosts: 7,430Member Uncommon

    The very fact that you deemed it necessary to write an article asking your readers to "ignore" the score and read the words - are you kidding me?  Come on, be serious.  You know and I know that most people will look at the 8.7 and the word GREAT! as meaning just that.

    Instead of writing a review that is critical and than polishing it with an total overblown, un-related score - score it honestly.

    If the game is missing basic features, is buggy, and whatever...score it accordingly.

    Tell you what...be more honest and upfront with your readers and they'll respect you more.  They may not all like you and hate you, but at least they'll respect you for being honest.   Besides, I can't be the only one on this website that gets hate mails - help me out here and take some of the heat off me.  :)

    image
  • Whiskey_SamWhiskey_Sam Lynchburg, VAPosts: 294Member Uncommon

     


    Originally posted by zhivik



    OK, Bill, the main critcism to your scoring is the following: quite often, there is a very big gap between what a reviewer has written and the actual score. Taking once again SWTOR , about which I guess your article was written, here is an example:

    In the section "Innovation", the reviewer had said that there weren't really many new things, as this is your standard MMO, even if very polished. Yet, the score was 8. If 8 means "not much innovation", then one may wonder, what the grades from 1 to 7 mean? In effect, it appears that an 8 score means "ok", and then you are left with only 9s and 10s for better performance.

    So what's the point of all this? Why use a 10 scale in the first place, if you are only going to use the top 3-4 scores? Wouldn't it be better if you just move to a 5 scale, and save yourself all the trouble? I mean, it's simple math - if you have a scale from 1 to 10, you could presume that scores from 1 to 5 means "not good" in different intensity, and scores from 6 to 10 mean "good", in different scale.



    Going back to the "Innovation" example, I would be more accommodating if the reviewer had written that the introduction of fully voiced conversations and group conversation was a truly groundbreaking innovation, which is why the score is 8. However, the reviewer didn't write that, and you kind of wonder, does he have any idea what he is reviewing at all?



    So it's not like everyone is trying to be negative here, it's just there is a huge discrepancy between what you are writing and what you are scoring, and I think you need to remedy that. Because, after all, scores may not be more important than the review, but it's what people see first, so it's important the two not to differ too much.



     

    Beautifully stated.  This is the crux of the problem.  By saying an 8 is a B-, they're effectively saying anything below, what, a 5? is all the same.  They say they're scoring on a scale of 0-10 but only rating on a scale of 6-10 which means by default they are inflating all scores and confusing their readers.



     

    ___________________________
    Have flask; will travel.

  • ClassicstarClassicstar rotjeknorPosts: 2,690Member

    If you have OPEN mind and can be objective its not that hard.

    Also NUMBER ONE RULE never let someone review a game thats his favorite genre.

    You get almost guarenteed biased review.

    And dont let the hype and big budget influence your review you'll be alright.

    Im prolly asking to much here right mmorpg.com staff?

    MB:MSI Z97XPOWER AC
    CPU:Intell Icore7 4790k
    GPU:MSI 2x AMD 290X
    MEMORY:Corsair PLAT.DDR3 1866MHZ 16GB
    PSU:Corsair AX1200i
    OS:Windows 8.1 64bit)not yet sure i upgrade to windows 10 need to know alot more with integrated cloud and other maybe spy stuff)

  • ClassicstarClassicstar rotjeknorPosts: 2,690Member


    Originally posted by Teala
    The very fact that you deemed it necessary to write an article asking your readers to "ignore" the score and read the words - are you kidding me?  Come on, be serious.  You know and I know that most people will look at the 8.7 and the word GREAT! as meaning just that.
    Instead of writing a review that is critical and than polishing it with an total overblown, un-related score - score it honestly.
    If the game is missing basic features, is buggy, and whatever...score it accordingly.
    Tell you what...be more honest and upfront with your readers and they'll respect you more.  They may not all like you and hate you, but at least they'll respect you for being honest.   Besides, I can't be the only one on this website that gets hate mails - help me out here and take some of the heat off me.  :)

    Behonest and dont flip flap around themepark-sandbox-themepark-sandbox then hate mails(all tho i agree should not happen whatever opinion you have its stilla free country right?, there losers who sent them)stop, its simple like that. Analyse yourself be open and objective then see what you all write here on forum then it prolly solve it self hehe.

    MB:MSI Z97XPOWER AC
    CPU:Intell Icore7 4790k
    GPU:MSI 2x AMD 290X
    MEMORY:Corsair PLAT.DDR3 1866MHZ 16GB
    PSU:Corsair AX1200i
    OS:Windows 8.1 64bit)not yet sure i upgrade to windows 10 need to know alot more with integrated cloud and other maybe spy stuff)

  • TealaTeala SomewherePosts: 7,430Member Uncommon

    I think the 10 point system is fine, it is how it is being used that is not fine.   You shoud remove a point if the game is missing basic MMO features or mechanics.  You shoud remove a point if it has bad features.  You should remove a point if the  game is buggy.  That right there means that a game like SWTOR would have gotten a 7/10 from the start.

    Unfortunately, I do not think anyone will do that except some of us.  I do not have anything to lose by scoring a game a 5/10, a website like this that is run on advertising dollars - it can have an impact.

    Who is going to advertise with a website if their game is not given a decent review?  Not many.

    So places like MMORPG.com are between a rock and a hard place.   So who can  blame them?

    The only way a website like MMORPG.com could hand out reviews that give games proper scores would be for them to go as a paid service.  That isn't going to work because nobody is going to pay to see reviews or use a website like this.  Where does that leave us gamers who read reviews?

    Personally I am more likely to take a players review, that isn't paid to write reviews, over someone that is paid.   ::shrugs::

    Again that leaves MMORPG.com between a rock and a hard place.

    You know, I am rambling here, and I am not sure there is a clean solution to this whrere everyone can win.   This sorta sucks now doesn't it?   :(

    I just wish for once we'd see a paid review write a proper review and score a game accordingly.   I wonder what  the game industry would do if reviews started actually scoring games appropriately.  Imagine if all the "official" game websites scored SWTOR a 6/10 or a 7/10.   I bet Bioware would be scrambling to fix their game like no tomorrow.

    image
  • QuizzicalQuizzical Posts: 14,784Member Uncommon

    I like the idea of moving the scores to the end.  I was actually thinking that as I read the article before I got to where you said it.

    For a large fraction of games, a good review should simultaneously convince some readers that they should try the game, and other readers that they should not try the game.  (I only say a large fraction of games and not all games, because some gamse are truly awful.)  If the focus is on 10/10 or 6/10, then you can't do that.  If the focus is on saying things that are true about the game and not true about most other games, then that's a lot more useful.  A review that says "this game does X" can make some people think "I like games that do X" and others think "I don't like games that do X".

  • SovrathSovrath Boston Area, MAPosts: 18,462Member Uncommon

    Originally posted by Yamota



    For example, someone made a list on the SW:TOR forum about all the things this game was lacking. From open world PvP to guild functions to UI customizability to character customizability and found alot of things that are lacking, which are basic in most MMOs these days.




     

    Ok, but most of that list was not something I liked, thought important or thought was a big deal.

    So now how does one create a rating? I would have to hack out most of that list to make it palatable. Yet, for him, that list was his gold standard.

    Once again, reviews are subjective.

    Reviews should be based on what the game is trying to do and how well it does it. It should either be a letter grade (if we must have some sort of letter or number system) or only on a scale of one to five.

    Or, maybe just a yay or nay. Give the reasons why the game is successful/fails and then your opinion of yay or nay. Then others can read what the reviewer said and make their own decisions knowing that the bullet points that the reviewer outlined were what led to ther decision.

    So no the case of that SWToR list, that person would give a nay and we would know that those were his reasons. I could then look at his reasons and say "most of what he doesn't like are non-issues therefore I need to do more research.

    One of the movie reviewers that I really like is "movie bob" on the escapist. Not because I always agree with his opinion but because he backs up his opinion with thought and examplse. It gives a good foundation for his "yay or nay" and that has greater meaning than listenign to someone bitch about something (or praise something) because they just "feel" it should be this way.

  • MorcotulconMorcotulcon LisbonPosts: 262Member Common

    I have to agree with you Bill when you say that we should keep more attention to your (MMORPG staff) words then to the score you give. But sometimes your words aren't that good either. Even more important is that in every review you do, wether the words are "neutral" enough or not, the score is ultimately what EVERYONE sees and will compare the game to. AND scores are what will make new players get interested in the genre, not words. If they see a high score and see it as a fail, they will get interested in other stuff for a long time (or forever).

    Loke666's way of scoring games is a good one and I took that idea from him before in other posts about scores.

    If you, MMORPG staff, won't leave the numbers scoring system then why don't update it to 0-5 scoring system? You are an MMORPG dedicated staff so you KNOW this is true: games need real innovation and the genre is lacking progress, that can only become possible by listenning to their fans and possible clients opinions, following their ideas the best they can with something big and bold, and the most important thing is to fix the major flaws in the genre, like KS, griefing, etc. 

    I think you have the capacity and the writers to fix your "flaws" too! To fix the score system and fix the things people like less/hate more in MMORPG site. Listen to us, try following the opinions of your clients and make  a scoring system where everyone will agree or slightly disagree, instead of keeping this one and having the same enormous quantity of bad "feedback" everytime you make a review.

    It's true there are haters and lovers for games, but the scoring systems we have in todays main sites and forums are like wood to fire, the more you give the more it will burn.


  • someforumguysomeforumguy HomePosts: 3,540Member Uncommon

    I see some problems with the reviewing on this website:

    If a reviewer loves the game (clearly noticable in the review), then he tends to make excuses for things missing or not good enough. He/She mentiones them, but it doesnt get reflected in the score of a category. This is because if you feel that the game is so tremendously fun and it should get a 9 for that, you tend to work back from that so you can give the respective categories also high scores.

    If a reviewer doesnt seem to care for the game (review doesnt reflect either way) , these flaws get reflected in their scores. I really think that you guys should be more aware of this.

    The reason for this imo, is that 'fun' isnt a category that can be treated objectively. But it does show that the used different categories (graphics, sound etc) dont make up how fun the game can be. This is why the endscore based on the different categories is nonsense and why it gets flamed so much.

    So, while I think that the good and bad lists are a good idea, the endscore should go. Or it should be put seperately. By making sure that the reader understands that it isnt an overall score based on the different categories, but a score based on the reviewers personal experience (subjective). Otherwise it wont work.

    Another reason for this is, that a lot of players dont give each category the same weight. Which also means that the endscore is useless for them.

    So, please get rid of the overall endscore or turn it into a personal score from the reviewer. Then, list games in the gamelist based on overall reader's score. That average makes more sense for readers that just want to know the score, then the score from one reviewer. The readers that take the time to read the review care less about an endscore anyway, but probably more about the pro/con list and the review itself.

     

  • NBlitzNBlitz ZwollePosts: 1,904Member

    Originally posted by Sovrath

    Originally posted by Yamota





    For example, someone made a list on the SW:TOR forum about all the things this game was lacking. From open world PvP to guild functions to UI customizability to character customizability and found alot of things that are lacking, which are basic in most MMOs these days.




     

    Ok, but most of that list was not something I liked, thought important or thought was a big deal.

    So now how does one create a rating? I would have to hack out most of that list to make it palatable. Yet, for him, that list was his gold standard.

    Once again, reviews are subjective.

    Reviews should be based on what the game is trying to do and how well it does it. It should either be a letter grade (if we must have some sort of letter or number system) or only on a scale of one to five.

    Or, maybe just a yay or nay. Give the reasons why the game is successful/fails and then your opinion of yay or nay. Then others can read what the reviewer said and make their own decisions knowing that the bullet points that the reviewer outlined were what led to ther decision.

    So no the case of that SWToR list, that person would give a nay and we would know that those were his reasons. I could then look at his reasons and say "most of what he doesn't like are non-issues therefore I need to do more research.

    One of the movie reviewers that I really like is "movie bob" on the escapist. Not because I always agree with his opinion but because he backs up his opinion with thought and examplse. It gives a good foundation for his "yay or nay" and that has greater meaning than listenign to someone bitch about something (or praise something) because they just "feel" it should be this way.

    I would be totally down with that! Understated and doesn't take over the review.

    Because it works so well I never paid much attention to it and simply when straight to the pros/cons list. Unless something scored a 1 or 2 out of 5 then I'd really pay attention why.

     

    4.5 out of 5 image for you, Sovrath. And everyone else using a similar system (deducting 5 from current scoring norm etc).

  • black_isleblack_isle IstanbulPosts: 239Member Uncommon



    Originally posted by SBFord

    @nblitz





    Welcome to the American grading system employed at EVERY school in the country.










     

    Just because it is used in american high schools doesnt mean that it's sensible, applicable to all situations or it has to make sense to the wide international audience you have here on the site. I mean that was a pretty ignorant thing to say and is pretty ignorant way of looking at things.

    And the point of scores is that they are supposed to be an objective reflection of a game merits in  numbers, not  "yea there are flaws and it's unoriginal and bla bla but omgomg i LOOVVVEE this game and SW franchise, this game is so FUNNN!!11!!!".  Noone really cares about particular person's biased opinion about a game when they read a review. They want a professional and objective evaluation of the product.

    This is not high school. There is a product made by a company for profits and people are wondering what's the deal with it and if they should spend their hard-earned money on it. So it is your responsibility as professionals in this business to be more serious about it and give an objective look at things.

    Cheers.



     

  • MimzelMimzel KristiansandPosts: 375Member

    Bill, your article came across as saying: "Read our words because we try to be honest, but dont look at the score. We all know its just BS."

    I think we all anticipate a game receiving a 8.7 would be almost flawless. Not saying it cant have any signs of being a new fledged game, but it cant have much faults. This ISNT the case with SWTOR, and we all know it.

    Giving SWTOR 8 in innovation, yet saying it isnt very innovative... That removes the scoring systems credibility right there.

    JBFord saying "welcome to the USA high school scoring system" doesnt help your case much. What are you really saying? You like it dumb over there?

    1-5 are hardly EVER given. Even the worst games dont go that low. Which obsoletes HALF of the scoring scale - leaving you with less wiggle room to score a game. Further more, 10 is NEVER given because it is a utopic score. Its been like this forever.

    So on a scale seemingly from 1-10, you are in reality left with ONLY 6, 7, 8 and 9. Its not even a 5 point scale... 9 is in reality the highest you go, and giving swtor 8.7 strips away your credibility to a whole new level of nonsensical.

    Your problem is

    a) You get advertising money to pay your staff. Bias. Minus point for cred.

    b) You cant backpeddal and change the review now without admission of guilt. It will probably be best for you to keep your current score on swtor. At least some of us will be able to keep our rosetinted glasses on a bit longer.


  • VhalnVhaln Chicago, ILPosts: 3,159Member

    I don't understand the comparison to the American school grading system - there are good reasons you need at least a 60 to pass.  It's a percentage, and if you get below that, it means you're getting half the material wrong.  That is abysmal.  If you study, its actually not that hard to get over 90% - that basically just means you're paying attention.

     

    Compare that to making a game, or any other form of entertainment.. and its really not comparable.  It's not a pass/fail system.  Getting a 6.0 doesn't mean the game is 40% wrong.  It really doesn't make any sense to grade games the way you grade school kids.

     

    Most importantly, Consider the way the numbers are used.  A student needs to learn the material to progress, ideally doing so at as close to 100% as possible.  There's no reason to aspire to grade disparity.

     

    An review rating system, OTOH, is more about comparing different products.  The more disparity, the better.  The more you cram all the ratings into a narrow top margin, the more meaningless they become, because they no longer function as a basis for comparison.

    When I want a single-player story, I'll play a single-player game. When I play an MMO, I want a massively multiplayer world.

  • krawllkrawll portland, ORPosts: 18Member

    Bill,

     

    Please don't micro manage the site...you're above that.

     

    We live to snivel about the scores, nothing is going to change that. We read what you say, we just disregard it, and form an opinion for ourselves, and even if yours is similar to ours, you'll still be an idiot. Thats the only thing we can all really count on...not the scores...not the games... not the red light at the Krispy Kream... its the fact that we are all idiots, but we have opposable thumbs, so we got that much going for us.

    VIVA LA SCORE-ITA-CHALUPA!!

     

    No...really... don't touch the scores... or ...trunk monkey!

  • LawlmonsterLawlmonster Dallas, TXPosts: 954Member Uncommon

    Originally posted by Vhaln

    I don't understand the comparison to the American school grading system - there are good reasons you need at least a 60 to pass.  It's a percentage, and if you get below that, it means you're getting half the material wrong.  That is abysmal.  If you study, its actually not that hard to get over 90% - that basically just means you're paying attention.

     

    Compare that to making a game, or any other form of entertainment.. and its really not comparable.  It's not a pass/fail system.  Getting a 6.0 doesn't mean the game is 40% wrong.  It really doesn't make any sense to grade games the way you grade school kids.

     

    Most importantly, Consider the way the numbers are used.  A student needs to learn the material to progress, ideally doing so at as close to 100% as possible.  There's no reason to aspire to grade disparity.

     

    An review rating system, OTOH, is more about comparing different products.  The more disparity, the better.  The more you cram all the ratings into a narrow top margin, the more meaningless they become, because they no longer function as a basis for comparison.


     

    It's all right here. If reviewers don't understand why they need to match writing to score, or why there needs to be full use of the ten point scale instead of 8 - 10, I'm not sure there's any credibility to find in that particular professionals' perspective. I'm sorry, I really want to believe MMORPG.com is doing right by their readers and the work they author, but there needs to be a greater attention to providing an honest, credible, and evaluative score in reviews.

    "This is life! We suffer and slave and expire. That's it!" -Bernard Black (Dylan Moran)

  • SovrathSovrath Boston Area, MAPosts: 18,462Member Uncommon

    Originally posted by NBlitz

     

    4.5 out of 5 image for you, Sovrath. And everyone else using a similar system (deducting 5 from current scoring norm etc).

    Darn, I always lose points in the swimsuit portion of the competition. image

  • ClerigoClerigo MatosinhosPosts: 400Member Common

    Originally posted by BillMurphy

    Let me just say that I find it very funny that I ask you to put the scores secondary to the words, and the comments quickly became all about the scores.  :)

    I get your concerns folks, really I do.  But we are trying to do our best service and honest recommendations on every game we review, which means we're writing about the good and the bad equally, which makes the scores seem unjust.  That's sort of my point.  But rather than us NOT write about the bad, I'm asking you to realize that a score of 8/10 can mean a game's a lot of fun despite its faults.

    Additionally, the stance we take on scoring in general is akin to a Grade Scale in any American High School.  Anything below 6 is likely considered failing, a 7 is a C, an 8 is a B, and an A (90/100+) is damned near perfect. 

    Again, you may not agree with this method, but it IS how we do things and we ARE always evaluating things.




     

    I dont want to start anything close to a hate war with my question, but i do have to ask, and please say if you find me to be out of line.

    Taking in consideration the rate system you just gave, can you please make the bridge between reviews mmorpg.com gave to games that were already sinking ships at the time of theirs launches (AoC, WAR, AION) by being full of bugs, not completed content wise, game design flaws, etc, and the recent review this same site gave to SWTOR?

    Do we have to expect a possible thunderstorm eveytime a reviewed game doesnt get a perfect 10?

Sign In or Register to comment.