In a lot of ways review scores are saying either "buy", "try" or "avoid" the subject they are reviewing. Past that point, a lot of review systems are window dressing.
Personally I appreciate a review that has a short list of pros and cons around a title, plus I think you could also have a "Better than..." and "Worse than..." comparison (and that will still generate lots of site traffic as fans of all games come in to tell you how wrong you are!).
On the subject of bias: be open. Have a Bias Box in every review saying, "This review was written during Patch 1.0.1.1.3. We were provided with 5 complimentary copies of this title for review, the author is best friends with the lead developer and they had a beer every Friday on the company dime." Bias might exist, but be open about it so that readers know they aren't being misled. If they choose to disregard a review because of "bias", then I'm sure they will find a reason anyway.
Agree about needing different reviews for different periods. A Day One review, A Month One review and a Year One review could cover the different experiences / evolution of a MMO.
Why not use a system that professional analysts use for stocks, specifically the sliding bar of sell, hold, and buy. But in a reviewing case, do not try, try the game out, must get NAO!!!
Then with the use of multiple reviewers can get a fairly consistent and hopefully accurate review of the game itself across a relatively large sample.
i dont think having no scoring system at all is a good idea. ill be honest and state that unless its a game im extremely interested in, i'll rarely read a full review. ill normally skim read or go straight to the bottom and read the conclusion and look at the score. im sure there are many more ppl out there who do this. w/o an at glance review score.
i would probably start looking to other websites for my reviews. im lazy and dont have enough computer times to spend 15 mins reading an entire review
If someone is talking in general chat in a language you dont understand, chances are they're not talking to you. So chill out and stop bitching about it!
The problem, as I see it, is that it seems MMORPG.com doesn't really have any idea at all what makes a good MMORPG. Maybe some of the folks working here do...but the criteria you use to review, and the various things I've seen on this site (polls, blogs, etc) all make me think that your perspective on MMORPGs is dictated completely by the modern conventional wisdom in the genre. Sadly, the modern conventional wisdom leads to failed games or games with niche player bases. It's actually no surpise that the games you guys give the highest scores to end up losing most of their subscribers after three months. It's completely predicatable if you are looking at the right things.
Think of it this way... Say you are a site that reviews new automobiles. Say everyone in the industry is talking about the new paint colors, new door locks, and new interior fabrics that are being used in modern cars. Now say you base your review of new cars SOLELY on those things. Paint color, Interior fabric, and door locks.
Obviously, the score you gave to a car would very often be completely irrelevant, and not predictive at all of how successful that car would be in the market. Why? Because you aren't reviewing the things about a car that really matter. How reliable is it? How powerful and fuel efficient is the engine? How well do the breaks and steering work? Without reviewing the aspects of a car that really make it a good car...you can't possible write a useful review.
The exact same holds true for MMORPGs. You don't review based on the elements that really matter. In fact, you often don't even mention them...and if you do it's in a passing way that doesn't seem to end up affecting the overall score at all. Things like depth, challenge, replayability....the realism, internal consistency, and immersiveness of the world...the scope and size and ambition of the design. How invested does the game make you feel in the world and in your character? How much of a sense of danger or excitement or adventure does the game create moment-to-moment? How meaningful are the rewards (how much investment and risk are involved in obtaining them)? Is the game accessible, but still DEEP...are you still learning and mastering new and interesting elements even hundreds of hours into your play, or do you have the game down in the first 10 minutes and never really have to do anything different? What new elements and new ideas is the game useing to IMPROVE the above things over previous games?
The conventional wisdom in the post-WoW genre leads you to believe that all the wrong things are important. People have drawn the wrong (or at least incomplete) conclusions about what made WoW successful, and run with them as fast as they could. All we get now are watered down, ever-more-shallow, ever-quicker and simpler and less involved games. You're reviews are based on these exact criteria...so the worst offenders in terms of creating shallow and short and meaningless experiences end up getting the highest scores on your site. You might think of it this way (although it isn't a direct cause-and effect, it might as well be), developers are creating games specifically for your review style...instead of making good games.
If we go back to the car analogy, car manufacturers are now making cars with really awesome paint jobs and the very best fabrics in the interior...but they have extremely wimpy and inefficent engines, terrible brakes and steering, and are basically death traps. Since you only review the paint and the interior, you'd give these cars high scores....and that's exactly what you're doing. Giving games that are doomed to lose most of their players in a short time the highest scores as being the best the genre has to offer. It's not reallly that you're being intentionally deceptive, it's really just a matter of priorities and a deeper understanding of what makes a good MMORPG. This lame and over-simplified idea of "fun" which centers around instant gratification and constant reward...the idea that a good MMORPG is one you can play entirely solo and complete entirely in 15 minute intervals from your smart phone... It really just seems like you guys and gals don't have a clue of what this genre is supposed to be about, what sets it apart from other types of games, and what makes the core players in this genre excited and keeps them playing.
The timing of your review really shouldn't matter. I could tell you after a few hours of play whether or not an MMORPG is going to last. If you are looking at the right criteria, it's almost immediately apparent which games are created to be long-term virtual worlds that will draw in players for years, and which are just shallow little dances that get big box sales and then disappear. First you have to recognize what the genre should be shooting for...long-term truly social experiences where you make friends, build a community, and get wrapped up in the world for years. Where you have memorable experiences and meaningful rewards rather than loot pinatas and a forgettable grind. You have to agree on what an MMORPG really is...then you'll be able to spot the great games without much trouble, and you'll probably be able to provide a solid review even before the game launches.
Are things like polish, playability, smoothness, and accessibility important? Sure they are...and a truly great game is going to provide those things on top of the elements that really matter. You need to recogize that it's perfectly possible to have a great MMORPG that launches with some bugs...but it's impossible to have a bad MMORPG succeed in the mass market simply because it's polished.
If you need help figuring out what's important... just ask. I'm sure there are plenty of folks who would be glad to help, me included.
The only problem I have with the reviews is that they are way too generous. Professional critics have a job to do, they have a responsibility. If they say something is okay, they're making it okay.
Nowadays we have a slew of identical MMOs churned out to make a quick buck at launch, with almost no attempt for innovation. Just throw together some instances, battlegrounds and a gear grind. Done. Stop saying it's okay.
Having to review and MMO is darn near impossible. With all the different classes and combinations, and crafting classes how can anyone get a real idea of what the game fully offers (end game and all)? With that said here is my 2 cents:
When I read a review what am I looking for? Why would I the reader even bother reading the review unless I am trying to get something out of it. Here is what i am looking for:
1 - Is the game worth the purchase price (whatver it may be) and will it keep me entertained for the initial 30 days?
If I read a review and it can tell me the above, we are off to a good start. What does the game initially do well, what does it do poorly. Can I get on a server and play for more than 20 minutes without being kicked off for maintance? Is 30 days all I am going to need to get the most out of the game?
2 - Is the game worth the monthly fee moving forward?
Game could be great, but is it worth the $14.95 per month? Is there more to see that is worth 14.95 after the 30 days is up? This question is more tricky because instead of comparing it to other games charging the same price, there are lots of free MMO's out there are really good and cost nothing. How does it stack up to those games as well.
Also let's not assume just because it's an MMO and we are all fans of MMO's that just because it does one thing different or has a few different classes it's worth 14.95 a month. What does it do that hasn't been done before? What does it do that has been done a hundrend times before that it does better than anyone else? Is the world unique and worth the investment? Is the company committed to the product or is it a 2 month experiement to see if it hits?
3 - What are the different play styles if offers? Is it PvP centric, PvE centric, a REAL mix, does it have crafting?
All of these questions need to be answered, I have read reviews on other sites that have said games are a mix of PvP and PvE only to find the game was primarily PvP with little if any PvE content. I want to know if I like playing a melee class or range class what does the game offer to me? Is it solo, duo or group friendly or is it hardcore raid centric? is it a mix (been fooled here alot as well). Here is where to take fun into account. Each person has their own idea of what fun is, but here is where you can say if raiding is your thing and something you find fun, this game is for you. If you like to duo with the wife, maybe you stear clear of this. If your thing going solo is fun, maybe this isn't for you. this sort of information is very important.
4 - Going along with #3 what is the long term outlook with the game (taking company history into account)?
The last thing I want to do as a gamer is waste months building a character only to have the servers shut down and the game go belly up 6 months down the line. Does the company have any history behind it? What is the buzz on the game? Do the servers look alive? Or is this one of those games that after the first 30 days is up people will be packing up and leaving? As a fan and as people who review these games, it shouldn't be THAT difficult to get a first impression like that. Now granted, no game is going to have WoW's players base, but is the game sustainable?
These are the main points I look for in a review, the number/letter overall score really doesn't matter to me all that much. As you mentioned what is the difference in game that gets a 6.5,7.0 or a 7.5? Are they all worth 14.95 a month or just the 7.5?
If you wanted to give each catagory a seperate score than tally it up at the end, that could work. That way at least we could go to specifics and look for what is important to us. For example if the game is worth the 14.95 a month but had a rough launch, perhaps for #2 they get a good grade but for #1 they don't pending some updates or patches.
Thanks for asking for feedback, it's always great to see sites being well engaged with their community. I put some thoughts together and had trouble getting them into a comment post without losing the thread - so decided to stick them in a presentation instead.
I can't embed the presentation here so hosted it on my own server: can access it <here>.
Good luck sorting out your approach - it's actually a pretty fun problem to be wrestling with imho.
I made an extensive suggestion here, but wanted to stress the point that you need to use sub-genres. In other gaming sites people quickly see "Racing Game" or "Platformer" and the like and it sets their expectations. WIth sub-genres I mean "Sandbox", "Themepark" and the like, maybe you find better sub-genre names.
The burden (and joy) is usually on journalists to come up with genre names, hence I think you guys (and others) are still a bit clueless about your role. Grow up.
1) MMORPG.com could setup an online questionaire form using their assessment criteria. They could get people to take a bartle test as part of the questionaire and then use the result to rank the game for different player archetypes.
2) They could also work with Beta testers to do the same before launch with help from the developers.
3) MMORPG.com is in a unique position to hold developers to account and to be a useful source of information for development teams - if they set up 1) and 2) then they can use this information to bargain for involvement during development and so be a real "voice" for the community.
I think that behind the answer to the review methodology lies a lot opportunity to do good and secure a key position in the development cycle. If it was my organisation I'd be thinking a lot more strategically and figuring out ways to leverage such a committed readership.
I think Fozzik nailed it right on the head. Too much emphasis is put into the wrong parts of the games and too much bonus is given to certain concepts that really are just function.
Honestly when it comes down to it, I read reviews for sake of something to read. If I want to decide whether or not I'm going to play a game I check out as much information as I possibly can before hand. I've read some reviews that never give any information that wasn't already available.
First, thank you for reading our feedback on other topics to address the rating system. Yes, the rating system used for MMORPGs is currently messed up. Given the complexity of MMORPGs, here are some ways you can address the situation:
Give a grade score (A to F, N/A or Incomplete) for different aspects of the game (Graphics, PvE, PvP, Crafting, Content, Sound, Long Term Playability, ect.) and a final socre (1-10) for the initial preview and then another for full review. Clearly some of the grades for the game aspects can be either N/A (for example, game doesn't have PvP) or incomplete (meaning that it cannot be determined in the initial preview). Then you can complete some of these grades when the final review comes out. Players should also be able to vote on these aspects like we vote for the overall score of a game.
The benefit of such a system allows people to know how you feel about different aspects of the game and we know what you haven't had time to experience (content, PvP and Long Term Playability probably the main things incomplete on initial release review). Furthermore, we can figure out if the game is good at certain things. As a PvP player, I'm more interested in the PvP score and Long Term Playability in an MMO, and less concerned about Sound, Content and PvE. I'm sure others are more interested in PvE, Content, Graphics. Right now, I have to search and read your reviews to learn about these issues, and it would be just easier if I could see a grade rating on these, then read up on it if it interests me.
As for the 1-10 scoring system, G4 TV uses the 1-5 system but I think they stated how the scores are granted (and could easily be applied to MMORPG.com). A score of 1 (or 1-3 for MMORPG) was simply terrible and/or unplayable with bugs. Shadowbane's sb.exe error disaster around launch comes to mind as deserving of this score. A score of 2 (or 4-5) is a bad game, or doesn't do what was advertised but doesn't crash because of bugs (e.g., Age of Conan perhaps at release since the PvP was broken and DX10 wasn't included and beyond Tortage the game lacked content). A score of 3 (6-7) is average game, and MOST games should fall into this rating (e.g., RIFT, etc). A score of 4 (8-9) should be for exceptionally good games with lots of replayability and a score of 5 (10) should be the rare game that just rocks on all factors.
The downside is that most games will have a score of 6-7, which right now MMORGP.com keeps most ratings at 7.0-8.9. But I think having a more formal rating system would be better. If you read the re-review of AION at MMORPG.com, you'll see the difference. That re-review said the game was better than it was at launch, yet gave it a lower score ("because it feels like an 8.0" if I remember the quote). It was this that eventually turned me off to your scoring system, because it depends on the whims of the review and had no standard.
This website's reviews on games that have PvP should be done by people who enjoy PvP (or allow people who enjoy PvP score that part of the game). I'm absolutely sick of hearing how the person doing the review doesn't like to PvP, and then goes ahead and discusses the good and bad of a game's PvP system. I don't remember exactly which review that was, but I read it on this site. I personally hate PvE and I know there is no way I could review that part of a game for people who enjoy PvE.
Finally, when a game receives a relatively high score, a follow-up review should be sooner if possible when the game changes rather quickly. DCUO gameplay changed DRAMATICALLY 1 month after the initial review, as population levels dropped dramatically making it a very different experience. Leaving the very positive review score unchanged despite a dramatically different game experience I thought was not professional. This website then had a follow-up article saying that everything was great except lack of population. Well, lack of population is a major problem, and it almost sounded like it was the players fault for not recognizing how great the game was. I believe that when a game's population declines dramatically after launch that received such a high score, there should be some kind of managerial control or intervention at MMORPG.com to see what went wrong in the review or what was missed, and if necessary correct the score in a re-review.
I know this was a long post, but I like MMOs and I'd like a good rating system so I know what to buy and try and what to avoid. Right now, I can't do that with the current system.
I like a combination of your ideas. Use the letter system, as its easier to understand what constitutes an A over a B and then use the rating of different areas of a game and average those findings for a final letter grade. Sort of like giving each game a GPA score.
this is exactly what I was thinking. Letter grade system is almost universally understood. Everyone has a general understanding of the difference between a A vs a B. But still make sure to incorporate aspects of the game into perhaps catergories giving an explanation of what you experience good or bad, so we know what to expect overall. I feel that once you pass below a 5 the game is pretty much crap at that point, labeling it with a 1 or a 2 isn't going to really make the damage any worse so you do in fact end up with half a scoring system that is useless.
I would still go with full out reviews, though Previews are nice, but I feel a review is better as i want to know if the game is worth my intitial purchase, and not only that, will it be worth a continuing subscription or perhaps I should wait a couple months. The more informed I am, the better decision I can make about where to put my money.
3.4ghz Phenom II X4 965, 8GB PC12800 DDR3 GSKILL, EVGA 560GTX 2GB OC, 640GB HD SATA II, BFG 1000WATT PSU. MSI NF980-G65 TRI-SLI MOBO.
Stick to the 10 Point scale. As you say, it works well with MetaCritic and you want your reviews to appear over there as well. But [1] enforce consistency and assign each number with a [2] proper meaning, which you should also [3] spell out right next to the number or by providing a link to a rating policy page, where you may also conveniently explain your criteria. The reviewer should first have a verdict, only then translate it to the scale. Here is a take with keeping in mind that 5 should be the true average.
10 — Perfect
09 — Excellent
08 — Very Good
07 — Good
06 — Above Average
05 — Average
04 — Below Average
03 — Bad
02 — Atrocious
01 — Abysmal
Categories...
I just wanted to quote this poster because THIS (pay attention all Games Reviewers!) is how the average person views the one to 10 scale!
NOT 1-5 plus 5! that you guys use... (see the quote here from the original article):
Next, I wanted to talk about scoring systems. over time, I’ve become more and more annoyed with the 1-10 scoring system. The trouble is that numbers 1-5 are very rarely used because they represent a failing grade. For me to give any game a 5 or below means that it probably didn’t even run on my machine or it so wildly missed the mark that it doesn’t even resemble a running game anymore. So why, if it’s never or rarely used, are we taking up half of our scale to say the same thing: This game totally fails.
Then there are the degrees that a 1-10 scale creates. Who, other than the reviewer, really knows the difference between a 6.5 and a 6.6? How does that difference stay consistent from reviewer to reviewer? Heck, how does the scoring system stay consistent from reviewer to reviewer anyway?
I know that 1-10 is how most gaming sites do things, and many of them do it because it puts the score into a nice little package for MetaCritic. So I’m wondering, where MMORPG.com is concerned, how important is it that we stay on par with everyone else? Do you all enjoy comparing the numbers from one site to the next? Is that helpful for you as a consumer?
Well to us consumers a game that doesn't run = 0... not 5.
And don't be so scared of saying that a game "fails"... some of them really do!
A score of 1 says "Abysmal - was this game written / programmed by primary school kids with no talent?"
A score of 10 says "You MUST buy this game - because if you don't you will be a social lepper when everyone else in the world is playing it because, yeah, it really is that good."
And you may never give out 1s or 10s, again, so what?
Ah... but then it's not fair is it? I mean a game that was an 8 in 2001 cannot be fairly compared to a game that is an 8 in 2011? Well no. So what. I want to know the date of the review and I judge it based on that. An 'average' 2011 game is bound to be vastly better than the original UO. I expect that. It's fine - really.
So, re-review the games that stick around every couple of years if there is a demand or you hear something has changed vastly improving the game.
Maybe even say "In 2005 this game was the trend setter with an overall score of 9/10 and 6 years on it's still a contender 7/10."
This is what we need to decide if we will actually invest our time and money.
If you want to break down by categories - please do - but overall score is where people look.
Other than that - some very good posts in this thread.
This isn't criticism of the review process, but I think this page is a great example of why the site needs a newer comment system with pages, sorting, thumbs up/down, search etc. Any site with a strong community that encourages discussion benefits from more organized comments and forums.
Just another thought -- my actual review comments are somewhere above, heh.
You guys aren't critical enough of game mechanics which don't work and too often praise a game's ideas instead of criticizing execution. How did Mortal Online's six month review get a 6.9? On any legitimate game reviewing site it would receive a 1.0 out of 10 and that's because it doesn't work, not because it's too l33t for everyone.
In addition, I often feel like reviews are given based on day one gameplay, like how Rift got an 8.7. Also, how does a game which so blatantly copies WoW get a good review on the site which is constantly calling for Ultima and SWG remakes and new sandbox games to be developed? It's like you don't want to hurt the feelings of the people that make these games.
I think the game list here is very helpful. being able to sort the game score. correct me if I'm wrong, but isnt the score posted there for a game an average of all submitted scores by registered members? I dont see how you can get a better idea of the "general" feel for a title than this...
The debate really comes down to:
1. The data used for the score
2. How to capture and display the data... 1-5 or A-F or 1-10 or 1-100...
On my first point... the scores should be cleared on a regular basis... using the current 1-10, a game that was a 6 at release +6 months and then 4 years later may have evolved into a 8 is still being dragged down by now obsolete reviews... maybe list a "lifetime score" and a last year score... For instance... I sucked at math in the 5th grade... but became a A student in 8th... if they still factored in how I did in 5th grade in 8th grade I might get a C which does not rate my "current" ability...
On my second point... 1-10 seems fine to me... it allows for easy averaging... maybe tweek the catagories... like customer service... its not really a valid listing unless you had to use thier CS... make the catagories so you can submit a rating or opt out, and your score in that catagory isnt used... most games Ive reviewed Ive never used thier CS... so I cant give any score... but am still forced to decide 1-10... it immediately skews the # no matter what I decide... I use 5... another uses 0...another 10... no consistancy...
......
As for staff reviews, have thier numerical review simply be part of the rest of the sites users... give a written headline review and let that stand as your verbal rating and have your numercal assessment fall into the pool with the rest of us... come on in... the waters fine... it just needs to be changed(see my first point) from time to time...
And the staff who is assigned to that game, like my suggestion for resetting the scores every X timeframe. Should also revisit the game and revisit thier review every X...
No system in opinion land is perfect, there are just better ways to collect and display data...
One other thing about this website reviews that annoys me, that unfortunately i thought of after my lengthy post. Some of your "reviews" actually state that they didn't have much time in the game, and then go on to review the game. Seriously? What's more important at MMORPG.com: getting a review published or doing a review correctly?
I swear, I have read that in the past couple of months a guy said he didn't have much time in the game because of personal issues, and then goes on to review the game given his limited playtime. WTF? If I had done that at my job, I would have been given an official Human Resources warning or something. Either do it right, or don't do it.
My idea relates to the site members' numerical ratings.
Members should be able to check some boxes about themselves as to their gaming preferences; e.g., PvP, PvE, etc.
That way, once the ratings are given, you could see how PvPers, for example, feel about a particular game versus PvEers.
The net result, then, would be a rating for a game that looks like this:
Darkfall Overall rating 8.7
PvP player rating 9.2
PvE player rating 5.1
And so on for other possible breakdowns. This would be hugely useful because a PvPer probably doesn't care about what a PvEer thinks about a particular game, and vice-versa. At a minimum, when a PvPer goes looking for a game, he or she is going to first want to see how other PvPers rated it.
I think this would be a lot more useful. Otherwise, all gamers of all types are thrown into a pot together on their ratings, which doesn't really tell you much.
EQ1, EQ2, SWG, SWTOR, GW, GW2 CoH, CoV, FFXI, WoW, CO, War,TSW and a slew of free trials and beta tests
I have not read all the comments so my apologies if my ideas have been presented earlier.
I think the best way to review MMOs would be to take a group of 5-10 people who would play the game together. Among all these people will be varying tastes and interests in MMOs. You'll have your raiders, crafters, chatters, AHers, etc.
By forming a kinship/guild the group will be able to test out that portion of the game. With enough people you can test group functions and dungeons. Some people will be looking at how "easy" money is to come across in the game and how much would be needed to perform various items/skills etc.
The review would be labeled as "on-going". The goup would continue all the way to "end game". Whatever the max level is. The goal being to explore all parts of the MMO with a group of people. Each person or persons would be responsible for parts of the review. Put all the pieces together and you would have a fully fleshed out review. All aspects of the game would be tested and reviewed, and enough time would be given to ensure a thorough look-over was done.
I don't know how feasible this approach would be, but I believe it would garner the best review possible.
I think that the best solution would be to have 3 reviews of the game. Preview review, a couple weeks after the release then a follow up after a good impression of the game is set.
Then to work out the 'scores' of a game, I think going to a total out of say 50 is best. However to combat the fact that gameplay>sound, rate gameplay out of 10 (because it is more important) and sound out of 5 (because it is less important). This way good gameplay has more of an impact than good quality sound.
Improvents to this method still are needed but is a nice foundation.
I'm of the opinion that reviews on MMORPG.com are quite reasonable.
Too many people think that if a game is not to their liking, that makes it a BAD game. This isn't the case. Many don't like certain things, while others do.
The reviews tell me what type of game it is and how it stacks up against other games of that style. For me this is helpful. If there's lots about the game that are things I don't find enjoyable then I'll skip further interest in the game.
Ken Fisher - Semi retired old fart Network Administrator, now working in Network Security. I don't Forum PVP. If you feel I've attacked you, it was probably by accident. When I don't understand, I ask. Such is not intended as criticism.
One other thing about this website reviews that annoys me, that unfortunately i thought of after my lengthy post. Some of your "reviews" actually state that they didn't have much time in the game, and then go on to review the game. Seriously? What's more important at MMORPG.com: getting a review published or doing a review correctly?
I swear, I have read that in the past couple of months a guy said he didn't have much time in the game because of personal issues, and then goes on to review the game given his limited playtime. WTF? If I had done that at my job, I would have been given an official Human Resources warning or something. Either do it right, or don't do it.
No - I disagree. When I see a review I like to know how long the reviewer spent with the game. Don't need to know why (keep your private life to yourself thanks... thats why its called private :P )
But the time spent with a game is very important - particularly with MMOs.
As we all know - most games have a 'honeymoon' period. It's easy to play many MMOs and think they are the best thing coming if you only get 24 hours access, limited access to certain features, certain areas, certain classes.
Indeed, we have seen this in the past and are still seeing it (SW:TOR anyone?).
I don't care if the reviewer cannot review the whole game so long as they are honest about what they did review and under what conditions.
Better that than reviewers who even lie (I caught a reviewer out completely once - implying he had tested "multiplayer" in a game where the multiplayer function had not been functional until after his review had been published.)
But, I must admit, any MMO review that spend less than a week on the game or is done based on any restricted conditions really isn't worth the reading time.
10 : Perfect (like if it was even possible from an MMO at release...)
I kinda like this more drastic scale.
To me the real issue is that your score (be it an aggregation of sub-scores or a plain overall grade) has few meanings if you don't specify which player profile the score applies to.
Not only would you need to refine scores as the game evolves (pre-release, at release, one months in, 3 months in, 6 months in, 1 year in, 2 years in, etc...), you also need to have one score per profile:
Casual
Sandboxer
Raider
etc
What you could do, is use grades per "topic" in the game (graphics, sound, combat, social, pvp, pve, craft, originality, freedom, accessibility, class balance, exploration, whatever makes sense really) and have a different set of weights to determine a final score for each of your identified player profile (Casuals won't care about freedom that much but accessibility will be very important, Sandboxers will value freedom, craft and social aspects, Raiders will be interested in combat mechanics, pve, ui and ui cutomization, PVPers won't care that much about Graphics and Sounds as long as the game is balanced and combat is engaging).
Of course, there is quite some work of analysis to be made to determine all these topics but, done properly, it should be easy to have areas that are prone to be graded objectively (interestingly, only a reviewer that likes crafting will be able to grade it properly in a game, making it mandatory to have different reviewers with different points of view and perspectives on the same game -- you can even average out the different reviewers note per area). Better yet, you can use the reviewer's player profile to determine the weight to give to each of his scores.
So, in short, each reviewer that plays a game will set scores on topics they feel like they have enough knowledge of to grade (someone that never PVP will simply not grade PVP, a reviewer that previewed combat at a convention will grade combat, gfx and sound). The site then polls these scores and, using the profile-specific weights, will compute a final score per profile (for instance 8.2 for someone into PVP, 4.3 for someone Casual). As time goes by, reviewers can come back and change their scores (with a little explanation note). The overall scores are then updated and we have a nice history of what reviewer felt changed over time (and, why not, with graphs showing how the game is evolving).
You could even go one step futher and have people state their own weight per topics in their profiles... and the site would be able to give them a personalized score just for them
All in all, I think you should be able to determine the topics of interest by your own experience or (why not) by polling your users. Once you have the list of topics, install the weight system on the profiles and, once enough people have filled them, a simple statistical analysis (classic classification) will allow you to extract player "profiles".
If you still need an overall score, it's then easy enough to build one from profile-specific ones with weights depending on the overall population of each profile
Comments
Taking out those massive pointless pictures that do nothing other than to make your 200 word articles look bigger would be a good start.
In a lot of ways review scores are saying either "buy", "try" or "avoid" the subject they are reviewing. Past that point, a lot of review systems are window dressing.
Personally I appreciate a review that has a short list of pros and cons around a title, plus I think you could also have a "Better than..." and "Worse than..." comparison (and that will still generate lots of site traffic as fans of all games come in to tell you how wrong you are!).
On the subject of bias: be open. Have a Bias Box in every review saying, "This review was written during Patch 1.0.1.1.3. We were provided with 5 complimentary copies of this title for review, the author is best friends with the lead developer and they had a beer every Friday on the company dime." Bias might exist, but be open about it so that readers know they aren't being misled. If they choose to disregard a review because of "bias", then I'm sure they will find a reason anyway.
Agree about needing different reviews for different periods. A Day One review, A Month One review and a Year One review could cover the different experiences / evolution of a MMO.
Why not use a system that professional analysts use for stocks, specifically the sliding bar of sell, hold, and buy. But in a reviewing case, do not try, try the game out, must get NAO!!!
Then with the use of multiple reviewers can get a fairly consistent and hopefully accurate review of the game itself across a relatively large sample.
i dont think having no scoring system at all is a good idea. ill be honest and state that unless its a game im extremely interested in, i'll rarely read a full review. ill normally skim read or go straight to the bottom and read the conclusion and look at the score. im sure there are many more ppl out there who do this. w/o an at glance review score.
i would probably start looking to other websites for my reviews. im lazy and dont have enough computer times to spend 15 mins reading an entire review
If someone is talking in general chat in a language you dont understand, chances are they're not talking to you. So chill out and stop bitching about it!
The problem, as I see it, is that it seems MMORPG.com doesn't really have any idea at all what makes a good MMORPG. Maybe some of the folks working here do...but the criteria you use to review, and the various things I've seen on this site (polls, blogs, etc) all make me think that your perspective on MMORPGs is dictated completely by the modern conventional wisdom in the genre. Sadly, the modern conventional wisdom leads to failed games or games with niche player bases. It's actually no surpise that the games you guys give the highest scores to end up losing most of their subscribers after three months. It's completely predicatable if you are looking at the right things.
Think of it this way... Say you are a site that reviews new automobiles. Say everyone in the industry is talking about the new paint colors, new door locks, and new interior fabrics that are being used in modern cars. Now say you base your review of new cars SOLELY on those things. Paint color, Interior fabric, and door locks.
Obviously, the score you gave to a car would very often be completely irrelevant, and not predictive at all of how successful that car would be in the market. Why? Because you aren't reviewing the things about a car that really matter. How reliable is it? How powerful and fuel efficient is the engine? How well do the breaks and steering work? Without reviewing the aspects of a car that really make it a good car...you can't possible write a useful review.
The exact same holds true for MMORPGs. You don't review based on the elements that really matter. In fact, you often don't even mention them...and if you do it's in a passing way that doesn't seem to end up affecting the overall score at all. Things like depth, challenge, replayability....the realism, internal consistency, and immersiveness of the world...the scope and size and ambition of the design. How invested does the game make you feel in the world and in your character? How much of a sense of danger or excitement or adventure does the game create moment-to-moment? How meaningful are the rewards (how much investment and risk are involved in obtaining them)? Is the game accessible, but still DEEP...are you still learning and mastering new and interesting elements even hundreds of hours into your play, or do you have the game down in the first 10 minutes and never really have to do anything different? What new elements and new ideas is the game useing to IMPROVE the above things over previous games?
The conventional wisdom in the post-WoW genre leads you to believe that all the wrong things are important. People have drawn the wrong (or at least incomplete) conclusions about what made WoW successful, and run with them as fast as they could. All we get now are watered down, ever-more-shallow, ever-quicker and simpler and less involved games. You're reviews are based on these exact criteria...so the worst offenders in terms of creating shallow and short and meaningless experiences end up getting the highest scores on your site. You might think of it this way (although it isn't a direct cause-and effect, it might as well be), developers are creating games specifically for your review style...instead of making good games.
If we go back to the car analogy, car manufacturers are now making cars with really awesome paint jobs and the very best fabrics in the interior...but they have extremely wimpy and inefficent engines, terrible brakes and steering, and are basically death traps. Since you only review the paint and the interior, you'd give these cars high scores....and that's exactly what you're doing. Giving games that are doomed to lose most of their players in a short time the highest scores as being the best the genre has to offer. It's not reallly that you're being intentionally deceptive, it's really just a matter of priorities and a deeper understanding of what makes a good MMORPG. This lame and over-simplified idea of "fun" which centers around instant gratification and constant reward...the idea that a good MMORPG is one you can play entirely solo and complete entirely in 15 minute intervals from your smart phone... It really just seems like you guys and gals don't have a clue of what this genre is supposed to be about, what sets it apart from other types of games, and what makes the core players in this genre excited and keeps them playing.
The timing of your review really shouldn't matter. I could tell you after a few hours of play whether or not an MMORPG is going to last. If you are looking at the right criteria, it's almost immediately apparent which games are created to be long-term virtual worlds that will draw in players for years, and which are just shallow little dances that get big box sales and then disappear. First you have to recognize what the genre should be shooting for...long-term truly social experiences where you make friends, build a community, and get wrapped up in the world for years. Where you have memorable experiences and meaningful rewards rather than loot pinatas and a forgettable grind. You have to agree on what an MMORPG really is...then you'll be able to spot the great games without much trouble, and you'll probably be able to provide a solid review even before the game launches.
Are things like polish, playability, smoothness, and accessibility important? Sure they are...and a truly great game is going to provide those things on top of the elements that really matter. You need to recogize that it's perfectly possible to have a great MMORPG that launches with some bugs...but it's impossible to have a bad MMORPG succeed in the mass market simply because it's polished.
If you need help figuring out what's important... just ask. I'm sure there are plenty of folks who would be glad to help, me included.
The only problem I have with the reviews is that they are way too generous. Professional critics have a job to do, they have a responsibility. If they say something is okay, they're making it okay.
Nowadays we have a slew of identical MMOs churned out to make a quick buck at launch, with almost no attempt for innovation. Just throw together some instances, battlegrounds and a gear grind. Done. Stop saying it's okay.
-Desagidit
Having to review and MMO is darn near impossible. With all the different classes and combinations, and crafting classes how can anyone get a real idea of what the game fully offers (end game and all)? With that said here is my 2 cents:
When I read a review what am I looking for? Why would I the reader even bother reading the review unless I am trying to get something out of it. Here is what i am looking for:
1 - Is the game worth the purchase price (whatver it may be) and will it keep me entertained for the initial 30 days?
If I read a review and it can tell me the above, we are off to a good start. What does the game initially do well, what does it do poorly. Can I get on a server and play for more than 20 minutes without being kicked off for maintance? Is 30 days all I am going to need to get the most out of the game?
2 - Is the game worth the monthly fee moving forward?
Game could be great, but is it worth the $14.95 per month? Is there more to see that is worth 14.95 after the 30 days is up? This question is more tricky because instead of comparing it to other games charging the same price, there are lots of free MMO's out there are really good and cost nothing. How does it stack up to those games as well.
Also let's not assume just because it's an MMO and we are all fans of MMO's that just because it does one thing different or has a few different classes it's worth 14.95 a month. What does it do that hasn't been done before? What does it do that has been done a hundrend times before that it does better than anyone else? Is the world unique and worth the investment? Is the company committed to the product or is it a 2 month experiement to see if it hits?
3 - What are the different play styles if offers? Is it PvP centric, PvE centric, a REAL mix, does it have crafting?
All of these questions need to be answered, I have read reviews on other sites that have said games are a mix of PvP and PvE only to find the game was primarily PvP with little if any PvE content. I want to know if I like playing a melee class or range class what does the game offer to me? Is it solo, duo or group friendly or is it hardcore raid centric? is it a mix (been fooled here alot as well). Here is where to take fun into account. Each person has their own idea of what fun is, but here is where you can say if raiding is your thing and something you find fun, this game is for you. If you like to duo with the wife, maybe you stear clear of this. If your thing going solo is fun, maybe this isn't for you. this sort of information is very important.
4 - Going along with #3 what is the long term outlook with the game (taking company history into account)?
The last thing I want to do as a gamer is waste months building a character only to have the servers shut down and the game go belly up 6 months down the line. Does the company have any history behind it? What is the buzz on the game? Do the servers look alive? Or is this one of those games that after the first 30 days is up people will be packing up and leaving? As a fan and as people who review these games, it shouldn't be THAT difficult to get a first impression like that. Now granted, no game is going to have WoW's players base, but is the game sustainable?
These are the main points I look for in a review, the number/letter overall score really doesn't matter to me all that much. As you mentioned what is the difference in game that gets a 6.5,7.0 or a 7.5? Are they all worth 14.95 a month or just the 7.5?
If you wanted to give each catagory a seperate score than tally it up at the end, that could work. That way at least we could go to specifics and look for what is important to us. For example if the game is worth the 14.95 a month but had a rough launch, perhaps for #2 they get a good grade but for #1 they don't pending some updates or patches.
Again just my 2 cents.
Hi,
Thanks for asking for feedback, it's always great to see sites being well engaged with their community. I put some thoughts together and had trouble getting them into a comment post without losing the thread - so decided to stick them in a presentation instead.
I can't embed the presentation here so hosted it on my own server: can access it <here>.
Good luck sorting out your approach - it's actually a pretty fun problem to be wrestling with imho.
Cheers,
BitJunkie
I made an extensive suggestion here, but wanted to stress the point that you need to use sub-genres. In other gaming sites people quickly see "Racing Game" or "Platformer" and the like and it sets their expectations. WIth sub-genres I mean "Sandbox", "Themepark" and the like, maybe you find better sub-genre names.
The burden (and joy) is usually on journalists to come up with genre names, hence I think you guys (and others) are still a bit clueless about your role. Grow up.
Nice post Stormwatch.
I have a couple of further thoughts:
1) MMORPG.com could setup an online questionaire form using their assessment criteria. They could get people to take a bartle test as part of the questionaire and then use the result to rank the game for different player archetypes.
2) They could also work with Beta testers to do the same before launch with help from the developers.
3) MMORPG.com is in a unique position to hold developers to account and to be a useful source of information for development teams - if they set up 1) and 2) then they can use this information to bargain for involvement during development and so be a real "voice" for the community.
I think that behind the answer to the review methodology lies a lot opportunity to do good and secure a key position in the development cycle. If it was my organisation I'd be thinking a lot more strategically and figuring out ways to leverage such a committed readership.
Cheers,
BitJunkie
I think Fozzik nailed it right on the head. Too much emphasis is put into the wrong parts of the games and too much bonus is given to certain concepts that really are just function.
Honestly when it comes down to it, I read reviews for sake of something to read. If I want to decide whether or not I'm going to play a game I check out as much information as I possibly can before hand. I've read some reviews that never give any information that wasn't already available.
First, thank you for reading our feedback on other topics to address the rating system. Yes, the rating system used for MMORPGs is currently messed up. Given the complexity of MMORPGs, here are some ways you can address the situation:
Give a grade score (A to F, N/A or Incomplete) for different aspects of the game (Graphics, PvE, PvP, Crafting, Content, Sound, Long Term Playability, ect.) and a final socre (1-10) for the initial preview and then another for full review. Clearly some of the grades for the game aspects can be either N/A (for example, game doesn't have PvP) or incomplete (meaning that it cannot be determined in the initial preview). Then you can complete some of these grades when the final review comes out. Players should also be able to vote on these aspects like we vote for the overall score of a game.
The benefit of such a system allows people to know how you feel about different aspects of the game and we know what you haven't had time to experience (content, PvP and Long Term Playability probably the main things incomplete on initial release review). Furthermore, we can figure out if the game is good at certain things. As a PvP player, I'm more interested in the PvP score and Long Term Playability in an MMO, and less concerned about Sound, Content and PvE. I'm sure others are more interested in PvE, Content, Graphics. Right now, I have to search and read your reviews to learn about these issues, and it would be just easier if I could see a grade rating on these, then read up on it if it interests me.
As for the 1-10 scoring system, G4 TV uses the 1-5 system but I think they stated how the scores are granted (and could easily be applied to MMORPG.com). A score of 1 (or 1-3 for MMORPG) was simply terrible and/or unplayable with bugs. Shadowbane's sb.exe error disaster around launch comes to mind as deserving of this score. A score of 2 (or 4-5) is a bad game, or doesn't do what was advertised but doesn't crash because of bugs (e.g., Age of Conan perhaps at release since the PvP was broken and DX10 wasn't included and beyond Tortage the game lacked content). A score of 3 (6-7) is average game, and MOST games should fall into this rating (e.g., RIFT, etc). A score of 4 (8-9) should be for exceptionally good games with lots of replayability and a score of 5 (10) should be the rare game that just rocks on all factors.
The downside is that most games will have a score of 6-7, which right now MMORGP.com keeps most ratings at 7.0-8.9. But I think having a more formal rating system would be better. If you read the re-review of AION at MMORPG.com, you'll see the difference. That re-review said the game was better than it was at launch, yet gave it a lower score ("because it feels like an 8.0" if I remember the quote). It was this that eventually turned me off to your scoring system, because it depends on the whims of the review and had no standard.
This website's reviews on games that have PvP should be done by people who enjoy PvP (or allow people who enjoy PvP score that part of the game). I'm absolutely sick of hearing how the person doing the review doesn't like to PvP, and then goes ahead and discusses the good and bad of a game's PvP system. I don't remember exactly which review that was, but I read it on this site. I personally hate PvE and I know there is no way I could review that part of a game for people who enjoy PvE.
Finally, when a game receives a relatively high score, a follow-up review should be sooner if possible when the game changes rather quickly. DCUO gameplay changed DRAMATICALLY 1 month after the initial review, as population levels dropped dramatically making it a very different experience. Leaving the very positive review score unchanged despite a dramatically different game experience I thought was not professional. This website then had a follow-up article saying that everything was great except lack of population. Well, lack of population is a major problem, and it almost sounded like it was the players fault for not recognizing how great the game was. I believe that when a game's population declines dramatically after launch that received such a high score, there should be some kind of managerial control or intervention at MMORPG.com to see what went wrong in the review or what was missed, and if necessary correct the score in a re-review.
I know this was a long post, but I like MMOs and I'd like a good rating system so I know what to buy and try and what to avoid. Right now, I can't do that with the current system.
this is exactly what I was thinking. Letter grade system is almost universally understood. Everyone has a general understanding of the difference between a A vs a B. But still make sure to incorporate aspects of the game into perhaps catergories giving an explanation of what you experience good or bad, so we know what to expect overall. I feel that once you pass below a 5 the game is pretty much crap at that point, labeling it with a 1 or a 2 isn't going to really make the damage any worse so you do in fact end up with half a scoring system that is useless.
I would still go with full out reviews, though Previews are nice, but I feel a review is better as i want to know if the game is worth my intitial purchase, and not only that, will it be worth a continuing subscription or perhaps I should wait a couple months. The more informed I am, the better decision I can make about where to put my money.
3.4ghz Phenom II X4 965, 8GB PC12800 DDR3 GSKILL, EVGA 560GTX 2GB OC, 640GB HD SATA II, BFG 1000WATT PSU. MSI NF980-G65 TRI-SLI MOBO.
I just wanted to quote this poster because THIS (pay attention all Games Reviewers!) is how the average person views the one to 10 scale!
NOT 1-5 plus 5! that you guys use... (see the quote here from the original article):
Next, I wanted to talk about scoring systems. over time, I’ve become more and more annoyed with the 1-10 scoring system. The trouble is that numbers 1-5 are very rarely used because they represent a failing grade. For me to give any game a 5 or below means that it probably didn’t even run on my machine or it so wildly missed the mark that it doesn’t even resemble a running game anymore. So why, if it’s never or rarely used, are we taking up half of our scale to say the same thing: This game totally fails.
Then there are the degrees that a 1-10 scale creates. Who, other than the reviewer, really knows the difference between a 6.5 and a 6.6? How does that difference stay consistent from reviewer to reviewer? Heck, how does the scoring system stay consistent from reviewer to reviewer anyway?
I know that 1-10 is how most gaming sites do things, and many of them do it because it puts the score into a nice little package for MetaCritic. So I’m wondering, where MMORPG.com is concerned, how important is it that we stay on par with everyone else? Do you all enjoy comparing the numbers from one site to the next? Is that helpful for you as a consumer?
Well to us consumers a game that doesn't run = 0... not 5.
And don't be so scared of saying that a game "fails"... some of them really do!
A score of 1 says "Abysmal - was this game written / programmed by primary school kids with no talent?"
A score of 10 says "You MUST buy this game - because if you don't you will be a social lepper when everyone else in the world is playing it because, yeah, it really is that good."
And you may never give out 1s or 10s, again, so what?
Ah... but then it's not fair is it? I mean a game that was an 8 in 2001 cannot be fairly compared to a game that is an 8 in 2011? Well no. So what. I want to know the date of the review and I judge it based on that. An 'average' 2011 game is bound to be vastly better than the original UO. I expect that. It's fine - really.
So, re-review the games that stick around every couple of years if there is a demand or you hear something has changed vastly improving the game.
Maybe even say "In 2005 this game was the trend setter with an overall score of 9/10 and 6 years on it's still a contender 7/10."
This is what we need to decide if we will actually invest our time and money.
If you want to break down by categories - please do - but overall score is where people look.
Other than that - some very good posts in this thread.
Nothing says irony like spelling ideot wrong.
This isn't criticism of the review process, but I think this page is a great example of why the site needs a newer comment system with pages, sorting, thumbs up/down, search etc. Any site with a strong community that encourages discussion benefits from more organized comments and forums.
Just another thought -- my actual review comments are somewhere above, heh.
rate each catagory
You guys aren't critical enough of game mechanics which don't work and too often praise a game's ideas instead of criticizing execution. How did Mortal Online's six month review get a 6.9? On any legitimate game reviewing site it would receive a 1.0 out of 10 and that's because it doesn't work, not because it's too l33t for everyone.
In addition, I often feel like reviews are given based on day one gameplay, like how Rift got an 8.7. Also, how does a game which so blatantly copies WoW get a good review on the site which is constantly calling for Ultima and SWG remakes and new sandbox games to be developed? It's like you don't want to hurt the feelings of the people that make these games.
I think the game list here is very helpful. being able to sort the game score. correct me if I'm wrong, but isnt the score posted there for a game an average of all submitted scores by registered members? I dont see how you can get a better idea of the "general" feel for a title than this...
The debate really comes down to:
1. The data used for the score
2. How to capture and display the data... 1-5 or A-F or 1-10 or 1-100...
On my first point... the scores should be cleared on a regular basis... using the current 1-10, a game that was a 6 at release +6 months and then 4 years later may have evolved into a 8 is still being dragged down by now obsolete reviews... maybe list a "lifetime score" and a last year score... For instance... I sucked at math in the 5th grade... but became a A student in 8th... if they still factored in how I did in 5th grade in 8th grade I might get a C which does not rate my "current" ability...
On my second point... 1-10 seems fine to me... it allows for easy averaging... maybe tweek the catagories... like customer service... its not really a valid listing unless you had to use thier CS... make the catagories so you can submit a rating or opt out, and your score in that catagory isnt used... most games Ive reviewed Ive never used thier CS... so I cant give any score... but am still forced to decide 1-10... it immediately skews the # no matter what I decide... I use 5... another uses 0...another 10... no consistancy...
......
As for staff reviews, have thier numerical review simply be part of the rest of the sites users... give a written headline review and let that stand as your verbal rating and have your numercal assessment fall into the pool with the rest of us... come on in... the waters fine... it just needs to be changed(see my first point) from time to time...
And the staff who is assigned to that game, like my suggestion for resetting the scores every X timeframe. Should also revisit the game and revisit thier review every X...
No system in opinion land is perfect, there are just better ways to collect and display data...
PeAcE... R2K
One other thing about this website reviews that annoys me, that unfortunately i thought of after my lengthy post. Some of your "reviews" actually state that they didn't have much time in the game, and then go on to review the game. Seriously? What's more important at MMORPG.com: getting a review published or doing a review correctly?
I swear, I have read that in the past couple of months a guy said he didn't have much time in the game because of personal issues, and then goes on to review the game given his limited playtime. WTF? If I had done that at my job, I would have been given an official Human Resources warning or something. Either do it right, or don't do it.
My idea relates to the site members' numerical ratings.
Members should be able to check some boxes about themselves as to their gaming preferences; e.g., PvP, PvE, etc.
That way, once the ratings are given, you could see how PvPers, for example, feel about a particular game versus PvEers.
The net result, then, would be a rating for a game that looks like this:
Darkfall Overall rating 8.7
PvP player rating 9.2
PvE player rating 5.1
And so on for other possible breakdowns. This would be hugely useful because a PvPer probably doesn't care about what a PvEer thinks about a particular game, and vice-versa. At a minimum, when a PvPer goes looking for a game, he or she is going to first want to see how other PvPers rated it.
I think this would be a lot more useful. Otherwise, all gamers of all types are thrown into a pot together on their ratings, which doesn't really tell you much.
EQ1, EQ2, SWG, SWTOR, GW, GW2 CoH, CoV, FFXI, WoW, CO, War,TSW and a slew of free trials and beta tests
I have not read all the comments so my apologies if my ideas have been presented earlier.
I think the best way to review MMOs would be to take a group of 5-10 people who would play the game together. Among all these people will be varying tastes and interests in MMOs. You'll have your raiders, crafters, chatters, AHers, etc.
By forming a kinship/guild the group will be able to test out that portion of the game. With enough people you can test group functions and dungeons. Some people will be looking at how "easy" money is to come across in the game and how much would be needed to perform various items/skills etc.
The review would be labeled as "on-going". The goup would continue all the way to "end game". Whatever the max level is. The goal being to explore all parts of the MMO with a group of people. Each person or persons would be responsible for parts of the review. Put all the pieces together and you would have a fully fleshed out review. All aspects of the game would be tested and reviewed, and enough time would be given to ensure a thorough look-over was done.
I don't know how feasible this approach would be, but I believe it would garner the best review possible.
I think that the best solution would be to have 3 reviews of the game. Preview review, a couple weeks after the release then a follow up after a good impression of the game is set.
Then to work out the 'scores' of a game, I think going to a total out of say 50 is best. However to combat the fact that gameplay>sound, rate gameplay out of 10 (because it is more important) and sound out of 5 (because it is less important). This way good gameplay has more of an impact than good quality sound.
Improvents to this method still are needed but is a nice foundation.
I'm of the opinion that reviews on MMORPG.com are quite reasonable.
Too many people think that if a game is not to their liking, that makes it a BAD game. This isn't the case. Many don't like certain things, while others do.
The reviews tell me what type of game it is and how it stacks up against other games of that style. For me this is helpful. If there's lots about the game that are things I don't find enjoyable then I'll skip further interest in the game.
No - I disagree. When I see a review I like to know how long the reviewer spent with the game. Don't need to know why (keep your private life to yourself thanks... thats why its called private :P )
But the time spent with a game is very important - particularly with MMOs.
As we all know - most games have a 'honeymoon' period. It's easy to play many MMOs and think they are the best thing coming if you only get 24 hours access, limited access to certain features, certain areas, certain classes.
Indeed, we have seen this in the past and are still seeing it (SW:TOR anyone?).
I don't care if the reviewer cannot review the whole game so long as they are honest about what they did review and under what conditions.
Better that than reviewers who even lie (I caught a reviewer out completely once - implying he had tested "multiplayer" in a game where the multiplayer function had not been functional until after his review had been published.)
But, I must admit, any MMO review that spend less than a week on the game or is done based on any restricted conditions really isn't worth the reading time.
Nothing says irony like spelling ideot wrong.
I kinda like this more drastic scale.
To me the real issue is that your score (be it an aggregation of sub-scores or a plain overall grade) has few meanings if you don't specify which player profile the score applies to.
Not only would you need to refine scores as the game evolves (pre-release, at release, one months in, 3 months in, 6 months in, 1 year in, 2 years in, etc...), you also need to have one score per profile:
Casual
Sandboxer
Raider
etc
What you could do, is use grades per "topic" in the game (graphics, sound, combat, social, pvp, pve, craft, originality, freedom, accessibility, class balance, exploration, whatever makes sense really) and have a different set of weights to determine a final score for each of your identified player profile (Casuals won't care about freedom that much but accessibility will be very important, Sandboxers will value freedom, craft and social aspects, Raiders will be interested in combat mechanics, pve, ui and ui cutomization, PVPers won't care that much about Graphics and Sounds as long as the game is balanced and combat is engaging).
Of course, there is quite some work of analysis to be made to determine all these topics but, done properly, it should be easy to have areas that are prone to be graded objectively (interestingly, only a reviewer that likes crafting will be able to grade it properly in a game, making it mandatory to have different reviewers with different points of view and perspectives on the same game -- you can even average out the different reviewers note per area). Better yet, you can use the reviewer's player profile to determine the weight to give to each of his scores.
So, in short, each reviewer that plays a game will set scores on topics they feel like they have enough knowledge of to grade (someone that never PVP will simply not grade PVP, a reviewer that previewed combat at a convention will grade combat, gfx and sound). The site then polls these scores and, using the profile-specific weights, will compute a final score per profile (for instance 8.2 for someone into PVP, 4.3 for someone Casual). As time goes by, reviewers can come back and change their scores (with a little explanation note). The overall scores are then updated and we have a nice history of what reviewer felt changed over time (and, why not, with graphs showing how the game is evolving).
You could even go one step futher and have people state their own weight per topics in their profiles... and the site would be able to give them a personalized score just for them
All in all, I think you should be able to determine the topics of interest by your own experience or (why not) by polling your users. Once you have the list of topics, install the weight system on the profiles and, once enough people have filled them, a simple statistical analysis (classic classification) will allow you to extract player "profiles".
If you still need an overall score, it's then easy enough to build one from profile-specific ones with weights depending on the overall population of each profile
Anyway, I hope I make sense.