Metacritic Founder Talks About LittleBigPlanet Low User Score, Fanboys, and the Site’s Future
by Chad Lakkis on October 23, 2008 at 1:05 am
After reading about the suspiciously low Metacritic user score of 6.3 for Media Molecule’s LittleBigPlanet, I decided to dig into the story a little deeper. For those unfamiliar with the situation, allow me to fill you in.
Metacritic, who by and large is considered the main source for aggregated scoring on the web, offers two score flavors (one for critics and one for users). This is the case across most sites. The problem is, when you are considered the “go to” resource for pretty much anything, there’s a chance that people will make a big deal out of something on your site that they would normally overlook on another.
That said, this specific instance pertains to the scores currently presented on Metacritic for the LittleBigPlanet game. The first, a 9.5 at the time of this article, represents the aggregated score based on several well accepted gaming review sites such as GamePro, IGN, and EuroGamer. The second, more controversial score, is a 6.3, and is compiled via the average score given by a pool of 2,225 user votes.
Gaming news site N4G, which pulls related news from various websites around the web, features a blog post titled “Fanboys pull LittleBigPlanet’s Metacritic score down”, discussing this very topic. Reading some of the furious responses to the score piqued my interest, so I visited the Metacritic site to see first hand what all the fuss was about. Clicking on the “Read user comments” link near the overall user score brings you to the bottom of the page, where the following statement by Marc Doyle, the sites founder, awaits:
“My advice for our faithful users is to focus your attention on the Metascore for this game and not the thousands of user votes, most of which have been submitted before said users have played the game. This is a gaming community, and if people want to stuff the ballot box, there’s not much I can do at this point. When we upgrade the registration requirements for participation on the site in the near future, this type of thing won’t happen. We’ll post the full legitimate user reviews upon the game’s release. As always, thanks for using the site.”
This prompted me to contact Marc directly in order to get a more complete picture of the situation from his end. Specifically, I felt that gamers would want some clarity regarding the portion of his comment stating that most of the user reviews submitted for LBP were made “before said users [had] played the game.”
Firstly, I wanted to know if he would consider eliminating user reviews submitted before the full release of the game, i.e. reviews clearly submitted based on the open beta alone, or even worse, no experience with the game at all, to which he responded:
” … we won’t be clearing those old scores from LBP when it launches, but we will only post reviews from people who have clearly played the game. All the reviews from fanboys and haters who haven’t played it will just be left out. That is true for all products on Metacritic. Humans actually read all of those user reviews (so we can delete profanity, spoilers, etc.), and those humans are told not to post reviews from people who clearly haven’t played the game.”
He then added that the burden of proof they demand from users submitting reviews prior to a game’s release date is be much higher than those doing so after the fact.
Slightly confused, I asked Marc for some clarification as to how the game could have such a low score, if his team was manually looking over each user review submitted. He stated:
“All the [numeric] scores “count” in the overall user average and total votes tally, but we only post a subset of those as [written] user reviews.”
When talking about the future of the site, as well as other things they are doing to help eliminate this type of fanboy nonsense in the future, Marc had the following to say:
“There will be a Metacritic re-design in the future (can’t say exactly when), and as a part of that process, we’re going to beef up our user registration and community functions so that “stuffing the ballot box” won’t be nearly as easy.
The original idea behind allowing users to write user reviews (and vote on them) was to allow those people (like me) who see advanced screenings of movies and sometimes play games ahead of their true release date to write reviews and have them posted on Metcritic. That’s still the case now.
Over the years, people have, by and large, not exploited the fact that you can vote early. Only recently have people started voting en masse for (and against) games like they’ve done with LBP.
Metacritic’s primary “product” is the Metascore, which is the weighted average of scores from professional critics. We include user reviews for obvious reasons, but because that score can never be 100% reliable, it’s given secondary importance on the site. For example, you can’t sort or search by user scores for any products. That will likely change with the redesign.”
Marc obviously has a solid understanding of the space, its evolution, and the challenges that lie ahead. The fact of matter though, is that currently, anyone can visit Metacritic and leave an abnormally low user score, skewing the average, without necessarily saying anything of value at all. I asked him if this was something the site would be revisiting in the immediate future.
” … nothing should change in this regard until we do the redesign. That tally will remain a raw tally, and only the printed user reviews will be “vetted” as useful reviews from people that we think actually played the game … “
He closed the topic by adding that the specifics of “whatever [they] do with the voting system moving forward [have] yet to be established.”
With October 27th just around the corner, LittleBigPlanet is four LittleBigDays away from being officially released. So while the current 6.3 Metacritic user score has been getting most of the attention on the net, I (like Marc himself), kindly ask that you shift your focus to the 95 out of a 100 Metacritic score the game is receiving from the mainstream media. Everything else will sort itself out in due time.
 |
Tags: aggregated scoring, controversy, fanboys, founder, LittleBigPlanet, Marc Doyle, Media Molecule, Metacritic, N4G, SackBoy, SCEA, scee, user score |
|
|
- PS3 Fanboys Strike Back: Gears 2 Metacritic User Score Lowered to 4.1
- Resistance 2 Latest Victim in Metacritic Douchey User Flames
- Despite Fanboy Accusations, Metacritic Does Not Exclude In-Network Review Scores
- Sessler Psycho-Analyzes the Fanboy Condition
- N4G Being Bombarded with Gears 2 Spoiler Articles Posing as News
- N4G Cares Not for Contributors, Only Bottom Line
- Joker’s T-Rated Gag Gun Fatality All Gag No Gore
- Gamespot’s Gerstmann Gets The Axe Over Kane & Lynch Review
16 Comments » |
on October 23, 2008 3:50 am
how about the reviews which have no legitimate score, but they read the review and base a score off of it. (i.e. variety) which to me is absurd, only accept reviews that have numeric values already.
on October 23, 2008 4:46 am
I am pretty sure that the site (i.e. Variety) willingly accepts the converted score prior to Metacritic adding it to their site. I will however ask about that if it helps to ease your concerns.
on October 23, 2008 5:45 am
bah, like i said, this game was only gonna be SO great.
on October 23, 2008 5:56 am
Well, like all sites that fall behind or out of favor, Metacritic seems to let all their shortfalls shine through. Let this and other obvious problems with this site be motivation for a better site to take it’s place that is more committed and professional.
on October 23, 2008 6:16 am
Here is my opinion on the Variety issue. If Variety is submitting a score to meta critic, then it should be posted on their web site. If there is no physical number given at the actual web site doing the review then there is no way to prove who created the score. Was it really Variety, or was it metacritic. This lack of reference cast doubt into the score and causes the illusion of bias, even if there is no bais at all. If metacritic wants to be taken seriously they should not post a rating that can not be verified via their reference links.
The other problem with metacritic is that they seem to be selective in who they use as reference. For example why is it that Variety, a news paper, has made it into metacritics scores for Little Big Planet, yet D+Pad, Gamesradar, SPOnG, Eurogamer Portugal, Eurogamer Germany, GameDaily and quite literally a dozen plus other sites that are far more gamer centric and reputable with scores have not? If you doubt this then follow this link to n4g’s links to Little Big planet reviews: http://www.n4g.com/ps3/games/g-28444/tt/metareviews.aspx
on October 23, 2008 6:29 am
” … we won’t be clearing those old scores from LBP when it launches”
That’s is just stupid.Metacritic is a product of Microsoft of course
on October 23, 2008 6:35 am
Good article Chad. I too am a little curious about the variety article. Metacritic rated the variety LBP review as an 80, which was perhaps fair enough on the basis of the text (which was positive but definitely not as glowing as other LBP reviews) but was a full ten points lower than the next lowest score (ie. a BIG differential); on that grounds it was certainly a little bit suspicious. The big thing was the fable II review: although the variety review was (i’d say) more positive than the LBP review I certainly don’t think it was a 98 point type review. He had some fairly major criticisms at the end. 98 was also a higher score than the highest of the sites that mark out of a hundred, by a whole three points: eurogamer and g4tv both gave the fame full marks but they mark out of ten and five respectively so it’s not so uncommon to see them giving out full marks. It’s a bit of a sad fanboy thing to get upset about it but nonetheless I’m a little curious, it does seem a bit odd.
on October 23, 2008 7:02 am
I love how 360 fanboys have to make themselves feel better by giving a 10/10 game a horrible review…I guess thats fun to them, IDK.
They really are pathetic, you didn’t see people going around giving Halo 3 a horrible score (atleast I didn’t) because could care less about the User Score of a video game….You think your score is gonna chnage a consumers mind when a “95″ is right next to it?…Come on now 360 people, we know your all not that big of a dumbass
on October 23, 2008 7:11 am
Maybe the user score is correct. Maybe the 2,225 beta players found the game somewhat boring. Which is how LBP looks to me. Kinda boring.
on October 23, 2008 7:48 am
LittleBigPlanet is a fun game, but from videos and the demo (probable purchase later on), gets to be a little repetitive. Can’t assume that the poor reviews are from 360 fanboys, because the Wii would be jealous (maybe) and the PC would also be jealous, so assuming one console to bring down the score would seem a bit out-of-reach don’t you think?
Hopefully the change in the website will help ease out these problems.
on October 23, 2008 11:35 am
This was the work of XBOTS, we all know that. WIIWEES not PC GAMERS, XBOTS!. This as much as they can do to make LBP look mediocre. Anyone that ever plays this game, will not in fact degrade it, its an instant hit, no matter what genre you prefer, you are still going to love it.
Variety obviously dont know sh** about games. When everyone sees that lame ass score next to the rest, its going to look ridiculous and unreliable. What pisses me off the most is that Variety doesnt grade reviews, so why even include it on MEtacritic. Why dont they post that 10 GamesRadar gave it instead. Why is GAmes Radar not being considered for this particular game? Too many 10’s?
on October 23, 2008 4:26 pm
Hey Chad,
JUst by the way, it’s ‘piqued’ my interest, not peaked.
WOOO smartassery ftw! Right?
on October 23, 2008 5:17 pm
Doh! That’s what I get for posting late at night. Thanks for the catch Kyle.
on October 23, 2008 7:09 pm
Maybe it’s gamers protesting the stupidity of pulling the game because of two stupid lines that might potentially have offended some dude on some forum somewhere. Why is it okay to “stuff the ballot” sometimes but not others. I didn’t see anyone complaining when people were going to Amazon bringing down the review scores of the chick’s book after she said something stupid about games on TV. I don’t remember anyone freaking out when people brought down the score of Spore because it had stupid copy protection. If you can adjust the score based on copy protection, why not adjust your “user review” score based on the fact that the game was delayed because of two lines.
But then, I too think LBP looks stupid. It’s like, let’s play an SNES platformer only now we can make the levels ourselves. I have the most advanced and complicated piece of gaming hardware on the planet so that I can play Super Mario World with puppets? Uhh… Woo Hoo?
on October 23, 2008 10:15 pm
I honestly think this is the first side scrolling platformer to get raved about so much since the early 90s. That being said, I find it funny that hardcore gamers are lashing out at innocent and cutesy LBP because of hype that it’s receiving.
Disclaimer: NO! I have never voted on Metacritic.
on October 24, 2008 6:15 am
Great write up…and the image is classic Sackboy. Has anyone been on metacritic to look at the Gears of War 2 user score? PS3 fanboys strike back lol.