or… “Why Playing With Double-Edged Swords Just Means Twice as Many Cuts”

An Image That Should Haunt Gamemakers Everywhere

An image that should haunt gamemakers Everywhere

I loathe MetaCritic.  It’s inaccurate.  It’s unfair.  It’s arbitrary.  And it wields way too much power.  What really gets me though, is how little MC seems to care about these issues.

I admit that this is well covered territory.  Actual journalists and journalists and industry bigwigs have been here.  Developers and publishers have lamented MC’s power.  Others have questioned whether MC has any power at all.

And yet, after all this chatter, nothing has changed.  So what’s going on here?

It comes down to a simple fact — MetaCritic casts a powerful shadow but it is constituted of editorial inaccuracy, factual distortions and unchecked subjectivity.

As PR people, we’re partly to blame for the role the site plays in our industry.  We love to leverage sites like MetaCritic when it suits our needs.  How many releases have you seen where publishers crow about their success in terms of overall Metacritic scores and the number of games they have “over 80?”

For a long time, I was just naive.  I never took the time to look under the hood.  I just saw that the average score was pretty close to what I projected it would be, and I was happy that I was so damn smart.  I just assumed that everything behind the scenes was a neat and straightforward process.

But  it’s not – under the hood of MetaCritic is ugly.  It’s distorting, subjective and, sometimes, flat-out misleading.  It’s the ugly little secret that everyone in our industry — especially PR people — need to understand and should take action against.

So what’s the problem?  Let’s take a look at three key areas:

1. RESISTANCE IS FUTILE!: APPLYING SCORES TO UNSCORED REVIEWS

A little tidbit from MetaCritic’s site hides just above the reviews for a game:

“If a critic does not indicate a score, we assign a score based on the general impression given by the text of the review.” – MetaCritic.com

Now, I know quite a few reviewers who, after years of struggling with review scores and systems and complaints from publishers about an 80 versus a 79, for example, simply removed scores all together.  They believe that the TEXT of a review is the most important element, especially for a medium like video games.  Those editors don’t feel a singular number can be applied to indicate the overall quality of something as complex as a video game.

However, MC seems to have awarded themselves the power to add their own review score to someone else’s review, based on a subjective interpretation of review text.  This is probably the most glaring infraction against editorial integrity on the site.

For example, if a reviewer does not give a score but says in the review that a game is fun, with innovative mechanics, but is mildly repetitive at times and lacks replay value, how is it possible for someone other than the reviewer to assign a score?  How could anyone but the original reviewer determine whether the fun, cool elements outweigh the repetition?  Is it a 70?  An 80?

In this case, the person making the decision is not the reviewer, but MetaCritic.  And there is ZERO CHANCE that the score will be accurate.

2. REVIEWING REVIEWERS: “SHE’S SMARTER THAN HIM BECAUSE HER SITE IS BIGGER”

Another lesser known aspect of MetaCritic’s scoring system is the “weighting” of a review’s importance.  Simply put, some reviews count more than others.  And how does MetaCritic make this distinction?  It is simply up to them to decide.  From their site:

“Why a weighted average? When selecting our source publications, we noticed that some critics consistently write better (more detailed, more insightful, more articulate) reviews than others. In addition, some critics and/or publications typically have more prestige and weight in the industry than others.” – MetaCritic.com

So, the better you write (in the eyes of MC, that is) or if you have a bigger, more popular site, or if people generally think you’re prestigious (according to MC staff again) then your score will count more when computing the average.  Lucky you.

Of course, I can probably name off the top of my head 3 reviewers from smaller web sites that write better than reviewers at the big guys.  Does their skill in writing overcome the major site’s numbers and create a balance?  I don’t know and neither do you.  And that’s the problem.

It shouldn’t matter.  One reviewer’s opinion is not more valuable than another simply because they happen to write for a “major” web site.  This is essentially saying mob rules, and independent thought is not important or valuable for criticisms.

3. OBEY:  RETRO-FITTING SCORES

Metacritic forces all scoring systems – no matter their intent — into a MetaCritic-defined “0 – 100″ system.

For instance, MC chooses to ignore the generally accepted North American academic scoring A-F system, in which an “A” represents a 95 and a B represents an 85, and so on.  When a web site that scores games A-F awards a game a “B-” (which we would think equals an 80), Metacritic retrofits that score to THEIR private scoring device, assigning that “B-” a numerical value of “67.”

A C+ receives a 58, a full 20 points under what is intended by the reviewer!  This is an outrageous distortion of editorial integrity.

This is what site manager Marc Doyle has said on the subject in an interview with The Guardian:

One of the more controversial aspects of what I do is the conversion of the A – F scale to Metacritic’s 1 – 100 scoring system. For example, when I convert a “B-” score to 67 on Metacritic, many users/fanboys/publishers/developers chime in to argue that a “B-” should be converted to the low 80s because that’s how American schools tend to interpret a “B-“.

As such, I feel that ANY scale simply needs to converted directly with its lowest possible grade equating to 0, and the highest to 100.

Most importantly, every score is converted exactly the same way on my site, so all games are subject to the same treatment. This consistency is key.

I understand the priority on maintaining “consistency,” and that IS important.  But the MC’s twisted drive for consistency only works in a vacuum, where the overall score is only compared AGAINST OTHER GAMES.  It is consistent though – consistently unfair for everyone.

Essentially the argument presented by MC is that since every game gets screwed , it all evens out.  But game scores aren’t compared against one another by publishers.  If a 67 (that should be an 80) pulls your average score down to a 7.4 instead of a 7.5, do you think your publisher gives a crap that another game’s average fell from 7.9 to 7.7?  NO.  Your score is a 7.4. Period.

Why should any game have to suffer through an unfair review conversion? Why can’t the “consistent” model be that all A-F review scores are just converted in the same way?  Wouldn’t that not only be fair to both the game and reviewing site’s intent, but also more accurate and representative of the title’s quality?

The sample answer, of course, is to pick a score for each letter grade and just stick to it.  DUH!  A=95. B=85. C+=78.  Done and done. To NOT do that is pure laziness.  It’s criminal apathy.

Even more frustrating — there is simply nothing you can do about.  I’ve personally worked with an Editor in Chief of a site that grades on an A-F scale who wrote a note to MC clearly stating: “Please be aware that a B- on our site should be properly scored as an 80.”   You know what happened?  Nothing.

The allegiance to the MC 0-100 “system” trumps allegiance to editorial integrity.

Wouldn’t the world be a better place if all sites scores on the 100 point scale?  Sure.  And then all sites could look the same too and think the same.  Why not just have one site do a review?

So MetaCritic sucks… Why should Gamers care?  Why should PR care?  Why should anyone care?

Plainly put, the industry is getting screwed.  Game Revolution has already done a fascinating piece on this issue.  But I couldn’t disagree more with this finding of theirs:

We talked to several PR people, designers, and brand managers during our research for this piece, and by and large they use these sites the right way: as macro resources rather than accurate converters. – Game Revolution

These averages matter A LOT and and if they are willingly  inaccurate and artificially lower than they should be, they become deceitful and dangerous.

I spent years on the publisher side and I can tell you firsthand that quite a bit is determined by aggregate numbers.  This includes, in some cases:

  • PR and Marketing employee bonuses and performance reviews (never in my case, for the sake of full disclosure)
  • A developers’ end-game company performance bonus
  • The future “greenlight” status of a video game franchise and developer contract
  • A developers’ ability to sign their next project.
    • I’ve sat in many greenlight meetings where the developer was being evaluated — at least partially — by looking at their previous performances on MC.

But don’t take my word for it, check it out:

MICROSOFT: Xbox Live general manager Marc Whitten said in an interview that Microsoft will enact a new policy of delisting underperforming titles on its Xbox Live Arcade downloadable gaming platform. Whitten explained that candidates for delisting are those XBLA games which have been out for six months, scored less than 65% on Metacritic, and have a conversion rate of less than 6%.”

WBIE: “[Jason] Hall’s strategy now is to turn to game review Web sites — such as GameRankings.com, Metacritic.com, and GameStats.com — that aggregate scores given to games by critics at game sites and magazines. Games based on Warner Bros. licenses must achieve at least a 70% rating, or incur an increase in royalty rates.

ELECTRONIC ARTS: “John Riccitiello, CEO of Electronic Arts, uses Metacritic as one of the most important measures of the performance of his company and is unhappy when they go down, telling Wall Street analysts ‘Our core game titles are accurately measured and summarized by these assessments, and that is a very big deal.‘”

So what’s the result of all this shenanigans?

If a system of aggregating scores is artificially lowering overall averages — say from a 78 to a 76 — that translates into a negative impact across many facets of the industry.  Does this mean a small developer misses a deal?  Does this mean a borderline franchise doesn’t get a second incarnation?

It’s possible.  It’s happening now, in fact.  So fewer new IPs survive.  Fewer independent developers survive.  The industry suffers.

The hard truth on the media side is that the time will come when there is no benefit to sending review copies to anyone who scores on a letter, star, thumb or anything other than a factor of 10.  Smaller sites like GamingNexus and GameRevolution would get no review copies.  Game fans would get their scores from a smaller pool of opinions who wield ever-more power.

And don’t blame the PR person. A PR person’s primary responsibility is to do what is best for the game.  Not for the editorial process.  Not for your small site.  The game is the end goal; we are the game’s friend before we are the press’ friend.

What PR person do you know who wants to work hard to do what they can to position a game to crack 80 only to have it brought down by an inaccurate interpretation of what a B- is? It’s completely ridiculous.  It wouldn’t be smart PR to advise running reviews with these sites unless you were abundantly confident in receiving an “A.”

The new mantra that’s coming will be “don’t send that site a copy early, we’ll get fucked on Metacritic.”

Actually –to be honest — for aware PR people, it’s already HERE.

What Can We Do?

I don’t know.  Nobody has done anything that has changed anything yet.

All I can say is that you can contact the site and express your displeasure with some of their policies, politely try to express your frustration that a score isn’t accurately reflected on the site.  Tell them you worked for 18 months on a game that got a B- and MC’s 67 ruined your chance at a sequel.  Tell them.  Go ahead.

Click here. http://www.metacritic.com/about/contact.shtml

I’ll give you a B- if you do.

- PR_Flakavatarprflak-whole