Making the grade: Why tying anything to aggregate review scores is bad

The culture around aggregate review scores, people being so obsessed with them and companies often tying developers livelihoods around them needs to change.

It’s an all-too-familiar tale and yet we heard it yet again this past week. Jason Schreier of Kotaku published a lengthy report on the behind-the-scenes troubles of developing the game Mafia III. Hearing reports of behind-the-scenes troubles developing games is certainly nothing new. Some games still turn out fine, even great, despite facing some major development problems before finally getting to retail.

A particular point in the article stood out (beyond the mention of a new Bioshock game that may or may not be currently in development). Regardless of sales, the financial well-being of Hangar 13, the development team that worked very hard just to get the game out to market, as well as the future of the Mafia franchise, was seemingly tied to an aggregate score. Specifically, the aggregate score of review compilation site Metacritic:

"Some people remembered the threshold being an 85; others remembered it being 80—either way, it didn’t matter. Mafia III did not earn an 80 on Metacritic. It didn’t even get close. Today, the open-world action game sits at 68, which is described as “mixed or average” on the aggregation website but is considered a critical disappointment among big-budget video games. Reviewers liked some aspects, but knocked the game for feeling grindy, repetitive, and buggy. Some developers still got bonuses, but they weren’t nearly as big as they would have been if Mafia III’s review scores had been higher."

Granted, the score is not great. But it would hardly have been the first time a long-running series stumbled on one entry and came back stronger the next time. According to publisher 2K, as stated in the article, the game sold phenomenally well. In fact, it was the “fastest-selling game” in the publisher’s history at the time. Granted, there’s likely some spin there, but it seems like Mafia III sold some solid numbers and the general reaction from developer Hangar 13 was to take the positive and negative criticisms and learn from them for the next game, whatever it may be.

Fast forward to now, and most of what made up Hangar 13 has been drastically reduced or and shuffled around to other projects, so the likelihood of even a spiritual successor to Mafia III from that same team seems slim at best.

This is hardly the most egregious example of aggregate review scores influencing the financial well-being of employees of a development studio. That’s most likely Fallout: New Vegas. Despite the game shipping over five million copies and generally regarded as one of the better entries in the series, developer Obsidian lost out on bonuses because the game didn’t get an aggregate score of 85 on Metacritic specifically. What was the average? 84. Let that sink in for a minute.

The purpose of this piece is not to rail against Metacritic. App Trigger’s own reviews are published on that site as well as dozens of other outlets big and small. While any aggregate site has its issues, there’s nothing inherently wrong with a site that collects reviews and gives them an average score based on the reviews. It’s not perfect, but it’s not a bad way to get a snapshot of the general opinion of the media in question. If you want to dig deeper, especially for any opinions that are outside the general consensus, it’s not hard to do so.

The problem comes from both publishers and a percentage of small but often extremely vocal fans placing so much emphasis on such an arbitrary measure as an aggregate score, especially from one aggregate site in particular. There can be any number of reasons a game’s average score might vary wildly, especially on a divisive title. Maybe the story or mechanics just didn’t work for some reviewers, while others loved them.

Either way, it shouldn’t be the end-all be-all it is treated as. I doubt reviewers often feel any pressure to take into consideration if someone might miss out on a bonus while in the process of reviewing, nor should they. That should never be a consideration when handing out a score on a game. It’s a critic’s responsibility to just say what they thought of the game: what worked for them and what didn’t. Nothing else should influence review scores.

Regardless, it’s not uncommon to be harassed on social media for giving a game what is perceived as an “unacceptable” score by some fans of the game or the developer and “wrecking” the aggregate score. The culture and dependence on such things is a bad and vicious cycle that in the end really benefits nobody except publishers hoping to save some money by not having to pay the developers what they are owed even in the event of a very successful game.

Next: 50 best RPGs of all time

Again, this is not a slight on Metacritic or aggregate sites like it. They are tools that when used correctly, are a perfectly valid way to judge how to spend your valuable time and money on entertainment. But certain fans being obsessed with them and especially publishers tying employees’ financial well-being to them is a terrible practice.

The type of fans being obsessed with review scores might not stop, those types have been around since long before aggregate sites were a thing and will continue to be. But publishers need to stop putting so much emphasis on this. More than anything else it is not only giving those fans more justification in their mind, it also just seems like an excuse to not pay very hard-working employees what they are due or publishers promising developers something they at least hope they don’t have to follow through on, especially as the very nature of criticism has evolved over time and gets more nuanced. Hopefully, it’s a practice that will end sooner rather than later.

The views expressed in this article explicitly belong to the author, and do not necessarily reflect the views of, nor should be attributed to, App Trigger or FanSided as an organization.