Reviewing the Reviewers: Killzone: Mercenary
If you cast your minds back to the recent past (and yes, we know gamers have the worst short term memories ever) you should recall that we quite liked Killzone: Mercenary, even going so far as to say that it was “the best shooty handheld game ever made”. Hyperbolic stuff indeed. Anyway, fast-forward a couple of months and we still think it’s a top Vita experience, replete with bells, whistles and bullets. But we’ve had something burning in the back of our mind. You see, we were a little sneaky during the review process and we wanted to share some of that information with you.
Killzone: Mercenary provided quite an interesting pre-review period. It’s fairly common for us to receive games close to release with a comically short time until a review embargo ends, but this time Sony went a little crazy and sent over the review code on the 14th August; that was a full three weeks before the embargo for the game was due. Combined with the allocation of preview code and a monster public multiplayer beta it was clear that this was a product that Sony were looking to push, push, push – regardless of some hardcore griping that it was being sent out to die.
In a bit of a perfect storm we were able to play a lot of Killzone: Mercenary; I’m pretty much a Vita-only reviewer and with not much else waiting in the queue we could afford to go back and forth a fair few times in the game. And, by back and forth, I mean I was able to play through the single player campaign twice (once on Recruit and once on Veteran) and then complete every contract mission the game had to offer, effectively meaning that I played through the game five times. I should point out now that playing through a game five times before reviewing it is not normal (unless it’s some crazy score attack or roguelike), and for the sake of the review in question wasn’t entirely necessary. Like I said, perfect storm.
Anyway, one thing that struck me during all this singleplayer play was how unfortunate it was that we didn’t really have access to the multiplayer section. The servers were turned on during the pre-release period and as you may have noticed you can run around the maps solo in a private mode, but that’s clearly not the same as actually getting a feel for the multiplayer. In some respects we were lucky as there was the massive open multiplayer beta ongoing, and we know a lot of reviewers who were happy to base their opinions of the multiplayer aspect of the game on that alone. For a genre (and franchise) for which online play is so important we weren’t quite so happy with that option, especially with so many maps and modes missing from the beta.
We then started to poke around some of the leaderboards in Mercenary. The game offers three sets of leaderboards – campaign, career and multiplayer. Each of those has two sections – a lifetime and a past week. After we’d captured a bunch of names (we reasoned that they were all probably reviewing the game) we fired off a whole load of messages trying to organise a multiplayer session. We had four responses, two of whom then managed to make it to an arranged session. Needless to say we weren’t able to get started, and all seemed lost. However, that wasn’t the case for everyone – a few days before the embargo was due the IDs of four brave souls popped up on the multiplayer leaderboards, making them the only people who had played multiplayer in the full copy of the game. That’s four people not out of our sample of twenty-four, but four people out of everyone that had received a review copy.
It seemed as though that would be it, and that the majority of the reviews (including ours) would have to go out with only impressions of the multiplayer beta to draw on. However, luck was on everyone’s side – it seems a continental retailer went a bit crazy with posting out, and from the 2nd of September some players (we believe mainly French) gained access to the game. Which was utterly fantastic as we then got to play some multiplayer games – in fact, thank that nameless store as otherwise in all probability every review you read of Killzone: Mercenary would have either not mentioned multiplayer or could have only talked about the beta (perhaps even misrepresenting it as an opinion on the release multiplayer).
Once the dust settled we realised we were left with twenty-four PSN IDs (including our own), all of which we were fairly certain were those of games journalists/reviewers, although clearly the odd anomaly may have crept in. Coupled with the fact that Killzone: Mercenary has quite an illustrative set of trophies that demonstrate progress through much of the game we thought we were in the perfect position to check out whether these reviewers had made it into the last minute multiplayer and how far they had progressed in the single-player campaign. Isn’t this exciting? We checked the trophy lists several times for each ID, and we counted each trophy won before the review embargo date of 4th September. Feel free to click the link – it’ll take you to a Googledoc spreadsheet of the data we captured.
Amazing Spreadsheet of WIN
Back? Cracking stuff huh? Let’s break down a little bit of what you just saw. First of all, and hopefully self-explanatory, columns B to J represent the single-player campaign missions, all eight of them. As you can see, only one reviewer failed to actually complete the game. Naughty reviewer! For a game where you could run through the campaign in a fairly short amount of time that’s a bit of a copout right there. At least they got to the final mission – problems with figuring out how to kill the boss maybe? Of more interest than this anomaly is the fact that eleven out of the twenty-four played through the entire campaign at least once on the hardest difficulty available – that’s roughly 46%. Seeing as how the hardest difficulty in Mercenary wasn’t exactly super-taxing I’m not really sure how to feel about this, only to say that hopefully people running only a single playthrough didn’t do so on Recruit.
Now, one thing Killzone: Mercenary does is provide replayability for the campaign in the contract missions which all emphasise different ways to approach each of the scenarios, as well as requiring you to use specific weapons as you do so. Take a look at the stats and it’s clear that if the reviewers we captured (and remember kids – the leaderboards here pretty much gave us only the heavy duty players) didn’t really go crazy in attempting everything. Ignoring me, we have only two other reviewers who completed all eight of one of the alternative contract missions – unfortunately we can’t break that down any more to show how many people actually tried some of them, but what we can do is throw (even more?!) speculation into the mix and look at some other trophies, namely ‘Preferred Customer’ and ‘Master at Arms’.
What these trophies tell us could be quite interesting – ‘Preferred Customer’ is awarded to players who buy everything that Blackjack has to offer in his shop, and ‘Master at Arms’ is awarded when you’ve actually used everything that you can purchase from the shop. A quick check shows that a third of all reviewers managed to earn enough credits (that is, played the game enough) to buy everything. I’ll go out on a limb here and say that those reviewers played a good amount of the game, probably playing around in the single player campaign and the contracts enough to count as two to three playthroughs in total. We can’t say that the other reviewers didn’t do that of course – they may have just not bought one final piece from the shop for instance. An interesting aside to this is that I was the only reviewer to bother actually using all of those purchases – you clearly don’t need to use every single weapon or item in a game to appreciate the overall package, but it seems strange that you would bother to play the game enough to get to a position where you have unlocked everything and then not try certain items.
Let’s take a look at multiplayer now, the question that kicked off this whole shebang. Glancing at the trophy awarded for completing a single multiplayer round we see that nine reviewers out of our sample managed to complete at least one round within the shipped version of the game, giving us a figure of 37.5%. As we’ve already stated there was a large public multiplayer beta ongoing and it’s probable that the other reviewers took their experiences from that and fed it into their work. But, whatever way you look at it that figure is staggeringly poor and is probably more representative of other games than you would think. We tend to not be massively driven by meeting embargo dates (which is mainly a product of our status as an enthusiast site more than anything) but ‘driving clicks’ is a very big deal for those with commercial concerns. The publishers set short-period embargo dates and keep multiplayer servers switched off until late in the day, most review readers gorge on reviews and click into a load the day the embargo releases which then drives sites to do everything possible to deliver to those dates, even if it means not experiencing a portion of the game as fully as potentially possible. It’s almost as if reviewers are incentivised from all corners to not engage fully with online multiplayer, and this can quite clearly affect the end review score a fair amount in genres where the online portions are key.
We’ve called this ‘Reviewing the Reviewers’, so the question really has to be how we would score the group here. We’re thankful that twenty-three of the twenty-four managed to finish the single-player campaign entirely (as well they should!) but this is very much balanced against the small numbers who achieved any kind of multiplayer experience. Killzone: Mercenary struck us as a game with missions designed to be played over and over, the various difficulties, alternative contracts and money/rank system all feeding into this. With that in mind it’s hard to award more than a 5/10 to the group due to the combination of low numbers trying the full multiplayer and our presumption that only around a third engaged with the single-player elements as designed. We’ve caveated our opinions several times throughout with the fact that we have no sight of activities undertaken during the public multiplayer beta, so it’s very possible that each and every one of the remaining reviewers played on that for many hours. Regardless, try harder next time everybody, please!
Finally, let’s make a sombre note for the hardcore amongst you out there. If you wanted to read a review from a reviewer that had played the campaign on Veteran difficulty and managed to play some multiplayer then our sample only provides four reviewers that would meet your requirements – a paltry 16.6%. Keep to your OTs and your streaming, chaps, they serve you well.
We’ve waited a while before we published this piece; it’s the kind of feature that we’ve wanted to try to do for a while, and we were unsure whether our sample size was large enough or even whether it would look like we were trying out a bit of willy-waving having picked a game where we came out looking shiny. There are, obviously, quite a few presumptions made throughout, although I think we’ve pointed them out ourselves in the text. Hopefully it’s given a little bit of insight into how some things work behind the scenes and I hope you’ve enjoyed the read. Let us know if you would be interested in seeing these kinds of stats again in the comments section below, and we’ll look for future opportunities and wider sample ranges.