Testing Your Games Isn’t Optional
I suspect that the ability to actually recognize what’s already good and what still needs improving is just as rare as determination. -Thoth
Recognizing it at first glance is tough.
But we have a lot of tricks. The first is to simply play the game and remember what was annoying. It takes some effort to remain attentive, but not much. The second is to get *other* people to play the game.
This goes to a much broader question. Everything which can be tested should be tested. If you’re smart, very experienced, and paying attention, then yes – you can guesstimate. Yes, the master craftsman has a pretty good idea of his work and others and the local market. But anybody can test. You don’t need to be exceptionally smart, organized, or experienced to manage. Sure, we may be somewhat insulting ourselves to say that, but isn’t honestly accepting that we’re not always as bright as we pretend better than staking everything on pride?
Further, it does not matter whether it’s an MMORPG, a shooter, or the next version of Dungeons and Dragons. We interact with the game in different ways but for the same goal (fun), so we can and should test the game.
Shamus Young took a quick look at this in his blog, the inestimably-valuable Twenty Sided. The article is located over here:
The short version is that the more playtesting – and more inclusive of people outside your company – the better. Insiders have it way too easy, because they know. They made the thing, and that makes it tough to have an objective eye. Thoth and I have a devil of a time seeing our own personal spelling errors. Case in point: Spelling in the preceding sentence originally went down as “sapellign“, because I’m an awful typist. I don’t easily *see* it, because I made it. I see what I meant to have happen, not what actually occurred.
And that’s with a very specific, noticeable, error that everyone knows is wrong. It’s a lot less easy with a wide, complex system like a game. Shamus or Jay Barnson of The Rampant Coyote blog, I believe, wrote of a story from one consultant testing a game. She saw the players were going down a blind passageway and falling into the insta-death pit.
The game designers thought the players were just being stupid. They wanted to know why anyone was going down that way at all! They didn’t realize the obvious: that their own definition of “obvious” was omniscient. They forgot just how much knowledge they’d developed over the source of building a game for the last year. They KNEW that passage was a trap, and couldn’t see that everyone else didn’t magically share that info.
Or in another anecdote, Final Fantasy 12 features one of the most egregious abuses of sanity in games. In order to get one particular weapon, you had two options. First, you could not open four unidentified, perfectly normal chests. Then the spear would appear when you finally went to one end-game area. Or you could run that dungeon thousands of times for a random drop. The spear’s existence was known, since it was listed as an equippable item. But the fact that those chests had anything special was not mentioned and was completely impossible to determine from the player’s perspective. In fact, everything the game did pushed you to take the items. And this kind of nonsense is hardly unique: just look up the article on “Guide Dang It” at tvtropes. Dozens of entries include all kinds of weird puzzles which no reasonable person could solve, and only were solved because thousands upon thousands tried.
What’s both best and most frustrating about this is that it often requires only simple fixes. Much of it involves taking things out, or at worst adding small items. It’s *easy* to do. At the same time, it’s not popular. Companies don’t like having anyone not under a non-disclosure agreement, or who hasn’t been specifically invited by the marketing, department touching the game. They definitely don’t want people giving bad feedback, because those people might give bad word of mouth.
They’re over-thinking and under-thinking it. Games don’t sell well because somebody didn’t talk about them. Old bad publicity is a lot less important than new good publicity (because your game rocks when it’s actually released). When a new Blizzard game arrives, everyone forgets how much it sucked to wait for two years while they perfected it.
Unfortunately, the games industry in particular is a hotbed of terrible, terrible business practices. I mean, it’s bad. It’s embarrassing. They venture into madness that no big, established corporation or a small, eager startup would dare conceive, let alone implement.
There is one absolutely clear reason for this: game companies are curiously isolated from their customer base and are often started by people with little business experience. Further, growing industries can paper over their flaws, and games are still a growth industry. In short, they have an awful record of responding to customer patterns and a worse one for predicting them. More often, they ham-fistedly chase patterns which may not exist and security they can’t really have.
The big genres see years of repetitive releases, precisely because publishers are flying blind. They constantly fret over their established markets and often want to discard innovation entirely, leading to a narrowing of genres and styles (a crisis equaled only in the pharmaceutical industry which operates under unimaginably tougher conditions). This is one reason Indy companies have risen to become serious concerns themselves – the big publishers are already sclerotic and risk-averse beyond all reason.
I recognize that like Hollywood, it’s an industry where output quality (how good da flick be) is always somewhat random. But it’s also a great deal more controllable. Once you start making a movie, the situation is largely out of control. It succeeds based on your ability to quickly adjust to challenges and how well you’re prepared; once you start filming, you can’t really change anything. Once you stop, re-shoots are a pricey last resort. It’s far more feasible to tweak an existing game and incorporate feedback, especially during production. And once it’s “finished”, you can always delay a month and polish accordingly. It’s only an issue if you schedule your release for Christmas and delays might miss the deadline (which is a very good argument for September targets or no target at all).
So, in short, the games industry is laughable in terms of skill. Management is mostly by rank amateurs who may be trained for something entirely different, and have no concept of HOW to manage a project. Now, it’s pretty well established that business school doesn’t teach you to manage. Don’t listen to any of the schools’ declaring otherwise. It does help you understand what kinds of things you need to do. The games industry is so young and fast-growing that it can skimp on that. It can have no decent appreciation of customer service, force dubious legal concepts on customers, and often get away with abusing their suppliers and/or customers directly. It’s a great time to be incompetent, or a jerk, or both.
On training: Harvard MBA’s do well because they get all the connections who also went to Harvard, not because they’re especially well-taught. It’s been more or less mathematically demonstrated as much as anything can be. And if any Harvard grads read this… tough. The data is quite clear. Connections, training, and even academic intelligence have almost nothing to do with managerial competence. What that can do is introduce ideas. But what people get is more in line of an opportunity. They must choose to make the most of it.
This is why to me it’s not surprising that Guest Writer Daniel at The Rampant Coyote talks about how wearying dealing with customers is. Towards the end he notes that “You just lose heart, and stop reading forums. There are a few who soldier on, taking the flak, and trying to get to the bottom of the feedback. There are even those BioWare hired to do so. But most developers eventually lose heart and just tune it out.”
Tuning it out is a mistake. It’s a huge mistake, and it probably explains why Bioware has made some titanically stupid decisions in recent years. Fortunately, they haven’t been fatal yet, because the company still has strong foundations. If you’re not listening, you’re flying blind.
There’s an old argument about whether games are art. Some of them can be, but not every game can or should be called art. It’s a group product, and artistic sense is less important than making a fun game. If you’re not listening – if you’re not sucking it up and trying your hardest to split good feedback away from the useless or trivial – you’re not doing your job. You will always have nitpickers. They’re not important. What is important is polishing before you dump a product on the market. Dragon Age, which Daniel is justly proud of, still had hugely obvious flaws, which could have been easily corrected. It’s sequel is much, much worse.
The internet is a problem, true. Game designers love using the internet themselves – they just don’t want their detractors on it. You get angry forum posts and customer complaints. Who wants to sort through those?
Tuning out the bad silences every voice which could help you, when you’ve never had more powerful tools for feedback. Consider that in the 80’s and 90’s, pen and paper and video games often shipped with postcards inside just so designers could get some kind of feedback or understanding.
People tend to be a lot more pleasant when they think you’re listening. Second, on the internet, you can actually get responses. It shouldn’t surprise you either that Blizzard takes a keen interest in their forums. They often read deeply into long threads, and I guarantee those forums have gotten a lot more posts than anything you’ve ever produced. And they listen. They really do. If enough people demand something, Blizzard will oblige, or resolve the issue with another solution. If they decide not, they’ll usually explain their reasoning and why they think it’s either necessary or the best practical answer.
No, customer feedback isn’t, and shouldn’t be, the answer to everything. It’s rarely very innovative. But it contains a wealth of information about your weaknesses, even if it’s often less clear on how you resolve them. I recently read some articles on how Bioware collected statistics (poorly, and obviously not using good statistical methodology) on Dragon Age: Origins. Of course, rather than admit the obvious – that the game had a weak Ostagar area and dragged too much in what should have been the real opening – they simply decided that people “didn’t like numbers” and tried to simplify DA2. And not surprisingly, their claimed statistical results don’t actually back them up, because they failed to clearly identify cause and effect.
Yes, it’s a lot more work. That’s the price you pay for endless reams of data and a hundreds of thousands of customers. My heart just bleeds for your troubles.