Game Testing and the Thinking Designer

Photographs of several "Final Fantasy XII...

Take at least one...

Pop Quiz!

Testing Your Games Isn’t Optional

I suspect that the ability to actually recognize what’s already good and what still needs improving is just as rare as determination. -Thoth

Recognizing it at first glance is tough.

But we have a lot of tricks. The first is to simply play the game and remember what was annoying. It takes some effort to remain attentive, but not much. The second is to get *other* people to play the game.

This goes to a much broader question. Everything which can be tested should be tested. If you’re smart, very experienced, and paying attention, then yes – you can guesstimate. Yes, the master craftsman has a pretty good idea of his work and others and the local market. But anybody can test. You don’t need to be exceptionally smart, organized, or experienced to manage. Sure, we may be somewhat insulting ourselves to say that, but isn’t honestly accepting that we’re not always as bright as we pretend better than staking everything on pride?

Further, it does not matter whether it’s an MMORPG, a shooter, or the next version of Dungeons and Dragons. We interact with the game in different ways but for the same goal (fun), so we can and should test the game.

Shamus Young took a quick look at this in his blog, the inestimably-valuable Twenty Sided. The article is located over here:

The short version is that the more playtesting – and more inclusive of people outside your company – the better. Insiders have it way too easy, because they know. They made the thing, and that makes it tough to have an objective eye. Thoth and I have a devil of a time seeing our own personal spelling errors. Case in point: Spelling in the preceding sentence originally went down as “sapellign“, because I’m an awful typist. I don’t easily *see* it, because I made it. I see what I meant to have happen, not what actually occurred.

And that’s with a very specific, noticeable, error that everyone knows is wrong. It’s a lot less easy with a wide, complex system like a game. Shamus or Jay Barnson of The Rampant Coyote blog, I believe, wrote of a story from one consultant testing a game. She saw the players were going down a blind passageway and falling into the insta-death pit.

The game designers thought the players were just being stupid. They wanted to know why anyone was going down that way at all! They didn’t realize the obvious: that their own definition of “obvious” was omniscient. They forgot just how much knowledge they’d developed over the source of building a game for the last year. They KNEW that passage was a trap, and couldn’t see that everyone else didn’t magically share that info.

Or in another anecdote, Final Fantasy 12 features one of the most egregious abuses of sanity in games. In order to get one particular weapon, you had two options. First, you could not open four unidentified, perfectly normal chests. Then the spear would appear when you finally went to one end-game area. Or you could run that dungeon thousands of times for a random drop. The spear’s existence was known, since it was listed as an equippable item. But the fact that those chests had anything special was not mentioned and was completely impossible to determine from the player’s perspective. In fact, everything the game did pushed you to take the items. And this kind of nonsense is hardly unique: just look up the article on “Guide Dang It” at tvtropes. Dozens of entries include all kinds of weird puzzles which no reasonable person could solve, and only were solved because thousands upon thousands tried.

What’s both best and most frustrating about this is that it often requires only simple fixes. Much of it involves taking things out, or at worst adding small items. It’s *easy* to do. At the same time, it’s not popular. Companies don’t like having anyone not under a non-disclosure agreement, or who hasn’t been specifically invited by the marketing, department touching the game. They definitely don’t want people giving bad feedback, because those people might give bad word of mouth.

They’re over-thinking and under-thinking it. Games don’t sell well because somebody didn’t talk about them. Old bad publicity is a lot less important than new good publicity (because your game rocks when it’s actually released). When a new Blizzard game arrives, everyone forgets how much it sucked to wait for two years while they perfected it.

Unfortunately, the games industry in particular is a hotbed of terrible, terrible business practices. I mean, it’s bad. It’s embarrassing. They venture into madness that no big, established corporation or a small, eager startup would dare conceive, let alone implement.

There is one absolutely clear reason for this: game companies are curiously isolated from their customer base and are often started by people with little business experience. Further, growing industries can paper over their flaws, and games are still a growth industry. In short, they have an awful record of responding to customer patterns and a worse one for predicting them. More often, they ham-fistedly chase patterns which may not exist and security they can’t really have.

The big genres see years of repetitive releases, precisely because publishers are flying blind. They constantly fret over their established markets and often want to discard innovation entirely, leading to a narrowing of genres and styles (a crisis equaled only in the pharmaceutical industry which operates under unimaginably tougher conditions). This is one reason Indy companies have risen to become serious concerns themselves – the big publishers are already sclerotic and risk-averse beyond all reason.

I recognize that like Hollywood, it’s an industry where output quality (how good da flick be) is always somewhat random. But it’s also a  great deal more controllable. Once you start making a movie, the situation is largely out of control. It succeeds based on your ability to quickly adjust to challenges and how well you’re prepared; once you start filming, you can’t really change anything. Once you stop, re-shoots are a pricey last resort. It’s far more feasible to tweak an existing game and incorporate feedback, especially during production. And once it’s “finished”, you can always delay a month and polish accordingly. It’s only an issue if you schedule your release for Christmas and delays might miss the deadline (which is a very good argument for September targets or no target at all).

So, in short, the games industry is laughable in terms of skill. Management is mostly by rank amateurs who may be trained for something entirely different, and have no concept of HOW to manage a project. Now, it’s pretty well established that business school doesn’t teach you to manage. Don’t listen to any of the schools’ declaring otherwise. It does help you understand what kinds of things you need to do. The games industry is so young and fast-growing that it can skimp on that. It can have no decent appreciation of customer service, force dubious legal concepts on customers, and often get away with abusing their suppliers and/or customers directly. It’s a great time to be incompetent, or a jerk, or both.

On training: Harvard MBA’s do well because they get all the connections who also went to Harvard, not because they’re especially well-taught. It’s been more or less mathematically demonstrated as much as anything can be. And if any Harvard grads read this… tough. The data is quite clear. Connections, training, and even academic intelligence have almost nothing to do with managerial competence. What that can do is introduce ideas. But what people get is more in line of an opportunity. They must choose to make the most of it.

 This is why to me it’s not surprising that Guest Writer Daniel at The Rampant Coyote talks about how wearying dealing with customers is. Towards the end he notes that “You just lose heart, and stop reading forums. There are a few who soldier on, taking the flak, and trying to get to the bottom of the feedback. There are even those BioWare hired to do so. But most developers eventually lose heart and just tune it out.”

Tuning it out is a mistake. It’s a huge mistake, and it probably explains why Bioware has made some titanically stupid decisions in recent years. Fortunately, they haven’t been fatal yet, because the company still has strong foundations. If you’re not listening, you’re flying blind.

There’s an old argument about whether games are art. Some of them can be, but not every game can or should be called art. It’s a group product, and artistic sense is less important than making a fun game. If you’re not listening – if you’re not sucking it up and trying your hardest to split good feedback away from the useless or trivial – you’re not doing your job. You will always have nitpickers. They’re not important. What is important is polishing before you dump a product on the market. Dragon Age, which Daniel is justly proud of, still had hugely obvious flaws, which could have been easily corrected. It’s sequel is much, much worse.

The internet is a problem, true. Game designers love using the internet themselves – they just don’t want their detractors on it. You get angry forum posts and customer complaints. Who wants to sort through those?

You do.

Tuning out the bad silences every voice which could help you, when you’ve never had more powerful tools for feedback. Consider that in the 80’s and 90’s, pen and paper and video games often shipped with postcards inside just so designers could get some kind of feedback or understanding.

People tend to be a lot more pleasant when they think you’re listening. Second, on the internet, you can actually get responses. It shouldn’t surprise you either that Blizzard takes a keen interest in their forums. They often read deeply into long threads, and I guarantee those forums have gotten a lot more posts than anything you’ve ever produced. And they listen. They really do. If enough people demand something, Blizzard will oblige, or resolve the issue with another solution. If they decide not, they’ll usually explain their reasoning and why they think it’s either necessary or the best practical answer.

No, customer feedback isn’t, and shouldn’t be, the answer to everything. It’s rarely very innovative. But it contains a wealth of information about your weaknesses, even if it’s often less clear on how you resolve them. I recently read some articles on how Bioware collected statistics (poorly, and obviously not using good statistical methodology) on Dragon Age: Origins. Of course, rather than admit the obvious – that the game had a weak Ostagar area and dragged too much in what should have been the real opening – they simply decided that people “didn’t like numbers” and tried to simplify DA2. And not surprisingly, their claimed statistical results don’t actually back them up, because they failed to clearly identify cause and effect.

Yes, it’s a lot more work. That’s the price you pay for endless reams of data and a hundreds of thousands of customers. My heart just bleeds for your troubles.

3 Responses

  1. Hello, guest writer Daniel here. I saw your link over at Rampant, and just finished reading the article above. While I agree with a lot of what you say above, I felt I should clarify the quote you attribute to me.

    I don’t want to give the impression that BioWare, as a policy, ignores forums. It’s quite the contrary. Not only does it have staffers who work with the fan community as a full-time job, many of the developers voluntarily do so as well. You’ll find designers, programmers, producers, and many others interacting with the community on feedback and bugs.

    I guess what I was getting at, and probably did a bad job explaining, was my own self-discovery on the road from gamedev newbie to gamedev vet. I learned that it takes a very special type of person to navigate forums on the internet without getting emotionally hurt. Not everyone has that kind of emotional armor, and I certainly still need to work on mine.

    I shouldn’t have generalized in my post, as it does make it sound like I’m advising developers to ignore fans. As both you and many of the commenters on that post point out, completely ignoring forum banter isn’t the way either.

    Anyway, thanks for the post, and the reality check. It looks like I may have to come back later and check out some more of your material. AD&D, Shadowrun, White Wolf…you’ve got a stable of blog topics to make my heart swoon!

  2. Please do come back. And if you ever have a shot at making a Shadowrun game (or something similar), I will sell my stuff and move to the Bioware basement offering foot massages if it gets me in the door, despite my many, *many* differences with the company’s direction. I had a mini-design-document for a Shadowrun RPG laying about this site somewhere.

    I’m just praying we don’t lose yet another RPG company if the new Star Wars RPG implodes. While Bioware has a reasonable track record and early reviews are positive, I have some real doubts as to the profitability of the result. Bioware has been biting off an awful lot recently, and quality noticeably suffers for it. I merely hope there’s enough to spread liberally over the MMORPG.

    One thing you mentioned: “I learned that it takes a very special type of person to navigate forums on the internet without getting emotionally hurt.”

    I must disagree. Emotional distance is as much as skill as any other bit of professionalism. I don’t say you won’t get annoyed, but it starts with one simple rule: never respond. If a response is needed, wait a while. You usually can ignore the obnoxious people, of course. They’re not really going to help. You might note what bothered them and leave it at that.

    But sometimes if you come back later, you’ll realize that the one person who angered you most was the one person who really offered something worthwhile – because they they hit closest to a tender nerve and a real weak spot.

  3. Waiting a while is a good suggestion. I’ve often found difficult situations easier to deal with after sleeping on them, so I’ll keep that trick in mind.

    And while I’m not sure if a Shadowrun-like RPG is in BioWare’s near future, I know *I* love the setting, and I wouldn’t be surprised if elements of that setting make it into my upcoming games ;-)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: