The Evils of Focus Testing

What’s your opinion on focus testing?

Here’s something I found recently on something like that: http://www.designer-notes.com/?p=1170

I think focus testing and test audiences in general are a tool, and like any other tool, they can be helpful, but also misused. Testing a product with actual people who are unfamiliar with the product is of incredible importance. Especially in the case of a game. Making a game is about creating a possibility space which extends far beyond what you can easily imagine just from looking at the code. You might set up the rules with one intention, then find people completely ignore it, or that your intended method of play is actually boring, or that it’s optimal, or the game suggests to play a different way entirely. Getting first impressions repeatedly is important throughout the development cycle. Getting experienced player impressions is important too.

The downside is that focus group testing is used as a type of insurance in the entertainment industry. When you invest a lot of money into something, you feel a greater pressure to make it a hit. Once you drop something, you can’t take it back. So focus testing is used to gauge whether something will be a hit with a general audience before it actually comes out.

The trouble is that first, a focus group is not a general audience. They might not be representative of the general public. Second, focus groups don’t always word their issues with developers the best. And third, executives have a tendency to take the word of the focus groups over the word of the designers. Working with focus groups effectively is something Valve has always been good at, for better or for worse. It requires watching their behaviors and sussing out troubles that they don’t necessarily vocalize. Take the example of point 5 from this blog post: https://www.platinumgames.com/official-blog/article/1924

Of course, for an example of focus testing gone wrong, we have Fuse (formerly Overstrike)

Kids apparently complained that the game looked too kiddie and said they wouldn’t play it. End result is that we got something totally generic looking that nobody was interested in on launch.

Another arguably negative example might be Hideki Kamiya not listening to feedback on Viewtiful Joe. It wasn’t exactly a big success, but nothing from Clover was. I remember finding the game too hard as a kid. I think it’s a bit rough around the edges even now.

I remember Itagaki once said that when he got feedback from some game testers saying Ninja Gaiden was too hard, he made the game even harder? Would that also count as a focus testing negative?

I don’t really know. I’ve definitely heard that quote before, and at the time (even with the Kamiya quote) I was like, “Haha, yeah, show ’em!” but on reflection, it might not have been the best idea even if it’s clearly a creator showing authority over the uneducated masses.

Honestly, I’m not even sure about the Kamiya example, except that I remember my experiences as a kid, and replaying it, it’s annoying to slow down time right when enemies shoot at you. I wasn’t used to that arcade type of difficulty as a kid, especially the unforgiving lives system, so I just played other games. Now as an adult, this is all manageable. I was progressing fine through the game last I played it, but I’m an expert. I can quantify how the game works a lot better, so I might not be able to judge it as accurately as someone experiencing it for the first time. And this difficulty of adoption might have lead to its largely unsuccessful sales. Is this a result of focus testing? Who really knows?

As for Itagaki, I don’t really know. He certainly made the game harder, but look at this version comparison between Black and Sigma. Sigma was made easier in some ways, but harder in others, where black (presumably the game his quote was talking about, or more true to his original vision) has a lot more sections with awkward controls where you’re simultaneously assaulted by enemies, such as in the swimming/water walking sections or some areas where you need to wallrun. Also the tank and horse bosses that need to be killed with arrows that can only be shot in first person view in black.