Game Design #17: Playtesting Wargames - A "Fair Test"

For someone who spends so much time testing and reviewing rules, and chatting with rules designers, I do very little official playtesting.  Simple reason: good playtesting is too much like hard work.

It's also something almost no one does properly - in fact it's almost an impossible task.

According to the old science curriculum, there are a few elements that make a "fair test"of something.  Amongst them are some pretty tough hurdles for a game designer to overcome:

*All variables should be considered and controlled
*The experiment must deal with the question being studied
*Only one variable should be altered at a time
*The experimenters should not be biased
*Conclusions drawn must be better than those from chance
*Experiments must be able to be replicated

So to fairly or properly playtest something in a scientific manner, there are a few things to consider:

Playtesting is difficult, when the subject involves chance, and the testors themselves are both a "variable"and likely biased.

*All variables should be considered and controlled

This is almost impossible in a wargame where testers will themselves are directly involved and not independent observers - they vary in playing skill, experience and their comprehension of the rules - the testers themselves are an uncontrolled variable! 

*The experiment (game) must deal with the question being studied
Well obviously, the playtest deals with the game.  But what is the "question"the game dev is asking of his testers?  Is he wanting to know if the game is fun? realistic?  too slow?  to confusing? feedback on a particular mechanic? - to get specific feedback the designer needs to pose specific questions for the playtesters.

*Only one variable should be altered at a time
This article was inspired by the v3 Infinity rules. There was changes in "to hit" modifiers of weapons at different range bands.  As shooting is very important in Infinity, the designers took a very careful approach.  They would change the ranges on a single weapon (keeping all the other weapon ranges and rules the same) play a bunch of games, then change and test another weapon. In a game with dozens of weapons, this means a LOT of playtest games.  In addition, they broke it down further - they experimented with the modifiers within each range band, for each weapon.  That is hundreds of playtest games - something Infinity, with its established and large loyal fanbase can do; but a bit harder for the average self-published game designer.  Most games go through various "alpha" and "beta" incarnations but I doubt many companies would have so many playtest games simply to fine-tune a single weapon. 

The problem with changing many variables is you do not know which change had the most impact on gameplay.  Even companies with the capability to do this often miss this piece of common sense; i.e. in video game companies when they can easily "patch" a particular weapon by simply typing a few lines of code, when they "balance" things they often mess with several variables. For example: I played a PC game where armed dune buggies ran rampant.  They were so fast they could engage and flee at will, they could be repaired while moving, and they were tough. In addition, they wielded a powerful bazooka.  These buggies were death machines, able to take out MBTs and infantry, their self-repairing and agility making them nigh-immortal to all foes - even gunships and aircraft.  The company responded. But they changed multiple variables. They reduced the weapon power, made the buggy very fragile, reduced the ability to repair while moving, and lowered the speed.  The buggy then became useless.  And the company could not point to the exact reason, due to the mass of changes they made.  The company should have reduced ONE variable (such as speed), tested it, then reduced another variable (say weapon damage) until the buggy reached a state of "useful but not overpowered."

*The experimenters should not be biased
Most playtesters are drawn from local clubs and friends (even ones you only deal with via email may form an emotional attachment and investment to you and your project; the loyalty that makes them a "good" playtester may also bias their opinions).   

*Conclusions drawn must be better than those from chance
The very nature of our card-based and  dice based wargames means chance plays a large part in any game and lucky rolling can change gameplay outcomes a lot.  Whilst we can usually recognize unusually lucky/unlucky dice rolling, the strong element of chance in wargames makes feedback less reliable - which brings even more emphasis to the following:

*Experiments must be able to be replicated
I'd reword this simply as "experiments must BE replicated."  Wargame playtesting has problems -  the element of chance, and our playtesters are who likely biased; and themselves a "variable" differing in ability and understanding.    This is where repeating the "test" leads to more reliable results.  If you slip a coin once, and it lands on heads, you might conclude the coin will land on heads every time, or the majority of the time.  However flip the coin 100 times, and you will probably realize it is likely to be around 50/50.  Flip it 1000 times, and you will increase the accuracy of your experiment.  Obviously game designers are limited in the number of tests they can make - but the point is, the more tests (games), the more accurate the data.

TL:DR
As you can see, it is well-nigh impossible to test a wargame "scientifically."  Many elements of a proper test are outside the game designer's control.  However we can learn from the principles of a "fair test"- I'd like to draw attention to those aspects which ARE controllable: 

*Have a focus for feedback. The game dev should seek specific feedback, perhaps by posing focus questions that playtesters can respond to.
*Only change one thing at a time. Change one or two small things, then test.  Overhauling masses of features and mechanics can make it unclear what is causing changes/aspects of gameplay.
*Test repeatedly.  The more test games, the better the results.

Sadly, these often involve more effort than people are willing (or even able) to spend; it's a little wonder we have so many wargames which are confusing, frustrating or exploitable.  In fact it's remarkable we have so many good games, when you consider the difficulties.  Hats off to the valiant game devs and playtesters!