How A/B Testing Could Change Online Gaming Forever

Video games were once universal experiences. But constant player testing could redefine the genre.

How A/B Testing Could Change Online Gaming Forever

To this day, the entire planet knows the first level of Super Mario Bros. by heart. Watch out for that first goomba. Hit the second question mark block for the mushroom. Make your way through the level–wait–there’s a 1UP hiding in the sky somewhere around here. We can all remember this for two reasons: One, we played a lot of Super Mario Bros. And two, Super Mario Bros. was always the same thing. The power ups were carefully placed in repeated spots. Mario’s peak velocity was constant. And a turtle bite always hurt the exact same amount.


Mario was a vastly different experience than the Internet today. The Internet is always changing and measuring our response to change. It’s called A/B testing, and its most famous example was in Google’s 41 shades of blue. Treating customers as unconscious lab rats, a company offers two (or way more) variations of a single controlled element, then they see how, and if, that affects user behavior. A few weeks back, one of the biggest gaming companies in the world–the ~$8 billion powerhouse Activision–signed a relatively unknown startup called Swrve to handle “Continuous A/B testing and Mobile Game Optimization.” Whatever might not have been obvious a year or two ago is certain: A/B testing is about to take over game design with all the ferocity of web engineering. Gaming may never be the same–but how?

Swrve is an analytics company. Their pitch is statistical analysis easy enough for creatives to use, and indeed, their dashboard is similar to any other you’ve seen. But what makes their toolset a bit different is that it’s actually controlling metadata within games for iOS, Android, Facebook and any other Internet connected platform. And with the tweak of metadata values, almost any parameter of a game can be adjusted and measured. Despite being just six months old, Swrve is processing a billion of these game events a day.

“We want to give creative folk a blank canvas to marry big data without going back to school and learn statistics,” says Swrve CEO Hugh Reynolds. “In the old scenario, you build something for two years, push it out, and it’s a huge success or a failure for reasons unknown for you. That’s a dreadful creative space to be in. [With Swrve], you can push it out, get it to a B-. Now put your heads down, make it into a B, a B+ and an A-, and we can do this by the sweat of our brow and inspiration, hard work.”

Tools like Swrve allow the designers themselves to rapidly test hypotheses about their games on just 5% of their users–everything from tweaking level difficulty (by adjusting enemy health or character physics), to seeing if users were more apt to select (and buy) a protagonist with a red or purple headband. Anything with a metadata value is adjustable in real time, all within an app the customer has already downloaded, all without any additional downloads.

It’s easy to see the benefits, especially in the lucrative free-to-play space that Swrve is targeting. Small development teams don’t have hundreds of game testers to painstakingly discover the elusive sweet spot of addictive game difficulty, and even when they do, professional game testers are a pretty specialized cohort compared to grandmas on Facebook. “Instead of having this huge team, we allow people to run these experiments against their live user base,” says Reynolds. “The ultimate adjudicator is your real users.”

Real users aren’t just the ultimate adjudicator; they’re also the ultimate customers. And a skeptic (make that, realist) might point out that the way to best monetize a free-to-play game is to know exactly how to coerce customers to buy the most in-game upgrades before walking away in frustration. With Swrve data, developers can actually identify different types of players, even suggesting different hints and selling different items based upon the way they play. “It’s nothing to do with trying to trap people into buying something,” argues Reynolds. “It’s very pure, the reality of what’s happening. Instead of people needing to pay up front with entertainment as a service, they pay as they enjoy. So the key is to make sure people enjoy the product.”


Swrve also has the potential to fundamentally shift the decision structure within game development companies. As in any sector, creatives often answer to suits, and they need to constantly justify artistic decisions within market context that make sense to bottom-line-hungry investors. If there’s a silver lining to what some may see as overly calculating big data in gaming–in constant, widespead focus testing of creative decisions–it’s that the artists will have the tools to prove the value of their decisions to their bosses, by showing data that 5% of real users were more likely to buy items of a particular aesthetic or play a game longer with a certain soundtrack.

“It definitely can [get in the way]. Taken to the extreme, where it becomes a mechanistic process, that would take the heart out of the creativity. But what we’ve seen is, the creativity still comes from the team,” says Reynolds. “It enables them to take bigger risks. Removing five features is something the creative team would have never been allowed to do. But being able to put a safety net under the process and be able to back it up with hard facts, creatives can be bigger risk-takers.”

It’s not a bad pitch: More power to creatives! But what would Mario have been with A/B testing? Would that fireflower always be there? Would only some players find the hidden 1UP? Would Mario have even been Mario, or would his taller, svelter brother Luigi have garnered more appreciation from 5% of the populace, thereby taking over the entire franchise? Would Luigi have ditched the green overalls for a black leather jacket?

They’re odd thoughts to contemplate–just as odd a pair of plumbers who fight giant reptiles to rescue toadstools and princesses.

[Image: Yurly Chetok/Shutterstock]

About the author

Mark Wilson is a senior writer at Fast Company. He started, a simple way to give back every day.