How A/B Testing Could Change Online Gaming Forever

Video games were once universal experiences. But constant player testing could redefine the genre.

To this day, the entire planet knows the first level of Super Mario Bros. by heart. Watch out for that first goomba. Hit the second question mark block for the mushroom. Make your way through the level—wait—there’s a 1UP hiding in the sky somewhere around here. We can all remember this for two reasons: One, we played a lot of Super Mario Bros. And two, Super Mario Bros. was always the same thing. The power ups were carefully placed in repeated spots. Mario’s peak velocity was constant. And a turtle bite always hurt the exact same amount.

Mario was a vastly different experience than the Internet today. The Internet is always changing and measuring our response to change. It’s called A/B testing, and its most famous example was in Google’s 41 shades of blue. Treating customers as unconscious lab rats, a company offers two (or way more) variations of a single controlled element, then they see how, and if, that affects user behavior. A few weeks back, one of the biggest gaming companies in the world—the ~$8 billion powerhouse Activision—signed a relatively unknown startup called Swrve to handle "Continuous A/B testing and Mobile Game Optimization." Whatever might not have been obvious a year or two ago is certain: A/B testing is about to take over game design with all the ferocity of web engineering. Gaming may never be the same—but how?

Swrve is an analytics company. Their pitch is statistical analysis easy enough for creatives to use, and indeed, their dashboard is similar to any other you’ve seen. But what makes their toolset a bit different is that it’s actually controlling metadata within games for iOS, Android, Facebook and any other Internet connected platform. And with the tweak of metadata values, almost any parameter of a game can be adjusted and measured. Despite being just six months old, Swrve is processing a billion of these game events a day.

"We want to give creative folk a blank canvas to marry big data without going back to school and learn statistics," says Swrve CEO Hugh Reynolds. "In the old scenario, you build something for two years, push it out, and it’s a huge success or a failure for reasons unknown for you. That’s a dreadful creative space to be in. [With Swrve], you can push it out, get it to a B-. Now put your heads down, make it into a B, a B+ and an A-, and we can do this by the sweat of our brow and inspiration, hard work."

Tools like Swrve allow the designers themselves to rapidly test hypotheses about their games on just 5% of their users—everything from tweaking level difficulty (by adjusting enemy health or character physics), to seeing if users were more apt to select (and buy) a protagonist with a red or purple headband. Anything with a metadata value is adjustable in real time, all within an app the customer has already downloaded, all without any additional downloads.

It’s easy to see the benefits, especially in the lucrative free-to-play space that Swrve is targeting. Small development teams don’t have hundreds of game testers to painstakingly discover the elusive sweet spot of addictive game difficulty, and even when they do, professional game testers are a pretty specialized cohort compared to grandmas on Facebook. "Instead of having this huge team, we allow people to run these experiments against their live user base," says Reynolds. "The ultimate adjudicator is your real users."

Real users aren’t just the ultimate adjudicator; they’re also the ultimate customers. And a skeptic (make that, realist) might point out that the way to best monetize a free-to-play game is to know exactly how to coerce customers to buy the most in-game upgrades before walking away in frustration. With Swrve data, developers can actually identify different types of players, even suggesting different hints and selling different items based upon the way they play. "It’s nothing to do with trying to trap people into buying something," argues Reynolds. "It’s very pure, the reality of what’s happening. Instead of people needing to pay up front with entertainment as a service, they pay as they enjoy. So the key is to make sure people enjoy the product."

Swrve also has the potential to fundamentally shift the decision structure within game development companies. As in any sector, creatives often answer to suits, and they need to constantly justify artistic decisions within market context that make sense to bottom-line-hungry investors. If there’s a silver lining to what some may see as overly calculating big data in gaming—in constant, widespead focus testing of creative decisions—it’s that the artists will have the tools to prove the value of their decisions to their bosses, by showing data that 5% of real users were more likely to buy items of a particular aesthetic or play a game longer with a certain soundtrack.

"It definitely can [get in the way]. Taken to the extreme, where it becomes a mechanistic process, that would take the heart out of the creativity. But what we’ve seen is, the creativity still comes from the team," says Reynolds. "It enables them to take bigger risks. Removing five features is something the creative team would have never been allowed to do. But being able to put a safety net under the process and be able to back it up with hard facts, creatives can be bigger risk-takers."

It’s not a bad pitch: More power to creatives! But what would Mario have been with A/B testing? Would that fireflower always be there? Would only some players find the hidden 1UP? Would Mario have even been Mario, or would his taller, svelter brother Luigi have garnered more appreciation from 5% of the populace, thereby taking over the entire franchise? Would Luigi have ditched the green overalls for a black leather jacket?

They’re odd thoughts to contemplate—just as odd a pair of plumbers who fight giant reptiles to rescue toadstools and princesses.

[Image: Yurly Chetok/Shutterstock]

Add New Comment


  • Chris Heywood

    Such an amazingly cringe-worthy video. To quote Demetri Martin: "It's so youthy."

    I play loads of games and this looks likely to provoke one irksome trait of modern gaming: perpetual updates. Games that are patched post release because people found it too difficult, or buggy, or just didn't like it. It removes the confidence of designers and developers throwing a challenging idea to the community and have them wrestle with it and embrace it.

    The major area of danger here is that it inevitably panders to the majority, creating less unique experiences as funders capitalise on the more profitable majority. An agency senior once lamented "Focus groups are where new ideas go to die. Most of the time, people will choose something they've seen before."

    That's why you can release the same Mario, Zelda or COD game year after year. Sure, they're reliable and comfortable, but they rarely challenge with new concepts or push the medium in new directions.

  • Patrick Higelmire

    "Games that are patched post release because people found it too difficult, or buggy, or just didn't like it. It removes the confidence of designers"
    It also gives a green-light to developers to release bug-ridden, barely-functioning messes like Skyrim because they knew they could just patch it up later.  

    "The major area of danger here is that it inevitably panders to the majority"

    I would say they pander not necessarily to the majority, but just the group that's complaining the loudest.  Of the people I knew that played the new Massive Effect, nearly everyone said that while they were disappointed in the ending(s), while only one seemed completely disgusted by it.  Guess who was the one who actually made complaints online?

    User feedback is always a good thing, but gamers have developed such a bloated sense of entitlement that it is starting to affect the overall direction a game takes, and it's a bit sad to see the vision of a developer get squandered to appease those that shout the loudest.  

  • Guest

    So if you could build Mario - you think could make it better? Hmmm. I question whether that is actually true. There is a reason why you do not get to shoot fireballs the whole game - you would get bored of it.

    The true geniuses of game design are masters of capturing your attention with the unexpected and rewarding adventurous behavior. An example of a pitfall of user-controlled design is when a player discovers an exploit that provides an unfair game advantage. While the discovery is entertaining for a short while, they inevitably walk away because the game no longer provides any challenge to continue.

    However, I do feel like this service could be a valuable tool for testing new mechanics with a broader audience. 

  • Krzystoff

    we love SMB (I don't mean the Philippine beer) because it was an original and wildly different form of escapism to other platformers around at the time.  subsequent versions did little to alter the feel and the simplistic kitchy graphics and sound are part of it's nostalgic appeal today.  
    none of these factors are going to apply to the next COD, Assasin's Creed, Halo or Forza.  this is a great initiative and a win for the gaming industry and gamers alike, it makes games more about what we want them to be, whilst still allowing complete creative control from the designers.  bring it on!