In a past life, Mark Walsh created the lovable character Dory in Finding Nemo, and served as the supervising animator on Ratatouille.
More recently, he found himself trying to con his wife out of her lunch. He was testing some new material, the dialog of a fast-talking seagull named Gary. Gary wasn't destined for a Pixar blockbuster, but a virtual reality simulation called Gary the Gull, developed by Walsh's new company, Motional.
"As an artist, VR is the ultimate empathy tool," Walsh says. Theoretically, VR is the most immersive platform ever built. But the problem is that the pixelated characters from video games today feel even more fake in VR, because they lack basic social etiquette—they don’t tend to make proper eye contact, for example, or know if you’re laughing or crying. They’re nothing more than coded animatronics, waiting to be queued for their next scripted line of dialog.
To build truly empathetic, interactive VR characters, Walsh needed new techniques. He turned to former Pixar colleague Tom Sanocki—creator of Mater, the country bumpkin tow truck in Cars, and the lead character designer in Bungie’s recent mega-game, Destiny. Sanocki is the founder of Limitless, a company that’s building powerful character technology for VR. Limitless isn’t a standalone game engine—you can’t create physics and textures and graphics for a complete virtual world inside. Instead, Limitless sits on top of a game engine to collect data about the user and make characters' reactions more realistic, based on the subtle social dynamics of the real world.
"Mark and I are looking for, what’s the next step, what’s VR really about?" Sanocki says. "The essence of VR to us is interactivity. You can pick up a pencil in VR, and that makes the experience 10 times more immersive, just to pick something up. That’s good for storytelling, but what’s really good is, bringing interactivity to characters."
In video games today, many characters look and sound quite realistic. But they lack core social graces, and that makes them feel like strange mannequin robots. If you walk by a character in a game, they don’t momentarily meet your eyes—then look away—as a person might on the street. If you venture too close, most will just stand there, not acknowledging their own personal space. These points might sound small, but try imagining those behaviors in the real world. They’re vital to human social dynamics.
"If things are not reacting to you, like characters talk and you walk around them and they still stare in the same spot, it doesn't make people feel good. They feel alone and isolated. They feel like a ghost," Sanocki says. "If your story is about being a ghost, that’s fine! But most stories are not about ghosts."
Limitless is designed to recognize the social cues being generated by the player in VR—using motion and sound data already being collected by a Playstation or HTC Vive headset—so that guys like Walsh can create characters that respond in a way that feels real.
In Gary the Gull, Gary knows if you move too close. He knows if you nod yes, shake your head, or turn the other way. He can recognize a lot of what you say, too. And in the case of Gary—a con-man seagull who really wants your picnic basket—these reactions are a powerful means to play out a story.
"If Gary is trying to take something from you, he’ll be shocked by something behind you, saying, ‘oh my gosh!’ And by him staring over your shoulder, it’s human nature you’ll look over your shoulder at what he’s looking at," Walsh says. If you look, you lose your lunch. If you don’t, Gary can keep talking, like any con-man would. "Limitless gives me the data to know if you’re falling for it or not."
I haven’t had a chance to try Gary the Gull yet—it premiers next week at the Game Developer’s Conference in San Francisco—but it’s easy to imagine how these little bits of interaction could add up to a much richer, more realistic character. Research has found that we react to VR avatars of real people, speaking with their own voice and moving with their own motions, very comfortably. More specifically, we tend to think of them as our friends—just wearing costumes.
Gary the Gull is a relatively simple four-minute experience, one in which anything you say or do will more or less lead you through the same narrative arc. But from early hands-on experiences, Gary is also an early proof-of-concept for a new generation of VR characters—whose stories really do depend, and evolve, based on your own reactions.