At age six, Greg Madison realized his potential. While his dyslexia made some school lessons difficult, it gave him what he equates to “perfect pitch” for magic tricks. Just by watching a magician at work, he could deconstruct the illusion. “I never saw the magic. I only saw the trick,” he says. By age 12, he’d learn the other side of the equation–how to work a crowd–and was sharing the stage with some of the most famous magicians in the world.
Today, Madison admits that his sleight of hand is a bit out of practice. Instead, he applies his unique perspective to a new topic: User interface. At Unity, a company that makes the fundamental technologies behind games like Pokémon Go and Google’s upcoming Daydream virtual reality headsets, Madison works in the Labs division, where he experiments with the virtual and augmented interfaces of tomorrow.
While Madison is certainly a futurist, who imagines a mixed reality world where pixels float all around us, he never forgets that, ultimately, the audience must buy into the magic. With that in mind, he shared some of the biggest misconceptions about where our Oculus Rifts and Hololens systems will take us next.
Companies like GoPro and Facebook are investing untold dollars into 360-degree video cameras, and Google recently hired Justin Lin, director of three The Fast and the Furious films, to shoot a high-octane short that allows you to look around a live-action monster film. And why not? On paper, the idea sounds amazing: “Imagine if you could be on the beaches of Normandy inside Saving Private Ryan!” But truth be told, they’re inherently difficult to watch–where do you look?–and there’s still no must-see movie that’s causing untold masses to grab virtual reality headsets.
“Because it’s possible to have 360-degree movies, people think we need to make 360-degree movies. But I think that’s wrong,” says Madison.
He points to the tradition of live theater, concerts, and Hollywood movies themselves, as proof that an audience wants to look one direction. (Even theater in the round has a stage! Theater is, only very rarely, literally around you.) “If you’re looking at how people will really watch a movie, in reality, they’re not looking everywhere. It’s nonsense,” he says. “You are looking forward, and sometimes left and right, but not behind you. It’s really a waste of pixels.”
As a result, Madison believes the future of VR movies is something more like 180-degree video, with the best content filling your field of view and peripheral vision, essentially a wide-screen made a touch wider. Instead, he thinks the real revolution will happen inside the Z plane–the depth plane of a film–rather than the wider X/Y axis.
Much like how the Lytro camera uses light field photography to allow an image to be refocused at various depths within a frame (thereby allowing someone to look at a recorded image much like they would the real world), Madison believes that technology will evolve to allow the viewer to focus on different parts of a frame like they are half-directing their own movie. And in fact, there is research working on such technology at the current moment. You’d look at the screen, focus your eyes on an object in the background, and it could literally sharpen before you, just like it would in real life.
I’ve tried working all of an hour in virtual reality, and I found the experience exhausting. Madison isn’t surprised. He never thinks we’ll work our day jobs in VR. For all of the endless possibilities of virtual reality–do spreadsheets while summiting Everest!–you’re still stuck inside a machine that puts blinders on the real world, and you’re still inside your own body, which needs you to be cognizant of simple ergonomics.
So when designing the Unity VR platform Carte Blanche–which allows you to build cities from miniature models, and will be released next year–management approached him about building it on the HTC Vive, a system that could allow the user to walk around an entire room and build things at a perceived giant scale. Madison agreed to use the Vive, but he had a condition: The experience would be seated anyway.
“[Ergonomics are] really important. Of course you can stand up. But when I worked at an IT company, you have to observe people who use a machine for an entire day at a task. It was the same thing for me. If I need to spend eight hours [in VR], how can I have a better experience?” he recounts. “I have to bring the best of the reality to put into this experience, to be seated, have your hands at the edge of a table, to keep you from being fatigued. And thanks to that, you get a new way to interact: At the end of this table, you can have a keyboard, a slider, whatever you want!”
By crafting VR around traditional workplace ergonomics–which, however unhealthy sitting at a desk may be, has proven to be comfortable for most people–rather than the endless possibilities of locomotion, his version of virtual reality was inherently more comfortable. At the very least, it gave the user the option to sit, and a spot to rest their arms, for longer stints inside the machine.
That said, he thinks ergonomics are only one issue with virtual reality as a container for your eight-hour workday. The other is that any way you slice it, it’s not reality. It’s not like reading your Kindle. It’s like being inside your Kindle, unable to escape, which cuts you off from a necessary, peripheral view of the world that our fundamentally human perception requires.
“I think that VR is really just a step to reach alternative reality,” says Madison, who uses “augmented reality” and “alternative reality” interchangeably. And it’s AR where we’ll waste away our lives–from work hours to our recreational ones.
Oculus is getting ready to debut its landmark controllers, which capture the subtleties of the human hand. Other startups have toyed with force feedback suits, gloves, and even omnidirectional treadmills to give humans their full range of motion in the virtual world. Why? Everyone wants to build the equivalent of the mouse and keyboard for virtual reality.
“For me, laziness is the better way to predict what will happen in the future,” says Madison. Playing off the same principles that limit VR’s usefulness during an eight-hour work day, he just doesn’t imagine anyone will want core interfaces that take that much physical energy. “We haven’t replaced the mouse and keyboard in 30 years, because with minimal effort, you can do lots of things. In VR, we need to think, it’s not because we can, but need to do that.”
As a result of human laziness, Madison is still bullish on systems like Microsoft’s Kinect (even though Kinect has had difficulty taking off on the Xbox), which can track you in your environment without requiring special controllers and wearable electronics.
He also designs as much laziness as possible into his VR interfaces. Obviously, people expect to be able to pick up and lift virtual objects in VR. That’s fine. He lets them. But in his demo Carte Blanche, he enables alternative options. You can gaze at an object long enough that you can then move it with your eyes (and neck), or you can stare and say something like “copy object.” “For any action, you always have several ways to do it,” says Madison. “I count on the user to find his own way to use interfaces. What I’m trying to do is offer him a lot of possibilities to find his natural way to move and feel his [own] reality.”
I’m reminded of a friend who used to play Wii bowling not by standing in front of the television and miming bowling with the rest of us, but by leaning back on the couch and flicking her wrist to get a strike. It blew the illusion that any of us were actually bowling, sure, but it was also the exact sort of gesture simplification you’d desire if you worked inside a virtual bowling alley. And she never had to put down her beer to play.
Along the same lines, Madison has made it a personal rule to never design virtual interfaces that require two hands, unlike apps like Google’s superb Tiltbrush, which rely on you to hold a palette in the left hand and a brush in your right.
“For me, [the goal of] VR is for everyone to do what you want to do and live as you want to,” he says. “It’s not [to recreate] the same issues as you have in reality.”