In Microsoft’s HoloLens, A Glimpse At The Future Of Interaction Design

Playing around with Microsoft’s mixed reality headset, we discovered some clever interactions that could become ubiquitous.

After taking the HoloLens “mixed-reality” headset around the country to demo to eager developers, Microsoft is opening the doors to a much richer experience: a showroom, secreted on the fifth floor of Microsoft’s Fifth Avenue flagship store in New York, where developers can dabble with three versions of HoloLens in action. If you’re a developer interested in virtual reality, augmented reality, and interaction design in general, you should sign up now.


The three demos are, all at once, simple, short, and even a little primitive—and at the same time, totally eye opening. The three depict core use-cases of a holographic headset: using it to create virtual objects, à la Photoshop for the 3-D world; using it to display a holographic presentation, à la Powerpoint; and using it to play a holographic game—à la Xbox.

A short refresher on HoloLens: it’s a surprisingly comfortable, ergonomic, and well-designed headset that slips on with just a few quick adjustments. It has eye tracking and gesture sensing; it also projects hologram-like images onto the visor. In short, you can wander around the real world, seeing holograms that range from toolboxes to chat windows to spaceships placed all around you, albeit through a small portal.

Gaming is perhaps the most immediately fun of the three demos. You see virtual alien robots busting through the walls, and to evade attack, you bob and weave your way around the room. To fire your own laser canon, you use an Xbox controller.

Other demos are more consequential. In the presentation demo, you see a hologram of a watch, which explodes into many layers, exposing its constituent parts. In the creation demo, you can wander around a cartoonish reef. Gaze at a fish, and it can be cut-and-pasted, scattering piscine clones into an ocean around you.

Okay, Great! What’s All This For?
Unless you’ve really been immersed in the whole emerging augmented reality/virtual reality space, you’re probably wondering what the big deal is. You might even be thinking Microsoft has HoloLens, Facebook has Oculus, and Google’s funding the Magic Leap—so that means all these guys are gonna duke it out with each other over who gets the right to put a computer on our faces?

Yes and no. The Oculus is a far more all-consuming device than the HoloLens. Strap on a full-immersion headset like the Oculus and the experience is—to use current virtual reality parlance—extremely present. When an arrow flies at you in VR land, you can’t help but duck because holy crap there’s an arrow flying at you.


Yet Microsoft, like a would-be presidential nominee enmeshed in a chaotic primary, is stridently trying to point out some subtle, perhaps decisive differences between HoloLens and Oculus. And they have a point. Oculus and other full-immersion headsets are solitary, personal experiences. You strap it on, and you’re in another world, without much sense about what the people around you are doing. Full immersion makes a lot of sense for anything you do on your computer solitary: Playing Call of Duty or watching a movie.

For more social applications—giving a presentation or showing someone how to fix a busted faucet—you have to see what someone else is looking at. You have to share an environment in real time, with no lag. Microsoft argues that while HoloLens isn’t as immersive as Oculus, it’s also not meant to be. HoloLens is additive—reality plus. Medical students could be guided through a medical procedure, while their professor monitors exactly what they’re seeing. (Case Western is working on this now.) Product teams could build a jet engine by manipulating a virtual replica, giving detailed feedback and minimizing the need for multiple, iterative prototypes. (Adobe is working on this now.)

The Main Problems You Notice
As you’re reading all these words about disembodied holograms floating before your eyes, it is a bit hard to convey how cool all this might be one day. The situation is similar to that when the first graphical user interfaces were introduced. Plenty of computer geeks at that time understood that presenting a computer program graphically would be nicer to use. Fewer of them understood why GUI’s might be transformative, because that had less to do with capabilities, than in the way transformed processes that were hard to understand into very simple graphics. The GUI needed Steve Jobs, who saw the possibility, and sweated every single detail until the Macintosh GUI felt amazing.

I was thinking of this as I explored that virtual product demo, fussing with machinery of an expensive watch. As a sensory experience, HoloLens still feels like an early version of what it should become. The field of vision is restricted; you’re not seeing holograms laid out all across your full field of vision. Instead they hover across a rather small windowpane floating in front of your eyes. The resolution of the display itself is still low, meaning that the visuals are less Princess Leia begging for help, more Minecraft.

The hardware exposes nascent, if promising capability, but the most interesting facet of HoloLens is the interaction design. That’s where the bets are being made. Microsoft has already conducted hundreds of hours of user testing to figure our just how we might interact in this new hybrid reality. They’ve already come up with some very clever interactions, like making your gaze function as a mouse pointer: there’s a dot floating at the centerpoint of your vision, and to pick up or interact with any virtual object, you first must gaze directly at it, aligning a dot. To pick something up or otherwise make it do something, you “finger click” it, holding up your hand roughly in the shape of making a “Number 1” gesture, then flicking your finger down. The motion tracking cameras on the front of the HoloLens capture your hand movement and turn intention into interaction.


There’s a subtle cleverness in that action. The main problem with any interface that deals with mixed reality is that it’s all too easy to accidentally do something that you didn’t intend. Say, for example, that the designers had made the click gesture into a mere hand wave. If that were the case, every incident gesture you made would be liable to trigger an accidental click. The tension in creating the right gesture is in inventing something like the finger click which is intuitive once you’d taught—but also unlikely to ever happen by accident.

Already, Microsoft has suggested other gestures that may reach ubiquity. To summon a menu, hold out a fist, with the back of your hand facing down, and quickly unfurl your fingers, almost like you’re revealing a hidden coin in the palm of your hand. That gesture makes a contextual menu bloom, like summoning a list of other people in a video chat.

All these potential gestures will evolve. They will also need to be taught. Just as there was once a race to define the most intuitive way to use a mouse cursor, so too will there be a race to define the most intuitive way to interact with virtual reality. The gestures and interfaces are at least as important as the hardware platforms themselves. But the thing is, even developers don’t yet know how to interact in virtual reality.

Which brings us back to the fifth floor of Microsoft’s flagship store. Microsoft realizes that you can’t just hope that ingenious developers will make a new technology intuitive. You’ve got to teach them what you already know, and you have to teach them one demo at a time. From those few hundred first adopters, the hope is that millions more will come.

About the author

Cliff is director of product innovation at Fast Company, founding editor of Co.Design, and former design editor at both Fast Company and Wired.