Last year, Microsoft Research revealed Illumiroom—which used projectors to stretch the image on your TV to take over a whole wall. Now comes the sequel. Called RoomAlive, this new system can turn every surface in a room—from the floor to the couch cushions to your own skin—into a glowing screen that reacts to your movement. It’s as if your whole analog world has been digitized. And that digital world can see where you’re looking, pointing, and touching, and adapt itself accordingly.
Microsoft hasn’t yet built Star Trek’s fabled Holodeck, but as project researcher Brett Jones quips, "we joke that until we discover matter to energy conversion, this might be the closest we can get."
In the minds of its creators, RoomAlive is the antithesis of the Oculus Rift and other virtual reality headsets that isolate you, and pull your attention from the people and world around you. Instead, it's a means to keep play physical and social as we enter an era where virtual reality is plastering pixels straight on our eyes.
But let’s back up for a moment. What is RoomAlive made of, exactly?
RoomAlive is a hive of really smart projectors.
The core unit of RoomAlive is called a node, and each node has three parts: A projector, a Kinect camera for depth sensing, and a computer. The node is installed to the ceiling of a room, where it creates a 3-D map of what it sees through the Kinect. Then, it can "projection map" the space, applying a coat of digital paint to the room.
Using as few as three of these nodes, Microsoft Research can map the average living room with a RoomAlive skin. Projection mapping is a somewhat well-trodden technology. What makes RoomAlive enticing is that it can effortlessly scale without limit. As nodes are added, they recognize one another and calibrate themselves to work together automatically. You could hypothetically map a whole house in RoomAlive.
RoomAlive also has all of the real-time body-tracking cleverness of the Kinect.
This is where things get really wild. RoomAlive can take your position into account all of the time, which means that any surface can become a makeshift touch screen because the projectors see what you’re touching. RoomAlive can also project parallax images images that dynamically shift in response to your precise perspective, allowing you to peek over and around the images. With parallax technology, 2-D projections of spikes sticking out of a wall can actually appear 3-D. RoomAlive can even create the illusion of an object floating in three-dimensional space by projecting at an exact spot on you wall or floor with just the right scale and shadows.
"It’s not completely like a 3-D stereo hologram, but it’s a very compelling illusion," Benko tells Co.Design. "If you covered one eye, the experience is exactly what you see with a camera."
But RoomAlive's tricks of perspective don't stop with just one person. Microsoft researchers expanded the project through technology they call Mano-a-mano to offer two people in the same room a unique perspective. "If we’re looking at each other, face-to-face, most of the surfaces I see are things you don’t see," Benko explains. "Things on my body, things behind you—we’re using this trick." This trick manifested in a Street Fighter-esque fighting game, in which two people fired energy beams at one another, each seeing their shots fire in an illusion of 3-D.
Video games are a natural extension for RoomAlive, but not the only extension.
It was clear, when talking to multiple researchers on the project, that gaming, and ideas of play, were what immediately came to mind with RoomAlive. Several ideas—ones that you don't see here—were prototyped. Even more were discussed.
"You can imagine playing a tower defense game in your living room. You run over, grab the pillows and build a pillow fort that’s your base," Jones says. "A lot of this is about bringing back the physical world, and we think that’s important with kids playing games."
Another concept proposed was a Dora the Explorer style of game, where a child would don a map and a backpack, and that backpack would magically glow and talk to its wearer on their adventure (a very cool idea, if you disregard the fact that you can't see your own backpack!). But of course, there is plenty of less G-rated gameplay potential RoomAlive, too. The researchers demoed a system for projecting bullet wounds onto players that you could imagine in a future Call of Duty.
"If you had gunfire, and the bullet hit someone’s body, that wound would actually follow your body from one projector to the next," explains researcher Raj Sodhi. "Each projector takes over and makes that transition completely seamless."
"Once, Raj had the blood dialed up way too high, so as you walked around, the blood would splatter on the floor and you’d leave a trail of blood in the room," Jones says.
Video games are the most obvious use cases, but beyond that application, the research team is exploring RoomAlive's potential for the business world, like advanced telepresence and augmenting conference room meetings with digital information.
With its prototype nodes costing $500 to $800 a pop to assemble, there are no plans to bring RoomAlive to market yet—though maybe you'll see it in your next-next-next Xbox?—but Microsoft Research is already exploring what it can do with RoomAlive next.
"We’re trying to push the boundaries of what is possible and see the potential of what’s possible, not cost reduce the particulars of one device," explains senior researcher Hrvoje Benko. "What’s interesting to us is merging the real and virtual worlds, in a contextually relevant sense, so the virtual things behave in relationship to real things," Becko continues. "I think we have a lot of work to do to make that a reality—which is good, it keeps us busy."