A Mind-Blowing UI That Could Finally Make Group Work Intuitive

This experimental app from the MIT Media Lab combines gestural interfaces, augmented reality, collaborative workspaces, and general sci-fi amazingness.

We’ve seen “magic-window” augmented reality interfaces, Minority Report-style gestural interfaces, and computer-vision-powered collaborative display interfaces. But what about an iPad app that combines all three? That would be T(ether), an experimental design from the MIT Media Lab.


Creators Matthew Blackshaw, Dávid Lakatos, Hiroshi Ishii, and Ken Perlin call T(ether) “a tool for spatial expression” that “acts as a window affording users a perspective view of three-dimensional data through tracking of head position and orientation.” In English, that means you can hold the iPad up with one hand to reveal a shared virtual space that you can manipulate with the other hand using a special glove. You can draw 3-D shapes, rotate, pinch, zoom, and spin them as if they’re real–but unlike most augmented reality apps, which silo the “world” into the screen of a single user, T(ether)’s is collaborative: Two people peering through their iPads at a grid of virtual cubes can both reach in independently and fiddle with them at the same time. Just like in real life.

T(ether) opens up mind-boggling possibilities for creating interactive digital art or exploring novel scientific visualizations. But how effectively can you really interact with a virtual world when you have one hand tied behind your back (that iPad may get awfully heavy after a few minutes)? “We designed the system without head-mounted displays or goggles to preserve the natural human-to-human communication,” Lakatos tells Co.Design. “The trade-off is that we lose one hand for the iPad, but in exchange the users can interact with each other naturally–for example, they can point in space somewhere, and the other user will be able to reference what he is pointing at. This isn’t possible with the current head-mounted display technologies.”

I also asked Lakatos what T(ether) offers the user that plain-old touch-screen manipulation doesn’t–in other words, why do you need the glove if you’re still just pinching and zooming pseudo-3-D objects that are only visible on a flat screen? “We tested it with a bunch of people, and they feel a kinesthetic connection between what they see on the screen and where their hand is,” Lakatos says. “On a touch screen alone you wouldn’t do much better than sitting at your desk with a mouse. The gloves give you the literally free-hand spatial positioning. We think this is super powerful.”

Okay, sounds good. But what would anyone actually use T(ether) for in the real world? Animators and filmmakers of the future will love it, says Lakatos. “Today, animators work at their desk individually and merge their designs through a version control system to a final scene. The iteration cycle between constructed scenes can take hours. With T(ether), animators can collaborate on virtual scenes in real time, while the director would observe the scene and provide feedback on-the-fly to them. These studios already have enormous rooms equipped with the motion capture systems–the same ones we use in our project–so this would be a very natural application. But architects, industrial designers, and everyone who designs 3-D systems on a computer could benefit from the viewing and manipulation capabilities of T(ether).”

I don’t know if I’d want to use it for more than a few minutes at a time, but I definitely want to use T(ether)–if only to feel like I’m living in a sci-fi movie. Let’s hope that the team can refine it and bring it to the iTunes App Store sooner rather than later.

About the author

John Pavlus is a writer and filmmaker focusing on science, tech, and design topics. His writing has appeared in Wired, New York, Scientific American, Technology Review, BBC Future, and other outlets.