Watch: A Crazy Floor Senses And Mirrors Your Every Move

What if everything in your living room had a digital shadow?

When we think about the screen, we think about it as something in front of us–a television mounted on the wall or a cellphone held at arm’s length. What if screens could be all around us, and sensing our every move?


That’s a question being explored by the Hasso Platner Institute, with support from Microsoft Research. Through a series of research projects, they’ve developed a smart floor (called Multitoe) that can sense, identify, and distinguish the presence of individual people and furniture. Then, using software and a projector placed well beneath a glass floor (called GravitySpace), they can extend the real world into a sort of subterranean digital shadow.

“We think the functionality will become integrated into regular carpets,” researchers Dominik Schmidt and Patrick Baudisch tell Co.Design, “so when you move into a new place you simply roll out a carpet.”

The floor itself uses a technology called FTIR, a network of invisible, IR-based light sensors that can measure the footprints of any physical entity. It’s so sensitive that software can actually extrapolate pressure information, which can imply the movements and poses of the human bodies standing on it. As for those digital shadows themselves, the models do have to be built ahead of time, but all of the movement animations are being rendered in real time. So several different people can automatically have their pre-built avatars sucked from a library when they walk into a room, and the experience is seamless.

“There is certainly fascination in watching the floor analyze the very little input it has, just a few pressure imprints, to reconstruct all of what you have seen the system reproduce” the team writes. “[But] a lot of the true feel of the system will be created by applications running on top of GravitySpace.”

This is an earlier demo of just Multitoe without the full GravitySpace experience running.

So far, those applications don’t really exist because there aren’t many developers working on this platform (which requires a special sensor floor and a projector hidden a floor below). But researchers have already played with some interesting foot-gesture UIs that would allow interaction through some clever toe tapping. And indeed, it’s not hard to imagine the possibilities for dance games or any other sort of full-body experience.

However, the most interesting parts of such technologies are their potential to be interoperable with existing systems. A floor like this could give a system like Kinect incredible movement fidelity, and a few real cameras placed in a room could mean that the shadow effect could serve as a high-resolution, texture-mapped mirror of the real world. Just imagine mixing GravitySpace with the 3-D projection of Illumiroom. You wouldn’t just be playing with some augmented-reality app; reality as you know it would simply be augmented.


About the author

Mark Wilson is a senior writer at Fast Company. He started, a simple way to give back every day.