An App That Turns Any Surface Into An iPhone Keyboard

An astonishing app uses an iPhone’s accelerometer to sense the location of a tap on any surface and translate it into typed letters on a keyboard.

We have virtual keyboards, clip-on keyboards, magnetic keyboards and even laser keyboards. But what if your keyboard wasn’t a keyboard at all?

Florian Kräutli has developed an ingeniously simple alternative called the Vibrative Virtual Keyboard. By placing an iPhone on any surface, that surface becomes a keyboard. Technically, the phone’s accelerometer is measuring vibrations on that surface. Kräutli’s software maps those vibrations to a point of origin on the table. And when the phone can “see” where you’re tapping, you can have a QWERTY keyboard on any tabletop.

“I wondered how a normal surface might become interactive, without using an elaborate combination of projectors and cameras, as Microsoft is currently doing in their research,” Kräutli explains. “There are already tons of sensors in current smartphones, so I thought there must be more that we can do with them.”

Kräutli was right. By using an Apple-approved iPhone app called SensorMonitor, he can access the raw sensor outputs of an iPhone via a network connection. The software he coded analyzes this sensor output on a networked MacBook. All the user needs to do is train a new surface—tap a few points and let the software know what letter those taps are supposed to be—and Kräutli’s software will number-crunch the positions for the rest of the keys. A user can then save this surface so the software won’t need a calibration for it again.

So how could this possibly be accurate? Machine learning is a powerful tool used successfully in many industries, but truth be told, the measurements still aren’t 100% clear. Instead of achieving flawless data, Kräutli has cleverly designed the software around this shortcoming.

“The important bit is that the software needs to deal with the fact that this recognition is not perfect,” Kräutli explains. “Therefore it also uses a kind of spell checker.”

Kräutli invisibly autocorrects typing, much like Apple’s own spell checker fixes mistakes, though at a deeper level of the application. Unfortunately, this design breakthrough won’t carry over very well to another killer application—gaming.

“When you play a game, you want every interaction to be recognised correctly,” he says. “Typically, you also want this when typing, but because the software ‘knows’ that you are writing with a certain vocabulary, it can correct errors more easily.”

For the same reasons, music creation is a no-go as well, since a computer could hardly predict your next move (unless you wanted every song to sound the same). Even still, Kräutli’s creation is a remarkable statement about the future of user interfaces, where conceivably, every surface becomes a conduit for digital input. But sadly, the OS X software isn’t available for download at this time.

See more of Kräutli’s work here.

[Hat tip: Designboom]

Add New Comment

28 Comments

  • Alex - My Love Story Photograp

    Cool idea but say you on a train...... the vibrations around you would affect it.

  • Vongsawat Wongkijjalerd

    Isn't this somewhat old news? I remember it being mentioned as an example in Avi Rubin's TedTalk back in Oct 2011 or so

    http://www.ted.com/talks/avi_r...
    (around the 13th minute or so for those interested)

    The mention was relatively low on visual content but its basically talking about the same thing, of how using the current iPhone accelerometer coupled with machine learning software someone could listen in on your keystrokes just by placing their iPhone on the same table as the one you are working on (Rubin's talk was more focused on security concerns)

  • J

    This doesn't make any sense. The "inventor" doesn't seem to have a grasp on how the accelerometer works. I could come up with much cooler imaginary tech - so as long as I make a "concept" video, I can be featured on Fast.Co too?

  • Vongsawat Wongkijjalerd

    This tech was mentioned in a TedTalk (Avi Rubin's) about a year ago and mentioned only using the the iPhone's accelerometer and a machine learning software to listen in on someone typing on a keyboard on the same table (which seems a lot further away than what is being shown above) 60-80% accuracy, enough to be a security concern anyways (which was the topic of the TedTalk)

  • gbacoder


    CHRISTIAN K. NORDTØMME:-
    "I'm blown away by the comments, though! It must be sad not to be able to enjoy something like this, see the opportunities, and let it trigger your imgaination and creativity, without looking for drawbacks or finding something to complain about? Please look inside yourself for some happiness, and see if you can't also find room in your life to give Mr. Kräutli and Mr. Wilson a compliment. I believe they have deserved it."I saw BongBong's comment's, and just thought, boy, you just don't get it. Won't be hiring you! lol Definitely don't allow BongBong in a brainstorming room. Thankfully much better comments have since come in!BongBong's comment :-"Only an engineer/hacker could come up with such a solution looking for a problem... but I commend him on his ingenuity.I recommend an Apple Bluetooth keyboard instead."Complete lack of imagination. And as a hacker / coder - who also appreciates the importance of design; I could not believe how snobish this comment was towards engineers, who have, done major innovations with massively creative insights and imagination in computers. Without them it would not be possible. And there is still plenty of room for them to drive it more forward I feel. Computing / software is a great place to be for the creative engineer right now. Ideas and this sort of thinking that we see here, and trying many mad things like this guy did, are what we need. It is an attitude that has achieved so much. 

  • Nick W

    I've heard about this concept in the past using optics and projection, but this way using audio makes it way cooler. Does make you think about the capabilities of spy equipment though....

  • hypnotoad72

    This is truly impressive.  The accelerometer, based on distance and angle, to calculate which letter is desired.

    This might eliminate the need for $100 bluetooth gadgets, which eat up the iPad's battery life while requiring batteries of their own.  Just print out a template keyboard and you're done. 

    True, the phone/tablet has to be on a stable, hard surface with limited interruption from ambient sources of vibration, but that's hardly a nitpick.  It's part of the laws of physics. if/when honed, this could be a wireless keyboard killer, but as it stands it sounds pragmatic already.  Apart from games and, arguably, being able to type 80 words per minute, which few people can do.  :)

  • lytnus

    Great idea; how exactly does it work though? with only one accelerometer, I would have thought that the device can only work out how far away a tap occurs. However, multiple keyboard keys would be the same distance from the accelerometer.

  • Amy

    I so much want this to be an app!

    And forget using a bluetooth keyboard, if I can get the input to be on my iPhone (instead of the phone itself just measuring the vibrations for my MacBook), I would download it in an instant. Imagine just taking it anywhere and plopping it down and being able to notes.

    Furthermore, could this be expanded to recognize vibrations for, say, a piano layout instead of just a keyboard layout? Then input could be read and played back as hitting musical notes on a piano keyboard. Or even beats tapped out could be registered for a drum.

  • BongBong

    Only an engineer/hacker could come up with such a solution looking for a problem... but I commend him on his ingenuity.

    I recommend an Apple Bluetooth keyboard instead.

  • hypnotoad72

     For now, such a keyboard - warts and all - is probably more practical.  And I'm sure third party BT keyboard creators might offer similar or better functionality for a lower price...

    But an engineer's job is to, well, engineer, and this ingenuity - I hope - gets patented by the creator first.

  • aiham dib

    the mobile phone can apply two techniques:
    1- the sonic  detection 
    2- the infra-red detection
    both by making a kind of cross zone generated by in-planted sensors and activators around the mine  micro laser projector that project the keyboard on any surface.   all those are aligned on the side of the mobile.

    the importance of the infrared over the sonic  .or the vibration thing mentioned above 
    is that it is accurate since the calculations   are done  based on the blocking of light by the finger on certain area- letter tab-  so neither noise nor random vibrations will interfere .
    if Apple not take this ..i gonna offer to her best enemies  : O )
    Aiham Dib
    beology@gmail.com

  • Mark Wolgemuth

    It occurred to me to try this out a while back after a BlackHat conference. It's an analogue to identifying electromagnetic patterns of displays from a separate device, but with vibration as the information conduit. The same methodology that makes the "keyboard" idea possible can be used to snoop on someone typing on the same table as you-- just lay the phone near enough.

    Of course, it needs some very smart training to sort out noise (proximity will help out here). Maybe future devices will have multiple accelerometers to better identify which direction the vibration is expected from. As in, the phone will pick the movement up as Z-axis motion-- but if it had multiple sensors, it could detect differences in intensity in the Z axis from one side of phone to other, and thus guess which side the vibration is coming from.

    - Mark Wolgemuth ( mark at rescuetime dot com / at node dot to )

  • Christian K. Nordtømme

    Fantastic! I am so glad to know I live in a world alongside people like Florian Kräutli. I believe his is the kind of thinking that enabled people like Steve Jobs (and Jony Ive, et al.) to develop the iPhone and its siblings in the first place.

    I'm blown away by the comments, though! It must be sad not to be able to enjoy something like this, see the opportunities, and let it trigger your imgaination and creativity, without looking for drawbacks or finding something to complain about? Please look inside yourself for some happiness, and see if you can't also find room in your life to give Mr. Kräutli and Mr. Wilson a compliment. I believe they have deserved it.