A Smartwatch That Recognizes What You Touch

Researchers have hacked a stock smartwatch to usher in a future of UI we could only otherwise dream about.

Smartwatches are still pretty dumb. One of the biggest reasons is that they can’t understand what you’re doing at any given moment, and offer you assistance in the task.


But researchers from CMU have debuted a fascinating proof-of-concept capable of transforming any off-the-shelf smartwatch into a high-fidelity gesture recognition tool that doesn’t just recognize subtle finger motions like flicks and waves. It’s capable of identifying whatever you’re holding in your hand–from hand saws to electric toothbrushes.

“Today’s approaches for detecting objects center around sticking RFID tags or QR codes,” says PhD student Gierad Laput, who worked on the project alongside fellow student Robert Xiao and advisor Chris Harrison. “But the holy grail for object detection is being able to detect an object at the moment of touch.”

Called ViBand, it’s actually just an LG G smartwatch with a souped-up accelerometer. Most accelerometers capture movement at 100 Hz (or 100 times a second). But just a bit of new code can overclock the accelerometer up to 4000 Hz (reading movement 4,000 times a second). In practice, this allows a far richer understanding of the slightest minutae in movement, especially in the realms of micro-vibrations that pass up your arm. This allows the ViBand to recognize if you’re rubbing two fingers together, or instantly discern the buzz of a Dremel tool in your hand from a power screwdriver. ViBand can even understand how you’re holding an object, so long as it’s on, because these micro-vibrations can differ by positioning.

In the video above, researchers gave several use cases–my favorite of which is a tuning app that feels the vibrations of your guitar and can actually tell you if a note is flat or sharp–and it’s easy to imagine how many apps could take advantage of contextual knowledge of what you’re touching at any given moment.

But there are catches. For an object to be recognized, it needs to vibrate somehow. That doesn’t mean it needs to be electronic, but it’s gotta shake. Secondly, the examples you see here are based upon software calibrated to each individual user. Machine learning might be able to construct software that understands what it looks like when anyone is holding a stick blender, but until then, it’s person-to-person training. Finally, overclocking these sensors takes a lot of extra power, and smartwatches have tiny batteries that deplete quickly. To make this power efficient, new chips would have to be designed with this hyper-tracking goal in mind.

This UI is no less impressive with those caveats in mind. CMU’s firmware update could turn watches including Moto 360, Samsung Gear 2, and Samsung Gear Fit into object-identifying wristbands overnight. And there’s no reason that Apple couldn’t enable the same thing on an Apple Watch (the company’s software is just too locked down for CMU to attempt it). It’s rare that the future feels so close that you might literally touch it today.


[All Images: via Gierad Laput]

About the author

Mark Wilson is a senior writer at Fast Company. He started, a simple way to give back every day.