How Flutter Hopes To Kill Gesture Control’s Point-And-Click Crutch

A promising software startup asks: If gesture-based interfaces like the Xbox Kinect are the future, why are we still using them like mice?

How Flutter Hopes To Kill Gesture Control’s Point-And-Click Crutch

Down to the familiar white arrow cursor, the gesture-control feature found in high-end Samsung TVs works just like a mouse. Moving the cursor to choose menu items requires a wave of the arm, and a flap of the hand stands in for a click. Xbox Kinect’s gesture-based user interface uses a similar pointer scheme and hand-hover click.


A startup gesture-controlled app called Flutter, however, has no vestigial point-and-click paradigm. To pause a song in iTunes, you simply hold your hand flat in front of you like a stop sign. To skip forward, you point to the right. To skip back, point left.

The app’s creators argue that their commands, though not yet as complex, are more intuitive. How would they know? Because they paid 1,000 workers on Amazon’s crowdsourced labor marketplace Mechanical Turk to send in photos of the gestures they’d make during conversation to indicate simple software commands.

“We as humans already have body language,” Flutter cofounder Mehul Nariyawala tells Co.Design. “If we teach the computer to understand it, then it becomes natural.”

Nariyawala compares Flutter’s treatment of gesture control to Apple’s treatment of touch technology. Before the iPhone, generations of touch devices were stuck in a point-and-click paradigm that often required a stylus. Apple, he says, was the first company to build a user interface specifically for the intuition of touch. Before the iPhone, devices came with a user manual. It was the user’s responsibility to learn the machine. After the iPhone, devices were built to respond to human intuition.

Flutter has the potential to inspire the same paradigm shift for gesture control.

The startup’s app for Mac and Windows uses gesture controls to pause, play, and skip forward and backward in programs such as iTunes and Spotify. It’s a simple use case for a technology that took three years to develop.


Using a simple webcam rather than separate hardware, the software is designed to calibrate for any background situation–-even in diverse light, what Flutter cofounder and CEO Navneet Dalal calls “the kryptonite of computer vision.” It’s a feature that makes good use of Dalal’s PhD in computer vision, imaging, and robotics.

But the team didn’t spend three years working through these problems just to improve iTunes navigation. Flutter’s free app is meant mostly as a way to prove a point: That the startup’s approach to a gesture-based user interfaces is superior to that of the translated point-and-click mouse approach.

So far, it’s doing a good job. Flutter has recorded more than 4 million gesture controls in its first four months in the Mac App Store. The startup’s hope is this popularity will eventually convince hardware manufactures to pre-install Flutter’s technology on their devices. By that point, the UI could become so intuitive that it doesn’t require any effort on your part to pick up and quickly learn.

Walking away from your computer while in a coffee shop could, for instance, be an implicit gesture command to lock your computer. Returning could be the gesture that unlocks it. Picking up the phone could be a command to pause any videos or audio playing on your computer, and putting it down again could be a command to play where it left off.

“We can think of [the webcam] as the eye of the machine,” Dalal says, “and that eye should understand what a human does.”

[Image: Hand via YuriyZhuravov on Shutterstock]

About the author

Sarah Kessler is a senior writer at Fast Company, where she writes about the on-demand/gig/sharing "economies" and the future of work.