Mesh Experience, a project by three German interaction design students, visualizes guitar licks in real time.

It’s a good deal more sophisticated than your average rock concert light show--the constellations of circles are determined by the notes being played, and by how the guitarist is playing them.

"A big thing for us was always legibility," Florian Friesinger, one of the creators, says, "but at the same time we were curious about integrating as much informations as we could."

They intended for the system to be used with live performances.

An infrared camera tracks the guitarist’s movement around stage; two projectors handle visualization duty.

"Just imagine someone like Jack White on guitar, playing and creating visual feedback with a visualization like this," Friesinger says.


A Rig That Turns Guitar Licks Into Light Constellations, In Real Time

Mesh Experience is one part rock concert light show, one part data viz.

Part of what made Guitar Hero so accessible was that you could see what you were playing in addition to hearing it. Okay, the fact that you only had five chunky plastic buttons to worry about instead of some 140-odd notes probably helped, too. But getting that visual feedback was essential for people who weren’t used to following along by ear.

Mesh Experience, a project by three interaction design students in Germany, takes on the ambitious task of visualizing the output of a real guitar, in real time. And even if it doesn’t help newcomers learn how to play—though I suspect it might—it certainly succeeds at turning more expert chops into a dazzling visual display.

For the rig, Florian Friesinger and his collaborators, Fabian Fischer and Sven Stumm, hooked a guitar up to a computer running VVVV, a robust piece of graphics software. As an infrared camera tracks the guitarist, two projectors mounted overhead surround him in a flurry of colored circles.

The students, who created the project for an audio-visual design course they were auditing at the University of Applied Sciences in Schwäbisch Gmünd, were aiming to make something that would serve as a visual supplement for live performances. But instead of just creating a crowd-pleasing light show, the team pursued something a bit more sophisticated—a system that actually keeps up with the performer, note for note.

The key of the song determines the color palette—"synaesthesia always kept in mind," Friesinger adds. Notes get assigned to a position around the guitarist according to their pitch, and the distance from the performer depends on the note’s velocity. Bending a note results in its circle getting a little twist, and notes that are tonally similar get connected on the fly with lines of light. The result is a floor full of stars, constantly shifting, connecting, exploding into life and fading away.

"A big thing for us was always legibility," Friesinger says, "but at the same time we were curious about integrating as much information as we could." It’s data viz as much as ambiance. But even without focusing on individual circles and clusters, the general appearance of the light show is directly affected by the style of music producing it—and the style in which that music is being played. The slow jam at the beginning of the clip above results in a sort of chilled-out, disco-ball effect, while the solo toward the end produces a dense spiderweb of light.

Currently, Friesinger says, the system works best if you optimize the software for the song’s particular style beforehand. But even in this early stage, it’s easy to see how it could turn into something a bit more engaging than the standard rock-concert light show. "Just imagine someone like Jack White on guitar, playing and creating visual feedback with a visualization like this," Friesinger says. "Awesome."

But of course, there’s always the issue of visibility. Getting a line of sight to a musician’s head from a packed show floor is already hard enough. Soon you might need to get an angle on their feet.

See more on the project page.

[Hat tip: Under Consideration]

Add New Comment