Part of what made Guitar Hero so accessible was that you could see what you were playing in addition to hearing it. Okay, the fact that you only had five chunky plastic buttons to worry about instead of some 140-odd notes probably helped, too. But getting that visual feedback was essential for people who weren’t used to following along by ear.
Mesh Experience, a project by three interaction design students in Germany, takes on the ambitious task of visualizing the output of a real guitar, in real time. And even if it doesn’t help newcomers learn how to play—though I suspect it might—it certainly succeeds at turning more expert chops into a dazzling visual display.
For the rig, Florian Friesinger and his collaborators, Fabian Fischer and Sven Stumm, hooked a guitar up to a computer running VVVV, a robust piece of graphics software. As an infrared camera tracks the guitarist, two projectors mounted overhead surround him in a flurry of colored circles.
The students, who created the project for an audio-visual design course they were auditing at the University of Applied Sciences in Schwäbisch Gmünd, were aiming to make something that would serve as a visual supplement for live performances. But instead of just creating a crowd-pleasing light show, the team pursued something a bit more sophisticated—a system that actually keeps up with the performer, note for note.
The key of the song determines the color palette—"synaesthesia always kept in mind," Friesinger adds. Notes get assigned to a position around the guitarist according to their pitch, and the distance from the performer depends on the note’s velocity. Bending a note results in its circle getting a little twist, and notes that are tonally similar get connected on the fly with lines of light. The result is a floor full of stars, constantly shifting, connecting, exploding into life and fading away.
"A big thing for us was always legibility," Friesinger says, "but at the same time we were curious about integrating as much information as we could." It’s data viz as much as ambiance. But even without focusing on individual circles and clusters, the general appearance of the light show is directly affected by the style of music producing it—and the style in which that music is being played. The slow jam at the beginning of the clip above results in a sort of chilled-out, disco-ball effect, while the solo toward the end produces a dense spiderweb of light.
Currently, Friesinger says, the system works best if you optimize the software for the song’s particular style beforehand. But even in this early stage, it’s easy to see how it could turn into something a bit more engaging than the standard rock-concert light show. "Just imagine someone like Jack White on guitar, playing and creating visual feedback with a visualization like this," Friesinger says. "Awesome."
But of course, there’s always the issue of visibility. Getting a line of sight to a musician’s head from a packed show floor is already hard enough. Soon you might need to get an angle on their feet.
See more on the project page.
[Hat tip: Under Consideration]