It’s hard to believe that the project began with a hack, combining the Kinect sensor with an SLR to create 3-D avatars. To do something with the footage, the team built a nonlinear editing system to accommodate this unique video feed, and then they started recording massive amounts of interviews with new media artists, critics, and designers. Originally, the plan was just to cut a traditional, albeit very cool-looking documentary from the interviews. But instead, realizing that they had a whole series of mapped 3-D figures on their hands, the team decided to build the documentary as an interactive piece of software—a whole narrative world drivable by the user and infinitely explorable.
"We want the experience of Clouds to feel like a continuous revelation," Minard tells Co.Design. "It’s kind of like the difference between Rothko making a painting and Brian Eno coding a system that makes nine million paintings." The viewer will explore the film like a philosophical videogame, flying through a polygonal universe of thought. They’ll encounter decision trees, powered by technology lying at the documentary’s heart, the Story Engine. It’s a piece of AI that’s not so dissimilar from a web-crawler that will analyze keywords in each scene’s conversations and generate optional pathways through the "film" in response.
"While the viewer is watching the sequence, the system presents digressions as the discussion touches on related topics," George explains. "We’ll lead viewers on tangents, letting them stumble onto new ideas and raising new questions to start the cycle over." Additionally, each discussion or scene will have a whole other layer of metadata—think humor or skepticism—which the software will consider in order to balance the tone of any given playthrough. So while the navigation decisions are your own, many of the choices will be algorithmically curated, ensuring that you’ll be satisfied by the ride.
"We respect the craft of editing, and in building this interactive documentary, will program the system to have a sensitivity to the content to "think" like an editor, to compose sequences with rising action, with real-time sound design to inflect a certain mood," Minard explains. And if you’d prefer, that software editor can make every decision for you, allowing you to view Clouds as lazily as you would any standard documentary.
No doubt, Clouds is a heady project that’s as technologically confounding as it is visually stunning. But at the same time, it’s spearheading a new direction for documentary filmmaking. Much like Bear71, Clouds will be an experience that’s constantly concerned with the viewer as an individual thinker, rather than one head in a mass audience. And, okay, it looks really cool, too.
[Hat tip: Creative Applications]