Inception is like a Pollock made with a straight edge. The Matrix is a child’s doodle. Annie Hall looks like someone winding yarn. And 2001: A Space Odyssey looks, quite suitably, like the cover art off a 1980s science book.
This is Computer Watching Movies, by new media artist Ben Grosser, a project in which he wrote software for a computer to illustrate its view of famous movie scenes in real time. Grosser gave the computer a level of autonomy via algorithms and artificial intelligence programs that he wrote, and then he screened clips of several well-known films. The result is a series of real-time sketches that allow us to watch a movie the same way a computer might—a piece of art/research right in line with Grosser's ongoing interest of exploring the cultural implications of software.
What’s mind-bending is that the computer-filtered scenes, while completely unrecognizable to a human, are still each distinctive in their own right. Even amidst what appears to be computerized nonsense, you can make out the visual signatures of various films.
“For example, in American Beauty, the visuals of that clip shift back and forth between a dark room illuminated by a TV to the scene being shown on the TV, which features a slowly floating plastic bag against a stationary brick wall. The software finds the bag and tracks it, leading to the light, curvy, sketch-like marks,” Grosser tells Co.Design. “[While] in the Inception scene, there's a lot more for the software to ‘watch’: face close-ups, walking people at distance, landscape, etc. When the scene shifts into the exploding buildings sequence, the software is attracted to the fast moving bits and pieces of those structures as they fly across the frame."
Ultimately, what Grosser’s software is lacking is the narrative context of a film. So films shot with a relatively stationary camera, like Taxi Driver, tend to have their rich mise en scène glossed over while the computer simplifies any action to a single line. Meanwhile, polar opposite moments like the famous, 360-degree pans of The Matrix, appear as total chaos, even though the human brain easily makes the association that its simply seeing multiple perspectives of the same, stationary scene.
“What is different about our vision that finds more to look at than the computer does?” Grosser asks. “Certainly our narrative construction of the scene and its role within the larger film plays a part here. But Computers Watching Movies shows me just how much my brain ‘fills in’ with these clips, and how my mind is left with some visual space that asks me to construct more of the narrative on my own.”
Indeed, without narrative context, software transforms Robert DeNiro's unforgettable "You talkin' to me?" scene into the disappointing oversimplification more akin to "Travis looks in the mirror." But given how much smarter our algorithms are getting so quickly—how much better our silicon is getting at recognizing trends that humans miss—it's not hard to imagine a day when Computers Watching Movies could be more enlightening than our human critics.