The MIT-Developed Tech That Could Make Pokémon Go Even Better

A new technique called Interactive Dynamic Video could make augmented reality apps far more realistic.

Without a doubt, Pokémon Go is the smash hit of the summer, and for good reason: As half-formed as the gameplay is, there’s something truly thrilling about pointing your phone at the world around you and seeing a Caterpee crawling across your girlfriend’s hair, a Ghastly standing behind your dog, or a Pikachu sitting on a park bench.


Still, these augmented reality illusions are crude. That Caterpee won’t ever change your girlfriend’s hairstyle, and the Ghastly won’t pull your dog’s tail–yet. But a new technique from MIT’s Computer Science and Artificial Intelligence Lab could make augmented reality apps like Pokémon Go more interactive, by allowing what’s happened virtually to appear to affect what’s happening in the real world.

It’s called interactive dynamic video (IDV) by its creator, PhD student Abe Davis, who is publishing his dissertation on the research this month. Davis wisely decided to apply IDV to the wildly popular app–showing how the technology can make Pokémon which appear to alter the world around them.

In most forms of augmented reality, if you want to make a “real” object move, you need to turn it into a 3D model. The problem is that creating a 3D model of an object from a 2D video, while certainly not impossible, is something that’s usually beyond the computational power of our smartphones.

Instead, Davis came at the problem from a different tack. His approach looks for tiny vibrations in as little as five seconds of videos to calculate the different frequencies with which an object can move in real life: the way a cable might swing, how a bobblehead wobbles, or how a branch might bounce in the wind. The technique provides a computationally cheap way to move those objects in a realistic way in video.

Beyond allowing Pokémon to realistically bounce through the real world, IDV has more practical uses. In other augmented reality apps, it could allow people to manipulate “real” objects in natural-feeling ways. But IDV could even be useful in filmmaking: for example, movies in which CGI and real-world characters interact could use IDV as a way to get more realistic results without blowing out the budget.

IDV could even save lives, according to Davis, who imagines his algorithm having real-world applications, too. For example, it could be possible to diagnose old buildings or bridges in need of replacement by taking video recordings of them, then inserting them into simulations of extreme weather events like hurricanes or earthquakes. “The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able to test the stress in a safe virtual environment, in a way that doesn’t harm the real-world counterpart,” Davis says in the press release. And hey, if that bridge can handle a hurricane, a Charizard is probably a piece of cake.


You can read Davis’ dissertation on Interactive Dynamic Video here.