Skip
Current Issue
This Month's Print Issue

Follow Fast Company

We’ll come to you.

3 minute read

What Happens When You Let A Neural Net "Curate" Art

Artificial intelligence built for the Tate digs into the gallery's century-old collections to bring old works new life.

  • <p>A Reuters photo of sunbathers in Wales (left) and <em>Bathers</em> by Richard Hamilton, 1969 (right) were strong matches based on context and composition.</p>
  • <p>The entrance to the <em>Recognition</em> exhibition, now up at London's Tate museum.</p>
  • <p>A visitor to the exhibition watches a projection of the Recognition computer program searching through the Tate's collection to find matches.</p>
  • <p>Recognition uses neural networks to analyze a photo's objects, faces, composition and context and match it with a piece of art at the Tate.</p>
  • <p>A photo of the DP World London Gateway container port (left) and a collage by Eduardo Paolozzi (right)</p>
  • <p>A photo of people playing in a fountain in Paris (left) and Venice, the <em>Piazzetta with the Ceremony of the Doge Marrying the Sea</em>(right)</p>
  • <p>Photo of the festival of Janmashtami in Mumbai, India (left) and <em>Parental Palette</em> by Ian Stephenson (right).</p>
  • <p>Photo of the International Military Music Festival "Spasskaya Tower" in Red Square in Moscow, Russia (left) and <em>Fates</em> by Gilbert & George (right)</p>
  • <p>A photo of vacationers swimming in Bassin d'Arcachon (left) and <em>August Blue</em> by Henry Scott Tuke have strong object and composition matches.</p>
  • <p>A photo of Changi Airport in Singapore (left) and <em>Industrial Landscape</em> by L.S. Lowry match strongly on the object analysis.</p>
  • <p>A photo of a merry-go-round in Stockholm (left) and <em>Structure</em> by Richard Hamilton (right).</p>
  • <p>A photo of Notting Hill Carnival in London (left) and <em>The Roundabout</em> by Sir Stanley Spencer (right).</p>
  • <p>Another photo of the Notting Hill festival (left) and <em>The Harvey Family</em> by Sir Godfrey Kneller (right).</p>
  • <p>Photo of attendees at the Afropunk Music Festival in Brooklyn (left) and  <em>Venus and Adonis</em> by Stephen McKenna (right).</p>
  • <p>Photo of a person taking shelter from the sun before seeing Pope Francis in Saint Peter's Square at the Vatican (left) and <em>Morning</em> by Dod Proctor (right).</p>
  • 01 /15

    A Reuters photo of sunbathers in Wales (left) and Bathers by Richard Hamilton, 1969 (right) were strong matches based on context and composition.

  • 02 /15

    The entrance to the Recognition exhibition, now up at London's Tate museum.

  • 03 /15

    A visitor to the exhibition watches a projection of the Recognition computer program searching through the Tate's collection to find matches.

  • 04 /15

    Recognition uses neural networks to analyze a photo's objects, faces, composition and context and match it with a piece of art at the Tate.

  • 05 /15

    A photo of the DP World London Gateway container port (left) and a collage by Eduardo Paolozzi (right)

  • 06 /15

    A photo of people playing in a fountain in Paris (left) and Venice, the Piazzetta with the Ceremony of the Doge Marrying the Sea(right)

  • 07 /15

    Photo of the festival of Janmashtami in Mumbai, India (left) and Parental Palette by Ian Stephenson (right).

  • 08 /15

    Photo of the International Military Music Festival "Spasskaya Tower" in Red Square in Moscow, Russia (left) and Fates by Gilbert & George (right)

  • 09 /15

    A photo of vacationers swimming in Bassin d'Arcachon (left) and August Blue by Henry Scott Tuke have strong object and composition matches.

  • 10 /15

    A photo of Changi Airport in Singapore (left) and Industrial Landscape by L.S. Lowry match strongly on the object analysis.

  • 11 /15

    A photo of a merry-go-round in Stockholm (left) and Structure by Richard Hamilton (right).

  • 12 /15

    A photo of Notting Hill Carnival in London (left) and The Roundabout by Sir Stanley Spencer (right).

  • 13 /15

    Another photo of the Notting Hill festival (left) and The Harvey Family by Sir Godfrey Kneller (right).

  • 14 /15

    Photo of attendees at the Afropunk Music Festival in Brooklyn (left) and Venus and Adonis by Stephen McKenna (right).

  • 15 /15

    Photo of a person taking shelter from the sun before seeing Pope Francis in Saint Peter's Square at the Vatican (left) and Morning by Dod Proctor (right).

The question of whether life imitates art is a philosophical one, posed most notably by Oscar Wilde in a 1891 essay promoting Romanticism over Realism. But ask a computer, and it will inevitably take the rationalist approach: what does the data have to say?

That's the premise of Recognition, a computer program created for the Tate's IK Prize, a digital art competition held by the Tate and Microsoft which asked designers to use AI to explore the art in the gallery's collection. It uses several types of machine learning to match art in the Tate's extensive British art collection to photos of current events from Reuters. By analyzing both visual and thematic similarities between two images, Recognition can come up with a fine art doppelganger to any event or figure in popular culture.

Coralie Gourguechon, Monica Lanaro, Angelo Semeraro and Isaac Vallentin.[Photo: Marco Zanin]

Recognition, which won this year's competition, was created by four designers at the Benetton-backed research center, FabricaCoralie Gourguechon, Angelo Semeraro, Isaac Vallentin, and project manager Monica Lanaro. The idea for the platform—and subsequent Tate exhibition—is both to resurface older works in the collection and experiment with how emergent forms of machine learning like deep neural networks can intersect with images and art.

"It's a fascinating but unexplored territory with lots of potential," says Semeraro. "How can we apply rational, objective thinking to a subject topic like art? [The Tate] collection inspired the idea that we can link our everyday to works of art."

Recognition does this by analyzing an image in four different ways—isolating any identifying objects, or faces, as well as the image's composition and its metadata—and searching the Tate's collection to find an image that closely matches. For the program to "see" each image, the Fabrica team utilized several forms of artificial intelligence that sit behind the platform's interface. They worked with web developers at JoliBrain on the AI, who developed some of their own algorithms and also built on existing technology.

Take the object recognition, for instance: using the open source deep learning servers DeepDetect and DenseCap, JoliBrain developed a deep neural network that finds an object from an image—an apple on a table, or a man wearing a suit—and labels it creating a short sentence. A similarity search engine then looks for a similar object in the Tate collection. The algorithms for matching image composition do something similar, but look for artistic elements like line and color instead of specific objects. For facial recognition, the team used Microsoft Cognitive Services's Computer Vision and Emotion APIs, which analyze any existing faces for age, gender and general emotion state.

Analyzing photos for context is more complicated: the team used a variety of deep neural networks to process both the images and their metadata to find semantic matching among words or sentences between the two images. Because of this mix of visual and textual factors, sometimes Recognition's matches are striking in their visual similarities, and other times the similarity is less obvious. Gourguechon points toward a photo of swimmers that the program pulled up during the Olympics as an example of this: the image was matched with a painting of people sitting around a table. The similarity wasn't clear until looking at the data, which Recognition displays alongside each image in the match.

"The algorithm saw on the torso of one of the swimmers a tattoo of a woman’s face, and the face was at the same angle as the people around the table," she explains. In this way, the tool's limitations reveal connections and similarities that human beings might not make. "Right from the beginning we wanted to combine the subjective point of view and the algorithm's rational point of view," says Gourguechon.

Exploring how a computer "sees" art can be entertaining—analyzing a photo of Donald Trump, the computer interprets his hair as a "man's hand on head"—but it also highlights how far machine learning has to come. In the meantime, projects like Recognition, and designers willing to present emerging AI technologies in compelling ways, are helping the field along.

You can play around with Recognition here, or visit the Tate, where Recognition is being displayed in its own exhibition.

[All Images: courtesy of Fabrica]

ARE YOU REGISTERED TO VOTE?
Register now to make sure you have a voice in the election.
loading