Robots turn music into paintings of light at the University of Waterloo

Researchers at the University of Waterloo are using robots and music to create moving trails of light. CTV’s Spencer Turcotte has more on the artistic mashup.

Robots and the arts don’t often collide.

But, at the University of Waterloo, researchers are turning music into coloured light through the motion of robots.

“The two arts that we are interested in connecting are painting – visual arts, and music – performing arts,” said Gennaro Notomista, an assistant professor in UW’s electrical and computer engineering department.

Once the robots’ lights are turned on and the lights in the room go off, they are ready to work.

Researchers developed a system in which a swarm of robots listened to music and converted it into moving light trails.

“These emotions are going to be mapped onto the canvas to places and times where they should release paint,” Notomista said.

The robots move within a fixed area on the floor, which acts as a canvas of sorts. Their movements are in response to the tempo and chord progression they have analyzed.

“We work with multi-robot systems,” Notomista explained. “Naturally, we asked the questions, ‘How do we coordinate them? How do we not make them run into each other?’”

The algorithm the researchers use controls up to a dozen robots but can be scaled to handle any number of units.

Technology was then created to extract and analyze musical features that express emotion.

“[These] lines lead to sadness, like the dark blue and this kind of violet,” said Notomista while showing the colourful light trails on a computer screen.

Beyond the visual impact, the work also explores how an artist may be able to interact with robots while they are painting.

“The big picture is to get a sense from the artists, from the painters, on whether this could be an aid to paint,” Notomista said.

Next, the researchers plan to host an art exhibition where the public can get one of these unique robot-created paintings.

“When we saw what the robots did, we said ‘Oh, this looks like a Monet.’ OK, we were maybe a little bit too excited,” said Notomista, laughing.

The excitement is warranted when you consider the unlikely pairing of engineering and the arts.