Google’s DeepMind AI already creates a soundtrack by watching a video

Google’s DeepMind continues to improve every day. Now it is capable of creating soundtracks with music, sound effects and even dialogue, from a video.

It is incredible to realize that the first generative AI reached users at the beginning of 2023. In a year and a half we have gone from chatbots, to an AI that already does everything, like DeepMind.

ChatGPT and company started by generating text. Later, images. And finally video. Until now, silent video, without sound, from a text.

There are specialized AIs that create sound effects for videos, like the one from ElevenLabs. But now DeepMind takes a giant step by being able to create a soundtrack from a video, containing music, sound effects, and even dialogueif there are characters.

A soundtrack created by AI

Google has been improving the technology Video-to-audio (V2A) from DeepMind, to create soundtracks complete.

DeepMind only needs one video, to do it on its own: based on what it “sees” in the video, it creates the music, the sound effects, and the dialogues. But it is possible to add text prompts to get something more specific.

For example, this is the soundtrack that DeepMind creates when we type the prompt: “cars skidding, car engine revving, angelic electronic music”:

As we see, DeepMind has not only created a soundtrack that fits that promptbut others is responsible for synchronizing sound effects, like the car skidding, in the right place. The user does not have to do absolutely anything.

The result is surprisingly good. It sounds pretty good and doesn’t clash with the video. It can almost pass for a conventional soundtrack.

As Google explains on your blogthe advantage of DeepMind is that can generate infinite soundtracks from the same videofor the user to choose.

At the moment, this important improvement in DeepMind’s V2A technology is not available to the general public. Google says it first wants to make sure it is completely secure.

He also confesses that it still needs to be perfected. If the video is of low quality, it does not recognize well what is on the screen. He also makes mistakes synchronizing the dialogues with the characters.

The day when a generative artificial intelligence be able to create an entire movie on your own, it’s a little closer. You already have the base…

Known how we work on Computertoday.

Tags: Artificial intelligence