Disco with AI
The next level of music visual generation 🚀🎶🎨
Paper: Generative Disco: Text-to-Video Generation for Music Visualization [20 pages]
Researchers at Columbia University are developing visuals to synchronize with music through a text-to-music interface!
Mainly, they observed that music visuals are a core element of a musical experience and the current process of visual generation is time and resource intensive. They introduced GPT-4 language model based music visualization generation, involving techniques to account for transitions, holds, lyrical understanding and quality of visuals.
First, the researchers studied current trends in music and visual generation such as tempo understanding, symbolic representation and visual/audio consistency
Then, they created the generative disco interface with the following key features:
Ability to import any waveform/song
Ability to generate frames using AI and interpolate them for certain intervals
Stitch output with audio and generated frames with suggested styles
Finally, they gathered quantitative and qualitative feedback on Generative Disco
As per results, the researchers urge more users to generative experiences with generative disco and would like to allow more styles of generations.
So Essentially,
AI can make music concert visuals.
Next generation of research will develop more consistent experiences.