Music deepfake
How it works
A typical music-deepfake pipeline runs in several steps. 1) Data collection: gathering the target artist's recordings, often with source separation to extract clean vocal stems. 2) Training/fine-tuning the voice model: a voice-conversion model (e.g. VITS-based, so-called so-vits-svc) or retrieval-based voice conversion (RVC) learns to reproduce the artist's timbre and vocal mannerisms. 3) Vocal synthesis: a reference vocal (hummed or sung by another person) or text and melody is fed in, and the model converts it into the target artist's voice (voice conversion) or synthesizes singing from scratch (singing voice synthesis). 4) Music generation/arrangement: the backing track is made by a generative music model (e.g. text-to-music family) or produced conventionally. 5) Mixing and release of the track. Quality depends on the quantity and cleanliness of training data and on how well the model matches the vocal register.
Problem solved
A music deepfake does not solve a technical problem in the classical sense โ it is rather an application (and simultaneously a source of problems) of generative voice and music synthesis. The underlying technologies lower the barrier to producing realistic vocals imitating well-known artists, which in turn raises significant legal and ethical issues: unauthorized use of voice, infringement of likeness/publicity rights, and spam and impersonation on music platforms.
Components
A model that reproduces the target artist's timbre and vocal manner; converts a reference vocal into the artist's voice or synthesizes it from training data.
A variant of voice synthesis specialized for singing โ it preserves pitch, rhythm, and melodic expression, not just spoken content.
A model that produces the composition, instrumental backing, or an entire track (e.g. text-to-music models). In full deepfakes it supplies the musical layer under the vocal.
A tool that isolates clean vocal stems from finished recordings, used to prepare training data for the voice model.
Implementation
Using a recognizable artist's voice without consent can infringe the right of publicity (voice/likeness) and copyrights tied to samples and composition.
Detecting generated vocals is hard; detector accuracy drops for other languages, accents, and newer generative models.
Evolution
A model generating music from raw audio, including vocals, demonstrating that neural networks can synthesize whole tracks.
Google publishes a paper on a text-to-music generator (January 2023), setting the direction for generative music models.
The track by creator ghostwriter977 (April 4, 2023) with cloned voices of Drake and The Weeknd goes viral; on April 17, 2023 Universal Music Group files takedown notices across platforms.
A generator of full songs with vocals, lyrics, and instrumentation becomes widely available (December 2023).
Udio launches as a web app (April 2024); Universal Music Group and Sony Music sue Suno for copyright infringement in training their models.
Randy Travis's voice, unable to sing since a 2013 stroke, was recreated with consent from over 40 recordings (May 2024) โ an example of authorized use.
Artists (incl. Elton John, Dua Lipa) demand protection from AI (May 2025); platforms and labels introduce rules on voice cloning, and UMG and Google work on frameworks with royalties and attribution.