Robots Atlas>ROBOTS ATLAS
Speech & Audio Processing

Music deepfake

2023ActivePublished: 19 November 2026Updated: 19 November 2026Published
Key innovation
Combining voice cloning with generative music models made it possible to produce finished recordings that imitate a specific artist's voice and style without that artist's involvement or consent.
Category
Speech & Audio Processing
Abstraction level
Technika
Operation level
ApplicationInferenceLaw and governance
Use cases
Viral tracks with the "voice" of famous vocalistsAI covers of existing songs in another artist's voiceRecreating the voice of an artist who can no longer singParody and entertainment contentUnauthorized impersonation of artists (abuse)

How it works

A typical music-deepfake pipeline runs in several steps. 1) Data collection: gathering the target artist's recordings, often with source separation to extract clean vocal stems. 2) Training/fine-tuning the voice model: a voice-conversion model (e.g. VITS-based, so-called so-vits-svc) or retrieval-based voice conversion (RVC) learns to reproduce the artist's timbre and vocal mannerisms. 3) Vocal synthesis: a reference vocal (hummed or sung by another person) or text and melody is fed in, and the model converts it into the target artist's voice (voice conversion) or synthesizes singing from scratch (singing voice synthesis). 4) Music generation/arrangement: the backing track is made by a generative music model (e.g. text-to-music family) or produced conventionally. 5) Mixing and release of the track. Quality depends on the quantity and cleanliness of training data and on how well the model matches the vocal register.

Problem solved

A music deepfake does not solve a technical problem in the classical sense โ€” it is rather an application (and simultaneously a source of problems) of generative voice and music synthesis. The underlying technologies lower the barrier to producing realistic vocals imitating well-known artists, which in turn raises significant legal and ethical issues: unauthorized use of voice, infringement of likeness/publicity rights, and spam and impersonation on music platforms.

Components

Voice cloning / conversionReproducing the artist's vocal identity

A model that reproduces the target artist's timbre and vocal manner; converts a reference vocal into the artist's voice or synthesizes it from training data.

Singing voice synthesis / conversion (SVS/SVC)Generating sung vocals

A variant of voice synthesis specialized for singing โ€” it preserves pitch, rhythm, and melodic expression, not just spoken content.

Generative music modelCreating the musical layer

A model that produces the composition, instrumental backing, or an entire track (e.g. text-to-music models). In full deepfakes it supplies the musical layer under the vocal.

Audio source separationPreparing training data

A tool that isolates clean vocal stems from finished recordings, used to prepare training data for the voice model.

Implementation

Implementation pitfalls
Lack of artist consent and rights infringementCritical

Using a recognizable artist's voice without consent can infringe the right of publicity (voice/likeness) and copyrights tied to samples and composition.

Difficult and unreliable detectionHigh

Detecting generated vocals is hard; detector accuracy drops for other languages, accents, and newer generative models.

Evolution

2020
OpenAI Jukebox

A model generating music from raw audio, including vocals, demonstrating that neural networks can synthesize whole tracks.

2023
MusicLM (Google)

Google publishes a paper on a text-to-music generator (January 2023), setting the direction for generative music models.

2023
"Heart on My Sleeve" โ€” the music-deepfake breakthrough
Inflection point

The track by creator ghostwriter977 (April 4, 2023) with cloned voices of Drake and The Weeknd goes viral; on April 17, 2023 Universal Music Group files takedown notices across platforms.

2023
Suno

A generator of full songs with vocals, lyrics, and instrumentation becomes widely available (December 2023).

2024
Udio and label lawsuits

Udio launches as a web app (April 2024); Universal Music Group and Sony Music sue Suno for copyright infringement in training their models.

2024
"Where That Came From" โ€” Randy Travis's voice

Randy Travis's voice, unable to sing since a 2013 stroke, was recreated with consent from over 40 recordings (May 2024) โ€” an example of authorized use.

2025
Artist pushback and platform protection

Artists (incl. Elton John, Dua Lipa) demand protection from AI (May 2025); platforms and labels introduce rules on voice cloning, and UMG and Google work on frameworks with royalties and attribution.