AI music generation
How it works
A typical pipeline: (1) a neural audio codec (SoundStream/EnCodec) encodes a recording into several streams of discrete tokens via residual vector quantization; (2) a generative model learns to produce those tokens — autoregressively (next-token prediction, as in MusicGen/MusicLM/Jukebox) or via diffusion (iteratively denoising an audio representation); (3) conditioning — text (through a language encoder/embeddings), melody, or other signals steer generation; (4) the codec decoder turns the generated tokens back into a waveform. MusicLM additionally uses a hierarchy of semantic and acoustic tokens, while MusicGen simplifies this to a single transformer over EnCodec's parallel code streams.
Problem solved
Traditional computer composition worked on symbolic MIDI and could not produce a realistic, finished recording with vocals and production. AI music generation addresses creating full, sonically rich audio directly from user intent (e.g. a text description), without playing instruments or manual production.
Components
Encodes a waveform into compact discrete tokens and decodes them back; usually based on residual vector quantization (RVQ). Examples: SoundStream, EnCodec.
Conversion of a continuous signal into a sequence of discrete tokens (semantic and/or acoustic) that the generative model operates on.
A transformer predicting the next audio tokens conditioned on previous ones and on a control signal. Used in Jukebox, MusicLM, MusicGen.
An alternative generative approach: iterative denoising of an audio representation (in waveform or latent space) to produce a recording.
A language encoder/embeddings linking a text description to the audio representation space, steering genre, mood and instrumentation of the generated track.
Evolution
OpenAI introduces Jukebox — a generative model producing music with vocals directly on raw audio via a hierarchical VQ-VAE and autoregressive models.
Google publishes SoundStream — an end-to-end neural audio codec with RVQ, a foundation for later audio-token models.
Meta publishes EnCodec — a high-fidelity neural audio codec later used by MusicGen.
Google introduces MusicLM — generating music from text based on a hierarchy of semantic and acoustic tokens.
Meta publishes MusicGen — a single transformer over EnCodec's parallel code streams, with text and melody conditioning.
Mainstreaming of consumer music generators: Suno (widely available since December 2023) and Udio (public beta April 10, 2024).
In September 2026 Suno replaced earlier models with the v6 family, developed with industry partners (Warner Music Group, BMG, Believe), following settlements in copyright disputes.
Hyperparameters (configurable axes)
Number of quantization streams in the audio codec — a trade-off between audio quality and token-sequence length.
Sample rate of generated audio (e.g. 24 kHz, 32 kHz, 44.1 kHz) affecting bandwidth and fidelity.
Type of control signal: text, melody, genre, length — determines control over the output.
Parameter count of the generative model, affecting quality, coherence and generation cost.
Parallelism
Method-dependent: training of autoregressive models is parallel (teacher forcing), but autoregressive inference is sequential token-by-token; diffusion models allow more parallelism at inference.
Hardware requirements
Training and inference of large transformer/diffusion models over long audio-token sequences require high GPU throughput and memory.
Models such as MusicLM were trained on Google accelerators (TPUs).