Generative models learn (usually via self-supervised training) the data distribution from massive corpora by minimizing a loss specific to the model family. LLMs predict the next token autoregressively; diffusion models learn to reverse a gradual noising process to reconstruct a sample from random noise; GANs train a generator and a discriminator in an adversarial game; VAEs encode data into a latent space and decode new samples from it. After training, generation samples from the learned distribution, often conditioned on an input (prompt, class, image). Modern systems are built on foundation models pre-trained at scale and then fine-tuned or prompt-steered for specific tasks.
Classical machine learning focused on discriminative tasks (classification, regression, detection) and could not create new, coherent content. Generative AI addresses data synthesis — automatically producing realistic text, images or audio, filling in missing data, and generating synthetic training sets where real data is scarce or costly.
Generative models can produce plausible-sounding but factually incorrect content.
Models reproduce and amplify biases present in their training data.
The lack of a single objective quality metric makes generation hard to compare and control.
Training and serving large generative models demand substantial compute and energy.
Kingma and Welling introduce the VAE, a probabilistic latent-variable model for data generation.
Goodfellow et al. propose training two networks in an adversarial game, enabling high-quality image synthesis.
Self-attention becomes the backbone of scalable generative language models.
DDPM makes diffusion competitive with GANs, and GPT-3 (175B parameters) demonstrates large-scale few-shot learning.
Public generative products reach mass audiences, triggering a surge of interest and deployment.
Training and inference of large generative models rely on massive matrix multiplication, ideal for GPU tensor cores.
TPUs efficiently execute the large-scale matrix operations of generative models.
