Architecture

LDM

2022ActivePublished: 8 June 2026Updated: 8 June 2026Published

Key innovation

Moves the diffusion process from pixel space to a compressed latent space of an autoencoder, reducing compute cost by 1-2 orders of magnitude while preserving generation quality.

How it works

The LDM pipeline has two separate training stages and one inference stage. Training stage 1: an autoencoder (encoder E + decoder D) is trained on image data with a loss combining perceptual reconstruction (LPIPS), regularization (KL or VQ), and an adversarial discriminator. Its weights are then frozen. Training stage 2: for an image x compute latents z = E(x). The forward diffusion adds Gaussian noise: z_t = √(α̅_t)·z + √(1-α̅_t)·ε. A U-Net ε_θ(z_t, t, c) learns to predict the noise ε conditioned on an embedding c (text via cross-attention). Inference: start from pure noise z_T ~ 𝒩(0,I) and iteratively denoise using a scheduler (DDIM, DPM-Solver, Euler) over 20-50 steps, obtaining ẑ_0. The final image is x̂ = D(ẑ_0). Classifier-free guidance (CFG) strengthens conditioning: ε̃ = ε(z,∅) + w·(ε(z,c) − ε(z,∅)).

Problem solved

Classical pixel-space diffusion models (DDPM, ADM) require enormous compute and memory because the U-Net must operate on image-resolution tensors over hundreds of denoising steps. Training SoTA pixel-space diffusion at 256×256 takes hundreds of GPU-days. LDM reduces this by an order of magnitude by running diffusion in a 4×-16× downsampled latent space, enabling 512-1024 px generation on a single consumer-grade GPU.

Components

Perceptual autoencoderCompression x ↔ z between pixels and latent space

Encoder E: x → z and decoder D: z → x. Trained with LPIPS + KL/VQ + adversarial discriminator. In SD 1.x the downsampling factor is 8× (latent 64×64×4 for a 512×512 image).

INRGB image tensor.

OUTLatent tensor; f is the downsampling factor (typically 4-16).

KL-VAE (kontynuacyjne latenty)KL regularization — used in SD 1.x/2.x.

VQ-VAE (dyskretne kody)Vector-quantized — produces a discrete latent space.

Official

Denoising U-NetPredicts noise in latent space

U-Net with residual blocks plus self-attention and cross-attention layers. Inputs: noisy latent z_t and timestep t (sinusoidal embedding). Output: predicted noise ε.

U-Net (SD 1.x/2.x/SDXL)Classical attention U-Net.

DiT (Diffusion Transformer, SD3, Sora)Transformer instead of U-Net for better scaling.

Official

Conditioning encoderEncodes the conditioning c (text, segmentation, depth, pose)

External encoder (CLIP text encoder, T5, OpenCLIP) producing token vectors fed into the U-Net via cross-attention.

Cross-attentionMechanism injecting the conditioning into the U-Net

Q from U-Net activations, K/V from conditioning embedding c. Realizes text and multimodal conditioning.

Noise schedulerDefines the α̅_t schedule and the denoising method

Classical: linear, cosine, scaled-linear. Inference samplers: DDIM, DDPM, DPM-Solver, Euler, Heun, UniPC.

Implementation

Reference implementations

Latent Diffusion (CompVis, oficjalna)

Python

Official

Stable Diffusion (CompVis)

Python

Official

Generative Models (Stability AI, SDXL/SD3)

Python

Official

Diffusers (Hugging Face)

Python

Official

Implementation pitfalls

Weak VAE reconstructionsHigh

The autoencoder is a bottleneck — anything that cannot be reconstructed from the latent is lost regardless of U-Net quality.

Fix:Better autoencoder (more channels as in SD3 16-ch), task-specific VAE fine-tuning, smaller downsampling factor.

Zero-SNR and image bleachingMedium

Standard (linear) schedules do not reach full noise at T, causing the model to generate images with muted contrast.

Fix:Zero-SNR schedule and v-prediction (Common Diffusion Noise Schedules paper).

CFG artifacts at high wMedium

Excessive classifier-free guidance produces oversaturation, posterization, and strange textures.

Fix:Dynamic thresholding (Imagen), CFG rescale, smaller w (4-7) for SDXL/SD3.

Resolution mismatchMedium

A model trained at e.g. 512×512 generates repeated objects at 1024×1024 (the "double heads" issue).

Fix:Resolution conditioning (SDXL), MultiDiffusion, hierarchical sampling.

Evolution

Original paper · 2022 · CVPR 2022 · Robin Rombach

High-Resolution Image Synthesis with Latent Diffusion Models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer

2020

DDPM — foundations of diffusion models

Ho et al. introduce Denoising Diffusion Probabilistic Models in pixel space.

Diffusion Model (concept)Denoising Diffusion Probabilistic Models (paper)

2021

Classifier-Free Guidance

Ho & Salimans introduce CFG, the key conditioning mechanism in later LDMs.

Classifier-Free Diffusion Guidance (paper)

2021

LDM preprint (CompVis)

Inflection point

The first Rombach et al. preprint introduces the idea of diffusion in autoencoder latent space.

2022

Stable Diffusion 1.x — first open SoTA T2I

Inflection point

Stability AI / RunwayML release SD 1.4/1.5 based on LDM, democratizing text-to-image generation.

2023

SDXL — scaling LDM

Larger U-Net (2.6B), two text encoders, two-stage refinement; 1024 px natively.

SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (paper)

2023

Stable Video Diffusion / AnimateDiff

Extension of LDM to video sequences with temporal attention blocks.

2024

SD3 — Diffusion Transformer + rectified flow

Inflection point

SD3 replaces the U-Net with an MMDiT architecture and switches from DDPM to rectified flow matching.

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis (paper)

2023

Diffusion Policy in robotics

Chi et al. show that an LDM-style architecture effectively models the action distribution in robotic manipulation.

Diffusion Policy: Visuomotor Policy Learning via Action Diffusion (paper)

Hyperparameters (configurable axes)

Latent downsampling factor fCritical

Ratio of image resolution to latent resolution (typically 4, 8, 16). f=8 is the SD 1.x standard.

Latent channels cHigh

Number of latent channels (4 in SD 1.x/2.x, 16 in SD3 for better reconstruction).

Diffusion timesteps THigh

Forward chain length (usually 1000 in training, 20-50 at inference with DDIM/DPM-Solver).

Noise scheduleHigh

Linear, cosine, scaled-linear, zero-SNR — strongly affects quality and contrast.

CFG scale wCritical

Classifier-free guidance strength (typically 5-12 for image, 1-3 for video).

Parameterization (ε / x₀ / v)High

What the network predicts: noise ε, original x₀, or v-prediction (better for video and SD2).

Execution paradigm

Primary mode

Dense

The entire U-Net (or DiT) is active at each denoising step.

Activation pattern

All paths active

Parallelism

Parallelism level

Partially parallel

Training is fully batch-parallel. Inference requires sequential denoising steps (20-1000), but each step is a dense U-Net forward pass that fully exploits GPU parallelism.

Scope

TrainingInferenceAcross devices