Autoregressive Diffusion
How it works
Typical AR Diffusion flow for video generation: (1) Context conditioning - the network takes all previously generated frames (t-1, t-2, ..., t-N) and optionally action signals (camera pose, robot command, prompt); (2) Noise initialisation - for frame t, start from clean Gaussian noise of the target frame dimensions; (3) Conditional denoising - over K steps (typically 20-50), the network iteratively denoises, conditioning each step on the temporal context; (4) Frame emission - after the last denoising step, frame t is emitted; (5) Context shift - frame t joins the context, the model advances to t+1. Key variants: (a) block-wise - generating larger sequence blocks at once (e.g. 4-8 frames instead of 1); (b) MAR-style - generating image tokens without VQ, each token via a small diffusion network (Meta MAR, 2024); (c) Diffusion Forcing - training with independent noise levels per token, enabling flexible AR sampling; (d) MaskGIT / Muse - iterative token unmasking instead of classic left-to-right AR.
Problem solved
Classic diffusion models generate the entire sample (image, video clip) in one shot - excellent quality, but no temporal control and high cost for long sequences (the whole video must be generated at once). Classic autoregressive image/video models (PixelCNN, PixelRNN) require discretisation via VQ-VAE - losing quality to quantisation. AR Diffusion solves both: it keeps diffusion's continuous space (no VQ) and gains element-by-element sequential generation, enabling real-time interaction, long video sequences, and controlling generation with an action stream.
Components
A neural network (typically Diffusion Transformer / DiT or U-Net) learning to reverse the noising process - for a given noise step t, predicts the noise to subtract. Conditioned on both noise level and temporal context.
Official
Conditioning mechanism: encodes the sequence of previously generated elements (frames, tokens) and injects into the denoising network via cross-attention, prepending, KV-cache, or similar. Critical for temporal coherence.
Official
Deterministic or learned schedule of noise levels (β_t or σ_t) used both in training and inference. Variants: linear, cosine, karras, rectified flow. In Diffusion Forcing - independent per token.
Official
Extra control signal (camera pose, robot command, text prompt, audio for avatars) injected into the denoising network alongside the temporal context. Critical for world models and interactive avatars.
Official
Algorithm for iteratively invoking the denoising network in K steps: DDPM, DDIM, DPM-Solver, Euler, Heun, rectified flow ODE. Choice impacts the latency-quality-NFE (number of function evaluations) balance.
Official
Implementation
Model trained on context of ground-truth prior elements, but inferred on own generations - errors accumulate (drift). For video: after several hundred steps, frames drift in quality from the first one.
K denoising steps per frame × N frames give high real-time latency. For 30 FPS video with K=20 that is 600 forward passes per second - requires a powerful GPU/TPU.
Long temporal context (hundreds-to-thousands of frames) generates a multi-gigabyte attention KV-cache. Critical for long video / long horizon.
AR Diffusion mixes AR (exposure-bias noise) and diffusion (sample-level noise). The combination can be unstable - loss oscillates, model doesn't converge.
FID/IS metrics evaluate sample quality but not temporal coherence. No consensus on long-horizon rollout evaluation.
Evolution
Denoising Diffusion Probabilistic Models establish diffusion as an alternative to GANs. Foundation for AR Diffusion.
ICLR 2022 paper (arXiv 2110.02037) introduces the family of models combining AR and diffusion - including Order-Agnostic Autoregressive Diffusion Models.
Bidirectional Transformer generating images via iterative unmasking - a form of AR Diffusion for images.
Scaling MaskGIT to text-to-image, competitive with Imagen and Stable Diffusion.
Tian et al. (ByteDance) - next-scale prediction instead of next-token, scaling consistent with LLM scaling laws. NeurIPS 2024 Best Paper award.
Li et al. (Meta FAIR + MIT) - autoregressive generation of image tokens in continuous space, each token via a small diffusion network. arXiv 2406.11838.
Fan et al. (arXiv 2410.13863) - scaling laws for VQ-free autoregressive image generation, proof that AR Diffusion scales like LLMs.
NeurIPS 2024 (arXiv 2407.01392) - unifies next-token AR and full-sequence diffusion through independent per-token noise levels. Enables flexible causal masking and long horizons.
December 2025 - Runway releases the GWM-1 family (Worlds, Avatars, Robotics) built on an autoregressive video model on top of Gen-4.5. Frame-by-frame in real time with action conditioning.
Chinese and academic labs show scaling AR Diffusion for language as an alternative to classic LLMs. MMaDA - multimodal image+text variant.
Hyperparameters (configurable axes)
Number of diffusion sampler steps per sequence element. Key tradeoff: latency vs quality.
How many prior sequence elements enter conditioning. Impacts temporal coherence and memory.
How many elements generated at once in the outer loop. Balance between pure AR (block=1) and pure diffusion (block=sequence).
Trajectory of β_t / σ_t. Impacts training stability and sample quality.
Computational complexity
Time complexity: O(N · K · f). Space complexity: O(L · d + K_cache).
Compute bottleneck
Unlike pure diffusion (whole sample denoised in parallel), AR Diffusion generates elements sequentially - element t cannot start before element t-1 is fully denoised (K steps). Blocks full inference parallelisation on the temporal dimension.
Execution paradigm
'Stage-dependent' because of two stages: (1) AR outer loop - one sequence element per time step, (2) diffusion inner loop - K denoising steps per element. The network is fully active at each step.
No expert routing like in MoE - one monolithic denoiser (or DiT). Conditioning is via context and action signals, not sparse routing.
Parallelism
Speculative decoding and block-wise sampling ease the constraint: block-wise generates several elements at once with partial conditioning, speculative launches candidates in parallel and accepts matching ones.
Hardware requirements
AR Diffusion is dominated by dense matmul (Transformer forward pass × K steps × N frames). GPUs with Tensor Cores (H100, H200, B200, A100) offer a 5-10x speedup vs classic GPUs. Critical for real-time video.
Google TPUs handle large Transformer/DiT matmuls well. Historical ARDM/MaskGIT/Muse implementations were trained on TPU (Google-first).
AR Diffusion is too compute-heavy for CPU even with AVX-512. Realistic only for small demo models or distilled samplers (K=1-2) on the edge.