Latent Space
How it works
Latent space learning and usage process: (1) Encoder network - mapping from raw data (e.g. a 224×224×3 = 150K pixel image) to a latent vector z ∈ ℝᵈ (typically d=128-4096). In autoencoders and VAE this is an explicit encoder network; in Transformers - hidden states of successive layers; in GANs - the generator has the reverse direction (z → data), with the encoder trained separately; (2) Learning objective - the encoder learns so that z contains enough information to perform the task: reconstruction (autoencoder), prediction (classifier, LLM next-token), generation (VAE, diffusion), cross-modal matching (CLIP). Objectives force semantically similar inputs to yield nearby z; (3) Regularization - VAE imposes a Gaussian prior (KL divergence loss) so the space is smooth and sampleable. GANs enforce that z comes from a simple distribution (Gaussian, uniform). Without regularization the space can be 'shattered' - interpolation stops working; (4) Geometric operations in latent space: (a) similarity = cosine(z_a, z_b) - close = similar, (b) interpolation z_c = α·z_a + (1-α)·z_b - continuous transformation from a to b, (c) arithmetic z_king - z_man + z_woman ≈ z_queen - subtraction of semantic directions, (d) sampling z ~ N(0,I) - a new random sample, (e) directed traversal - finding feature 'axes' (smile, age, race in a face latent); (5) Decoder network - optional mapping z → raw data (for generative models: VAE decoder, diffusion decoder, GAN generator). LLMs do not have a decoder in the classic sense - hidden states are directly projected to dictionary logits.
Problem solved
Raw data (image pixels, text characters, audio samples) is high-dimensional, sparse, and does not reflect semantic structure - two images of the same cat differ in every pixel, though they are the 'same' concept. Classic hand-designed features (SIFT, HOG, TF-IDF) partially solved this problem but were rigid, domain-specific, and did not generalise. Latent Space solves this via a learned representation: the neural model itself discovers the optimal space structure where the 'meaning' of data becomes geometric - similar things close, different ones far, transformations continuous. This is the foundation of modern AI - without it, neither generative modelling (VAE, GAN, diffusion), nor semantic search (embeddings), nor transfer learning (pretrained representations), nor multimodal alignment (CLIP, image-text in a shared space) would exist.
Components
Neural network mapping raw data → latent vector z. Can be CNN (for image), Transformer (for text), RNN (for sequence), or a combination. For LLM: simply stacking Transformer layers produces increasingly rich latent representations in successive layers.
Official
The output vector itself - representation of a data point in ℝᵈ. Dimension d is a key hyperparameter: 128 for classic word embeddings, 768 for BERT-base, 1536 for OpenAI text-embedding-3-small, 4096 for LLM Transformer hidden state.
Function defining 'closeness' in latent space - most commonly cosine similarity (for normalised) or Euclidean distance (for non-normalised). Choice impacts how the model interprets 'similarity'. Cosine ignores magnitude, Euclidean does not.
Official
Distribution to which we regularise the latent space (typically N(0,I) Gaussian). Critical for generative modelling - from a known prior one can easily sample new points and decode them to raw data. In classic (deterministic) autoencoders, the lack of a prior yields 'holes' in the space.
Official
Network mapping z → raw data. Present in autoencoders, VAE, diffusion, GAN. Absent in representational models (BERT embeddings, CLIP encoder) where the latent space itself is the goal (retrieval, classification).
Official
Implementation
In high-dimensional spaces (d>1000) all points are 'far from each other' - cosine similarity compresses around the mean, nearest neighbours lose interpretability.
Latent space can 'collapse' - the generator ignores z dimensions (uses only a few), or the decoder ignores z entirely and generates from the prior. Symptom: little-varied generations.
In most latent spaces a single dimension does not correspond to one semantic feature - they are entangled. Makes interpretation and targeted editing hard.
A pretrained encoder (e.g. BERT on Wikipedia) yields weaker latent for a specialised domain (medical, legal, code). Cosine similarity does not preserve domain relations.
1M documents × 3072-dim float32 embeddings = 12 GB. 100M = 1.2 TB. Storage becomes the dominant cost of RAG infrastructure.
Evolution
Principal Component Analysis - a linear method for finding a lower-dimensional representation of high-dimensional data. Foundation of the latent space concept.
Publication shows that neural networks can learn nonlinear dimensionality reduction - foundation for modern autoencoders.
Science publication showing deep autoencoders discover semantic data structure better than PCA. Neural representation learning renaissance.
First publicly demonstrated 'king - man + woman ≈ queen' - latent space arithmetic. Popularises the notion of embeddings in NLP.
arXiv 1312.6114 - probabilistic latent space with Gaussian prior, enables sampling and smooth interpolation. Foundation of generative modelling.
Latent z ~ N(0,I) mapped to photorealistic images by a generator adversarially trained against a discriminator. New approach to generative latent.
Shows that GAN latent interpolation (faces) and arithmetic (glasses + man - glasses = man without glasses) work on images as they do on text.
Contextualized embeddings - the same word has different latents depending on context. Standard NLP embeddings.
Images and texts in a shared latent space via contrastive learning. Foundation for many multimodal models (DALL-E, Stable Diffusion cross-attention).
Rombach et al. show that operating in VAE latent (64x64x4) instead of pixels (512x512x3) gives 8x speedup while retaining quality. Latent space enters mainstream consumer AI.
Hyperparameters (configurable axes)
Key hyperparameter. Higher d = more informational capacity and better quality, but higher memory and downstream compute cost.
Type of network mapping data to latent. Impacts data type, representation quality, and cost.
Defines how the latent space is shaped. Different objectives yield fundamentally different spaces.
Definition of 'closeness' in latent space. The choice has serious implications for downstream tasks.
Computational complexity
Time complexity: O(f_encoder) + O(d) per operation. Space complexity: O(d) per data point.
Compute bottleneck
The encoder forward pass itself (CNN, Transformer) is typically O(1000-10000x) more compute than any latent space operation. Compressing d=4096 → 128 by training a smaller encoder is a significant quality-vs-cost tradeoff.
Execution paradigm
In a classic encoder all weights are active for every input (dense). Sparse latent space (top-k active dimensions) is a rarer specialisation - experiments with Sparsecoders.
The latent space itself has no routing - it is a static vector space. Routing appears only in models built on top (MoE - routing between experts, RAG - routing between documents).
Parallelism
Latent space operations (similarity search, arithmetic) are trivially parallel - massively GPU-parallelizable. Vector database ANN queries leverage GPU for batch retrieval.
Hardware requirements
Encoder network (Transformer, CNN) is dense matmul - GPU Tensor Cores give a 5-10x speedup. Batching multiple inputs into one forward pass exploits full SM width.
Retrieval over vectors (cosine, ANN) works great on CPU with SIMD (AVX-512). FAISS has optimised CPU kernels - often the choice for self-hosted vector DBs.
Google TPUs (v4, v5, Trillium) handle Transformer encoders well - used by Google for their own embeddings (Gecko, textembedding-gecko).