Embedding model
How it works
1) The input is tokenized and passed through an encoder (usually a Transformer) that produces contextual token vectors. 2) A pooling layer (mean pooling, CLS pooling or last-token pooling) aggregates those vectors into a single fixed-length vector. 3) The vector is optionally projected and L2-normalized so that cosine similarity equals the dot product. 4) The model is trained contrastively on pairs/triplets (positives pulled together, negatives pushed apart), often with hard negatives and query/document distinguishing prefixes. At inference, queries and documents are encoded into vectors and search becomes nearest-neighbor retrieval (ANN) in a vector database.
Problem solved
Classic lexical search (keyword matching) fails to capture synonyms, paraphrases or meaning and is hard to transfer across modalities. Embedding models solve this by mapping heterogeneous data into a shared vector space where proximity means semantic similarity, enabling fast, scalable comparison, retrieval and grouping by meaning.
Components
Usually a Transformer encoder (e.g. BERT) producing a context-aware vector for each token across the whole sequence.
Official
Typically mean pooling (attention-masked average), CLS-token pooling or last-token pooling.
Official
L2 normalization of the output vector, sometimes with an extra linear layer or Matryoshka (MRL) support for dimension truncation.
Official
Usually an InfoNCE or triplet loss with hard negatives; often multi-stage (contrastive pre-training plus supervised pair fine-tuning).
Official
Implementation
If vectors are not L2-normalized, the dot product does not equal cosine similarity, which skews ranking.
Many models (E5, BGE, EmbeddingGemma) require distinct query vs document prefixes; omitting them degrades retrieval quality.
After truncating a vector to a smaller dimension (MRL) it must be re-normalized, otherwise similarities are distorted.
Using CLS pooling for a model trained with mean pooling (or vice versa) yields poor representations.
Evolution
Mikolov et al. introduce efficient learning of dense, distributed word representations via a dedicated network model.
Pennington, Socher and Manning (Stanford) propose global word vectors based on co-occurrence statistics.
A siamese BERT model with pooling produces comparable sentence vectors, making semantic comparison fast and scalable.
Microsoft introduces general text embeddings via weakly-supervised contrastive pre-training.
BAAI releases open embedding resources and models (BGE) reaching top retrieval-benchmark results.
OpenAI ships text-embedding-3-small (1536-dim) and -large (3072-dim) with support for shortening the dimension.
An open, multilingual (100+ languages) ~308M-parameter embedding model, 768-dim with Matryoshka (512/256/128) and a 2048-token context, optimized for on-device use.
Hyperparameters (configurable axes)
Length of the output vector; trades off quality, memory cost and search speed. Matryoshka models allow truncating the dimension.
Maximum number of input tokens the model can encode into a single vector.
How token vectors are aggregated into one vector (mean, CLS, last-token).
Measure used to compare vectors; most often cosine similarity or dot product.
Computational complexity
Time complexity: O(n^2 * d). Space complexity: O(n^2 + n * d).
Execution paradigm
The encoder activates all parameters for every input; no sparse routing (unlike MoE).
Parallelism
A Transformer encoder processes all tokens in parallel; encoding many documents is embarrassingly parallel.
Hardware requirements
The Transformer encoder's dense matrix multiplications and large-corpus encoding benefit from tensor cores.
Small models (e.g. all-MiniLM, EmbeddingGemma) encode single queries with acceptable latency on CPU, including on-device/edge.