Cross-Attention
How it works
1. The "querying" sequence (e.g. the decoder) is linearly projected to a query matrix Q. 2. The "source" sequence (e.g. the encoder output) is projected to key K and value V matrices. 3. Dot products Q·Kᵀ are computed, scaled by 1/√d_k and passed through softmax, producing attention weights that show how strongly each querying position attends to each source position. 4. These weights multiply the values V, yielding context representations. 5. In practice the operation is multi-head: Q, K, V are split into h heads computed in parallel, then concatenated and projected by an output matrix. Key point: Q has a different source than K and V, the sequences may have different lengths, and no causal mask is applied (unlike the decoder's masked self-attention). During autoregressive decoding the encoder's K and V are computed once and cached across all steps.
Problem solved
How can one sequence (or modality) draw on information contained in another, possibly of a different length? Cross-attention solves conditional fusion of two representations — e.g. aligning a translation with its source sentence, an image with a text caption, or a robot action with a visual observation — without compressing the source sequence into a single vector, which was the bottleneck of earlier encoder-decoder architectures.
Components
A linear projection W_Q applied to the querying sequence. This is what distinguishes cross-attention from self-attention — Q comes from a different stream than K and V.
Projections W_K and W_V applied to the source sequence. In autoregressive decoding they are computed once and cached, since the source does not change.
softmax(Q·Kᵀ / √d_k)·V. No causal mask; an optional padding mask on the source side.
Official
Concatenation of the h heads and a linear projection W_O.
Implementation
Q must come from the querying sequence (decoder) and K, V from the source sequence (encoder). Swapping the sources breaks the conditioning.
Cross-attention uses no causal mask — only a source-side padding mask is needed. Adding a causal mask wrongly restricts access to the input.
In autoregressive decoding the source K and V do not change; recomputing them at every step wastes compute.
Evolution
Bahdanau et al. introduce attention between the decoder and encoder states in machine translation — the decoder attends over the source sequence.
Vaswani et al. define the encoder-decoder attention layer with scaled dot-product attention and multiple heads; Q from the decoder, K and V from the encoder.
Perceiver uses cross-attention to project very large inputs onto a compact latent array, decoupling cost from input length.
Rombach et al. add cross-attention to the UNet to condition image generation on text embeddings — the basis of Stable Diffusion.
Flamingo injects visual representations into a frozen language model through gated cross-attention layers.
Hyperparameters (configurable axes)
Number of parallel attention heads.
Per-head dimension used in the 1/√d_k scaling.
Dimension of the layer's input and output embeddings.
Computational complexity
Time complexity: O(n_q · n_kv · d). Space complexity: O(n_q · n_kv).
Compute bottleneck
The main cost is the dot products between queries and keys plus softmax normalization; with long source sequences it grows linearly with n_kv.
Execution paradigm
Dense attention: every querying position attends to all source positions (except masked padding).
Parallelism
In training all querying positions are computed in parallel. In autoregressive decoding querying positions are produced sequentially, but source K and V are computed once and cached.
Hardware requirements
The operation is dominated by dense matrix multiplications (Q·Kᵀ, ·V), ideal for tensor cores.
Systolic matrix-multiply units handle batched attention products well.
Runs on CPU and other accelerators, but with lower throughput for long source sequences.