Scaled Dot-Product Attention
How it works
The model projects inputs into three matrices: Q (queries), K (keys) and V (values). It computes similarities QK^T, scales them by √d_k, normalises with softmax across key positions, and multiplies by V to obtain a weighted sum of values for each query.
Problem solved
It computes attention quickly and in parallel as matrix operations, avoiding RNN sequentiality and costly MLP-based scoring.
Components
Representations of positions for which matches are sought.
Representations of positions against which similarity is measured.
Values aggregated by attention weights.
Computational complexity
Time complexity: O(n² · d). Space complexity: O(n²).
Execution paradigm
Parallelism
Within a layer, all positions can be processed in parallel as matrix operations.
Hardware requirements
Dominated by matrix multiplications QK^T and AV, which map well to GPUs/Tensor Cores.