Gram Anchoring
How it works
During the refinement phase, the Gram matrix of the student's normalized patch features (X_S·X_Sᵀ) is computed alongside that of the "Gram teacher" (an earlier checkpoint with better dense properties, X_G·X_Gᵀ). The loss L_Gram = ‖X_S·X_Sᵀ − X_G·X_Gᵀ‖²_F pulls the student's patch-to-patch similarity structure toward the teacher's. This loss is added to the overall refinement objective (alongside DINO, iBOT, and Koleo losses, weighted by w_Gram), computed on global crops, with the Gram teacher periodically refreshed to the current EMA teacher.
Problem solved
During long self-supervised training schedules for vision models, dense feature maps (per-patch representations) degrade even as global features keep improving. This harms dense downstream tasks such as semantic segmentation and depth estimation. Gram anchoring stabilizes the structure of local patch similarities, counteracting this degradation.
Components
A P×P matrix of all pairwise dot products of L2-normalized patch features. It encodes the local similarity structure that is regularized instead of the raw feature vectors.
An earlier iteration of the teacher network exhibiting higher patch-level feature consistency. It provides the target Gram matrix and is periodically (every ~10k iterations) refreshed to the current EMA teacher.
Squared Frobenius norm of the difference between the student and teacher Gram matrices: L_Gram = ‖X_S·X_Sᵀ − X_G·X_Gᵀ‖²_F. Added, weighted by w_Gram, to the overall refinement loss alongside the DINO, iBOT, and Koleo losses.
Implementation
The method's effectiveness depends on anchoring to a checkpoint with genuinely superior dense properties. Anchoring to a degraded or too-late teacher removes the benefit.
The Gram matrix scales as O(P²) with the number of patches, sharply increasing compute and memory cost at high resolution.
Evolution
Meta AI introduces Gram anchoring in DINOv3 to solve the degradation of dense feature maps during long self-supervised training, enabling scaling to 7B parameters and 1.7B images while preserving high-quality dense features.
Hyperparameters (configurable axes)
Weighting coefficient for the L_Gram loss in the overall refinement objective. Controls the strength of anchoring the similarity structure to the Gram teacher.
Number of main-training iterations after which the refinement phase with Gram anchoring is enabled. ~1M iterations in DINOv3.
How often the Gram teacher is refreshed to the current EMA teacher. ~10k iterations in DINOv3.
Optional variant: the Gram teacher receives 2× higher-resolution images and the feature maps are down-sampled (2×2 bicubic) to the student dimension, yielding smoother patch representations.
Computational complexity
Time complexity: O(P² · d) na obraz. Space complexity: O(P²) na obraz.
Parallelism
The Gram matrix computation (matrix multiplication) and Frobenius norm are fully parallel on GPUs; the technique applies only during training.
Hardware requirements
The Gram matrix computation is a dense matrix multiplication that efficiently uses GPU Tensor Cores; DINOv3 was trained on large GPU clusters.