The Newton-Schulz iteration is equivalent to extracting the left and right singular vectors (UVᵀ) from the update's SVD and discarding the singular values, which amplifies underrepresented directions. Overhead is only ~0.5–0.7% of typical LM training compute and it operates stably in bfloat16.
Momentum-based updates typically have a high condition number (dominated by a few directions), leaving important learning directions underrepresented.
