Training
Muon (optimizer)
2024ActiveUpdated: 28 August 2026Published
Key
innovation
Newton-Schulz orthogonalization of momentum updates for 2D hidden-layer matrices, speeding up neural-network training.
Category
Training
Abstraction level
Building block
Operation level
Architecture blockModel
Use cases
Faster training of neural networks and LLMsSpeed records (NanoGPT speedrun, CIFAR-10)Model training recipes (e.g. Qwen3.8-Flash-Next: Muon + AdamW)Efficient pretraining and fine-tuning
How it works
The Newton-Schulz iteration is equivalent to extracting the left and right singular vectors (UVᵀ) from the update's SVD and discarding the singular values, which amplifies underrepresented directions. Overhead is only ~0.5–0.7% of typical LM training compute and it operates stably in bfloat16.
Problem solved
Momentum-based updates typically have a high condition number (dominated by a few directions), leaving important learning directions underrepresented.