Ridge
How it works
An α·Σwⱼ² penalty term is added to the least-squares loss. Optimizing this objective yields modified normal equations whose closed-form solution is ŵ = (XᵀX + αI)⁻¹Xᵀy. Adding αI raises a 'ridge' (hence the name) along the diagonal of XᵀX, guaranteeing invertibility and improved conditioning. The parameter α governs the tradeoff: as α→0 the estimator approaches OLS, while large α shrinks coefficients strongly toward zero. Features should be standardized because the penalty is scale-dependent; the intercept is usually left unpenalized. The optimal α is selected via cross-validation (e.g. RidgeCV, generalized cross-validation / GCV).
Problem solved
Ordinary least squares (OLS) produces high-variance estimates when features are strongly correlated (multicollinearity) or when XᵀX is near-singular or non-invertible (e.g. when features outnumber observations), leading to unstable, overfitted models. Ridge addresses this via L2 regularization, which stabilizes the matrix inversion and curbs overfitting.
Components
A parametric linear predictor ŷ = Xw whose coefficients are estimated under regularization.
A regularization term penalizing the squared Euclidean norm of the coefficients, shrinking them toward zero.
Official
Solution of the modified normal equations ŵ = (XᵀX + αI)⁻¹Xᵀy.
Official
Implementation
The L2 penalty is scale-dependent; without standardization, large-scale features are under-regularized.
Including the intercept in the penalty introduces a target-shift-dependent error.
Too-small α fails to curb overfitting; too-large α causes underfitting.
Evolution
Andrey Tikhonov formulates the regularization method for ill-posed problems — the mathematical equivalent of Ridge.
Two Technometrics papers introduce ridge regression in statistics and its applications to nonorthogonal problems.
Robert Tibshirani introduces the Lasso, yielding sparse solutions — a contrast to Ridge's proportional shrinkage.
Zou and Hastie combine L1 and L2 penalties, reconciling Lasso sparsity with Ridge stability.
Hyperparameters (configurable axes)
Non-negative L2 penalty coefficient; controls the bias–variance tradeoff. α→0 approaches OLS, large α strongly shrinks coefficients.
Solution algorithm: SVD/Cholesky decomposition, lsqr, sparse_cg, sag/saga, lbfgs.
Ridge is not scale-invariant; standardization is usually required for meaningful regularization.
Whether to fit and (usually) leave unpenalized the intercept.
Computational complexity
Time complexity: O(n · d²). Space complexity: O(d²).
Compute bottleneck
The main cost is solving the normal-equations system with the regularized d×d matrix.
Execution paradigm
All coefficients are active in every prediction; no routing or conditional activation.
Parallelism
Dense linear-algebra operations (matrix multiply, factorization, dot product at prediction time) parallelize well in BLAS/LAPACK libraries.
Hardware requirements
Built on basic linear algebra, it runs on any hardware.
For small-to-medium datasets, a CPU with BLAS/AVX is fully sufficient.
A GPU accelerates matrix operations on very large data.