Robots Atlas>ROBOTS ATLAS

LPIPS

2018ActivePublished: 10 April 2018Updated: 17 July 2026Published
Key innovation
The discovery that activations of deep convolutional networks (VGG, AlexNet, SqueezeNet) pretrained on ImageNet classification form an excellent feature space for perceptual similarity โ€” far better than any hand-designed metrics. Key is the emergent nature of this property: it holds across many architectures and supervision types (even self-supervised and unsupervised), suggesting that a perceptually meaningful feature hierarchy is a general property of deep visual representations. Training linear weights w on the BAPPS dataset with human judgments further calibrates the metric.
Category
Abstraction level
Building block
Operation level
ApplicationEvaluation (runtime)TrainingSystem
Use cases
Evaluating image quality generated by GANs (StyleGAN, BigGAN, ProGAN)Evaluating diffusion models (DDPM, Stable Diffusion, Imagen, DALL-E) alongside FIDSuper-resolution metric (ESRGAN, Real-ESRGAN, SwinIR)Perceptual loss in generator training (image-to-image translation, style transfer, in-painting)Evaluation of frame interpolation, video super-resolution, and denoisingComputational design optimisation (e.g., Phantom Twist UAV, Northwestern 2026) โ€” minimising LPIPS as a proxy for visual detectabilityAdversarial attacks โ€” generating perceptually minimal perturbations that fool a classifierLossy compression โ€” optimising bitrate vs perceptual quality

How it works

1) Choose the base network: VGG-16 (default, highest quality), AlexNet (faster, mobile), or SqueezeNet (lightest). 2) Outputs of selected convolutional layers of the network (typically 5 layers for VGG) are extracted for both compared images. 3) Activations are unit-normalised per channel per spatial position. 4) Element-wise squared L2 difference of activations is computed. 5) For each layer, differences are multiplied by a learned weight vector w (per-channel, trained on BAPPS). 6) Per-layer differences are averaged over spatial positions (spatial mean). 7) The sum across all layers gives the final LPIPS. Value range: 0 (identical) to ~1 (very different).

Problem solved

Classical image quality metrics such as PSNR and SSIM are simple, shallow, hand-designed functions for classical distortions (JPEG compression, Gaussian noise, blur). They cannot capture human perceptual similarity for modern artifacts (GAN outputs, super-resolution, style transfer) โ€” two images perceptually identical to a human can have dramatically different PSNR. An objective, reproducible similarity metric calibrated on actual human perception and operating across a broad spectrum of distortions was missing.

Components

Backbone CNN (VGG-16 / AlexNet / SqueezeNet)Extract hierarchical visual features from images. VGG-16 (default) used for the highest correlation with human perception; SqueezeNet for the lightweight version.
Activation unit normalizationPer-channel per-position L2 normalisation of activations โ€” removes the influence of absolute activation scale, isolates directional information.
Learned linear weights (w)Per-channel per-layer weight vector optimised on BAPPS. Calibrates each feature's contribution to the final metric, aligning it with human perceptual judgments.
L2 distance in feature spaceElement-wise squared difference weighted by w, averaged over spatial positions, summed across layers โ€” final LPIPS scalar.
BAPPS dataset (Berkeley-Adobe Perceptual Patch Similarity)Dataset with triplet judgments ("which of these two patches is more similar to the reference?") collected from humans. Used to train weights w and validate the metric.

Implementation

Reference implementations
richzhang/PerceptualSimilarity (reference implementation)
TorchMetrics (Lightning AI)
kornia (PyTorch computer vision)
PIQ (PyTorch Image Quality)
Implementation pitfalls
High
Fix:LPIPS measures perceptual similarity โ€” it does not replace semantic metrics (FID, IS) for generative content. Always report alongside FID and others. High LPIPS can be caused by semantically different content even for perceptually realistic images.
High
Fix:LPIPS with VGG-16 gives different numbers than AlexNet or SqueezeNet โ€” do not compare directly. Always report the backbone in publications ("LPIPS-VGG", "LPIPS-Alex").
Medium
Fix:The backbone expects input in the [-1, 1] range (ImageNet mean-std normalization). Feeding images in [0, 1] or [0, 255] yields useless results. Many users forget the preprocessing.
Medium
Fix:When used as a loss, the model may learn to exploit backbone idiosyncrasies ("VGG artifacts") instead of generating perceptually better images. Combine with L1/L2, adversarial loss, and other terms. Regularisation is critical.
Low
Fix:LPIPS-VGG is ~1000x more expensive than SSIM (full CNN forward pass). For real-time (video processing), use SqueezeNet-LPIPS or cache reference image activations.

Hyperparameters (configurable axes)

Backbone choice (VGG-16 / AlexNet / SqueezeNet)High

VGG-16: best correlation with human ranking, slower (~5-10x slower than SqueezeNet). AlexNet: compromise. SqueezeNet: lightest, for mobile and real-time applications.

Weights version (linearly trained vs vanilla feature distance)Medium

"Lin" โ€” weights w trained on BAPPS (recommended). "Lin-tune" โ€” additionally fine-tuned backbone. Version without weights (raw features) works but weaker.

Input resolutionMedium

Typically 64x64 or 224x224 (ImageNet default). Higher resolution increases memory and time O(n^2), but does not always improve correlation with human perception โ€” high CNN layers have large receptive fields anyway.

Training as perceptual loss vs read-only metricHigh

Using LPIPS as a loss term requires differentiability of the whole pipeline (gradients through the CNN backbone). Using only as an evaluation metric can be no-grad (faster).

Color / grayscale inputLow

The backbone expects 3 RGB channels. For grayscale, channel replication suffices. Value range [-1, 1] (ImageNet input normalisation).

Execution paradigm

Primary mode
Dense

LPIPS activates the full CNN forward pass for both images โ€” all layers, all channels, all spatial positions. No skip-layers or sparse computation.

Activation pattern
All paths active

Parallelism

Parallelism level
Fully parallel

LPIPS computation is fully parallelisable. CNN forward pass for a batch of image pairs โ€” identical to classification inference. Layer-wise reduction is independent โ€” each layer can be computed independently. Linear scaling with batch size until the GPU is saturated.

Scope
InferenceAcross devices

Hardware requirements

Primary
Possible
Good fit