04·GAN training issues: mode collapse, instability, and stabilisation techniques

Introduction

GAN training is famously brittle. This lesson catalogues the main pathologies: mode collapse (G produces only a few modes of p_data), oscillations and non-convergence (a non-zero-sum game has no fixed-point guarantees), vanishing gradients on disjoint supports (the motivation for WGAN), discriminator overconfidence, and memorisation. It then systematises remedies: feature matching and minibatch discrimination (Salimans et al. 2016), spectral normalisation (Miyato et al. 2018), gradient penalty (WGAN-GP, Gulrajani et al. 2017), TTUR (Heusel et al. 2017), R1/R2 regularisation (Mescheder et al. 2018), exponential moving average of G weights, and adaptive data augmentation (StyleGAN-ADA, Karras et al. 2020).