24 Generative Models and GANs

Learning Data Structure

The primary goal of generative modelling is to understand the underlying structure of high-dimensional data, such as images

• Data as Variables: An image is treated as a high-dimensional random variable $\mathbf{x}\in {\mathbb{R}}^{d\times d}$
• Complexity: Image follows a “complicated” distribution $p(\mathbf{x})$. Because most random configurations of pixels look like noise, we aim to learn a compressed representation that captures the meaningful structure of the data.
• The Generative Task: Use a neural network to approximate the true distribution $p(x)\approx {\hat{p}}_{\theta }(\mathbf{x})$. Once learned, we can sample (generate) new data points: $\mathbf{x}\sim {\hat{p}}_{\theta }(\mathbf{x})$

Density Estimation vs. Sampling

• Density Estimation: Fitting a probabilistic model $p_{\theta }(\mathbf{x})$ to data to learn its parameters $\theta$ such that $p_{\theta }(\mathbf{x})\approx p_{\text{data}}(\mathbf{x})$
• Sampling: Training a system that allows us to generate new samples that look like they came from the training set.
• Challenges: Data is extremely high-dimensional and dimensions are correlated in complex, non-linear ways ($p(\mathbf{x})\ne \prod p(x_i)$)

Generative Adversarial Networks (GANs)

Introduced by Goodfellow et al. (2014), GANs use a game-theoretic approach to generative modelling.

Architecture

The system consists of two networks competing against each other.

1. The Generator $(G_{\theta })$: A “counterfeiter” that learns to map a latent noise vector $\mathbf{z}$ (from a simple prior distribution like $N(0,I)$ to the data space).
   • Goal: Create “fake” samples that are indistinguishable from real data.
2. The Discriminator $(D_{\varphi })$: A “detective” or binary classifier.
   • Goal: Predict the probability that an input $\mathbf{x}$ is “real” (from the training set) rather than “fake” (from the generator).
   • $D_{\varphi }(\mathbf{x})=1$: Certain the input is real
   • $D_{\varphi }(\mathbf{x})=1$: Certain the input is fake

The Min-Max Game

The training is Two-player Min-Max game. The objective function $J_{\text{GAN}}(\theta ,\varphi )$ is defined as:

$$\underset{\theta }{\min }\underset{\varphi }{\max }V(D_{\varphi },G_{\theta })={\mathbb{E}}_{\mathbf{x}\sim p_{data}(\mathbf{x})}[\log D_{\varphi }(\mathbf{x})]+{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[\log (1-D_{\varphi }(G_{\theta }(\mathbf{z})))]$$

• Discriminator Objective: Maximise the probability of assigning the correct label to both real and fake samples.
• Generator Objective: Minimise the probability that the Discriminator identifies its samples as fake.

Training GANs: Theory vs. Practice

Theoretical Algorithm

1. Train $D$ for $K$ iterations (usually $K$ = 1) to maximise the log-likelihood of correctly classifying real vs. fake.
2. Train $G$ for $1$ iteration to maximise the probability that $D$ correctly identifies fake samples.

Practical Generator Loss

In early training, $G$ is often very poor, and $D$ can reject fake samples with high confidence. This leads to vanishing gradients for theoretical loss.

• Theoretical Loss: ${\min }_{\theta }{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[\log (1-D_{\varphi }(G_{\theta }(\mathbf{z})))]$
• Practical Loss: ${\min }_{\theta }{\mathbb{E}}_{\mathbf{z}\sim p(\mathbf{z})}[-\log D_{\varphi }(G_{\theta }(\mathbf{z}))]$
  • This provides much stronger gradients early in training.

Theoretical Optimality

• Optimal Discriminator $(D^{\star })$: For a fixed Generator, the optimal Discriminator is:

$$D^{\ast }(\mathbf{x})=\frac{p_{data}(\mathbf{x})}{p_{data}(\mathbf{x})+p_g(\mathbf{x})}$$

• Global Optimum: When $p_g(\mathbf{x})=p_{\text{data}}(\mathbf{x})$, the optimal Discriminator is $D^{\star }(\mathbf{x})=0.5$ everywhere, meaning it can no longer distinguish between real and fake.
• Divergence Minimisation: Theoretically, the GAN objective minimised the Jensen-Shannon(JS) Divergence between the data distribution and the model distribution.

Conditional Generative Models (cGANs)

While basic GANs learn the marginal distribution $p(\mathbf{y})$, Conditional GANs learn the posterior distribution $p(\mathbf{y}\mid \mathbf{x})$.

• Application: Image-to-image translation (e.g., turning a sketch into a photo, restoration, or un-cropping).
• Why cGANS over Regression (MSE) ?
  • Regression with Mean Squared Error (MSE) finds the Expected Value ($E[\mathbf{y}\mid \mathbf{x}]$), which acts as a “compromise” between all possible outcomes. this results in blurry images.
  • Conditional models account for uncertainty and can sample from the full distribution, producing sharper, more realistic results.

Evolution of GANs

• GAN (2014): Original multilayer perceptron approach.
• DCGAN (2016): Deep Convolutional GANs. Introduced architectural constraints (e.g., removing pooling, using strided convolutions) to make training stable.
• Allows Latent Space Interpolation: Decoding values of $\mathbf{z}$ between two points results in “smooth” transitions between generated images.
• BigGAN (2019): Scaling GANs for high-fidelity natural image synthesis at high resolutions (e.g., $512\times 512$).