Module ml::gan
generative adversarial networks.
An MLP generator G maps latent noise z ~ N(0, I) to data rows; an MLP discriminator (critic) D scores rows as real or generated. They train in alternation: D on a real and a generated batch, then G through D. The objective is picked in the config:
GAN_STANDARD Goodfellow et al. 2014, with the non-saturating generator
loss: D minimizes BCE(D(x), 1) + BCE(D(G(z)), 0), G
minimizes BCE(D(G(z)), 1).
GAN_LSGAN least squares (Mao et al. 2017): D minimizes
(D(x) - 1)^2 / 2 + D(G(z))^2 / 2, G (D(G(z)) - 1)^2 / 2.
GAN_WGAN Wasserstein (Arjovsky et al. 2017): the critic maximizes
D(x) - D(G(z)) with its weights clipped to [-clip, clip],
'nCritic' critic steps per generator step, RMSProp.
GAN_HINGE hinge loss (Miyato et al. 2018): D minimizes
relu(1 - D(x)) + relu(1 + D(G(z))), G -D(G(z)).
GAN_WGAN_GP Wasserstein with gradient penalty (Gulrajani et al.
2017): the critic minimizes D(G(z)) - D(x) + gpWeight *
(||grad_x D(x^)|| - 1)^2 on random interpolates x^ of
the real and generated rows, no clipping, Adam with
betas (0, 0.9); use nCritic 5. The penalty's parameter
gradient is a mixed second derivative; it is computed as
a central difference of the critic along each row's
normalized input gradient (two extra critic passes, exact
to O(eps^2), eps 1e-2) rather than by differentiating the
backward pass. The critic must treat rows independently:
no batch norm in it (group / layer norm are fine).
classes > 0 makes it a conditional GAN (Mirza & Osindero 2014): the one-hot label is appended to G's noise and to D's input, and G learns the class-conditional distribution.
emaDecay > 0 keeps an exponential moving average of the generator's weights (Karras et al. 2018) and Gan::sample samples from it: the adversarial game oscillates, and the average sits nearer its centre than any single step's weights. The decay ramps up as min(emaDecay, (1 + t) / (10 + t)) over the first steps t.
The GAN owns both optimizers (Adam, beta1 0.5, as DCGAN recommends; RMSProp for GAN_WGAN) and a tensor arena for its steps.
Dcgan (below) is the same game with convolutional networks from convnet.h (Radford et al. 2016).
Structs
| GanConfig | |
| Gan | |
| GanLosses | |
| Dcgan | Gan with a ConvNet generator and discriminator you build (convnet.h). G takes the latent batch as a latent x 1 x 1 input and must output the data's size; D takes a data row ([CHW], CHW order) and must output one score. Reads latent, loss, lrG, lrD, nCritic, clip, classes, emaDecay and seed from the config (hidden, hiddenLayers, genAct and outAct describe the nets you built and are ignored). classes > 0 makes it a conditional DCGAN: G gets the one-hot label appended to its noise (latent + classes inputs), D gets it as 'classes' extra input channels, one constant plane per class (tensor_append_label_channels) -- so for C-channel images D takes C + classes channels. Takes ownership of both nets. Training steps run the nets in training mode (batch norm on batch statistics); Dcgan::sample and Dcgan::score in inference mode. |
Traits
| GanModel | What a trainable GAN offers (Gan and Dcgan; CpuGanTrainer is generic over it): its two training entry points, its tensors and sampling. |
Macros
| GAN_STANDARD | |
| GAN_LSGAN | |
| GAN_WGAN | |
| GAN_HINGE | |
| GAN_WGAN_GP |