std :: ml::convnet
Struct std::ConvNet
struct ConvNet {
struct Vec<struct ConvLayer> layers;
struct Vec<&FloatTensor> params;
struct Vec<&FloatTensor> buffers;
struct Vec<struct Rng> rngs;
unsigned long slots;
int training;
unsigned long inC;
unsigned long inH;
unsigned long inW;
unsigned long outC;
unsigned long outH;
unsigned long outW;
}Start with the input geometry, then append layers; each one's input is the previous one's output. Weights are initialized like PyTorch's: U(-1/sqrt(fanIn), 1/sqrt(fanIn)) for weights and biases; the norms start at gamma 1, beta 0. params holds every layer's trained tensors, all requiring grad, on the heap; buffers holds what is not trained (batch norms' running statistics, spectral norms' u). 'training' (1 from ConvNet::new) picks batch statistics in batch norm and updates the running ones, iterates spectral norms and applies dropout; set it to 0 for inference with ConvNet::set_training. The models in autoencoder.h and gan.h do that in their inference functions (encode / decode / sample / generate / score).
struct ConvNet net = ConvNet::new(1UL, 28UL, 28UL);
net.conv(32UL, 3UL, 1UL, 1UL, ACT_NONE, &rng); // 32 x 28 x 28
net.batchnorm(ACT_RELU);
net.residual_block(64UL, 2UL, 1, ACT_RELU, &rng); // 64 x 14 x 14
net.global_avgpool(); // 64 x 1 x 1
net.dense(10UL, ACT_NONE, &rng); // logits
&Tensor logits = net.forward(batch); // [m, 784] -> [m, 10]
Fields
not trained by the optimizer
one per dropout layer
saved activations (CONV_LAYER_SAVE)
the last layer's output geometry
Methods
void conv(
unsigned long outC,
unsigned long kernel,
unsigned long stride,
unsigned long padding,
int act,
&Rng rng,
)source · definitionvoid deconv(
unsigned long outC,
unsigned long kernel,
unsigned long stride,
unsigned long padding,
int act,
&Rng rng,
)source · definitionvoid avgpool(
unsigned long kernel,
unsigned long stride,
unsigned long padding,
int countIncludePad,
)source · definition-> C x 1 x 1
nearest, integer factor
Norms over the current output's channels, then 'act': place them between a convolution (ACT_NONE) and its activation. Batch norm: momentum 0.1, eps 1e-5. Group norm: eps 1e-5; layer norm is 1 group, instance norm one group per channel.
Spectral normalization of the last CONV / DECONV / DENSE layer's weight (one power iteration per training forward pass): the usual way to keep a GAN critic Lipschitz.
Dropout with probability p (byChannel: whole planes); its random stream is seeded from rng.
channelsheightwidth must equal the current output size.
Skips. ConvNet::save keeps the current activation and returns its slot. ConvNet::add_skip adds a slot to the current output, then 'act': through a learned 1x1 convolution (stride = the size ratio) when the channels or size differ -- ResNet's projection shortcut. ConvNet::concat_skip appends a slot's channels (same height and width) to the current output -- a U-Net skip. ConvNet::residual_block is ResNet's basic block: conv 3x3 (stride) -> [batch norm] -> act -> conv 3x3 -> [batch norm] -> + shortcut -> act. Output side (dim - 1)/stride + 1.
void residual_block(
unsigned long outC,
unsigned long stride,
int batchNorm,
int act,
&Rng rng,
)source · definitionAn activation on the last layer's output (replaces the one it had).
Training (1) or inference (0) mode; returns the previous mode.
For inference: folds every batch norm that directly follows a CONV / DECONV / DENSE layer (no activation or spectral norm between) into that layer's weight and bias, using the running statistics, and turns the batch norm into a no-op. Inference output is unchanged; do not train the net afterwards. Returns how many were folded.
inCinHinW
outCoutHoutW
[batch, in_dim] -> [batch, out_dim]
Same layers, a copy of the parameters and buffers (e.g. a GAN's averaged generator).
Parameterized (module.h): the trained parameters; the state adds the buffers (running statistics, spectral norms' u), so model_save_file / model_copy carry them too.
= drop()
static struct ConvNet new(
unsigned long channels,
unsigned long height,
unsigned long width,
)source · definition