Module ml::convnet
trainable convolutional networks on std::ml's Autograd::get core.
cnn.h runs convolutions forward on FeatureMaps; this file makes them differentiable on Tensors so they train with Tensor::backward and the optimizers in nn.h, alongside Mlp. Images travel as rows, the same as every other model in std::ml: a batch is a [batch, CHW] tensor, each row one image in CHW order (row[cHW + h*W + w]). The geometry (C, H, W) is passed to every op explicitly, since a 2D tensor does not carry it.
* tensor_conv2d / tensor_conv2d_ex — convolution with stride, zero
padding, dilation and groups (depthwise when groups == channels),
weight [outC, (inC/groups)*kH*kW] (a PyTorch Conv2d weight,
flattened), bias [outC].
* tensor_conv_transpose2d / _ex — its transpose (the gradient of conv2d
with respect to its input, run forward), weight [inC,
(outC/groups)*kH*kW] (a PyTorch ConvTranspose2d weight, flattened),
bias [outC], with output padding to reach every output size. The
usual learned upsampler of decoders and generators.
* Pooling: max / avg with any window, stride, padding and ceil mode;
global average / max.
* Resizing: nearest (integer factor, fast path), and nearest / bilinear
/ bicubic to any size, with or without aligned corners.
* Normalization: batch norm with running statistics; group norm (layer
norm and instance norm as its two ends); spectral norm of a weight.
* Dropout, elementwise or by whole channels.
* ConvNet — a stack of those layers plus fully connected ones, with
saved activations for residual (add) and U-Net (concat) skips, built
layer by layer and parallel to Mlp. autoencoder.h (ConvAutoencoder,
ConvVae) and gan.h (Dcgan) take ConvNets as their encoders,
decoders, generators and discriminators.
Output sides along each axis: conv2d (dim + 2pad - dil(k-1) - 1)/stride
- 1 (rounded down, as PyTorch); conv_transpose2d (dim - 1)stride - 2pad
- dil*(k-1) + outPad + 1; pooling as conv2d with dilation 1, rounded up in ceil mode. Both convolutions compute through im2col + sgemm (3x3 stride-1 forward passes with 8+ channels through cnn.h's Winograd kernel); a large enough batch is split across std/sync/parallel.h's thread pool, one chunk of images per thread, with results independent of scheduling. Pooling, resizing and the norms split large inputs across the pool by plane, row or channel the same way.
Structs
| Conv2dSpec | Every field per axis; Conv2dSpec::new fills a square kernel with dilation 1, 1 group and no output padding. inC and outC must divide by groups; outPad (transposed convolution only) must be below stride or dilation. |
| Pool2dSpec | Windows are clipped to the plane: max pooling ignores the padding (as PyTorch's MaxPool2d does) and sends each window's gradient to its (first) largest input. Avg pooling with countIncludePad divides by the window's extent inside the padded plane (PyTorch's AvgPool2d default), else by the positions inside the plane. padH / padW at most half the window. ceilMode rounds the output side up, as PyTorch's ceil_mode (a last window must still start inside the input or its left padding). |
| ConvLayer | |
| ConvNet | Start with the input geometry, then append layers; each one's input is the previous one's output. Weights are initialized like PyTorch's: U(-1/sqrt(fanIn), 1/sqrt(fanIn)) for weights and biases; the norms start at gamma 1, beta 0. params holds every layer's trained tensors, all requiring grad, on the heap; buffers holds what is not trained (batch norms' running statistics, spectral norms' u). 'training' (1 from ConvNet::new) picks batch statistics in batch norm and updates the running ones, iterates spectral norms and applies dropout; set it to 0 for inference with ConvNet::set_training. The models in autoencoder.h and gan.h do that in their inference functions (encode / decode / sample / generate / score). |
Functions
| tensor_conv2d_ex | x is [batch, inCinHinW] (a 1D tensor is one image). Each returns [batch, outCoutHoutW] and records its backward edge to x, weight and bias, like tensor_add does. outC is bias's size. |
| tensor_conv_transpose2d_ex | |
| tensor_conv2d | The common case: kH x kW kernel, one stride and padding for both axes. |
| tensor_conv_transpose2d | |
| conv_out_side | Output side along one axis. |
| conv_transpose_out_side | |
| conv_out_side_ex | |
| conv_transpose_out_side_ex | |
| pool_out_side | |
| tensor_batch_norm | Batch normalization (Ioffe & Szegedy 2015) over x [batch, channelsplane]: each channel is normalized over the batch and its plane (plane = HW for feature maps, 1 for [batch, features] rows), then scaled by gamma and shifted by beta ([channels] each, trained). When 'training', the batch's statistics are used and runMean / runVar ([channels], no gradient) move toward them by 'momentum' (PyTorch: 0.1, with the unbiased variance); otherwise the running statistics are used. eps is added to the variance. |
| tensor_group_norm | Group normalization (Wu & He 2018): each image's channels split into 'groups' groups, each normalized over its channels and planes, then gamma / beta per channel. Batch-independent, the same in training and inference. groups 1 is layer norm over (C, H, W); groups == channels is instance norm. |
| tensor_spectral_norm | Spectral normalization (Miyato et al. 2018): weight / sigma, sigma its largest singular value (the weight seen as a matrix [rows, size/rows]), estimated by power iteration from u ([rows], no gradient). In training 'iters' iterations update u; in inference u is kept. The gradient treats u and v as constants, as PyTorch does. |
| tensor_append_label_channels |
Macros
| RESIZE_NEAREST | To any outH x outW (up or down), as PyTorch's interpolate(size=...): |
| RESIZE_BILINEAR | |
| RESIZE_BICUBIC | |
| CONV_LAYER_CONV | |
| CONV_LAYER_DECONV | transposed convolution |
| CONV_LAYER_MAXPOOL | |
| CONV_LAYER_AVGPOOL | |
| CONV_LAYER_UPSAMPLE | nearest, by 'factor' |
| CONV_LAYER_DENSE | fully connected over the flattened map -> outC x 1 x 1 |
| CONV_LAYER_RESHAPE | reinterpret the geometry (same size), no data change |
| CONV_LAYER_BATCHNORM | |
| CONV_LAYER_RESIZE | Tensor::resize2d to outH x outW ('mode', 'flag' = alignCorners) |
| CONV_LAYER_GLOBAL_AVGPOOL | -> C x 1 x 1 |
| CONV_LAYER_GLOBAL_MAXPOOL | |
| CONV_LAYER_GROUPNORM | 'factor' groups |
| CONV_LAYER_DROPOUT | 'prob', 'flag' = by channel |
| CONV_LAYER_SAVE | keep the activation in slot 'slot' |
| CONV_LAYER_ADD |
|
| CONV_LAYER_CONCAT | channel concat with slot 'slot' |
| CONV_LAYER_IDENTITY | no-op (a folded batch norm) |