Module ml::nn
trainable networks on std::ml's Autograd::get core.
tensor.h / tensor_nn.h carry the Tensor type, reverse-mode Autograd::get and the activations. This file adds what training a model end to end needs:
* Rng — a small deterministic random generator (splitmix64), so every
initialization, shuffle and sample is reproducible from a seed.
* More differentiable ops: bias add, exp / log / abs, clamp, minimum,
row sums, column concatenation, row gather, log-softmax, detach.
* Losses: MSE, Huber, binary cross-entropy on logits, softmax
cross-entropy on class indices.
* Mlp — a stack of fully connected layers with an activation.
* Optimizer — SGD, momentum, Nesterov, RMSProp, Adagrad, Adam, AdamW
and Lion over a list of parameter tensors, with gradient-norm
clipping.
trainer.h runs the training loop; autoencoder.h, gan.h and rl.h build their models from these pieces. Every model here works on rows: a batch is a [batch, features] tensor. Class labels are tensors of class indices stored as floats ([batch] or [batch, 1]), the same as datasets.h makes.
Structs
| Rng | |
| Mlp | inDim -> hidden (x hiddenLayers) -> outDim, 'act' after every hidden layer and 'outAct' after the last. hiddenLayers 0 is a single linear layer. Weights are [in, out] (transpose a PyTorch nn.Linear's), biases [out], initialized like PyTorch's nn.Linear: U(-1/sqrt(in), 1/sqrt(in)). params holds W0, b0, W1, b1, ... and every one requires grad. |
| Optimizer | Fields are public: change lr (a schedule), betas or weight decay between steps. State buffers are created when a parameter is added, always on the heap even inside a tensor arena. |
Functions
| tensor_add_bias | Shapes are [rows, cols] unless noted; every op records its backward edge like tensor_add does. |
| tensor_minimum | elementwise min |
| tensor_concat_cols | [m,p] ++ [m,q] -> [m,p+q] |
| tensor_gather_rows | out[i] = table[idx[i]]: [K,d] and m indices -> [m,d]. Gradients scatter-add back into the table's rows (embeddings, VQ codebooks). |
| tensor_mse_loss | |
| tensor_huber_loss | 0.5 d^2 for |d| <= delta, else delta (|d| - delta/2). Gradient to pred only. |
| tensor_bce_logits_loss | Binary cross-entropy of sigmoid(logits) against targets in [0, 1], computed from the logits so it never takes log(0). Gradient to logits only. |
| tensor_cross_entropy_loss | Softmax cross-entropy of logits [m, classes] against m class indices, averaged over rows. Gradient to logits only. |
| tensor_accuracy | Fraction of rows whose argmax equals the label. |
| tensors_save | A list of tensors' shapes and values to a file and back: the magic "SCTN", the count, then per tensor its rank, shape and float32 values (all little-endian, 64-bit counts). Loading fills existing tensors and requires the same count and shapes. Return 1 on success, 0 when the file can't be opened, is truncated or doesn't match. |
| tensors_load |
Macros
| ACT_NONE | |
| ACT_RELU | |
| ACT_LEAKY | leaky ReLU, slope 0.2 (the usual GAN choice) |
| ACT_TANH | |
| ACT_SIGMOID | |
| ACT_GELU | |
| ACT_SILU | |
| OPT_SGD | |
| OPT_MOMENTUM | heavy ball, 'momentum' 0.9 |
| OPT_NESTEROV | |
| OPT_RMSPROP | squared-gradient average 'beta2' (0.99) |
| OPT_ADAM | 'beta1' 0.9, 'beta2' 0.999, 'eps' 1e-8 |
| OPT_ADAMW | Adam with decoupled weight decay ('weightDecay' 0.01) |
| OPT_ADAGRAD | per-weight lr / sqrt(sum of squared gradients) (Duchi et al. 2011) |
| OPT_LION | sign of the interpolated momentum (Chen et al. 2023); 'beta1' 0.9, 'beta2' 0.99, decoupled 'weightDecay'; use ~3-10x smaller lr than Adam |