Module ml::norm

#include <std/ml/norm.h> source

trainable normalization: LayerNorm and AdaLN.

Both normalize each row of a [batch, dim] tensor to mean 0 and variance 1 and carry reverse-mode autograd (activations.h's Tensor::layernorm_rows is the forward-only version).

LayerNorm (Ba et al. 2016):  y = LN(x) * gamma + beta, learned
  per-feature gamma (init 1) and beta (init 0).
AdaLN (adaptive LayerNorm, as in DiT, Peebles & Xie 2023):
  (shift, scale[, gate]) = cond @ W + b, per sample
  y = LN(x) * (1 + scale) + shift
  A conditioning vector (timestep / class embedding) steers every
  sample's normalization. W and b start at zero ("adaLN-Zero"), so a
  fresh AdaLN is a plain LayerNorm, and with 'gated' the gate starts
  at 0, so a residual branch x + gate * f(y) starts as the identity.

Structs

LayerNorm
AdaLN
AdaLNOut

Functions

tensor_mul_row x [m,n] * g [n] on every row, with Autograd::get.