Module ml::lora

parameter-efficient fine-tuning: LoRA and QLoRA.

LoRA (Hu et al. 2021) freezes a pretrained network and learns a low-rank update for each linear layer:

y = x W + b + (alpha / r) * (dropout(x) A) B

with A [in, r] random (Kaiming-uniform) and B [r, out] zero, so the adapted network starts exactly equal to the base. Only A and B train: r * (in + out) values per layer instead of in * out. LoraMlp::merge folds the update into W for deployment (no extra cost at inference).

QLoRA (Dettmers et al. 2023) does the same on top of a base whose weights are stored in 4-bit NormalFloat (NF4): each block of 64 weights keeps one absmax scale and a 4-bit index into the 16 NF4 levels (the quantiles of a standard normal, so normally distributed weights use every level evenly). Double quantization also stores those scales in 8 bits. Weights are dequantized to float32 per forward pass, so the frozen base costs ~4.1-4.5 bits per weight at rest while gradients flow in float32 through it to the adapters.

Both wrap an Mlp (nn.h) and are Modules (module.h), so they train like any other model:

struct LoraMlp ft = LoraMlp::attach(&pretrained, &cfg);
model_add_params(&ft, &opt);       // adapters only
module_fit(&ft, TRAIN_LOSS_CE, &opt, &data, &tc);