std :: ml::gpu_cuda
Function std::cuda_mp_adam_many
int cuda_mp_adam_many(
const &Vec<struct CudaBuf> w,
const &Vec<struct CudaBuf> m,
const &Vec<struct CudaBuf> v,
const &Vec<struct CudaBuf> g,
const &Vec<struct CudaHalfBuf> wHalf,
float lr,
float beta1,
float beta2,
float eps,
float weightDecay,
int decoupled,
unsigned long step,
struct CudaBuf skip,
)cuda_mp_adam on every tensor of a model, 24 tensors per launch (w[i], m[i], v[i], g[i] of w[i].len values; wHalf empty for no 16-bit copies, else one per tensor, all of one dtype). The same arithmetic per value; one launch instead of one per tensor matters for models with many small tensors (biases, norms): GPT-2 small's 148 tensors take 1.11 instead of 1.84 ms on an H200 (fused torch.optim.AdamW: 1.37 ms, 1.61 with the bf16 copy).