std1.0.0-beta-7 SafeC docs

std :: ml::gpu_cuda

Function std::cuda_mp_adam_many

#include <std/ml/gpu_cuda.h> source · definition

int cuda_mp_adam_many(
    const &Vec<struct CudaBuf> w,
    const &Vec<struct CudaBuf> m,
    const &Vec<struct CudaBuf> v,
    const &Vec<struct CudaBuf> g,
    const &Vec<struct CudaHalfBuf> wHalf,
    float lr,
    float beta1,
    float beta2,
    float eps,
    float weightDecay,
    int decoupled,
    unsigned long step,
    struct CudaBuf skip,
)

cuda_mp_adam on every tensor of a model, 24 tensors per launch (w[i], m[i], v[i], g[i] of w[i].len values; wHalf empty for no 16-bit copies, else one per tensor, all of one dtype). The same arithmetic per value; one launch instead of one per tensor matters for models with many small tensors (biases, norms): GPT-2 small's 148 tensors take 1.11 instead of 1.84 ms on an H200 (fused torch.optim.AdamW: 1.37 ms, 1.61 with the bf16 copy).