Function std::cuda_mp_sgd

int cuda_mp_sgd(
    struct CudaBuf w,
    struct CudaBuf g,
    struct CudaBuf vel,
    struct CudaHalfBuf wHalf,
    float lr,
    float momentum,
    float weightDecay,
    struct CudaBuf skip,
)

Optimizer steps on fp32 master weights w with gradient g, as nn.h's Optimizer::step (OPT_SGD / OPT_MOMENTUM when vel is valid, OPT_ADAM / OPT_ADAMW when decoupled; 'step' is the 1-based step count for the bias correction). The updated weights are also written, rounded, to wHalf when it is valid. Skipped entirely when skip is valid and skip[0] != 0. Pass an invalid buffer (dev == 0) for any optional argument.