Function std::cuda_half_convert

#include <std/ml/gpu_cuda.h> source · definition

int cuda_half_convert(struct CudaHalfBuf src, struct CudaHalfBuf dst)

The usual layout: fp32 master weights and optimizer state, a 16-bit copy of the weights for the forward / backward passes, 16-bit or fp32 gradients with the loss scale in them. One step:

cuda_half_gemm ... (forward / backward in fp16 or bf16)
zero the flag; cuda_half_unscale_many(grads16, grads32, 1 / scale, flag)
cuda_mp_adam(w32, m, v, g32, w16, ..., flag)       // skipped on the GPU if flag != 0
read the flag (4 bytes) to back the loss scale off or grow it

so an overflowed step costs no host round trip before the update. All of these are queued and can be recorded into a CudaGraph.

dst = src converted between fp16 and bf16 (src.len elements).