Function std::rocm_half_convert

#include <std/ml/gpu_rocm.h> source · definition

int rocm_half_convert(struct RocmHalfBuf src, struct RocmHalfBuf dst)

The usual layout: fp32 master weights and optimizer state, a 16-bit copy of the weights for the forward / backward passes, 16-bit or fp32 gradients with the loss scale in them. One step:

rocm_half_gemm ... (forward / backward in fp16 or bf16)
zero the flag; rocm_half_unscale_many(grads16, grads32, 1 / scale, flag)
rocm_mp_adam(w32, m, v, g32, w16, ..., flag)       // skipped on the GPU if flag != 0
read the flag (4 bytes) to back the loss scale off or grow it

so an overflowed step costs no host round trip before the update.

dst = src converted between fp16 and bf16 (src.len elements).