Function std::device_loss_scale

float device_loss_scale(int device)

The fp16 loss scale on 'device' (waits): gradients are scaled by it during backward and unscaled when backward ends, a step that overflowed is skipped by the optimizer, and the scale halves on overflow and doubles after 2000 clean steps. 1 when the device isn't in fp16.