struct Nf4Tensor {
&heap unsigned char codes;
&heap float absmax;
&heap unsigned char absmaxQ;
&heap float absmaxScale;
float absmaxMean;
unsigned long rows;
unsigned long cols;
unsigned long blockSize;
unsigned long blocks;
int doubleQuant;
}Fields
two 4-bit level indices per byte, low nibble first
one per block (when not double-quantized)
int8 + 128 per block (double-quantized)
one per NF4_SCALE_BLOCK blocks
subtracted before the 8-bit quantization
Methods
storage, all parts
= drop()
§
static struct Nf4Tensor quantize(
const &FloatTensor w,
unsigned long blockSize,
int doubleQuant,
)source · definition[rows, cols], no grad