Function std::cuda_attention_use_decode

#include <std/ml/gpu_cuda_attention.h> source · definition

int cuda_attention_use_decode(int on)

Flash attention with few query rows per kv head ((heads / kvHeads) * seqLen <= 16, e.g. decoding one token against a KV cache) runs a split-KV ("flash-decoding") kernel pair in any dtype, ahead of cuDNN: the keys are spread over every SM and the partial softmaxes merged. It sizes a workspace on its first call of a shape (make that call before capturing a graph). on = 0 turns it off; returns the setting.