std1.0.0-beta-7 SafeC docs

std :: ml::gpu_cuda_attention

Function std::cuda_linear_attention

#include <std/ml/gpu_cuda_attention.h> source · definition

int cuda_linear_attention(
    struct CudaBuf q,
    struct CudaBuf k,
    struct CudaBuf v,
    struct CudaBuf out,
    unsigned long heads,
    unsigned long seqLen,
    unsigned long kvLen,
    unsigned long headDim,
    int causal,
)

out_i = phi(q_i) S / phi(q_i) z with S = sum_j phi(k_j) v_j^T and z = sum_j phi(k_j) over all keys (causal = 0) or over j <= pos (causal = 1, which needs kvLen >= seqLen). phi(x) = elu(x) + 1; no scale, since the division cancels it. The same result as attention.h's linear_attention_forward per head. s.window must be 0.