std1.0.0-beta-7 SafeC docs

std :: ml::attention

Function std::mha_forward

#include <std/ml/attention.h> source · definition

&FloatTensor mha_forward(
    const &FloatTensor Q,
    const &FloatTensor K,
    const &FloatTensor V,
    unsigned long numHeads,
)

Multi-head attention: splits Q/K/V's dModel columns into 'numHeads' equal headDim-wide slices (dModel must be divisible by numHeads), runs attention_forward() per head, and concatenates the per-head outputs back into one [seqLen, dModel] result — the standard MHA shape from "Attention Is All You Need", without a separate output-projection matrix (apply your own W_O via tensor_matmul on the result if needed).