std :: ml::attention
Struct std::MultiHeadAttention
struct MultiHeadAttention {
&FloatTensor Wq;
&FloatTensor Wk;
&FloatTensor Wv;
&FloatTensor Wo;
unsigned long numHeads;
int causal;
unsigned long window;
}The projections around mha_forward as one layer:
out = mha_forward_mask(X Wq, X Wk, X Wv, numHeads, causal, window) Wo
X: [seqLen, dModel]. The weights require grad (a Module: module.h).
Fields
[dModel, dModel]
0 from new(); 1 for a decoder (each token sees itself and earlier ones)
0 from new(); > 0 a sliding window
Methods
[seqLen, dModel] -> [seqLen, dModel]
Parameterized (module.h): Wq, Wk, Wv, Wo.
= drop()
§
static struct MultiHeadAttention new(
unsigned long dModel,
unsigned long numHeads,
&Rng rng,
)source · definitionWeights U(-1/sqrt(dModel), 1/sqrt(dModel)) from rng; numHeads divides dModel.