Struct std::EgMla

#include <std/ml/attention_advanced.h> source

struct EgMla {
    &FloatTensor Wq;
    &FloatTensor Wdkv;
    struct Vec<&FloatTensor> Wuk;
    struct Vec<&FloatTensor> Wuv;
    unsigned long headsPerGroup;
}

EG-MLA (Extended-Group MLA): shares one down-projected latent c_kv across 'numGroups' independent up-projection groups (grouped-query attention's KV-sharing idea layered onto MLA's low-rank compression). Group g owns WukGroups[g]/WuvGroups[g] ([dLatent, groupDim], groupDim = dModel/numGroups) and runs 'headsPerGroup' ordinary attention heads over its own up-projected K/V slice; results are concatenated back into one [seqLen, dModel] tensor. WukGroups/WuvGroups hold one tensor per group; their length is the group count.

Fields

§&FloatTensor Wqsource

[dModel, dModel]

§&FloatTensor Wdkvsource

[dModel, dLatent]

§struct Vec<&FloatTensor> Wuksource

per group [dLatent, groupDim]

§struct Vec<&FloatTensor> Wuvsource

§unsigned long headsPerGroupsource

Methods

§&FloatTensor forward(const &FloatTensor X) constsource · definition

[seqLen, dModel] -> [seqLen, dModel]

§void collect_params(&Vec<&FloatTensor> out) constsource · definition

Parameterized (module.h): Wq, Wdkv, then each group's Wuk, Wuv.

§void collect_state(&Vec<&FloatTensor> out) constsource · definition

§void drop()source · definition

§void free() consumesource · definition

= drop()

§

static struct EgMla new(
    unsigned long dModel,
    unsigned long dLatent,
    unsigned long numGroups,
    unsigned long headsPerGroup,
    &Rng rng,
)

source · definition

numGroups groups of headsPerGroup heads; numGroups divides dModel.