Struct std::GatedDeltaNet

#include <std/ml/attention_advanced.h> source

struct GatedDeltaNet {
    &FloatTensor Walpha;
    &FloatTensor Wbeta;
    unsigned long keyDim;
    unsigned long valueDim;
}

Linear-recurrent attention replacement (Yang et al. 2024): a [keyDim, valueDim] associative-memory state S is updated one timestep at a time with a delta rule (write the residual between the new value and what the current state already predicts for this key, rather than just accumulating) plus a scalar gate controlling how much of the old state decays away each step:

alpha_t = sigmoid(x_t . w_alpha)                 decay gate, scalar
beta_t  = sigmoid(x_t . w_beta)                   write gate, scalar
pred_t  = S_{t-1}^T @ k_t                         [valueDim]
delta_t = v_t - pred_t                            [valueDim]
S_t     = alpha_t * S_{t-1} + beta_t * outer(k_t, delta_t)   [keyDim, valueDim]
out_t   = S_t^T @ q_t                             [valueDim]

Scope note: alpha_t/beta_t are scalar-per-timestep here (a single gate value applied uniformly across the state), not per-channel as in the full paper — documented simplification, same spirit as this file's other scope cuts.

Fields

§&FloatTensor Walphasource

[dModel, 1]

§&FloatTensor Wbetasource

[dModel, 1]

§unsigned long keyDimsource

§unsigned long valueDimsource

Methods

§

&FloatTensor forward(
    const &FloatTensor X,
    const &FloatTensor Q,
    const &FloatTensor K,
    const &FloatTensor V,
) const

source · definition

X: [seqLen, dModel] (gate input only), Q/K: [seqLen, keyDim], V: [seqLen, valueDim] -> result: [seqLen, valueDim], computed via a sequential scan over time.

§void collect_params(&Vec<&FloatTensor> out) constsource · definition

Parameterized (module.h): every weight tensor.

§void collect_state(&Vec<&FloatTensor> out) constsource · definition

§void drop()source · definition

§void free() consumesource · definition

= drop()

§

static struct GatedDeltaNet new(
    unsigned long dModel,
    unsigned long keyDim,
    unsigned long valueDim,
)

source · definition