Struct std::AdamState

#include <std/ml/tensor_nn.h> source

struct AdamState {
    &heap float m;
    &heap float v;
    unsigned long size;
    float lr;
    float beta1;
    float beta2;
    float eps;
    unsigned long t;
}

Per-parameter state (first/second moment buffers, same size as the parameter) plus the shared hyperparameters and step count. One AdamState per parameter tensor (W1, W2, ... each get their own) -- unlike sgd_update (a pure function of data/grad/lr with no persistent state), Adam needs m/v carried across steps.

Fields

§&heap float msource

§&heap float vsource

§unsigned long sizesource

§float lrsource

§float beta1source

§float beta2source

§float epssource

§unsigned long tsource

step count, starts at 0, incremented by AdamState::step

Methods

§void step(&FloatTensor param)source · definition

§void drop()source · definition

§void free() consumesource · definition

= drop()

§

static struct AdamState new(
    unsigned long size,
    float lr,
    float beta1,
    float beta2,
    float eps,
)

source · definition

§static struct AdamState new_default(unsigned long size, float lr)source · definition

Convenience default (lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8 — the values from the original Adam paper and every framework's default).