std1.0.0-beta-7 SafeC docs

std :: ml::nn

Struct std::Optimizer

#include <std/ml/nn.h> source

struct Optimizer {
    struct Vec<&FloatTensor> params;
    struct Vec<&FloatTensor> m;
    struct Vec<&FloatTensor> v;
    int kind;
    float lr;
    float momentum;
    float beta1;
    float beta2;
    float eps;
    float weightDecay;
    unsigned long t;
}

Fields are public: change lr (a schedule), betas or weight decay between steps. State buffers are created when a parameter is added, always on the heap even inside a tensor arena.

Fields

§struct Vec<&FloatTensor> paramssource
§struct Vec<&FloatTensor> msource

first moment / velocity, one per parameter

§struct Vec<&FloatTensor> vsource

second moment, one per parameter

§int kindsource
§float lrsource
§float momentumsource
§float beta1source
§float beta2source
§float epssource
§float weightDecaysource

L2 for SGD / momentum / RMSProp / Adagrad / Adam, decoupled for AdamW / Lion

§unsigned long tsource

steps taken

Methods

§void add(&FloatTensor param)source · definition
§void zero_grad()source · definition
§float clip_grad_norm(float maxNorm)source · definition

Scales every gradient so their global L2 norm is at most maxNorm and returns the norm before scaling.

§void step()source · definition
§void drop()source · definition

frees the state, not the parameters

§void free() consumesource · definition

= drop()

§static struct Optimizer new(int kind, float lr)source · definition
§static struct Optimizer from(const &Optimizer o)source · definition

An empty optimizer with o's kind and hyperparameters (not its parameters or state).