std1.0.0-beta-7 SafeC docs

std :: ml::attention_advanced

Function std::windowed_attention_forward

#include <std/ml/attention_advanced.h> source · definition

&FloatTensor windowed_attention_forward(
    const &FloatTensor Q,
    const &FloatTensor K,
    const &FloatTensor V,
    unsigned long windowSize,
    unsigned long shift,
)

Swin's two core ideas — partition into fixed-size local windows and attend only within a window, then cyclically shift the partition so the next layer's windows straddle this layer's boundaries — carried over to this library's 1D [seqLen, dim] sequence shape rather than Swin's original 2D image-patch grid (this library has no 2D spatial tensor type; see cnn.h's FeatureMap for the closest thing, which this doesn't use). 'seqLen' must be divisible by 'windowSize'. 'shift' (0 for the unshifted variant, typically windowSize/2 for the shifted one) cyclically rotates rows before windowing and rotates the result back after, so windows at the boundary contain rows from both original neighbors on the next layer.