| cuda_available | 1 if a CUDA GPU is usable (initializes the backend on first call). |
| cuda_device_name | Name of the selected GPU ("NVIDIA H200"), or "" if none. |
| cuda_write | |
| cuda_read | |
| cuda_sync | cuda_write returns once 'host' may be reused. Copies of 1 MiB and up run
as 4 parallel lanes (3 helper threads started on first use plus the
caller), each staging its slice through pinned buffers with the DMA of
one chunk overlapping the copy of the next: ~2x the driver's own
pageable-memory path, which is bounded by one core's memcpy rate. |
| cuda_set_tf32 | TF32 tensor cores for every later cuda_gemm / cuda_gemm_relu (sm_80 and
newer): inputs are rounded to a 10-bit mantissa, accumulation stays fp32.
Several times the matmul throughput, ~1e-3 relative error. Off by default.
Returns 1 if the setting was applied. |
| cuda_gemm | c[m,n] = op(a)[m,k] * op(b)[k,n] (cuBLAS sgemm) |
| cuda_gemm_relu | c = relu(op(a) * op(b)) in one kernel (cuBLASLt ReLU epilogue). |
| cuda_add | out = a + b |
| cuda_sub | |
| cuda_mul | |
| cuda_div | |
| cuda_pow | |
| cuda_scale | out = a * k |
| cuda_relu | |
| cuda_exp | |
| cuda_log | |
| cuda_sqrt | |
| cuda_gelu | tanh approximation (gelu_pytorch_tanh) |
| cuda_rmsnorm | RMSNorm per row: y[r] = x[r] / sqrt(mean(x[r]^2) + eps) * w, over 'rows'
rows of 'n' values; w has n values, or dev == 0 for no weight. y may be x. |
| cuda_relu_grad | out = z > 0 ? g : 0. 'z' may be the ReLU input or its output (same sign test). |
| cuda_sgd | w -= lr * g, in place |
| cuda_sgd2 | Both SGD updates of a two-tensor model in one launch: w1 -= lrg1, w2 -= lrg2. |
| cuda_sum | out1[0] = sum(a) |
| cuda_mse_grad | Fused MSE gradient: dy = 2 (y - t) and loss1[0] = sum((y - t)^2), in one pass. |
| cuda_copy | dst[dstOff .. dstOff+n) = src[srcOff .. srcOff+n) (device-to-device copy) |
| cuda_copy_rows | Strided 2D copy, one queued kernel: for r < rows, c < rowLen,
dst[dstOff + rdstStride + c] = src[srcOff + rsrcStride + c].
Rows must not overlap (rowLen <= both strides when rows > 1). |
| cuda_upsample2x_nearest | Nearest-neighbour 2x upsample of 'planes' row-major h x w planes into
planes x 2h x 2w: out[p][y][x] = in[p][y/2][x/2] (cnn.h's FeatureMap::upsample2x_nearest). |
| cuda_capture_begin | Ops issued between begin and end are recorded instead of run. Don't call
cuda_read/cuda_write/cuda_sync/CudaBuf::alloc while recording. |
| cuda_capture_end | exec == 0 on failure |
| cuda_graph_launch | queued like any other op |
| cuda_half_write | |
| cuda_half_read | |
| cuda_to_half | Conversions over dst.len (to_half) / src.len (from_half) elements. |
| cuda_from_half | exact |
| cuda_half_gemm | c[m,n] = op(a)[m,k] * op(b)[k,n] (ta/tb as in cuda_gemm), a and b of the
same dtype; c in that dtype (_relu: then max(0, .) in the epilogue), or
fp32 with _f32. |
| cuda_half_gemm_relu | |
| cuda_half_gemm_f32 | |
| cuda_half_gemm_strided | 'batch' products c_i = op(a_i) op(b_i) in one cuBLAS call, a_i starting
at element i * strideA of a (likewise b, c); a stride of 0 reuses one
matrix for every product (e.g. a weight shared by a batch). One dtype. |
| cuda_half_gemv | Matrix-vector product for decoding one token at a time: y = w x with w an
[n][k] fp16/bf16 matrix (row-major, e.g. a linear layer's weight) and x,
y fp32 (x is not rounded to 16 bits). Memory-bound: on an H200 1.3-2.2x
faster than a one-row cuda_half_gemm_f32 on 1536-6144-wide projections
(e.g. 1536 x 2048: 3.2 vs 7.0 us), 4.47 TB/s on a 262144 x 1536 LM head.
Fastest when k is a multiple of 8 (16-byte loads; scalar otherwise). |
| cuda_half_relu | Training ops, as their CudaBuf versions above (loss stays fp32). |
| cuda_half_relu_grad | |
| cuda_half_mse_grad | |
| cuda_half_sgd | w = round(w - lr * g) |
| cuda_half_sgd2 | |
| cuda_half_convert | The usual layout: fp32 master weights and optimizer state, a 16-bit copy
of the weights for the forward / backward passes, 16-bit or fp32
gradients with the loss scale in them. One step: |
| cuda_to_half_many | Every src[i] -> dst[i] (lengths dst[i].len / src[i].len as in
cuda_to_half / cuda_from_half), 32 buffers per launch: a model's
parameters in one or a few launches instead of one per tensor. |
| cuda_from_half_many | |
| cuda_half_unscale_many | Gradient unscaling: dst[i] = src[i] * scale in fp32 (16-bit or fp32
sources; the fp32 form may work in place, dst == src), and flag[0] = 1
when any result is inf or NaN. The flag is only ever set: zero it
(cuda_zero) before the step. |
| cuda_unscale_many | |
| cuda_zero | b = 0 (all of b.len) |
| cuda_mp_sgd | Optimizer steps on fp32 master weights w with gradient g, as nn.h's
Optimizer::step (OPT_SGD / OPT_MOMENTUM when vel is valid, OPT_ADAM /
OPT_ADAMW when decoupled; 'step' is the 1-based step count for the bias
correction). The updated weights are also written, rounded, to wHalf when
it is valid. Skipped entirely when skip is valid and skip[0] != 0. Pass an
invalid buffer (dev == 0) for any optional argument. |
| cuda_mp_adam | |
| cuda_mp_adam_many | cuda_mp_adam on every tensor of a model, 24 tensors per launch (w[i], m[i],
v[i], g[i] of w[i].len values; wHalf empty for no 16-bit copies, else one
per tensor, all of one dtype). The same arithmetic per value; one launch
instead of one per tensor matters for models with many small tensors
(biases, norms): GPT-2 small's 148 tensors take 1.11 instead of 1.84 ms
on an H200 (fused torch.optim.AdamW: 1.37 ms, 1.61 with the bf16 copy). |
| cuda_half_add | 16-bit elementwise ops (fp32 arithmetic, one rounding per value; a, b and
out of one dtype and a.len elements). |
| cuda_half_sub | |
| cuda_half_mul | |
| cuda_half_scale | |
| cuda_half_axpy | a + k b |
| cuda_half_bias | y[r][c][p] += bias[c] (rows x channels x plane, row-major), then ReLU when
relu: a linear layer's bias (plane 1) or an NCHW convolution's. |
| cuda_half_maxpool2d | 16-bit max / avg pooling on NCHW (x: [batch, channels, height, width]),
kernel x kernel window, given stride, no padding: the same windows as
gpu_cudnn.h's cuda_maxpool2d. fp32 accumulation, one rounding; max
pooling copies the largest value's bits. |
| cuda_half_avgpool2d | |
| cuda_stream_handle | The CUstream every op above is queued on (a cudaStream_t to the runtime
API, cuDNN and cuFFT); null if there is no GPU. Work another library
queues on it is ordered with this backend's ops and recorded by graph
capture. |
| cuda_tf32_enabled | 1 if cuda_set_tf32(1) is in effect. |
| cuda_compute_capability | Compute capability of the selected GPU as major * 10 + minor (80 on an
A100, 90 on an H100/H200); 0 if there is no GPU. |
| cuda_sm_count | Streaming multiprocessors on the selected GPU (132 on an H100/H200 SXM); 0 if none. |
| cuda_load_module | Compiles CUDA C++ source with NVRTC for the selected GPU (-arch=sm_<cc>,
-std=c++17) and loads it as a CUmodule, for libraries that bring their
own kernels (gpu_cuda_attention.h). Null on failure, after printing the
compile log. Look kernels up with cuModuleGetFunction and launch them on
cuda_stream_handle(). |
| cuda_copy_to_device | Byte copies (host.len bytes) between host memory and a raw device address (e.g. a
CudaBuf's dev plus an offset, or memory from cuMemAlloc), through the
same path as cuda_write / cuda_read (pinned lanes from 1 MiB up).
cuda_copy_to_device returns once 'host' may be reused; cuda_copy_to_host
waits for queued work first. 1 on success. |
| cuda_copy_to_host | |
| cuda_add_f32 | |
| cuda_sub_f32 | |
| cuda_mul_f32 | |
| cuda_div_f32 | |
| cuda_pow_f32 | |
| cuda_relu_f32 | |
| cuda_log_f32 | |
| cuda_exp_f32 | |
| cuda_sqrt_f32 | |
| cuda_scale_f32 | |
| cuda_sum_f32 | |
| cuda_matmul_f32 | out[M,N] = a[M,K] * b[K,N] with this library's own tiled CUDA kernel
(64x64 tiles in shared memory, 4x4 per thread). Kept as a dependency-free
reference; cuda_matmul_f32_blas is faster at every size. |
| cuda_matmul_f32_blas | Same product through cuBLAS. |