Module ml::rl
reinforcement learning.
Environments:
Bandit k-armed Gaussian bandit.
GridWorld tabular grid with walls, pits and goals; GridWorld::cliff()
is Sutton & Barto's cliff walk (Example 6.6).
CartPole the classic pole-balancing control task (Barto et al.
1983, Gym's CartPole-v1 dynamics and limits).
Agents, grouped by technique:
Bandits epsilon-greedy, UCB1, Thompson sampling, gradient bandit.
Tabular TD Q-learning, SARSA, expected SARSA.
Value-based DQN (Mnih et al. 2015): replay buffer, target network,
Huber loss, epsilon schedule, optional Double DQN.
Policy-based REINFORCE with a learned baseline, A2C (advantage
actor-critic with GAE) and PPO (clipped surrogate,
several epochs of minibatches per rollout).
Observations are tensors (an agent reads obs->data[0 .. obsDim)), and states and actions of discrete problems are indices. A typical loop:
struct CartPole env = CartPole::new(1UL);
&Tensor obs = Tensor::new_1d(4UL, 0); &Tensor next = Tensor::new_1d(4UL, 0);
env.reset(obs);
unsigned long a = agent.act(obs, 1);
struct EnvStep s = env.step(a, next);
agent.observe(obs, a, s.reward, next, s.terminated);
Structs
| EnvStep | What a step returned: 'terminated' ends the episode in a terminal state (no future reward); 'truncated' cuts it off by a time limit (the state still has value, and learners bootstrap from it). |
| Bandit | |
| GridWorld | |
| CartPole | |
| BanditAgent | |
| TabularAgent | |
| ReplayBuffer | |
| DqnConfig | |
| Dqn | |
| PgConfig | |
| PolicyAgent | Separate policy (logits) and value networks, one Adam optimizer. The rollout collects transitions until PolicyAgent::update consumes them: call it at episode ends (REINFORCE) or every N steps (A2C / PPO). |
Macros
| GRID_EMPTY | |
| GRID_WALL | cannot be entered |
| GRID_PIT | pitReward; back to the start (or the episode ends when pitTerminal) |
| GRID_GOAL | goalReward, episode ends |
| GRID_UP | |
| GRID_RIGHT | |
| GRID_DOWN | |
| GRID_LEFT | |
| CARTPOLE_OBS | cart position, cart velocity, pole angle, pole angular velocity |
| CARTPOLE_ACTIONS | push left, push right |
| BANDIT_EPS_GREEDY | param: epsilon |
| BANDIT_UCB1 | param: exploration c (sqrt(2) in Auer et al.) |
| BANDIT_THOMPSON | Gaussian posterior per arm; param: prior std |
| BANDIT_GRADIENT | softmax preferences with a reward baseline; param: step size |
| PG_REINFORCE | |
| PG_A2C | |
| PG_PPO |