RL from scratch

building a Mario agent one algorithm at a time — DQN → PPO → ICM → RND → kNN novelty

PPO: on-policy policy gradients that climb past DQN's ceiling

From REINFORCE through A2C to PPO — the on-policy learner that optimizes the policy directly and overtakes DQN on Mario 1-1.

28 min read

ppo policy-gradients reinforcement-learning · rl-from-scratch
DQN: value learning that beats the floor (and then hits a ceiling)

The first learning agent — a Nature-DQN Q-network with replay and a frozen target network — dropped into the off-policy half of rl-factory. It beats the random baseline on Mario 1-1, then plateaus.

7 min read

reinforcement-learning dqn q-learning pytorch · rl-from-scratch
rl-factory: an RL codebase where a new algorithm is a plug-in, not a rewrite

The lean spine that lets DQN, PPO, ICM, RND, and kNN-novelty share one training loop — where adding a model means registering a learner, not branching the control flow.

10 min read

reinforcement-learning software-design factory-pattern pytorch · rl-from-scratch