RL from scratch

building a Mario agent one algorithm at a time — DQN → PPO → ICM → RND → kNN novelty

The last mile was a compiler flag: getting Procgen to build on Apple Silicon

Why I moved the RL work from Mario to Procgen — and why the first step was fixing a four-year-old build bug in the benchmark itself, then upstreaming the patch.

5 min read

procgen generalization open-source apple-silicon · rl-from-scratch
From Mario to Procgen: preprocessing, the emulator tax, and generalization

Three concrete reasons I moved the RL work from Super Mario Bros to Procgen — the preprocessing you don't have to write, the environments you can finally parallelize, and the generalization test Mario can't give you.

8 min read

procgen mario environments generalization · rl-from-scratch
kNN episodic novelty: the count that can't saturate

A count-based novelty signal that structurally cannot die the way distillation did — because every episode starts from an empty memory.

10 min read

curiosity exploration ngu reinforcement-learning · rl-from-scratch
RND: a simpler curiosity, and why simpler won

ICM asks a dynamics question, RND a visitation question — and on Mario the simpler visitation signal held up where a prediction-on-features variant collapsed.

10 min read

curiosity rnd icm exploration · rl-from-scratch
ICM: giving Mario curiosity (and the bug that ate it)

Bolting an Intrinsic Curiosity Module onto PPO as just a new learner — and the real bug where the curiosity signal silently collapsed to zero.

18 min read

curiosity icm exploration reinforcement-learning · rl-from-scratch

RL from scratch

building a Mario agent one algorithm at a time — DQN → PPO → ICM → RND → kNN novelty

The last mile was a compiler flag: getting Procgen to build on Apple Silicon

From Mario to Procgen: preprocessing, the emulator tax, and generalization

kNN episodic novelty: the count that can't saturate

RND: a simpler curiosity, and why simpler won

ICM: giving Mario curiosity (and the bug that ate it)