RND: a simpler curiosity, and why simpler won

Code: rl_ablations/models/rnd, built on the rl-factory engine.

You’ve read the ICM tutorial (curiosity_icm_on_ppo.md), so you know the big idea: to make an agent explore, hand it a bonus reward for surprise — for reaching states it couldn’t predict. Then we added a second curiosity method, RND, and the natural question is: is RND just ICM with a tweak?

No. They share one sentence of philosophy (“prediction error = novelty”) and nothing else of substance. This tutorial explains — for someone who has seen ICM but not RND — exactly where they diverge, why RND came after ICM but is simpler, and why that simplicity is what saved our Mario runs. Every new term is defined the first time it appears. The code lives in rl_ablations/models/icm/ and rl_ablations/models/rnd/; both plug into the same PPO training loop.

If you remember one sentence: ICM asks “can I predict what my action does here?”; RND asks “have I seen a state like this before?” — a dynamics question versus a visitation question.

• ICM: Pathak, Agrawal, Efros & Darrell, Curiosity-driven Exploration by Self-supervised Prediction, ICML 2017 (papers/curiosity_driven_exploration_pathak2017/).
• RND: Burda, Edwards, Storkey & Klimov, Exploration by Random Network Distillation, ICLR 2019 (papers/random_network_distillation_burda2018/).

Part 0 — The one thing they agree on

Both methods compute an intrinsic reward (a bonus the agent invents for itself, on top of the game’s extrinsic reward) as a prediction error: build a predictor, measure how wrong it is, pay the agent proportionally to the wrongness. High error = “I don’t understand this yet” = novel = go there.

In both of our implementations that idea shows up as the same two-method interface:

intrinsic_reward(transitions) -> per-transition bonus, (T·N,)   # score novelty
learn(transitions)            -> train the predictor            # make today's states cheap tomorrow

The total reward the policy optimizes is r = r_extrinsic + η · r_intrinsic in both (η is intrinsic_scale in each config). And in both, the raw error is EMA-normalized by a RewardNormalizer so the bonus stays ~unit-scale as the predictor improves.

That’s the entire overlap. Everything below is a difference.

Part 1 — The core split: what do you predict?

This is the whole ballgame. Both predict something and pay you the error. They predict different things.

ICM predicts the future — a dynamics model

ICM’s predictor is a forward model: given the current state’s features and the action you took, predict the next state’s features.

forward model: φ̂(s’) = f(φ(s), a). “If I’m here and press A, where do I end up?” ICM intrinsic reward: ‖φ̂(s’) − φ(s’)‖² — how wrong that guess was.

The signal answers: “Are the consequences of my actions still surprising here?” It’s inherently about transitions — it needs (state, action, next_state). Look at rl_ablations/models/icm/module.py: intrinsic_reward encodes both states and next_states, takes actions, and computes _forward_error(phi_s, actions, phi_next).

RND predicts a fixed random function — a visitation counter

RND has no future, no action, no dynamics. It sets up two networks over the current state only:

• a target network: randomly initialized weights, then frozen forever. It’s an arbitrary, meaningless-but-fixed function f(s).
• a predictor network: trained to reproduce the target’s output, f̂(s) ≈ f(s), on the states the agent actually visits.

RND intrinsic reward: ‖f̂(s) − f(s)‖² — how badly the predictor matches the frozen target at this state.

Why is that novelty? Because a neural network only gets good at inputs it’s been trained on. If the agent has visited states like s many times, the predictor has had many gradient steps there, so it matches the target closely → low error → familiar. If s is somewhere new, the predictor has never been fit there → high error → novel. The error is, in effect, a soft visit counter: “how many times have I been near here?” See rl_ablations/models/rnd/model.py — the target’s parameters get requires_grad_(False) and the predictor chases it.

This is the extension question, answered concretely: RND does not extend ICM’s forward model — it deletes it. Gone: the action input, the next-state prediction, the whole notion of dynamics. RND is strictly less machinery than ICM, not more.

Part 2 — The encoder: ICM’s clever bit that RND doesn’t need

ICM’s subtlety was where it predicts. Predicting raw pixels invites the noisy-TV problem: a patch of random flicker (TV static, swaying leaves) is forever unpredictable, so a pixel-predictor pays curiosity forever for staring at noise. ICM’s fix is to predict in a learned feature space φ trained by an inverse model — a second head that predicts which action took you from s to s'. To do that job, φ must keep only what the agent can control and discard uncontrollable noise. So the flicker never enters φ, and the noisy-TV trap closes.

That inverse model is real, load-bearing complexity in rl_ablations/models/icm/: an extra network head, a cross-entropy loss, and (as the ICM tutorial’s war story tells) a mandatory detach so the forward loss doesn’t collapse φ’s scale to zero.

RND sidesteps all of it. Its target is a fixed random function of the state, so there is no representation to shape, no inverse model, no encoder-collapse bug to guard against. RND’s only required preprocessing is observation normalization (whiten the pixels with a RunningMeanStd before the nets) — because a frozen random target is unlearnable on raw, arbitrarily-scaled pixels. That’s it. Compare the two module.py files: ICM juggles two losses and a beta blend; RND runs a single distillation MSE.

Part 3 — Why the difference mattered on Mario

This isn’t academic — it’s exactly why we have both in the repo. On our Atari-style Mario (4-stacked frames + frameskip), ICM’s forward model saturates in ~20k steps: consecutive frames are trivially predictable, so forward_loss → ~0.003 everywhere, even in genuinely new regions. The curiosity signal dies, and no amount of tuning entropy_coef revives a reward that has gone to zero. Pure-curiosity ICM gets Mario stuck around x≈400.

RND doesn’t have a “next frame” to find predictable, so it can’t saturate that way. Its error tracks visitation, which keeps rising as new areas appear. In our 80k-step pure-curiosity run (game reward off), RND’s intrinsic_reward stayed 0.04–0.13 and rose on reaching new areas, entropy stayed healthy, and Mario reached x≈1200 on curiosity alone. The mechanism that saved us is precisely the ICM machinery RND removed — dynamics prediction was the thing that saturated.

Takeaway for the exam: “RND is simpler than ICM” isn’t a footnote — the simplification (drop the forward model, count visitation instead) is the fix for the failure mode that sank ICM here.

Part 4 — So, sibling or successor?

Chronologically RND (2019) came after ICM (2017), from an overlapping group of authors, and was partly motivated by ICM’s known weaknesses (noisy-TV, and the saturation we hit). But it is not built on ICM — it’s a different, leaner answer to the same question. If you must draw a family tree:

• Shared parent: “intrinsic reward = prediction error” (curiosity as self-supervised surprise).
• ICM’s branch: predict dynamics in a learned, control-aware feature space. Powerful, but more moving parts and prone to saturating when dynamics are easy.
• RND’s branch: predict a fixed random function of the raw (normalized) state. A soft visit counter. Fewer moving parts, robust to easy dynamics, but tells you nothing about why a state is novel (no controllability signal).

Neither is strictly better. ICM’s controllability filter is genuinely useful when uncontrollable noise dominates the observation; RND’s visitation signal is more robust when the environment’s dynamics are too easy to stay surprising. In this repo they’re peers — two PPOLearner wrappers behind one interface — which is exactly how you should think of them.

Where to look in the code

Both are training-only (they never touch acting or evaluation — the policy is a plain PPO PolicyAgent) and neither is checkpointed. Run either with its config in configs/icm.yaml / configs/rnd.yaml; set extrinsic_reward_scale: 0.0 (RND) or reward_scale: 0 (ICM) to see pure curiosity, which is the only honest test that the intrinsic signal actually works.

Results on Mario 1-1

Tested as pure curiosity, the design question becomes empirical: put the novelty split on raw pixels (RND) versus on the policy’s learned features φ (rndf). The feature-space variant collapses — the predictor fits the frozen random target across the compact φ-manifold almost instantly, so prediction error → 0 everywhere and the novelty signal dies within ~20k steps. RND-on-pixels stays alive:

That collapse is the whole lesson: any distillation-based novelty can saturate once its target becomes easy to predict. Which raises the obvious question — is there a novelty signal that structurally cannot saturate? There is: stop predicting, start counting → kNN episodic novelty.