I spent fifteen million training steps teaching an AI to play Pokémon Red. It became a brilliant explorer, a shameless cheater, and — once — a total catatonic. It never won a single badge. Here’s everything that went wrong, and why that’s the interesting part.
Pokémon Red is a deceptively brutal problem for reinforcement learning. There’s no score ticking upward, no lives counter, no immediate feedback at all. The first real milestone — the Boulder Badge — sits behind thousands of coordinated decisions: navigate two towns, survive a maze of a forest, manage a party, win a gym battle. An Atari agent gets rewarded every few frames. A Pokémon agent has to behave coherently for the better part of an hour before the game acknowledges it exists.
So I asked a narrow question: how far can a bog-standard PPO agent get on raw pixels alone, if I compensate with an obsessively engineered reward function? No hand-fed game state in the observations. No cheating with RAM features. Just a grayscale screen, a convolutional network, and whatever reward signal I could compute behind the scenes.
The answer, after seven training runs: it gets to the edge of Viridian Forest. And it stops there, forever.
The Setup
The agent lives inside PyBoy, a Game Boy emulator, wrapped as a Gymnasium environment. It sees a 72×80 grayscale downsample of the screen — the last four frames stacked so it can perceive motion. Each action is held for 24 emulator frames, about 0.4 seconds of game time, and an episode lasts 8,192 actions: roughly 55 in-game minutes. Every episode resets to a save state just after picking Charmander, because forcing an RL agent to sit through the intro cutscene every episode is a war crime against your own compute budget.
Since pixels carry no progress signal, every reward comes from reading the game’s RAM: player coordinates, map ID, party stats, badge flags, story-event bits, battle state. New tile? Tiny bonus. New map? Big bonus. Level up, deal damage, win a battle? Bonuses. Badge? A massive +50 that, spoiler, never once fired.
Observation engineering vs. reward engineering. The famous Pokémon RL projects feed the network structured game state alongside pixels. I deliberately didn’t. The bet was that a dense enough reward signal could steer a pixels-only policy through the early game. The rest of this post is the story of losing that bet — informatively.
The Reward-Hacking Arms Race
Every penalty in my final reward function is a fossil — the scar tissue of an exploit some earlier agent discovered. Give a policy-gradient method a loophole and it will drive a truck through it. Three favorites:
The Infinite Heal Loop
Early runs rewarded healing at +2.5 per full-party restore. The agent’s response: walk to the Pokémon Center, take a little damage, heal, repeat. Forever. A perfectly legal perpetual-motion machine of reward. It wasn’t playing Pokémon; it was running a spa.
Menu Camping
Once penalties made the overworld expensive, the agent found sanctuary: the party-switch menu, a reward-neutral pocket dimension where time stops and nothing bad happens. It would open the menu mid-battle and simply… stay. The pause button as an optimal policy.
Wall Grinding
After exhausting local novelty, agents would walk into a wall and keep pressing into it for thousands of steps — the RL equivalent of a Roomba stuck in a corner. Related: the per-hit-point damage reward once exploded so violently against high-HP enemies that it destabilized the entire value function.
Reward shaping never eliminated degenerate behavior. It just redistributed it. Every patch closed one exploit, and every subsequent run found a new one.
Seven Runs, Three Fates
The seven runs weren’t clean ablations — they were honest, sequential debugging, each configuration a reaction to the last one’s failure. One caveat before the table: because the reward function changed between runs, the reward numbers are only comparable within a run, not across runs. The only cross-run metric is actual game progress. Which was zero, everywhere.
| Run | Steps | Mean reward, first → last | What happened |
|---|---|---|---|
| 1 | 4.1 K | −1.5 → +0.6 | Smoke test of the training loop |
| 2 | 455 K | −38.9 → −60.3 | Penalties dominated; reward drifted down |
| 3 | 18 K | — | Aborted configuration check |
| 4 | 5.00 M | −57.4 → −52.8 | Longest run. Perfectly, tragically flat |
| 5 | 625 K | −161.8 → −139.9 | Heavier penalties; same behavior, angrier numbers |
| 6 | 4.47 M | −158.0 → −1637.2 | Total policy collapse (see below) |
| 7 | 4.42 M | +125.8 → +446.7 | The one that worked — sort of |
The Night the Policy Died
Run 6 started as my best ever. By 131,000 steps it hit the highest mean reward of any configuration to that point. Then the policy’s entropy — the measure of how much randomness, how much curiosity, remains in its decisions — began to decay. And it didn’t stop. It fell to numerical zero. Not “low.” Zero, as in 10⁻²³ nats, as in the policy became perfectly deterministic.
A deterministic policy in Pokémon is a horror show. Whatever loop the agent’s single fixed behavior drives it into, it repeats until the episode mercifully ends — racking up stuck penalties and idle penalties for thousands of steps, every episode, forever. The optimizer’s own diagnostics flatlined too: KL divergence zero, clip fraction zero. Learning hadn’t slowed. It had ceased. The run kept burning compute for millions of steps as a corpse.
final mean episode reward in run 6 — two orders of magnitude worse than any other run, from a config that briefly peaked at +17.8
Entropy is the single best early-warning metric in this domain. The reward curve was still smiling while the policy was already dead inside.
The Agent That Learned to Be a Tourist
Run 7 got everything I’d learned: the Start button (runs 1–6 literally could not press Start — you can see it in the logs, where initial policy entropy is ln 7 in early runs and ln 8 in run 7), four-frame stacking, rebalanced penalties, an entropy bonus to prevent another collapse, and a gentler learning rate. It’s the curve you dream about: reward tripling from +126 to +447, entropy settling at a healthy 0.78 nats instead of imploding.
Watching the checkpoints play is where it gets bittersweet. The agent walks with purpose. It hugs unexplored edges, exits buildings promptly, no longer camps menus. It looks competent. But decompose that +447 and nearly all of it is exploration bonuses, movement rewards, and incidental battle damage. The +50 badge bonus? Never fired. The +8 catch bonus? Never fired. The agent mastered the reward function I wrote instead of the game I wanted it to play.
It had learned to be an efficient tourist, not a Pokémon trainer.
Where Exploration Goes to Die
The visitation heatmaps tell you exactly where the story ends. Pallet Town is saturated — every walkable tile visited, hot spots along the fences where earlier agents ground against walls. Route 1 shows a clean traffic funnel, heavy at the southern mouth and thinning north. Viridian City is a skeleton of corridors between route exits, side streets barely touched.
And then Viridian Forest: a single bright band along the southern entrance, a thin trail up the eastern edge, and the entire interior — the only path to Pewter City and the first badge — pitch black. Untouched.
Here’s the mechanism, and it’s the deepest lesson of the project. My novelty bonus reset every episode. So each episode, re-exploring the same comfortable tiles near spawn pays out again — while the marginal new tile deep in the forest is 8,000 steps away, behind wild encounters and a visually repetitive maze. The shaped return of pushing deeper is locally worse than milling around the explored core.
Episodic novelty
Exploration bonus resets each episode. The agent gets paid every day to rediscover Pallet Town. The frontier never becomes profitable.
Global novelty
Novelty tracked across episodes — the approach behind Go-Explore and the well-known Pokémon RL results. Only unexplored territory keeps paying.
The exploration bonus wasn’t failing to help at the frontier. It was actively pricing the frontier out.
The Playbook I’d Hand My Past Self
- Persist novelty across episodes
The single highest-leverage fix. An episodic exploration bonus provably re-rewards the explored core every episode. Track visited tiles globally, and the frontier becomes the only thing that pays.
- Put game state in the observation
Badges, party HP, and story progress are invisible in pixels. A four-frame stack spans 1.6 seconds of a 55-minute episode. Feed the network the RAM features you’re already reading for rewards — turn hidden state into visible state.
- Watch entropy like a smoke alarm
Reward looked healthy in run 6 long after the policy was unsalvageable. Entropy told the truth early. Set an alert on it; the default entropy coefficient is not a safe assumption in a menu-heavy action space.
- Only then, scale compute
My 15M total steps are one to two orders of magnitude below what comparable projects spent to reach mid-game. But compute poured into a broken exploration scheme just buys you a more confident tourist.
In Praise of the Negative Result
I set out to learn whether careful reward engineering could substitute for observation engineering in Pokémon Red. Over fifteen million steps, the answer was a clear, well-documented no. Dense RAM-derived rewards over raw pixels produced an agent that moves beautifully, explores exhaustively, improves steadily — and achieves none of the game’s actual objectives.
But the failures are legible: a reward-hacking arms race with receipts, a textbook entropy collapse caught on camera, and heatmaps that pinpoint the exact tile where ambition ends. Everything — the environment, training code, all seven runs of logs, checkpoints, and heatmaps — is released for anyone who wants to carry the torch into the forest. Bring a global novelty bonus.
“The agent learned everything I rewarded, and nothing I wanted. That’s not the agent failing the test — that’s the test working.” P.S. Here is the link if you want to read the Research behind it: https://shorturl.at/epbvJ
Leave a Reply