← Anson Chu

RL — Player of Games

2016–2026 · my journey into RL

AlphaGo came out right around the time I left Uber. I was so excited about AI when I joined Numerai, but honestly I knew very little about deep learning. My interview day was actually a company party (strange, right?). There was an ex-DeepMind researcher there, and I basically asked him: why can't we just do what AlphaZero did, and we'll have the best returns? He looked at me like I was stupid and crazy.

This page is the long answer to why.

I should be honest about games too. I love playing them, but I'm not that great. Not compared to the pros, not even compared to my friends. If I'm ever going to be good at a game, it's going to be through AI. And watching AI beat games never gets old for me. AlphaZero. The OpenAI Dota team. AlphaStar. Games have always been the dream path of RL.

OpenAI's hide and seek video was seminal for me too. But I saw it too early. I didn't yet understand what RL was, what deep learning was, what a neural net even is. I understand it now. Getting there is what the rest of this page is about.

YouTube · video Multi-Agent Hide and Seek OpenAI · 2019

The project is named after my favorite book.

the namesake · novel The Player of Games Iain M. Banks, 1988. Jernau Gurgeh, the greatest player of games in the Culture, versus an empire built on one. goodreads.com
“It’s insurance. You heard of that? No? Never mind. It’s like gambling in reverse.” — Iain M. Banks

The books

In 2023 I was on parental leave and went through Karpathy's Zero to Hero course. My notes are here. At the same time I was reading Sutton & Barto. The first real exercise in the book is tic-tac-toe, so I wrote it: a tabular TD learner playing against an imperfect opponent.

GitHub · repository kumikoda/rlbook coding and exercises from the reinforcement learning book — tictactoe.ipynb, committed Jan 2, 2024. github.com/kumikoda/rlbook
Average reward over 10,000 episodes for five trials of the tabular TD tic-tac-toe learner, climbing from about -0.75 to winning.
the TD learner over 10,000 episodes, five trials. It actually learned. Remember this for the next section.

I also watched all of Steve Brunton's RL lectures, though I didn't take detailed notes. Honestly the math was a bit beyond me, and I had a young baby 😅

YouTube · playlist Reinforcement Learning: Machine Learning Meets Control Theory Steve Brunton

And I kept watching people on the internet make AI beat their games.

YouTube · video Training an unbeatable AI in Trackmania Yosh

Somewhere in here I started investigating drone racing, which pulled me into a completely different hole: robotics and hardware. ALOHA, Physical Intelligence, the Tesla Bot, 1X. That hole leads out of software entirely, and it's where I landed. Today I build fabs.

YouTube · video Champion-level Drone Racing using Deep Reinforcement Learning UZH Robotics and Perception Group · Nature

One more thing from that year. At a party during leave I ended up in a corner with two people: a very senior, very celebrated AI person, and a peer I really respect. I asked the group: what are we most excited about in AI? I said RL. My peer said robotics. The elder said LLMs, applied inside applications — he believed the embedded agent within each application would win. I believed the opposite: increasing returns to a single, all-powerful agent that acts on your behalf, instead of me having to talk to your app's support agent. And on RL he didn't hesitate. Dude, nobody cares about RL anymore. It's all LLMs now.

I wasn't sure if I was right. It really did feel like RL winter. But I bet RL.

By hand

Then I wanted to do it myself. I gave myself one rule: no looking up answers. Read the environment, think it through, write the code.

obs(pos, vel)Linear 2→64ReLULinear 64→64ReLULinear 64→3Q(s,·) → argmaxε=0.1 randomMountainCar-v0 — reward −1 per step, 200-step cutoffactionobs, rewardReplayBuffer(100_000)sample minibatch 64(s, a, r, s′)MSE updatethe bug: target_q_value = reward + self.gamma + max_next_q_values # + where × should be
mountain_car.py — the network, the loop, and the bug that decided the outcome

Mountain car stopped me. The reward is −1 every step until you reach the flag, and the episode cuts off at 200 steps. Until you reach the flag once, every policy scores exactly −200. There is nothing to learn from.

I got around this in two ways, and I felt bad about both. I added a +200 bonus for reaching the flag, which is just reward shaping. And I hand-coded a momentum rule (push in the direction you're already moving) to feed successful episodes into the replay buffer. That rule solves the game by itself, so what exactly is the agent learning?

Even the observation bothered me. Position and velocity are features somebody already extracted for me. So really it was the same problem at three depths: the reward, the data, and the representation, all with human knowledge smuggled in. The learning wasn't doing the work.

Essay · the rabbit hole The Bitter Lesson Sutton's argument that general methods + compute beat human-knowledge methods. Rich Sutton · incompleteideas.net · 2019
-200-160-120-800250500750999episodehand-coded heuristic (even episodes) — 100/500 reached the flagDQN (odd episodes) — 0/500, exactly −200.00 every time
scores.csv, 1000 episodes. Even episodes: my momentum rule. Odd episodes: the DQN.

The DQN never reached the flag. Not once in 500 episodes. Exactly −200.00 every time. Going back this year I finally found out why: target = reward + gamma + max_q. A plus where a times should be. Also lr=1 on Adam. On a sparse reward, a broken learner and a slow learner produce identical output, so the bug was invisible.

So the tic-tac-toe learner from the book worked, my from-scratch DQN didn't, and I wasn't willing to fake the difference. Then a new job started, and this all went dormant.

LLMs play the games

When I came back to it at the end of 2025, LLMs had happened. I tried three things.

First, I had a local Qwen model literally pick the actions, one observation at a time. I did this on purpose as a “this is stupid” baseline. It's the wrong way to use an LLM and I wanted to see it fail for myself.

Second, I built an OaK agent after Sutton's architecture. Honestly, I vibe-coded it. It ran, but I didn't understand it deeply enough to be interested in my own code, which felt like a signal to read more before copying more.

envWorld Modelenc (s⊕a)→64→64heads: Δs · r · doneQ-Networks→64→64→|A|Dyna plannerimagined rolloutsexperience bufferreal + simulatedmodel of P(s′|s,a), R(s,a)simulated (s,a,r,s′)trainthree modules, learning continuously in parallel threads — after Sutton's OaK architecture
oak_agent.py — world model (transition · reward · done heads), Q-network, and Dyna planner in parallel threads
YouTube · talk The OaK Architecture: A Vision of SuperIntelligence from Experience Rich Sutton · RLC 2025

Third, I built a harness for League of Legends. Screen capture, input control, the works. Then I researched the anticheat and got convinced it was never getting past it. That dead-end sent me down a computer use research hole instead.

The insight that finally stuck came mostly from my day job, watching agents work all day: the model shouldn't act, it should write code that acts. Not think, then act. Think, then write code.

Autoresearch

Karpathy released autoresearch: an agent edits train.py overnight on a 5-minute training budget, keeps what improves the metric, discards what doesn't. I ported it to MLX so it could run on my Mac.

GitHub · repository karpathy/autoresearch AI agents running research on single-GPU nanochat training automatically. github.com/karpathy/autoresearch
1.701.751.801.851.901.95baseline (AdamW, default config) on M3 Max — 1.743 (keep)1remove logit soft-cap (worse, less stable) — 1.772 (discard)23x MLP (worse; also thermal-throttled ~48k tok/s) — 1.927 (discard)3raise matrix LR 0.04 -> 0.06 — 1.733 (keep)4raise matrix LR 0.06 -> 0.08 (overshoot) — 1.801 (discard)5longer anneal warmdown 0.7 (worse) — 1.790 (discard)6shorter anneal 0.35 (CONFOUNDED: only 51 steps, low throughput) — 2.431 (discard)7raise embedding LR 0.8 (worse) — 1.788 (discard)8experiment (overnight, M3 Max)filled = kept · hollow-gray = discardedval bpb
results.tsv — val bits-per-byte. Kept: raise matrix LR (1.743 → 1.733). Discarded: removing the logit soft-cap, 3× MLP (thermal-throttled).

The interesting part is that you don't edit the Python anymore. You edit program.md, the instructions. The research process itself becomes the thing you iterate on.

What learning actually costs

I wanted Atari next. So I wrote a PPO from scratch in one readable file, plus a loop where Claude proposes the next experiment's hyperparameters with a hypothesis. And for calibration, I ran the canonical SB3 baseline.

4 × 84 × 84stacked framesConv 328×8 / s4→ 20×20Conv 644×4 / s2→ 9×9Conv 643×3 / s1→ 7×7Flatten 3136FC → 512actor512 → |A| logitscritic512 → V(s)rl/networks.py — Nature CNN (Mnih 2015), orthogonal init, shared body + two heads
rl/networks.py — the actor-critic. 3 conv layers → 512 → policy + value heads.

The baseline beats Breakout: 33.85 mean reward, above human level. It needs 40.5 million environment steps to get there. Learning works. It just costs 40 million steps.

the PPO baseline playing Breakout after 40.5M steps

The rematch

The 2024 environments were still unbeaten, and that bothered me more than it should have. So: player-of-games. The agent doesn't play moves anymore. It writes solution.py. I write instructions.md and the grader. The grader trains the agent's code from a pristine copy under a wall-clock budget and scores it on 100 held-out seeds. No pretrained weights, no cached checkpoints. The same rule I gave myself in 2024, except now it's enforced by code.

librarygames/<name>/116 foldersorchestratorspawns researcher(claude -p program.md)solution.pythe only filethe agent may editgraderpristine copy →train(budget) →100 held-out seedsresults.tsvbest/ ratchet(git commit / revert)editsgrade attemptscorekeep bestnext attemptanti-cheat: grader owns the env + eval seeds; train() runs from a pristine copy under a wall-clock budget
the loop. I edit instructions.md; the agent edits solution.py; the grader is the referee.

All twelve classic control games fell in 13 days. Mountain car took one attempt: −99.08 against a −110 threshold, inside a 60-second budget. It did it by reading the physics out of gymnasium's source and running value iteration on the exact dynamics. My 2024 discomfort, inverted. Maximum prior, zero gradients, zero guilt. And completely legal under rules I wrote.

Classic control — 12/12 solvedacrobot — solvedbipedal_walker — solvedblackjack — solvedcar_racing — solvedcartpole — solvedfrozenlake — solvedlunarlander — solvedlunarlander_continuous — solvedmountaincar — solvedmountaincar_continuous — solvedpendulum — solvedtaxi — solvedAtari — 13/104 solvedatari_adventure — solvedatari_air_raid — untriedatari_alien — untriedatari_amidar — triedatari_assault — untriedatari_asterix — solvedatari_asteroids — triedatari_atlantis — triedatari_atlantis2 — triedatari_backgammon — triedatari_bank_heist — triedatari_basic_math — solvedatari_battle_zone — triedatari_beam_rider — solvedatari_berzerk — untriedatari_blackjack — triedatari_bowling — triedatari_boxing — triedatari_breakout — triedatari_carnival — triedatari_casino — solvedatari_centipede — triedatari_chopper_command — triedatari_crazy_climber — triedatari_crossbow — triedatari_darkchambers — triedatari_defender — triedatari_demon_attack — triedatari_donkey_kong — triedatari_double_dunk — triedatari_earthworld — triedatari_elevator_action — triedatari_enduro — triedatari_entombed — triedatari_et — triedatari_fishing_derby — triedatari_flag_capture — triedatari_freeway — solvedatari_frogger — triedatari_frostbite — triedatari_galaxian — triedatari_gopher — triedatari_gravitar — solvedatari_hangman — triedatari_haunted_house — triedatari_hero — solvedatari_human_cannonball — triedatari_ice_hockey — triedatari_jamesbond — triedatari_journey_escape — solvedatari_kaboom — triedatari_kangaroo — triedatari_keystone_kapers — triedatari_king_kong — triedatari_klax — triedatari_koolaid — triedatari_krull — triedatari_kung_fu_master — triedatari_laser_gates — triedatari_lost_luggage — triedatari_mario_bros — triedatari_miniature_golf — solvedatari_montezuma_revenge — triedatari_mr_do — triedatari_ms_pacman — triedatari_name_this_game — triedatari_othello — triedatari_pacman — triedatari_phoenix — triedatari_pitfall — triedatari_pitfall2 — triedatari_pong — solvedatari_pooyan — triedatari_private_eye — triedatari_qbert — triedatari_riverraid — triedatari_road_runner — triedatari_robotank — triedatari_seaquest — solvedatari_sir_lancelot — triedatari_skiing — triedatari_solaris — solvedatari_space_invaders — untriedatari_space_war — untriedatari_star_gunner — untriedatari_superman — untriedatari_surround — untriedatari_tennis — untriedatari_tetris — untriedatari_tic_tac_toe3_d — untriedatari_time_pilot — triedatari_trondead — triedatari_turmoil — triedatari_tutankham — triedatari_up_n_down — triedatari_venture — triedatari_video_checkers — triedatari_video_chess — triedatari_video_cube — triedatari_video_pinball — triedatari_wizard_of_wor — triedatari_word_zapper — triedatari_yars_revenge — triedatari_zaxxon — triedsolvedattemptednot attempted
library outcome after 530 graded attempts
cartpole — return 500.0 / 500 · trained in 0.2 s · linear policy by hill-climbing
mountaincar — return −104 · trained in 0.9 s · value iteration on the exact model
pendulum — swing-up + balance · trained in 12.8 s · value iteration, 256×256 grid
car_racing — return 913.9 · no training · road segmentation + pure-pursuit steering
atari_pong — scripted pixel tracker · predicts ball crossing, reflects off walls
atari_freeway — reactive pixel dodger · pauses under traffic, climbs in gaps

six solved games, replayed on the grader's held-out eval seeds — same path as a graded run

solved gamebestthresholdattemptsbest attempt, from results.tsv
cartpole499.984752robustness gate: only trust theta with perfect min over 30…
mountaincar-99.08-1101model-based value iteration (GRID=900), 1-step lookahead +…
mountaincar_continuous97.097903restrict candidates to failure-free band {0.4,0.45,0.5}; p…
pendulum-134.285-2007grid768 VI + dense 301-action policy lookahead; +0.2 over …
acrobot-81.94-1003larger CEM pop(32)/elite(8)/8 seeds per gen; marginal gain…
lunarlander247.5732002Double DQN MLP128x2, soft target, best-snapshot; 91% eps>=…
lunarlander_continuous273.9582004Double DQN on 3x3 (main x side) action grid; reuse of disc…
bipedal_walker303.9530011N=24/b=12: better gradient -> better linear optimum, SOLVE…
car_racing918.4490011v_max 16->18: speedometer caps at 16 but car goes faster; …
frozenlake0.740.71model-based VI: explore to learn P/R, value iteration for …
taxi7.972exact deterministic model by probing all 500 states x 6 ac…
blackjack-0.015-0.053exact model-based VI from known rules; deterministic optim…
atari_pong10.9013carry reflected vy across y-bounce seeds fit; fixes blind-…
atari_freeway27.7154reactive dodger ahead=11 hw=7 thr=0; NOOP when colorful ca…
all 14 solved games. Look at the right column: planning, scripted controllers, search. Almost never gradients.

Then the wall. Two out of 104 Atari games. Where you can't read the mechanism out of the pixels, the scripted-controller strategy dies. And the agent never switched to learning.

0102030agent, scripted CV — 17 graded attemptsattempt #best 24.3PPO baseline — 40.5M steps33.9020M40Menv stepsgraded attemptmean reward
the wall, in one game. Left: 17 scripted-CV attempts at Breakout under player-of-games budgets. Right: the PPO baseline, which needed 40.5M steps.
unsolved (most attempted)attemptsbestbest attempt, from results.tsv
atari_enduro19158.7dodge hysteresis: hold committed dead-ahead side vs frame …
atari_breakout1724.3loss-classifier + hold-during-bricks + serve delay 20; min…
atari_phoenix166519boss bomb-dodge hw 7->8 (interacts w/ loose detect); +774
atari_gopher1516864ALIGN 8->5 with velocity-lead active (lead positions preci…
atari_turmoil121103.5band-width 7..13 identical (enemies within +-7 of lane cen…
atari_galaxian102207EARLY_R0 130->129; seed7 1840->2170; mean 2174->2207. Turn…
atari_venture9600alternate macro A(R40,L64) and B(R48,L72) across room entr…
atari_journey_escape8-700aggressive UP-accel (thr=3.0): UP unless own-column hazard…
where the strategy ran out — the notes are the agent's own experiment log

The thread

2024: a network learns the policy. 2025: the LLM is the policy. March 2026: the LLM runs the experiments. June: it proposes them. July: it writes the solver. Every project moves the model one level up the stack.

Is this recursive self-improvement? No. I'm still the outer loop. My own README says so: instructions.md is “the spec — YOU edit this.” Nothing improves the improver yet.

But I think I finally understand the look that researcher gave me at the party. AlphaZero needs a perfect simulator, fixed rules, and a game you can self-play a billion times. Markets have none of those. Games have all of them. That's why games are the dream path of RL, and why they keep pulling me back.

And the bet from that party? The scoreboard is funny. My peer said robotics, and that's where I ended up working. The elder said LLM agents inside applications, and those agents built everything on this page. And somehow RL is back. Dwarkesh had Sutton on. Coding agents, the math results, RLVR. The entire industry is pivoting into RL wrapped around LLMs. Nobody cares about RL anymore, right?

YouTube · podcast Richard Sutton — Father of RL thinks LLMs are a dead end Dwarkesh Patel · from 56:41
5projects
116games in the library
530graded attempts
3,166agent session logs
12/12classic control solved
2/104Atari solved

what's mine and what's not: the 2024 code is all mine, bugs included. In player-of-games I built the harness (grader, orchestrator, contract, dashboard); the agent wrote every solution.py inside it, and quoted results come from its logs. The charts, diagrams, and gameplay recordings on this page were generated by Claude from those logs.