RL is back
At a party in 2023, during my parental leave, I ended up in a corner with two people I really respect. One is a very senior, very celebrated AI person. The other is a peer. I asked the group: what are we most excited about in AI right now? I said RL. My peer said robotics. The elder said LLMs.
And he didn't just say LLMs, he had a whole thesis. The winners would be embedded agents living inside each application. Your CRM gets an agent, your spreadsheet gets an agent, and whoever owns the application owns the agent. I believed the opposite: increasing returns to a single powerful agent that acts on your behalf. I don't want to talk to your app's support agent. I want my agent to talk to it for me.
On RL he was blunt. Dude, nobody cares about RL anymore. It's all LLMs now.
Some time later I met a Google DeepMind researcher, a PhD, along with another researcher friend. Same question. I said the thing I couldn't stop thinking about was the Bitter Lesson. He hadn't heard of it.
A DeepMind researcher who hadn't heard of the Bitter Lesson. I genuinely wondered if I was the crazy one. I'm over here reading Sutton, and the entire world is saying LLM, LLM, LLM.
To be fair, the world got me too. LLMs meant coding. And when Claude Code agents started actually working, in late 2025, I, like many software engineers, officially entered the singularity. That is where my whole think-then-write-code conviction comes from (more on that in Player of Games). I understand the pull. I lived it.
Here is what I got wrong though. When I first learned what RL even meant for LLMs, the answer was RLHF. This talk is what made the picture click for me:
And I kind of dismissed it. Human preferences in the loop felt impure to me. I was only interested in systems with no human in the loop. It's the same purist instinct that made me refuse reward shaping on mountain car, and honestly it has cost me both times.
Because RLHF wasn't wrong, it was early. People stuck with it, and the human in the loop kept getting replaced. Preference data became reward models, and reward models became verifiers. RLVR with GRPO, and the wave of Chinese open-source models that ran with it, was the unlock. You don't need a human to say which answer is better when the unit tests can tell you.
And now? Sutton went on Dwarkesh. Coding agents actually work. The math results keep landing. The entire industry is pivoting into RL wrapped around LLMs.
So the scoreboard from that party reads funny now. My peer said robotics, and robotics is where I ended up working. The elder said embedded agents, and an LLM agent built half of this website. And RL, the thing nobody cared about anymore, turned out to be what makes LLMs work at scale.
The singularity feels very intense from here. We can all feel it accelerating. I'm keeping my bet.