RLVR, the unlock
RLHF has a load-bearing weakness: the reward comes from a learned model of human preference, and any learned reward model can be gamed. Optimize against it hard enough and the policy finds the cracks — confident tone instead of correct answers, length instead of substance. That's Goodhart's law with a GPU budget, and it caps how much RL pressure you can safely apply.
RLVR — reinforcement learning from verifiable rewards — swaps the learned judge for a checker:
# RLHF: reward is a model's opinion
reward = reward_model(prompt, response) # learnable, gameable
# RLVR: reward is a fact
reward = 1.0 if check(response) else 0.0
# check() = the unit tests pass, the boxed answer equals 42,
# the proof verifies, the program compiles and runsThree things fall out of that swap, and together they're the unlock:
1. You can push as hard as you want. A verifier doesn't drift when you optimize against it. The tests pass or they don't. So the RL stage can run orders of magnitude longer without the reward collapsing under the policy's pressure.
2. Data generates itself. You don't need humans labeling preferences at scale. You need problems with checkable answers — math with known solutions, code with tests — and the model can generate, get graded, and learn in a loop. This is why the reasoning models (o1-style, DeepSeek R1) came from math and code first: those are the domains where the verifier already existed.
3. Long chains of thought become trainable. With an outcome-level verified reward, the model is free to discover how to earn it — longer reasoning, self-checking, backtracking. Nobody labeled those behaviors; they emerged because they help pass the checker.
The limit is honest to state: RLVR only works where a verifier exists. Poetry doesn't have unit tests. The open question of this era is how far verified-domain gains transfer to everything else — and the results so far suggest: further than expected.
One more thing, from the person who asked for this lesson. A grader that trains your code from scratch and scores it on held-out seeds against a fixed threshold — like the one in player-of-games — is RLVR-shaped. The environment is the verifier. The purist instinct that wanted no human in the loop was, it turns out, just early.