PPO vs GRPO, side by side

2026 · a claude-generated lesson · made to be read on a phone

Both are policy-gradient methods for the same job: nudge an LLM's weights so that higher-reward outputs become more likely, without drifting so far that the model collapses. The difference is almost entirely about where the advantage comes from — and what you have to keep in GPU memory to compute it.

PPO

PPO trains two models: the policy, and a value model (critic) that predicts, per token, how much reward is still coming. Advantages are computed with GAE — reward minus the critic's baseline — so every token gets its own credit assignment. The clipped objective then keeps each update inside a trust region, and a KL penalty against a frozen reference model keeps the policy from wandering off distribution.

The cost: the critic is a full copy of the LLM in memory, it has to be trained too, and a per-token value function for "did the final answer come out right?" is a strange thing to learn.

GRPO

GRPO (from DeepSeekMath, 2024) deletes the critic. Instead, for each prompt you sample a group of G completions, score each one, and define the advantage of a completion relative to its own group:

# PPO: per-token advantage from a learned critic
advantage[t] = GAE(rewards, values)          # needs a value model

# GRPO: one advantage per completion, from the group
rewards = [score(y) for y in group]          # G samples, same prompt
advantage = (r - mean(rewards)) / std(rewards)   # no value model

Every token in a completion shares that one advantage. The group mean is the baseline — you estimate "how good is a typical answer to this prompt" by just asking the model several times, instead of training a second network to guess it.

The comparison

	PPO	GRPO
baseline	learned value model	group mean of sampled rewards
models in memory	policy + critic + reference (+ reward model)	policy + reference (+ verifier)
credit assignment	per token	per completion
fits best with	dense / shaped rewards	outcome rewards (right answer? tests pass?)
canonical use	InstructGPT-style RLHF	DeepSeekMath, R1-style reasoning

the trade: PPO buys per-token credit with a second model; GRPO buys memory with sampling

Why GRPO won the reasoning era: when the reward is a verifier (the answer is right or it isn't), the reward only exists at the end of the completion anyway. A per-token critic adds cost without adding signal. Group-relative advantages are exactly the right shape for verifiable, outcome-level rewards — which is why GRPO and RLVR took off together.