Every reasoning model you have used this year — the ones that pause, think in long chains, and check their own work — owes a large part of its behavior to a single reinforcement learning algorithm. It is called GRPO, short for Group Relative Policy Optimization, and in 2026 it has become the default starting point for teaching models to reason.
GRPO is not new — it debuted in early 2024 — but it went from a clever trick inside one lab to the central reference algorithm for the entire "reasoning RL" wave. Here is how it works, why it beat the incumbent, and where it is already showing cracks.
The Problem GRPO Solves
Classic reinforcement learning from human feedback leaned on PPO (Proximal Policy Optimization). PPO works, but it carries a heavy passenger: a separate value model, also called a critic.
The critic's job is to estimate how good a given state is — a baseline the algorithm subtracts from actual rewards to compute the advantage (how much better an action was than expected). The catch is that this critic is typically another neural network roughly the size of the model you are training. You end up training two large models at once, doubling memory pressure and adding a second thing that can go unstable.
For math and code — where you can often just check whether an answer is right — carrying a full critic started to feel like overkill. That is the opening GRPO walked through.
How GRPO Actually Works
GRPO's core insight is almost embarrassingly simple: if you want to know whether a response is good, compare it to other responses to the same prompt.
The recipe:
- Sample a group. For a single prompt, generate a group of candidate responses — say 8, 16, or 64 of them — from the current policy.
- Score each one. Run every response through a reward signal. For verifiable tasks this can be a rule: did the math answer match, did the code pass the tests?
- Compute a group-relative advantage. Instead of asking a critic "how good is this state?", GRPO takes the rewards in the group and normalizes them by the group's own mean and standard deviation. A response that scored above the group average gets a positive advantage; below average, negative.
- Update the policy. Push the model toward the above-average responses and away from the below-average ones, with a PPO-style clip to keep each step from moving too far.
That single change — replacing the learned value critic with the statistics of a sampled group — is what makes GRPO critic-free. You delete an entire model-sized component and get your baseline for free from responses you were already generating.
The whole method reduces to one idea: the group is the baseline. The mean reward of the batch tells you what "expected" looks like, so you never need a separate network to predict it.
Where GRPO Came From
GRPO was introduced in the DeepSeekMath paper (arXiv 2402.03300) in February 2024, framed explicitly as a variant of PPO for improving mathematical reasoning. In that work, DeepSeekMath 7B reached 51.7% on the competition-level MATH benchmark without external tools or majority voting — approaching the level of far larger frontier systems at the time.
But GRPO's real fame came later, when the same team used it to train DeepSeek-R1-Zero and DeepSeek-R1. R1-Zero was trained with large-scale RL and no supervised fine-tuning stage first, and it spontaneously developed long chains of thought, self-verification, and backtracking. That result — reasoning behavior emerging almost entirely from GRPO-style RL on verifiable rewards — is what turned the algorithm into an industry template.
This dovetails with the broader shift toward RLVR (reinforcement learning with verifiable rewards): instead of relying on a learned reward model trained on human preferences, you train on tasks where correctness can be checked programmatically — math, code, logic, structured outputs. GRPO is the natural optimizer for that regime because the reward is cheap, objective, and abundant.
GRPO vs PPO vs DPO
It helps to place GRPO next to the two methods it is most often confused with.
| Method | Needs a critic? | Core idea | Best for |
|---|---|---|---|
| PPO | Yes | Learned value model estimates the advantage baseline | General RLHF when no cheap reward exists |
| GRPO | No | Group of samples supplies the baseline via normalized rewards | Reasoning RL with verifiable rewards (math, code) |
| DPO | No | Trains directly on chosen-vs-rejected preference pairs, offline | Offline preference alignment from a fixed dataset |
The distinction that matters: DPO is offline — it learns from a frozen dataset of preference pairs and never generates fresh samples during training. GRPO is online — it generates new groups of responses every step and scores them live. DPO is cheaper and more stable for aligning tone and helpfulness; GRPO is what you reach for when you want the model to get better at solving problems it can be graded on.
The Cracks — and the 2026 Variants
GRPO is dominant, not perfect, and a small industry of successor methods has grown up specifically to patch its known flaws:
- Dr. GRPO targets GRPO's length bias. The original normalization can quietly reward longer answers, inflating responses. Dr. GRPO corrects how advantages are normalized so shorter correct answers are not penalized and long reasoning traces are not artificially favored.
- DAPO (Dynamic Sampling Policy Optimization) keeps the group-comparison workflow but stabilizes training with decoupled clipping and by filtering for more informative prompts. It was released with an open large-scale RL system and reported strong results on AIME-style reasoning benchmarks.
- GSPO (Group Sequence Policy Optimization) moves the importance ratio from the token level to the sequence level, which turns out to be notably more stable when training Mixture-of-Experts models — a real pain point for token-level GRPO.
The proliferation is the tell. When a field produces a dozen named variants of one algorithm in eighteen months, it means that algorithm has become the shared substrate everyone is building on.
The Bottom Line
GRPO won because it threw away the most expensive part of PPO — the critic — and replaced it with a free baseline drawn from the very samples the model was already generating. Pair that with verifiable rewards, and you get a cheap, stable loop for teaching models to reason, which is exactly the loop DeepSeek used to make R1 think out loud. If you want to understand why 2026's models pause and reason before answering, GRPO is the two-page idea underneath it all — and the growing family of Dr. GRPO, DAPO, and GSPO variants is the clearest sign that it will be the foundation for a while yet.
