Before it
RL fine-tuning used PPO, which needs a separate value network alongside the policy.
The problem
The value model doubles memory and is the fiddliest part of the pipeline to stabilise.
What it introduced
Group Relative Policy Optimization: sample a group of answers and use their relative scores as the baseline, dropping the critic entirely.
Why it matters
GRPO is what made large-scale RL for reasoning affordable, and it is the algorithm behind DeepSeek-R1.
What modern models inherit
The RL algorithm most open reasoning training now uses.
What came next
DeepSeek-V3's post-training, then R1, then wide adoption across open reasoning models.