Before it
Code generation was evaluated by text similarity to a reference solution.
The problem
Similarity to a reference says nothing about whether the code runs correctly.
What it introduced
Fine-tune GPT on public code and evaluate by executing unit tests, via the HumanEval benchmark.
Why it matters
Functional correctness as the metric is why coding became the field's clearest measure of progress.
What modern models inherit
Execution-based evaluation, now standard for code models.
What came next
Copilot productised it; HumanEval led to SWE-bench and the agentic coding benchmarks.