NLP Lineage

7 September 2020 · Scale and data

MMLU

Measuring Massive Multitask Language Understanding

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt

Date
7 September 2020
What the date marks
arXiv submission
Kind
Dataset
Organisation
UC Berkeley / Columbia / UChicago
Access
MIT
Other dates
ICLR 2021
Threads
In the guided tour
Chapter 5. Scale becomes a science
Learning phases
2. How LLMs are trained

A milestone: 1 later milestone descends directly from it, and its descendants reach 2 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Benchmarks like GLUE were close to saturated and measured narrow skills.

The problem

No standard way to measure broad knowledge across many domains at once.

What it introduced

57 subjects, 15,908 exam-style questions, from elementary to professional level.

Why it matters

It became the headline number in every model release for five years, for better and worse.

What modern models inherit

The benchmark culture that now drives model marketing.

What came next

Saturated by 2024, prompting MMLU-Pro, GPQA and harder successors.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for MMLU without weak links.

No documented ancestors are recorded for MMLU without weak links.

No recorded ancestors are recorded for MMLU.

No recorded ancestors are recorded for MMLU.

After: what it led to

MMLU has 2 documented descendants across 1 chapter, reaching forward to Emergence mirage in 2023. Showing its direct descendants.

MMLU has 2 documented descendants across 1 chapter, reaching forward to Emergence mirage in 2023. Showing all of them.

MMLU has 3 recorded descendants, including indirect ones, across 2 chapters, reaching forward to Chatbot Arena in 2024. Showing its direct descendants.

MMLU has 3 recorded descendants, including indirect ones, across 2 chapters, reaching forward to Chatbot Arena in 2024. Showing all of them.

  1. 2020MMLUThis milestone.
  2. 2022Emergent abilitieswas influenced by MMLUChapter 5. Scale becomes a science
  3. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  1. 2020MMLUThis milestone.
  2. 2022Emergent abilitieswas influenced by MMLUChapter 5. Scale becomes a science
  3. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  4. 2024Chatbot Arenawas influenced by MMLU (indirect influence)Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

Developed alongside

  • GPT-3, 2020 (timing only, no documented link) Contemporaneous; MMLU was built to measure models of this class. Source: arxiv.org

What it led to

Influenced