Language Models are Few-Shot Learners
T. Brown, B. Mann, N. Ryder, et al. (31 authors)
- Date
- 28 May 2020
- What the date marks
- arXiv submission
- Kind
- Model
- Organisation
- OpenAI
- Access
- API only
- Other dates
- API beta 2020-06-11; NeurIPS 2020
- Learning phases
- 2. How LLMs are trained; 3. API usage and prompting
A turning point: 13 later milestones descend directly from it, and its descendants reach 10 of the 10 threads. How levels are decided
Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 4 (July 2020) of the paper also published at NeurIPS 2020).
What it changed
It showed that a very large language model can do a new task from a few examples written into its input, with no training on that task. The authors call this in-context learning. With GPT-3, a prompt became a practical way to use one model for many tasks.
The problem before it
In 2020, the standard method was to pretrain a model on plain text, and then fine-tune it on a labelled dataset for each task (BERT, T5). Each task needed thousands of labelled examples and its own training run.
The authors give three reasons this is a problem:
- Many useful tasks have no large labelled dataset.
- A model fine-tuned on a narrow dataset can learn shortcuts in that data, so its benchmark score can overstate its real skill.
- People can learn a new language task from a short instruction or a few examples.
GPT-2 had tried tasks written as text, with no fine-tuning, but its results were far below fine-tuned systems. For example, it answered only about 4% of Natural Questions correctly.
The idea
The mechanism. Keep the GPT-2 design and make it much larger. The authors expected in-context learning to improve with size, as the language-model loss does (scaling laws).
- The model. A Transformer decoder with 175 billion parameters, 96 layers and a context of 2,048 tokens. This is 10 times more than any earlier language model that was not sparse. Its layers alternate between full attention and a local, banded attention pattern, similar to the Sparse Transformer. The authors also trained 7 smaller versions, from 125 million to 13 billion parameters, to measure the effect of size.
- The data. About 300 billion tokens from five sources: filtered Common Crawl (60% of the training mix), WebText2 (22%), two book collections (8% each) and English Wikipedia (3%). To filter Common Crawl, a classifier kept pages that looked like the high-quality sources. Duplicates were removed. Higher-quality sources were sampled more often than their size alone would give.
- Three ways to give a task. No weights change in any of them. The model only reads the text and continues it.
- Zero-shot: a plain-language description of the task, such as “Translate English to French:”.
- One-shot: the description and one solved example.
- Few-shot: the description and as many solved examples as fit in the context, usually 10 to 100.
The notation. There are no new equations. The paper’s Table 2.1 lists the model sizes. Training the 175-billion-parameter model used about 3.14 × 10^23 operations, about 3,640 petaflop/s-days.
What it showed
Reported in the paper. We read version 4 of the arXiv paper. All GPT-3 results below are from the 175-billion-parameter model, with no fine-tuning.
- Size helps in-context learning most. Zero-shot results improved steadily with model size. Few-shot results improved faster, so the gap between them grew with size.
- LAMBADA (guess the last word of a passage). 86.4% few-shot, against an earlier best of 68.0%.
- Closed-book questions (TriviaQA). 71.2% few-shot. The model answers from what it learned in training, with no documents to search. This beat a fine-tuned T5 model of 11 billion parameters (50.1%). With one example, it matched an open-book system that searches 21 million documents (68.0%).
- Translation. Into English, few-shot GPT-3 beat earlier unsupervised translation work. For example, French to English scored 39.2 BLEU. Out of English it was weaker, and English to Romanian was more than 10 BLEU below earlier unsupervised work.
- SuperGLUE. With 32 examples per task, the test score was 71.8. A fine-tuned BERT-large scored 69.0. The best fine-tuned system scored 89.0.
- Arithmetic. With few examples: 100% on 2-digit addition and about 80% on 3-digit addition (the text gives 80.2%, the tables 80.4%). It fell to about a quarter for 4 digits and about 10% for 5 digits. The 13-billion-parameter model got 2-digit addition right only about half the time.
- News articles. People tried to tell human-written news articles of about 200 words from GPT-3’s. They were right 52% of the time, close to the 50% of guessing. For a deliberately bad control model, they were right 86% of the time.
- Clear weak spots. Tasks that compare two sentences were weak. On WiC (is a word used the same way in two sentences?), it scored 49.4%, the level of guessing. On the hard ANLI inference test, the smaller models stayed at chance and GPT-3 only began to rise above it. Some reading-comprehension sets, such as RACE and QuAC, were also weak.
Limits, and what came later
The limits come from the paper:
- Text quality. Long outputs can repeat, lose coherence or contradict themselves.
- Test data in training. A bug in the filtering left some benchmark text in the training data, and the model was too costly to train again. The authors measured the effect: it was small for most benchmarks. They removed several language-modelling results and marked the PIQA and Winograd results.
- One direction only. The model reads left to right. The authors think this may explain the weak results on tasks that compare two passages.
- Unclear mechanism. It is not known whether few-shot learning learns a task from the examples or only recognises a task seen in training.
- Cost. The model is expensive to run.
- Bias. The model reflects stereotypes in its training data. For example, 83% of 388 occupations were more likely to be followed by a male word than a female one.
- The objective. Next-word prediction weights every token equally and is not grounded in the world. The authors suggest learning the objective from people, or fine-tuning with reinforcement learning, as future directions.
Many later models built directly on GPT-3. InstructGPT fine-tuned it to follow instructions. Codex fine-tuned GPT models on code. GPT-3.5 descends from it. OPT openly reproduced a model of the same size. Chain of thought reported its results on models of this class.
What to know first
- The Transformer decoder and GPT-2.
- The pretrain-then-fine-tune method of BERT, which GPT-3 sets out to avoid.
- Scaling laws, which guided the size of the model.
- What benchmarks measure: fixed test sets, with a score that can be compared between systems.
How to read it
The paper is 75 pages. The main text is the first 41 pages, and the rest is appendices and references.
- Read section 1 with Figures 1.1 to 1.3. They define in-context learning and show the effect of size.
- Read section 2 with Figure 2.1 and Tables 2.1 and 2.2: the three settings, the models and the data.
- Read section 3 selectively. Start with 3.1.2 (LAMBADA), 3.2 (closed-book questions) and 3.9.1 (arithmetic).
- Read section 4 on test data in the training set.
- Read section 5, the authors’ own limits.
Skim section 6 on misuse, bias and energy. Expect about 3 hours for the steps above (an estimate).
Links
- The paper: arXiv 2005.14165
- Data and samples: openai/gpt-3, named in the paper. It holds test data and samples, not the model. The repository is archived on GitHub.
- Video: none is linked here yet.